top | item 7125464

A RESTful Micro-Framework in Go

234 points| Nogwater | 12 years ago |dougblack.io | reply

63 comments

order
[+] mitchellh|12 years ago|reply
This looks like a great resource! A bit of a rant (not directed at the person who wrote this, because the link at hand is great):

Writing APIs in Go is not the hard part. There are a ton of options for routing basic URLs to handlers. There are a few areas that are a nightmare, total total nightmare:

* SQL - The raw database/sql package in Go is very low-level. It is a terrible experience to use in a project that makes anything more than a handful of simple queries. I'm not asking for an ORM, but it would really benefit the community if there was a SQLAlchemy style (the non-ORM part of it) library for Go. It should be MUCH easier to make basic queries to SQL and get out results.

* Templating - The built in "html/template" package is just not good enough for a web app when compared to something like Jinja (Python). "html/template" gets points for being simple, and actually its not a complaint against "html/template": I think there just needs to be a more feature-packed templating package like Jinja for Go. "html/template" doesn't support things like inheritance, didn't even support comparison operators until Go 1.2, etc.

These two problems have made it so I've pretty much given up web apps in Go for now. Services in Go are _amazing_. But if you're trying to get a full blown web app (again, not the point of the linked article above), then you're gonna have a bad time.

[+] hierro|12 years ago|reply
We're about to release our Go web framework which, I think, implements all the features you're mentioning and a few more. We're aiming for a release in Q1, but getting everything cleaned up and writing good documentation is taking a bit longer than expected. Right now, our internal framework (codenamed Gondola, but subject to change), includes:

* A full ORM which generates tables from structs and allows relations between models, as well as a simple querying system based on Go functions which joins the tables automatically for you e.g.

   memeId := base36.Decode(param)
   var meme *Meme
   var template *Template
   err := ctx.Orm().One(orm.Eq("Meme|Id", memeId), &meme, &template)
Meme declares a FK to Template, so the ORM automatically joins the tables and populates both pointers. Note that the ORM does not require the backends to be relational, so even when ATM we only have database/sql based backends you could write a backend for mongodb or rethinkdb.

* Inheritance based template system, with a declarative assets pipeline supporting compilation (e.g. it compiles coffescript to JS and less to CSS on demand - also pluggable). This is the base template from one of our apps:

   {{/*
      jquery|if=~ie-gte-9: 2.0.3
      jquery|if=ie: 1.10.2
      bootstrap|fontawesome=4.0.3: 3.0.0
      styles|bundle: css/style.css, lightbox/css/lightbox.css
      scripts|top,bundle: js/modernizr.js, js/detectizr.js
      scripts|bundle: js/responsiveslides.js, lightbox/js/lightbox-2.6.min.js, js/site.js
      analytics|nodebug: {{ $Config.Analytics }}
   */}}
    <!DOCTYPE html>
    ...
    {{ template "main" }}...
This server query 1.10 and 2.0 from Google's CDN, bootstrap and fontawesome from bootstrapcdn.com and bundles all our styles and scripts a CSS and a JS file. Then other templates can extend this one and override its blocks e.g.

   {{/*
     extends: base.html
   */}} 
   {{ define "main" }}...
(note that base.html could also define its own "main" block which would act as a default if you loaded base.html and it would be replaced by main.html's "main" block if you loaded "main").

* Easy to use signed and encrypted cookies.

* Regular expression based URL routing.

* Functions for easily parsing input parameters.

* A development server which automatically rebuilds the source as it changes and reports compilation errors in the browser itself. While developing, runtime errors are also reported in the browser with complete backtraces and printing the values passed to each function. While running in production, those reports are sent by email to the project administrator(s).

* App-wide and low-level caching system with pluggable backends, currently supporting Redis and Memcache as well as a dummy backend for development.

* Form generator and validator from Go structs, with renderers for both Bootstrap and Foundation (renderers are also pluggable, so you could also write your own. We also support generating forms from multiple objects so you e.g. write an object which generates a captcha and include it in any form with a single line in code (in fact, we provide just that, but a simple numeric captcha a a full-blown recaptcha object). It also does automatic CSRF protection.

* A complete i18n system, which extracts strings from Go code and templates, generates .pot files which can be translated using any editor (or a specialized .po editor if you feel like it) and them compiles .po files back to Go code again. This makes packages with their translations "go-get-able".

* Periodic and background tasks which can either be scheduled or just started from any web request.

* A configuration package which takes a struct and fills it by parsing an optional configuration file as well as command line flags.

* Support for administrative commands, which can be executed from the command line or started remotely.

* Support for pluggable apps. e.g. we do have a "users" app which provides user registration, authentication and social integration. This can be included in any other app with a couple of lines of code (one line for the import, another line to tell our main app to include it). Included apps can do a lot of stuff, like inserting code in every template rendered by the parent app, including external assets like images, styles or scripts and even rendering their own templates contained inside your parent's app base template.

* Last but not least, a minimal overhead (in the range of 100ns per request) profiling framework which lets you know how much time a request spent in the cache, the orm, the template, e.g... as well as seeing how much each operation took (e.g. you can check how much time a SELECT took and also perform an EXPLAIN on it). Users can also define their own events and profile them by instrumenting their code (it usually requires just a line of code per resource, to indicate its name). You can also profile a live server by using a command line app included in the framework.

We also have helper and utility functions for the more mundane stuff and basically everything you could expect from a web framework. We took inspiration from Django, Tornado and Rails and tried to take the best of them while building this framework (I used to contribute to Tornado back in the day). We've been developing this internally for almost 2 years and it's now around 40KLOC of Go and a couple of K of coffescript, less and html.

[+] lann|12 years ago|reply
> SQL

I'm working on an arel-esque fluent SQL generator now: https://github.com/lann/squirrel

I'll release it once I'm a little more confident in the interface. Currently you can do things like this:

  rows, err :=
        Select("col").
        From("tab").
        Where("x = ?", 1).
        RunWith(db).Query()
[+] aktau|12 years ago|reply
I'm usually pretty anti-ORM and actually like low-level SQL interfaces (I guess because I did a short stint as an Oracle developer once, once you see what can be done with some custom SQL...). But for a recent sideproject of mine, I really didn't want to create a laborious interface to the database. I wanted something quick and dirty. I got something quick, and it doesn't feel dirty at all. It's called gorp: http://github.com/coopernurse/gorp.

By the way, my sideproject is about getting financial data from sources like Yahoo Finance, Bloomberg, et cetera. You can find it here: https://github.com/aktau/gofinance

[+] elithrar|12 years ago|reply
> * SQL - The raw database/sql package in Go is very low-level. It is a terrible experience to use in a project that makes anything more than a handful of simple queries. I'm not asking for an ORM, but it would really benefit the community if there was a SQLAlchemy style (the non-ORM part of it) library for Go. It should be MUCH easier to make basic queries to SQL and get out results.

I can highly recommend https://github.com/jmoiron/sqlx which allows you to easily unpack results into structs/maps, or pass in a struct/map and insert it (using struct tags + named queries). If you're writing a CRUD application this covers a lot of the basics. There's also https://github.com/eaigner/jet which is a 'micro ORM' (and very light on the ORM part).

> "html/template" doesn't support things like inheritance, didn't even support comparison operators until Go 1.2, etc.

I grew to like html/template over time, but the lack of comparison operators pre Go 1.2 was a huge sore point and I had I been working on this little project a year ago I'd have been extremely frustrated. The lack of inheritance was a bit of a pain for me too—I ended up splitting my templates into many (!) smaller pieces so I could at least re-use the portions I needed to. Hopefully something "Jinja-esque" comes in time.

There's an (unmaintained) Mustache port and a Mustache-alike in Mandira (https://github.com/jmoiron/mandira) but I've not seen anything like Jinja yet. If there's one thing html/template has going for it is that I can trust it (as much as one can) to be 'safe'.

I wouldn't say web apps in Go should be avoided at all costs, but I would say that you should consider the scope/scale and do a bit of research first. Sometimes Django/Rails or Flask/Sinatra might make it easier on you.

[+] latch|12 years ago|reply
FWIW, I started to work on an erb-inspired templating package:

https://github.com/karlseguin/gerb

I didn't get to finish it before leaving on a trip, but I'll be back on it Feb 1st and can hopefully wrap it up in the first week.

I did finish a liquid-inspired template: https://github.com/karlseguin/liquid

but I wasn't particularly happy with it (neither the implementation nor liquid itself).

[+] chimeracoder|12 years ago|reply
> "html/template" doesn't support things like inheritance

I don't want to get into a debate about what "inheritance" means, but html/template does support template composition and nested templates, which seem to address the same use case.

> These two problems have made it so I've pretty much given up web apps in Go for now. Services in Go are _amazing_. But if you're trying to get a full blown web app (again, not the point of the linked article above), then you're gonna have a bad time.

I'm sorry to hear that, especially since writing a "full-blown web app" is really not difficult in Go at all, once you know what you're doing.

I've actually been working on Jetpack[0], a sample web application in Go to demonstrate how straightforward this is, once one understands how it actually works. (Disclaimer: it's still a work in progress, so it's not complete/ready yet).

[0] https://github.com/ChimeraCoder/go-angular-jetpack

[+] johne20|12 years ago|reply
You mentioned "Services in Go are _amazing_", but how do you avoid the first issue you mentioned? I have found so far getting data in and out to be somewhat cumbersome. Also how do you handle data validations and logging?
[+] jamra|12 years ago|reply
I agree with the templating problem. Additionally, it is difficult to find proper documentation on how to do templating and not just basic examples.
[+] tptacek|12 years ago|reply
This looks like great work, so probably this comment is a little off topic, but here goes anyways:

Every web framework I've seen for Golang seems to do a competent job of tackling the easiest part of designing a web framework, which is routing requests to handlers.

Actually, the built-in net/http library already does a serviceable job of handling requests, and if you understand closures, filtering them and maintaining request state as well.

The thing no framework I have seen so far addresses is persistence. SQL persistence in Golang is a bit of a nightmare. It's raw SQL, and the database libraries are fussy.

The thing that makes Rails so easy to write in, especially for newcomers, is ActiveRecord. As far as I can tell, nothing like it exists (in a stable form) for Golang.

[+] carbocation|12 years ago|reply
I agree with you re: the lack of ORMs in golang. I actually enjoy writing and profiling my own SQL, but if I want to prototype, ORMs can speed up the process.

I will say that some of the parts of SQL in golang that I find most annoying, such as populating structs, are handled very nicely by jmoiron's sqlx library. [1] This is not an ORM, but it's an extension to sql that removes some of the most annoying boilerplate nevertheless.

1 = https://github.com/jmoiron/sqlx

[+] mbreese|12 years ago|reply
I find the lack of ORM a feature in a web framework. Personally, I find them frustrating and an extra layer of abstraction that just gets in my way. Then again, I also don't mind writing queries. I usually end up using my own lightweight set of helper functions to manage queries, regardless of the language.

I still have Hibernate flashbacks from back in the day.

[+] sdegutis|12 years ago|reply
ActiveRecord may be a stepping stone for those just starting out, but for serious, long-living, evolving applications I firmly believe it does more harm than good.

Back when I did Rails contracting, cleaning up code which conflated database access with business logic was quite possibly what was keeping us in business.

[+] bliti|12 years ago|reply
Yes, SQL is still spotty in Go. I wrote a medium sized program with it and the learning curve was fairly high. Documentation for the database libraries is mostly nill. Though I'm not sure if it would benefit from an ORM. This is a systems language. I'd rather have it use interfaces to access data from a database than have it deal with it itself. That way I keep it tidy and fast. Which is Go's aim.
[+] JonnieCache|12 years ago|reply
If anyone is planning to write a golang ORM, you should have a look at sequel: http://sequel.jeremyevans.net/

I'm not aware of a better ORM anywhere. The simplicity, and the way it separates the model classes from everything else, is sublime. It does pretty much everything activerecord does and lots more, without all the layers of abstraction.

[+] chimeracoder|12 years ago|reply
I'm going to chime in, since I've been using Go as my primary language for over a year, and I'm currently working on a major (production) Go project that others might see as well-suited for Rails (in fact, the early prototype was written in Rails[0])

> The thing no framework I have seen so far addresses is persistence. SQL persistence in Golang is a bit of a nightmare. It's raw SQL, and the database libraries are fussy.

> The thing that makes Rails so easy to write in, especially for newcomers, is ActiveRecord. As far as I can tell, nothing like it exists (in a stable form) for Golang.

I wouldn't recommend newcomers to Go use SQL right off the bat. Document-based storage (like CouchDB or MongoDB), or even key/value (Redis, etc.) is much more aligned with Go's concept of structs, and all three of those databases have strong library support in Go. If you're just looking to dabble in Go and just happen to need a database (and it doesn't matter which), I'd recommend playing around with those first. Even Neo4j works very well with Go, though as of January 2013 (when I last used it), the only Go library for Neo4j was third-party, and not very well-documented.

That said, once you know the Go language, using SQL (either with or without an ORM) is straightforward, if not as convenient as ActiveRecord. jmoiron has a good blog post[1] comparing the good, bad, and ugly of Go ORMs, so I won't try to top that here - rather, I'll share my experience.

(As pointed out elsewhere, ORMs are not universally liked as a general concept, so I'm writing this comparison assuming that one actually does want the features an ORM provides - naturally, if you prefer prepared SQL, Go already provides perfect support for that.)

Gorp is definitely not as "magical" as ActiveRecord, but it certainly does the job. A number of the ways in which it differs from ActiveRecord have more to do with the (strongly) static vs. dynamic difference in type systems, though some are specific to Go's architecture.

There are a few odd features here and there that are still missing (for example, I'm actually not sure if there's a wrapper function for "INSERT IF NOT EXISTS" yet, though it's not a huge bother).

I do end up spending more time thinking about the queries that I'm using, compared to Django or Rails. There is some additional verbosity - I have to write some helper functions that I wouldn't otherwise have to. On the other hand, I know exactly what queries are being executed at all times, so it's easy to identify potential bottlenecks.

In short - I agree that ActiveRecord is more convenient than anything that currently exists in Go. On the other hand, it's reasonably advanced enough that relative newcomers to the language[2] shouldn't have a problem with it. So far, the biggest bottleneck in development of this application has not been with the SQL bindings, even though it would be nice to spend a bit less time on that front.

It hasn't had as much time to develop and mature as ActiveRecord have, but those of us using it in production have accepted those tradeoffs.

[0] If you're wondering why we didn't continue it in Rails, we had to scrap the original codebase for a number of other reasons, so given that we weren't keeping any of the original code, there was no logical reason we needed to use the same language.

[1] http://jmoiron.net/blog/golang-orms/

[2] As opposed to newcomers to programming in general

[+] jamra|12 years ago|reply
What's wrong with using raw sql classes? A web framework should be database agnostic.

I agree with the usefulness of a better SQL library though.

Edit: Wow that was worded poorly. What I meant was, what do you find bad about the SQL library?

[+] icholy|12 years ago|reply
IMO having some ORM automagically construct queries for you is a pretty scary idea. Obviously it's not a good idea to have SQL scattered all over your codebase (we've all seen it). But including SQL in your code isn't hard if you keep it in one place and abstract it properly.
[+] natural219|12 years ago|reply
This post really highlights what I like about Go so far vs other languages -- the evolving community. Like Rails, there is an emphasis on abstraction and productivity. Unlike Rails, the blog posts I've seen so far seem to be centered around explaining how to implement the functionality instead of just wrapping the functionality and providing an easy API. Now that I've read this post, I could download his library, implement my own, or extend the parts I want to customize very easily. I feel more powerful.
[+] polymathist|12 years ago|reply
I would have taken a different approach for handling unsupported resource methods. I'd love to hear some feedback on whether or not this is considered idiomatic. Instead of this:

  type HelloResource struct {
     sleepy.PostNotSupported
     sleepy.PutNotSupported
     sleepy.DeleteNotSupported
  }
Something like this:

  type HelloResource struct {
     sleepy.DefaultResponder
  }
DefaultResponder is simply a struct which has all the methods of sleepy.PostNotSupported, sleepy.PutNotSupported, etc, and is defined:

  type DefaultResponder struct{}
  func (DefaultResponder) Get(values url.Values) (int, interface{}) {
     return 405, map[string]string{"error": "Not implemented"}
  }
  func (DefaultResponder) Post(values url.Values) (int, interface{}) {
      return 405, map[string]string{"error": "Not implemented"}
  }
  func (DefaultResponder) Put(values url.Values) (int, interface{}) {
      return 405, map[string]string{"error": "Not implemented"}
  }
  func (DefaultResponder) Delete(values url.Values) (int, interface{}) {
      return 405, map[string]string{"error": "Not implemented"}
  }
Now, if you want to implement resource methods, you define them on the top-level struct, like so:

  type HelloResource struct {
     sleepy.DefaultResponder
  }
  def (HelloResource) Get(values url.Values) (int, interface{}) {
     data := map[string]string{"hello": "world"}
     return 200, data
  } 
So now, if you call Get on a HelloResource type, it uses the method defined on HelloResource, not the one defined on it's embedded DefaultResponder type. But if you call HelloResource.Post, it uses the method defined on DefaultResponder, i.e., it returns a 405 error. This is effectively like "overriding" in Java or other object oriented languages and saves a few lines when people are using your library.

What do you think? Again, I'm really just looking for feedback here as I too am new to go. Is this something that would be considered idiomatic, or is the original solution better?

(Apologies in advance for any typos or syntax errors. Didn't check this with a compiler, but hopefully you get the idea.)

[+] optymizer1|12 years ago|reply
You could even go further and assume all user resources are logical extensions of sleepy.Resource:

  type HelloResource struct {
    sleepy.Resource
  }
This makes it clear that you're inheriting everything from sleepy.Resource, so you can provide reasonable implementations for all methods of resources, not just HTTP methods handlers, e.g. getBodyAsJSON(), redirect(resource), etc.

Also, depending on how your mux works, you could store app-level data in sleepy.Resource, to be accessed by all resources:

  sleepy.Resource.DB
  sleepy.Resource.Session
  sleepy.Resource.Root
and other things I can't think of right now.
[+] _ak|12 years ago|reply
Nice, easy to use, really straightforward code. The only thing I would have done differently is make sleepy.Api implement http.Handler so that it would easily integrate with other libraries that build upon and play nicely with net/http.
[+] dclara|12 years ago|reply
I haven't seen the full picture of Go yet, partially by some examples. I found it makes calling multi-threading functions a lot easier.

I consider Ruby, Python, Go and Julia are the new generation of programming languages aiming to be quickly picked up and made hands-on working.

Can anybody tell me if Ruby, Python, Go supports a full spectrum of functionalities for various network protocols, multi-threading, messaging (sync and async) and concurrent accessing etc. Those kind of functionalities don't only require additional libraries, but also frameworks and algorithms.

Maybe by using Jython, JRuby or something like that, they are able to achieve the complete functionalities like Java.

According to some research, Python evolved from Lisp, and Ruby is closed to Perl. So why aren't those script-based language more popular than Java/C++?

[+] mjallday|12 years ago|reply
Languages live or die on their libraries.

This is great. I sat down to write a quick api in go about three weeks ago but couldn't find a framework that I wanted to use. This looks like it has a decent abstraction and libraries like this really help language adoption.

Much thanks to the author of the library.

[+] jevinskie|12 years ago|reply
I appreciated the development narrative rather than a post-project summary. Thanks!
[+] jrockway|12 years ago|reply
Drive-by code review:

I don't see any reason for the Api type to exist. Just make them normal functions?

It would also be helpful to print the portstring rather than just "Hi.".

[+] cronos|12 years ago|reply
Why exactly do Resource's methods take variadic argument? In what scenario can there be more than one?

Also, small remarks about code: You define type HandleFunc in your package which is identical to http.HandlerFunc. Looks like it can be removed. Some error logging (rw.Write, json.Marshal) would be useful as well.

[+] dougblack|12 years ago|reply
Author.

Great points. They don't need to be variadic. I misunderstood the type of url.Values. And I didn't see http.HandlerFunc!

I eschewed error logging in the interest of simplicity.

Thanks for the corrections! Updating the article now.

[+] swah|12 years ago|reply
Kinda off-topic, but would you write a real-time web application in Go today?
[+] tptacek|12 years ago|reply
We did. Worked well.
[+] jchrisa|12 years ago|reply
The Go runtime is still being optimized. For instance we found that the upgrade to Go 1.2 made a big difference for some of our performance tests. So it's nice to get surprises like that but also shows that the ecosystem is still young.
[+] dark_ph0enix|12 years ago|reply
I'm yet to test this, but any idea how it compares against Martini?
[+] macu|12 years ago|reply
It doesn't have the same aim as Martini, which provides more of a full middleware manager.

Sleepy is a simple wrapper for RESTful resources that takes a few steps out of the handler work, e.g. internally handles the work of encoding a response to JSON and then sending it.

Nice to see new developers delving in, albeit a little clumsy at first :P

[+] jjsz|12 years ago|reply
I don't understand why, when questions like these are asked, they are never answered...
[+] skj|12 years ago|reply
Wonderfully small, easy to grok, and put into practice.