Managing Node.js Callback Hell with Promises, Generators and Other Approaches

[tgriesser]

Disappointing that most promise articles/tutorials reference Q as the preferred promise library, rather than bluebird - https://github.com/petkaantonov/bluebird

Bluebird is not only so fast that it's almost on par with callbacks[1], but also innovates on the ability to filter on catching different error types[2], support for generators[3], context binding[4] among other things.

[1]: https://github.com/petkaantonov/bluebird/blob/master/benchma...

[2]: https://github.com/petkaantonov/bluebird/blob/master/API.md#...

[3]: https://github.com/petkaantonov/bluebird/blob/master/API.md#...

[4]: https://github.com/petkaantonov/bluebird/blob/master/API.md#...

[angersock]

We just started using Q for our all--aaaargh, motherfucker.

Great to see progress being made, I guess. :(

[phpnode]

yes! I recently switched from Q to bluebird and it's been great, and really is exceptionally fast. Also the source code is a work of art - extremely highly optimised.

[inglor]

Bluebird is definitely the way to go, we recently introduced it to an existing 100K LoC code base and we couldn't be happier.

The stack traces and error management is amazing and everything is very fast.

[jbeja]

That library looks pretty cool and is fairly popular , i will give it a try.

[rdtsc]

Looking at it, in general, ignoring that I know about Node internals, you know what is the simplest way to solve this "problem"?

It is something like this:

---

    callback(find_largest(get_stats(read_files(dir)))

---

1 line.

If an error happens, an exception should be raised.

The fact that there are multiple such requests happening or that some of those operations have to wait for a select or epoll loop to return, should be handled by an underlying framework. The fact that even with promises you have to write 27 lines of code to do that, should be making you worried.

[inglor]

Please stop spreading FUD.

With promises (Bluebird) this sounds like very little code in practice, assuming promisifyAll has been called the FS module:

- One line to read all files in a directory

- Map them with .map to a .stat call on them, returning array of promises.

- Reduce them to the largest element.

Something like:

    var fs = Promise.promisifyAll(require("fs"));
    var largest =  fs.readdirAsync("/your/file/path").map(function(file){
         return fs.statAsync(file);
    }).reduce(function(x,y){
        return (x.size > y.size) ? x : y;
    });

With arrow syntax:

    var largest = fs.readdirAsync("/your/file/path").
                  map((file) => fs.statAsync(file)).
                  reduce((x,y) => (x.size > y.size) ? x : y);

Alternatively, with Generators using Bluebird promises

     var largest = Promise.coroutine(function *largest(dirname){
        var files = yield fs.readdirAsync(dirname);
        var sizes = yield files.map(name => fs.statAsync(name));
        return files.reduce((x,y) => x.size > y.size ? x : y);
    });

If one error occurs, the promise rejects. VERY far from 27 lines of code. Not to mention you can of course compose these in one line if you extract them to methods like in your code:

    var largest = fs.readdirAsync("/your/file/path").map(fs.statAsync).reduce(largest);

[gfxmonk]

You may find StratifiedJS pleasing: http://onilabs.com/stratifiedjs

You wouldn't even need that one callback, just:

return find_largest(get_stats(read_files(dir)));

Exceptions are propagated as you would expect from synchronous code, despite being async under the hood.

[tlrobinson]

The "promises" way of writing that concisely would be:

    read_files(dir).then(get_stats).then(find_largest).then(callback, errback)

Any exceptions in those functions would propagated to the "errback". Not perfect but much better than the examples in the article would suggest.

[jarrett]

> If an error happens, an exception should be raised.

For better or worse, Node's best practice seems to be to propagate errors via the err parameter in each callback. E.g. `function myCallback(err, data)` where data represents the result of a successful computation.

One could reasonably debate whether this was the best possible design. Some people like exceptions, for one thing. For another, JS's dynamic typing, combined with multiple layers of error propagation, could lead to mishandling of the errors--you expect your callback to be passed error A, but it actually gets error B from elsewhere in the callback chain.

But this pattern is ingrained by now. If people started deviating from the pattern and using exceptions, you'd end up with an even worse monstrosity. Every bit of calling code would have to handle both styles of errors.

[wtbob]

It seems to me that Node.js is just manually writing in continuation-passing style, which is PAINFUL. Why not simply use a language which supports continuations natively? Heck, writing a Scheme->JavaScript compiler really shouldn't be that difficult.

[shtylman]

The generator example has no error handling (at least not from what I can see).

Once the "callback hell" example was re-written with named functions, it is not much different than the generator example and depending on how you are testing, easier to test because it separates out IO from logic (Which you could still do in the generator example).

One thing callback style code has begun to make me think about is the nesting/IO complexity of the functions a I write. When you nest, the IO/event boundary is visible in the code (by the indentation). As you continue to nest, you can start to get a clear idea of the amount of IO being done and where a particular function may be doing too much. Anyhow, most of this is preference anyway :)

[sync]

A simple try/catch around the whole thing would take care of error handling for the generator example.

[pak]

As you read all these many lines of code to simply find the largest file in a directory, recall that in any language with friendlier blocking/non-blocking semantics, it could be as simple as (e.g. in Ruby)

    Dir.glob('*').select{|f| File.file?(f) }.max_by{|f| File.size(f) }

Plus, the error semantics of this code are sane: if something goes wrong I get an exception I can rescue. If there were no files in the directory, I get nil.

Statting a bunch of files is rarely so costly that parallelism is warranted, unless it is impressed upon you as in Node. The cost in readability and dreaming up generators and promises to solve such a mundane problem is likely much worse. Before you charge headlong into building your next webapp with Node, consider this article food for thought.

[ggreer]

It was a short, simple example to illustrate how yield will improve Node. In real life, you'd probably just do something like:

    fs.readdirSync(dir).map(function (f) {
      return {
        name: f,
        stat: fs.statSync(path.join(dir, f))
      }
    }).sort(function (a, b) {
      return b.stat.size - a.stat.size;
    });

Not going to win any code golf tournaments, but that's JavaScript for you. It was never designed to do this sort of thing. Its strengths lie elsewhere.

[rmgraham]

It's definitely a trade-off. What Node gives you is async when you need it at the cost of giving you async even when you don't.

The pain and constraints introduced by this forced use of async sure has spawned a lot of wheels of all sizes, colours, and compositions.

[spion]

With arrow function syntax, extra `maxBy` and `filter` methods on promises for arrays, and nicer file/dir libraries:

  Dir.glob('*').filter(f => File.isFile(f)).maxBy(f => File.size(f));

Asynchronicity isn't the issue there, the issue is that most JS / node libraries are minimal, not optimized for ergonomic (and of course the verbose lambda syntax)

[programminggeek]

I remember doing a deep dive into node and evented JS a couple years ago and I realized that I couldn't recommend it to my team because the code would become such a huge mess on an large system that whatever performance you gain, you lose in maintainability in a large team environment.

The best I could find at the time is IcedCoffeescript, which sort of replaces the whole existing coffee script tooling, so now you are dealing with custom tooling to install for each team member as well.

I find it sad that years later it looks like what IcedCoffeescript did is still the cleanest looking solution to the problem. It feels like the evented model is maybe great for smaller things, but horrible for building and reasoning about in larger systems on larger teams.

[DonPellegrino]

I'd like to mention my favorite tool to deal with Callback Hell: Streamline, a JavaScript (and CoffeeScript) language extension that can compile to either 1) callback spaghetti 2) Node-fibers or 3) generator code. It basically writes the mess for you. It lets you focus on the higher order concepts and provides configurable parallel array operations as well as many other convenient features. The callback "error parameter" is thrown as an exception to be caught with a basic try/catch. I highly recommend that any Node and JS enthusiast give it a try.

https://github.com/sage/streamlinejs

[porphyry3]

There is another nice feature of ES6 that helps a lot with callbacks and promises: proxies. Check out this article I wrote on Flippin Awesome (http://flippinawesome.org/2013/11/18/taming-asynchronous-jav...). The following is some sample code to show how code can be simplified with ES6 proxies, making async functions chainable (example in coffeescript):

    getUser = (user) ->
        _("https://npmjs.org/~#{user}")
          .get()
          .extractLinks()
          .filter( -> /package/.test(arguments[0]) )
          .map( -> "https://npmjs.org#{arguments[0]}" )
          .log()
    
    getUser("foo")

where `get` is THE jQuery get for Ajax requests returning a promise and is chained with the rest of the functions in a pipeline fashion. I've found that these techniques can work very well for form validation, as [you can check here](http://www.vittoriozaccaria.net/chained/demo.html) — Firefox required.

Edit: I wrote in markdown but it seems that only verbatim is supported. Edit2: some clarification

[pron]

The problem with all of these solutions (callbacks, promises, etc.) is that they make you adopt a certain programming style not because it's the most appropriate from a design perspective, but rather to work around scheduling limitations.

Lightweight threads first remove the problem, and then let you choose the most appropriate programming style for your domain. See https://news.ycombinator.com/item?id=7179144

[campbellmorgan]

an important omission here is the async.auto function which is a phenomenal relatively new addition that takes a amd approach to asynchronously running multiple tasks at the same time: https://github.com/caolan/async#auto if you haven't seen it already

[ilaksh]

Here is how I would do it using ToffeeScript (https://github.com/jiangmiao/toffee-script)

    fs = require 'fs'
    async = require 'async'
    path = require 'path' 
     
    module.exports = (dir, cb) ->
      err, files = fs.readdir! dir
      paths = (path.join dir, file for file in files)
      er, stats = async.map! paths, fs.stat
      largest = 0
      for stat, i in stats
        if stat.isFile and stat.size > largest
          fname = files[i]
          largest = stat.size
      cb fname

I use ToffeeScript for everything now, since it makes things much easier to read.

[shubhra51]

Promises have always been debatable. The new Generators is probably what I am looking for.

[onestone]

Generators and Promises play very well together.

[guiomie]

Why?

[inglor]

Why would one use Q promises in Node in 2014? They're very very very slow. Especially when there are alternatives like Bluebird promises that are almost as fast as callbacks.

[pekk]

When we had these things in other languages like Python, node's approach was better. Now that node has adopted them, node is even better still, I guess.

[davidw]

Some other camps ( http://www.erlang.org/ ) still strongly prefer their own approach, and I can't say I blame them.

[gfodor]

This is satire, right?

[ksherlock]

All that work and the findLargest routine doesn't seem to benefit whatsoever from fs.readdir or fs.stat being asynchronous.