Facebook open sources Haxl

[AaronFriel]

This is a fascinating project. Haxl is the brainchild of former Glasgow Haskell Compiler lead* Simon Marlow.

The tl;dr of Haxl: what if you could describe accessing a data store (a la SQL) and have the compiler and library work together to "figure out" the most efficient way to perform queries, including performing multiple queries in parallel? That's what Haxl does, it allows you to specify the "shape" of your query, the type checker verifies its correctness, and the library executes it in parallel for you, without the developer having to know about synchronizing access or anything.

Here's a link to their paper (PDF): http://www.haskell.org/wikiupload/c/cf/The_Haxl_Project_at_F...

* - I am not sure if he's still committing, or if he's only doing application development. His accomplishments in Haskell land though, are many.

Edited: I removed my comment about GitHub issues, seems it's a known problem. :)

[vamega]

He also wrote a book called Parallel and Concurrent Programming in Haskell[1], which I've heard is an excellent read[2].

[1] http://chimera.labs.oreilly.com/books/1230000000929

[2] http://www.serpentine.com/blog/2014/03/18/book-review-parall...

[simonmar]

He is still committing, but not quite so often :)

[thisisdave]

I feel like I'm missing something.

I haven't used databases much, but don't most SQL implementations already "figure out" the most efficient way to perform queries? Can't most implementations already perform queries in parallel?

[carlio]

GitHub is currently having problems - https://twitter.com/githubstatus

[emmanueloga_]

reminds me of http://ql.io/ (which does not seem to have taken off... last commit: Apr 24, 2013).

[pjc50]

Isn't that what an SQL query planner does?

[simonmar]

Here's our paper about the ideas behind Haxl: http://community.haskell.org/~simonmar/papers/haxl-icfp14.pd...

[seanmcdirmid]

Somehow I think this should have been upvoted to the top.

[vdijkbas]

Also see the slides of Simon's talk at ZuriHac 2014 last weekend:

http://www.haskell.org/haskellwiki/ZuriHac2014#Talk_by_Simon...

[lbrandy]

Hi. I'm one of the engineers who has worked on this so if anyone has any specific questions I can help answer and/or get someone to answer.

As said by @nbm, we also have a blog post up: https://code.facebook.com/posts/302060973291128/open-sourcin....

[Igglyboo]

I had no idea you guys used Haskell at FB. Other than Haxl, what are your main use cases for Haskell?

[evmar]

Have you replaced all use of FXL with Haxl? Or are both languages supported? In the latter case, what is the relative proportion of each in the live codebase?

(I appreciate that migrating code that already works to a new language often just introduces bugs for no gain, so please don't take my questions as trying to dig up dirt or anything. I'm genuinely just curious.)

[awda]

Why Haskell?

[nbm]

The release blog post is here - https://code.facebook.com/posts/302060973291128/open-sourcin...

It contains a lot more information about the problem it was originally created to solve, and potential other use cases.

[JonCoens]

I'm an engineer on the Haxl project and am really excited to launch this today. Ask me anything!

[X4]

Are you employing the ideas behind reactive programming? And can you explain the types of monads you used for what problem and why? I am writing a paper on Functional Reactive Programming and Haxl really made me curious. The paper (currently in german, but I'll translate it) proposes a new Hypothesis that tries to shred FRP in general, by showing a novel way that solves some of the problems automatically that naturally occur with FRP.

I am really interested in seeing how you solve problems for distributed systems with Haxl and how query sharding is handled etc..

I've wasted a whole day looking for Haxl online a few weeks ago, just to find out that it wasn't released yet. The release really makes me happy :)

[ciferkey]

How large has the haskell team at Facebook gotten (unless there isn't an official group and its on a project by project bases)?

[tveita]

I'm curious how this is executed.

Is it like a query engine, where you work with the entire query up-front, apply transforms and build a query plan?

Or is it more like an event loop, where you run as far as you can until the code blocks on IO, batch up and send all the pending IO requests, and run further when the tasks you're blocked on resolve?

[nusbit]

Is there any resources about why Facebook uses haskell? What's your experience?

[fiatjaf]

http://hackage.haskell.org/package/haxl

Why do Haskell libraries on Hackage doesn't come even with a single example, getting started, how to use, quick start, nothing, really, just function declarations? This scares Haskell newbies.

[m0nastic]

The "documentation" on Hackage is almost universally just the haddock-generated files (which is why it's mostly just function declarations and type signatures).

Most libraries list a "Home Page" that more often than not includes more useful documentation (Haxl's, for example, has the things you've mentioned).

I concur, that most of the time, the documentation on Hackage isn't really sufficient, but I've found that for the most part I just use it to find the homepage, and then go there to read the actual documentation.

I agree that it would be nice if everything was all in one place.

[tel]

It's definitely suboptimal, especially for newbies. The standard response (which I agree with completely) is that the types make API documentation so much more valuable that it's less commonly a concern for advanced users. So there's a bit of a squeaky wheel problem compared with, for instance, Ruby where docs are completely required to understand most libraries even slightly.

[imanaccount]

You mean "this library" not "Haskell libraries". Some libraries have real documentation (either on hackage or off) to go with the API reference.

http://hackage.haskell.org/package/pipes-4.1.2/docs/Pipes-Tu...

http://hackage.haskell.org/package/aeson-0.7.0.6/docs/Data-A...

[edofic]

Slides from Marlow from 9 months ago about Haxl and it's workings https://github.com/meiersi/HaskellerZ/blob/master/meetups/20...

[EGreg]

How do you guys batch requests in PHP? You don't, right? So this is an intermediate layer basically, and it sends requests every few millisecods and waits to batch things in between?

[nbm]

The systems described here isn't directly involved with any PHP code.

Like all Facebook services, they are communicated with over the Thrift RPC system, and may have PHP (or any other language) clients, and may talk to other services using Thrift (or occasionally other protocols), some of which may use PHP.

If you're asking a general question about batching requests in PHP, http://docs.hhvm.com/manual/en/hack.async.php may be informative.

[fnord_murks]

If you want to have similar query parallelization magic for PHP+SQL have a look at the standalone SQLTap service written by the guys from DaWanda.com (https://github.com/paulasmuth/sqltap).

[radnam]

Cool ! Are there any alternatives out there, especially to deal with fault tolerance especially when if we want to establish connection with 100's of varied databases?

[polskibus]

How does the functionality of Haxl differ from a mature ORM system? I'm thinking about .NET Entity Framework + LINQ in particular since it not only does the mapping but also assists in query generation, scheduling.

[mkesper]

It would be cool if you could transform those tables into HTML tables. They would look prettier (not nice JPG noise) and would also be more accessible.

[skyahead]

are there similar things for iOS?

[EGreg]

http://platform.qbix.com/guide/patterns ;-)

[ixmatus]

You obviously don't understand what Haxl is for.