Scaling to 1M active GraphQL subscriptions on Postgres

[aidos]

Just another little shout out for Hasura here. It’s a really lovely system to use and the team behind it are awesome to deal with. Super friendly and helpful.

We evaluated a lot of stuff recently before settling on Hasura. Prisma, aws appsync, postgraphile, roll our own socketio backend etc. I had a few hesitations about Hasura. In particular I was hoping to lean on Postgres RLS for the authorisation, and to be able to customise the graphql endpoints a bit more. After hearing the rational from the guys about how their system is better than RLS and seeing the patterns they have for implementing the logic I’m really happy we made the leap.

We’re still in early days, but it’s been really solid and a real pleasure to use.

(No affiliation, but happily paying for their support licence)

[ndarilek]

Out of curiosity, what is the rationalle for abandoning RLS?

In a previous project, I used Postgraphile and enjoyed it. Sure, RLS and PLPGSQL were a bit awkward, but I knew they were well-tested and had lots of eyes on them, so I felt comfortable that if an RLS policy blocked access to a row, then that particular role wasn't getting access. I also enjoyed working with a text-based format that more clearly fit within my other workflows of, well, working with other text-based formats. :)

Another part of it is that I'm blind, and Hasura seems to make doing things outside of its web interface somewhat painful. Sure I can write a YAML migration by hand, but the YAML migration format seems more machine-friendly than user-friendly. Yesterday I needed to create a function, and the SQL web interface wasn't showing me the line number on which my function had an error. Ultimately I dropped to the psql command line, cut-and-pasted the function in manually, and right away got back the line number and fixed the issue.

Please don't get me wrong. I'm not trying to hate on Hasura. But the fact that I can't just drop to a non-YAML text-based format and throw down a few checks to secure my tables has been an endless source of frustration for me. So if there is a non-NIH reason for abandoning RLS in favor of a separate security system, then maybe knowing that might help me be a bit less annoyed. :)

[marton78]

Hasura sure looks interesting, but two things make e hesitant:

1. The overly complicated schema it generates. For example, given an integer id, one can (and must) use various operators to query that id, e.g. `user(id: { _eq(1) })` rather than just `user(id: 1)`, where other operators are `_gt`, `_lt`, etc. To make this possible, hasura defines a comparison object, which results in a very long schema. However, often the only comparison that makes sense is equality, e.g. for IDs, so this should be optional.

2. More importantly, using Hasura establishes coupling between the DB and the GraphQL schemas. What if at a later stage one decides to remove information from the DB and move it to an external source? GraphQL was designed to hide such implementation details. With Hasura this is no longer possible.

[k__]

Does AWS AppSync even play the same game as Hasura?

[Thaxll]

I was curious about the code so I went to check it, it's written in Haskell, never seen Haskell before, it's really hard to follow for someone not used to it: https://github.com/hasura/graphql-engine/blob/master/server/...

Not sure how people enjoy FP.

[lpage]

Haskell and ML languages look like symbol soup when you're used to mainstream/C style languages. Once you understand some of the basic syntax they become quite readable and easy to reason about as a series of transformations.

Step one, which applies to most functional languages, is knowing how to read a Hindley-Milner type signature [1].

Armed with that you can often gain a surface level understanding of what some function does - and what the program does in aggregate - just by reading type signatures and seeing what calls what.

Hoogle [2] is a good resource for figuring out Haskell syntax, like what's up with all of those $ signs.

Converting OCaml to ReasonML syntax with sketch or reason-tools [3] makes OCaml less foreign.

With basic knowledge of Haskell and ML syntax you'll find many FP languages readable.

[1] https://drboolean.gitbooks.io/mostly-adequate-guide-old/cont...

[2] https://hoogle.haskell.org/

[3] https://reasonml.github.io/docs/en/extra-goodies

[robotmay]

I've said it a few times on HN in the past but Haskell is by far and away the hardest language I've tried to learn. I have no formal grounding in maths/CS and I think a lot of the people who enjoy it really struggle to communicate with those of us who don't come from that sort of background.

In contrast I picked up Rust and Elixir in a couple of weeks apiece, so I don't feel like I'm particularly dense; I just suspect that it needs a certain way of thinking to get the best from Haskell :)

[coldtea]

>it's really hard to follow for someone not used to it

So like anything else?

>Not sure how people enjoy FP.

By virtue of actually having tried it?

[mrkurt]

It's worth writing something in an ML or Haskell sometime. Functional programming is great, and it makes my code in things like JavaScript much more robust.

[tirumaraiselvan]

> Not sure how people enjoy FP.

See The Blub Paradox [0]

0 : http://www.paulgraham.com/avg.html

[jose_zap]

Anything you are unfamiliar with will obviously look alien and difficult.

I actually find their code quite readable, but that’s because I have experience writing Haskell.

[hota_mazi]

And FP developers in general tend to push conciseness to an extreme, especially with this kind of shenanigan (from the very source you linked):

    import qualified Data.ByteString.Lazy as BL

It's ironic that after all this obfuscation, the author of the source still finds it useful to do ASCII art and align everything neatly. Talk about misplaced effort.

[londons_explore]

> we’ve currently fallen back to interval based polling to refetch queries.

This fancy "subscribe to events when the results of this query change" system boils down to just polling the database...

[aidos]

Sometimes the simple option is the best. Having said that, it’s not that simple, for them. I believe it relies on being able to fetch and dissect different results for different clients in the same query (all while applying their own form of RLS).

[josephg]

Yep. I’d love to see an implementation of this in the wild built on top of logical decoding:

https://www.postgresql.org/docs/9.6/logicaldecoding-explanat...

[GorgeRonde]

    Future work:
    Reduce load on Postgres by:
    - Mapping events to active live queries
    - Incremental computation of result set

Fake it till you make it.

Meanwhile: https://opensource.janestreet.com/incremental/

[themccallister]

I’ve been using Hasura for around 3 months now. It has changed the velocity and way I approach our projects. I have a small team that works as a startup inside of a larger organization and this was crucial to delivering some key projects... plus very little code. The event trigger really help reduce our costs as well, using the Serverless.com framework with Lambda Go has been a huge success in our organization!

Thank you to the Hasura team for all you do and in turn allow me and my team to do!

[thegoleffect]

When I first saw Hasura, it was AGPL licensed which makes it unusable in a lot of use cases so we passed on it. Looks like they converted it to Apache license two months ago. I suppose it is worth another look, I'm glad they made the change.

[dmitryminkovsky]

Hasura is such a promising application platform. I tried it a few months ago and was really pleased. In a way it kind of felt like RethinkDB, except GraphQL and Postgres. Was very easy to turn on with Docker Compose and play around. Looking forward to 1.0!

[z3t4]

I guessed it would use the pub/sub capabilities of postgres. But it seems from the readme that it use polling, although using the same query which allows cache optimizations. The naive implementation can often be surprisingly effective.

[rishav_sharan]

As an Indian engineer, Hasura is one the few indian startups that i really am proud of. They are innovative, distinct, product focused and super young. The only other company which made me feel like this was Notion Ink, but those guys seem to have faded away.

I wish Hasura the best and would be cheering for them from the sidelines

[gunnarmorling]

I'm impressed by the numbers, but I'm having some doubts about the general idea of exposing your database via GraphQL.

It seems to expose your internal data structures, and any change to those will immediatly impact that public GraphQL API. Also it seems that this approach would only work well for applications without any business logic, or with business logic solely implemented in the database. At least the applications I've been working on wouldn't fall into this category.

But in any case, impressive work and demos!

[aleksei]

You aren't limited to a single database though, so you could stream your main database to a public facing one with a more stable structure.

But you'd have to handle changes through another API, or at least route them differently, and then worry about consistency so probably not ideal for all business cases.

[agrippanux]

Hasura is fantastic. I am using it for several projects atm. Coming from Prisma it's refreshingly easy to use and well thought out. The security/ACL model is esp. nice and easy to grok.

[pier25]

We evaluated Hasura and the only problem we found is the integration with our own authorization system. Otherwise Hasura is quite frankly amazing.

Our auth system has many roles per user, then each role has its permissions, pretty common setup. The problem is that Hasura expects a single default role per request to then evaluate against its permissions. The Hasura team has been looking into accepting an array of roles and merging permissions and whatnot, but AFAIK this hasn't been solved.

If you start with Hasura from scratch this is not really a problem, it happened to us because we had to figure out how to integrate Hasura with our current permissions.

[contrasti]

> the only problem we found is the integration with our own authorization system

But integration with external authorization systems is an extremely basic requirement for every piece of software, certainly on the top 5 on the requirements list for every serious solution - I can not understand how did you forget about this?

[lubesGordi]

I'm curious about the underlying push tech. Is Hasura using firebase to handle that? Then firebase relies on different push services depending on the client, like Apple's push service for iOS and http/2's push for web (which is supported by what percentage of browsers)?

[pier25]

Firebase? No, not at all. It's all Hasura and Postgres.

[StreamBright]

Nice scaling numbers. Can you use Hasura without Kubernetes or Docker?

[tango12]

You can compile the binary yourself, but otherwise docker is the way to go.

What’re you looking at?

[jazoom]

I know there's an issue about this but it doesn't really indicate Hasura's intentions regarding it:

Is supporting CockroachDB something Hasura is actively working on?

[tango12]

It’s on the cards!

Don’t want to spread ourselves too thin too early. Also working with the yugabyte folks closely to have Hasura supported with their 2.0 release.

https://twitter.com/karthikr/status/1128003465370693636?s=21

[machbio]

Write Ahead Logs would have been amazing - hate that AWS RDS does not support WAL - could have saved so much time for some of the applications we build.

[gunnarmorling]

WDYM by "RDS does not support WAL"? I wouldn't know how Postgres works without them, and they are accessible using logical decoding. wal2json is supported on RDS and used by tools such as Debezium for streaming data changes into Apache Kafka (Disclaimer: I work on Debezium).

[he0001]

How does it handle transactions and locking? It seem to support multiple databases, does it handle distributed transactions?

[rj5]

I can’t wait to try this out!