A Graph-Based Firebase

[fhur]

Very interesting. I came to many similar conclusions completely independently, even attempted to build a typesafe in memory datalog in typescript.

I also came to the conclusion that just exposing datalog triples as a query language would never feel right and tried to expose a graphql like language that generated the datalog triples.

IMO react relay offers a great similar offering with their normalized cache. Relay has great DX too and can be totally type safe. To my knowledge datalog is way too dynamic for static analysis.

That being said, I would love to try Instant out. I'm really happy to see innovation in this area.

[stopachka]

Thank you for the kind words! Both Joe and I are around if you any feedback. In terms of type-safety, we're thinking of it as an added layer. I'm optimistic, that since InstaQL provides objects as the interface, we could have more idiomatic types down the road.

[habosa]

I used to work on Firebase. I also use Firebase heavily in my work and personal projects.

I am normally quite skeptical of “Firebase but X” projects because they often seem to be running head first into the issues that the Firebase team so skillfully avoided.

This isn’t one of those projects. The author of this essay clearly gets what you need to actually build a backend for a modern app that’s also simple enough to get started with. I’m very excited about this project.

[stopachka]

Thank you for the kind words.

[pezo1919]

Are transactions supported in that solution or in the mentioned Datalog related ecosystem? I have built my own similar reactive inmemory triple store with typescripts type safety, but then I realized I still need transactions which are a bit of a pain, because transactions are bundling the otherwise separete triplets, so the atomic, independent logic of triplets and related effects breaks a bit. (I am sure it's solvable.)

[jitl]

The example code uses a `transact` function. But it really depends on what you mean by “transaction”. We don’t use a triple store at Notion, but we do use an abstraction that ensures a collection of operations either all succeed or all fail. We don’t support “interactive” transaction, where you can read, modify, write as an atomic group. This just isn’t desirable in a multiplayer or offline system - in cases we need that kind of consistency we use a normal HTTP API which is online-only.

[stopachka]

As jitl points out, in multiplayer settings, what you need is some way to commit a series of transactions all-together or not at all. We support this.

In the case where you want to read the database inside your transaction, we take inspiration from Datomic. Datomic runs all mutations in one high-memory box. You can provide functions that run in that box. This way, you can guarantee that the reads inside your transaction have the latest value. There's a lot of UX to figure out there, and this would be something to try to avoid in an offline-available setting.

[j-pb]

The basic intuitions they have are correct, but if feels like there is a serious lack of or disregard for the theoretical background needed to talk about this stuff properly.

> (pull db '[* {:team/task [* {:task/owner [*]}]}] team-id)

That's not Datalog. It's kind of a mix between a conjunctive query and a regular path query.

Datalog isn't even really a query language. It's a class of query languages with a specific expressive power.

Whereas SQL is essentially conjunctive queries with non-stratified Negation, Datalog is recursive conjunctive queries with no or only stratified negation.

Datalog also has nothing to do with triples, they are two completely orthogonal concepts.

I've been slaving away in this very space for years and this post heavily reminds my of my initial hubris. Building something truly foundational that can be universally implemented and perform in all potential target languages from Javascript, to WASM via e.g. Zig and Rust, while at the same time being both conceptually simple, and easy to implement, is really really really difficult with a lot of pain lurking in the details.

[blain_the_train]

do you write about your findings and ideas? i would be interested to read more.

[kevmo314]

> These triples say that the Layer with id 1 has a fontSize 20 and backgroundColor blue. Since they are different rows, there’s no conflict.

This sounds a lot like Bigtable (https://cloud.google.com/bigtable), which also does last-write-wins conflict resolution layer. So this is adding a GraphQL + frontend layer to it?

[stopachka]

I'm not as familiar with Bigtable's data model. Afaik they don't use a triple-store like system, but their data model does look interesting. [^1]

The concept you have is on point though. You can think that we've moved a graph-database over to the frontend, and introduces a GraphQL-like language for it.

[^1]: https://static.googleusercontent.com/media/research.google.c...

[jinjin2]

Another product that does all this (relations, reactivity, offline, permissions, sync, etc…) really well, is Realm.

It is beyond awesome for building mobile apps, but mobile only. Annoyingly they have never released a web version. No idea why.

I would still look into it for ideas. It is super powerful and a joy to work with. I would love to see something like that working in the browser.

[dustingetz]

One possible solution to the "sync" layer (how do you sync the local db shard with the global consensus) looks like this: https://www.researchgate.net/publication/359578461_Continuou... (2022)

also need to fit whole view updates into 16ms frame budget (not just one query but every query on the page that is impacted by a change as well as downstream reactive views). At some tipping point it can be faster to move relational queries to the cloud (sacrificing local first), and treat local first as an edge case (not all page components need to be live in offline mode - depending on the app, you may just need document edits not relations)

[stopachka]

The research paper looks intriguing. I'll look into it. I am not as convinced about moving queries to the cloud, when it comes to the north stars of Figma / Linear / Notion. It would be hard to get the same kind of UX.

[tlarkworthy]

At Firebase we sometimes pondered the feasibility of a SQL version, but the semantics of SQL seem littered with semantic footguns that don't lend themselves to offline, secure, scalable and event driven distributed applications. We know everybody wanted better query expressivity but delivering that in a mobile friendly, clienside, secure package was very difficult to see the path to.

tripletstore/datalog actually seems like a decent compromise between SQL and no-SQL that could actually work out! Awesome idea!

[stopachka]

I'm encouraged reading this. Thank you.

[joe_fishfish]

Does Instant have any plans for iOS and Android SDKs? One of the biggest wins with Firebase is that it's really easy to integrate across platforms.

[stopachka]

Yes! Right now one of our users implemented an RN integration. We'd love to support iOS and Android soon.

[ar7hur]

I would really use this for many of my projects! For project A, I used Firebase which was great initially but eventually ran into the limitations well described in Stopa's essay. For project B, I decided to write everything from scratch but the amount of work it required is crazy, and of course the result in quite brittle.

[stopachka]

Thrilled to hear this, thank you :).

[fernandohur]

Another project with similar goals in mind: https://paulbutler.org/2020/the-webassembly-app-gap/

Also backed by YC.

[paulgb]

Thanks for the mention!

For some context, that blog post discusses what I saw as missing pieces of the stack circa 2020. The startup I’m building (https://driftingin.space) is roughly the “lambda for websockets” part, whereas Instant is closer to the “generalized CRDT data layer part”.

[jitl]

Is this blog post a company? What company?

[mizzao]

This looks like an evolution of Meteor, doesn't it?

[stopachka]

Meteor certainly inspires us!

[antidnan]

Very cool! What're the differences between this and something like Replicache?

[stopachka]

I haven't looked too deeply at Replicache, but the two differences as I understand it:

1. Replicache is a layer you add over an existing backend. Instant handles the backend. 2. The expose a key-value store. We expose a graph store.

Both 1&2 have pros and cons each way. I think we're inspired by the same problems, and their docs are really well done.

[waveywaves]

Are there any plans to open source instant in the future ?

[stopachka]

We like the open-core mode. We have definite plans to open-source the client. Still think the best way to do this for the server.

[revskill]

I use Hasura for this purpose. With some hooks, you can achive offline mode, too.

You need a bit tricky hacks to use Hasura permission system the way you want, though.

[stopachka]

Thanks for taking a look at the essay. As you mentioned, indeed it's possible if you store all commands, and sync with IndexedDB. The edge cases get quite complicated though. We're optimistic that a local layer that handles transactions and sync could makes things a lot easier.

[alexashka]

Hasura does not offer syncing.

Syncing is hard :)

[kennymeyers]

Tell me more about this!