tomconnors's comments

Cool stuff as usual, Anders. One of the nice things about running a networked DB is that it makes redeploying the application a bit simpler. You can spin up a new EC2 instance or whatever and once it's online kill the old one. That gets 0 or close to 0 downtime. If the DB is on the same instance, replacing it requires loading up the DB onto the new instance, which seems more error prone than just restarting the app on the original instance, but in my experience that typically incurs downtime or some complicated handoff logic. Have you had to deal with anything like that running sqlite in prd?

I recently used c.a.flow for a program that reads files from s3, massages the data a bit, and writes to a database. The code came out very easy to understand. The challenges were:

- getting back pressure right. You need to tune the buffers for the input chans correctly, which takes some thinking, and in my case, some blowing up in prd.

- flow monitor (the UI for debugging flows) can't handle large process states, which makes it pretty much useless for my program.

- understanding the flow as a whole (rather than specific processes, which are easy to understand in isolation). For example, answering questions like "why hasn't this process received a message for a while?" was tricky.

It's always frustrating to me that the order matters at all. I understand SQL can be complex to parse, but surely we are smart enough to come up with a parser that doesn't care whether FROM is before or after SELECT?

HoneySQL lets us define queries with maps, like {:select [:col1 :col2] :from :table}, and turns that into SQL. In a better world, SQL would be structured data like HoneySQL, and the strange SQL syntax we know and love would be a layer on top of that, or wouldn't exist.

The band Protest the Hero has a song about this event: https://www.youtube.com/watch?v=sNcGbAQgZIg

This post could be summarized as "write as much of your project's tooling as you can in the project's main programming language and the project's main programming language should be Clojure" and I agree wholeheartedly.

Long ago, when Datomic was only recently released, I saw blog posts (not from Cognitect) claiming that Datomic's transaction log could be used to create a subscription system for arbitrary datalog queries. Basically "tell me when the result of this query changes". In practice, I've yet to find a general way to do that, except for wildly inefficient stuff like keeping query results in memory, requerying on each transaction, and comparing. How does 3DF handle a query like this?:

  [:find (max ?num) :where [_ :attr ?num]]

It seems like Datomic's transaction log can only tell us that the result of this query might have changed. Does differential dataflow solve that problem?

I've been guilty of this. Normally I like to blame CSS for all of my personal failures, but I think the definition of "overflow" is intuitive enough. I was hesitant to turn on visible scrollbars system-wide because I thought it might be ugly. I did and it's not, so I second the author's suggestion to do that.

Fair enough, hitting that bug would have pissed me off too.

On your last point, I agree that it still has a way to go. It's good for some (many?) production use cases now, as Nubank's success demonstrates, and hopefully with Nubank's resources it'll start to live up more to its promise.

This seems like an excessively uncharitable read of the situation. I've never used Nubank's software, but I have used (on-prem) Datomic and I certainly wouldn't say it has bugs everywhere. In fact, in my (admittedly low-volume and simple) usage of the system I haven't come across any bugs I can remember. Calling Cognitect a "burnout" company is inaccurate and rude.

I agree with you that the Datomic cloud stuff comes across as being frighteningly complex. I think they probably just need to work on the documentation, like making it more obvious what the differences and tradeoffs are between the deployment scenarios.

Did you inherit a Datomic system that was previously developed by a small team or a small company? Because inheriting a system that's hard to understand and change transcends languages and databases. It is the tie that binds us all as software developers.

It sounds like Nubank might have the biggest Datomic installation in the world (2000 servers). This is exciting news for people like me who think Datomic is the most interesting database out there. Putting more resources behind Datomic might finally allow it to go mainstream.

A Picardy third can be heard at the end of Avengers Infinity War as Thanos watches the sun rise on a grateful universe.

Everything Distributed Masonry does is very interesting. Wish I had more excuses to use your stuff at work.

Storing all data forever in a single source of truth is awesome until regulation like GDPR comes along. Do you have plans to support excision or is your guidance on personal data to avoid putting it into a system like Kafka/Pyrostore?