I have a few of these posts I've written coming out over the next few months that I want people to discuss. Would you prefer I add a disclaimer at the top? Easy enough to add
Keep in mind for what reason? The content doesn't change based on this knowledge. It either stands up for what it is or it doesn't. It wouldn't somehow be better if it were written by a random Joe user instead: It would be the exact same work. Attributing qualities of the author onto the work is a logical fallacy.
Use that precious mind space for understanding basic logic instead.
While I am a fan of TigerBeetle’s general stance on correctness, coding practices, and their desire to hyper-specialize, I have some critiques on the post.
I think the paragraph about multi-node is a bit misleading. Contrary to what cloud native folk will tell you, a single beefy DB, well-tuned and with a connection pooler, can serve a dizzying amount of QPS just fine. At a former employer, during a maintenance period, I once accidentally had all traffic pointed to our single MySQL 8 RDS instance, instead of sharing it between its read replicas. That was somewhere around 80-90K QPS, and it didn’t care at all. It wasn’t even a giant instance - r6i.12xlarge - we just had a decent schema, mostly sane queries, and good tuning on both ProxySQL and MySQL. At peak, that writer and two .8xlarge read replicas handled 120K QPS without blinking.
A DB hosted on a server with node-local NVMe (you know, what used to be normal) will likely hit CPU limits before you saturate its I/O capabilities.
For redundancy, all RDBMS designed for networked activity have some form of failover / hot standby capability.
My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
None of this should be taken as belittling their accomplishment; I remain extremely impressed by their product.
That a traditional modern RDBMS is very fast is not really useful here because the use case of Tigerbeetle includes work loads with high contention. Under that kind of load, in their own tests you can see in a recent conference talk, you can see the throughput drops dramatically in clusters where many transactions affect a single account.
>we just had a decent schema
Glad someone mentioned it. In the dev ORM culture now days it is all about how to make the db look like the code. DB is essentially reduced to a object store. Often a proper decent schema can handle most of these load. You can always throw in redis layer later if your traffic is really getting to require that level of optimization.
> My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
Isn't that the point? They're saying to separate out transactions workload from other workloads. They're not saying they'll replace your OLGP db, you remove transactionally important data into another db.
I really like the work Joran and his team have done around DST, distributed systems awareness, and performance practices. I especially love the craziness around no dependencies (well, could you consider an underlying OS a dependecy?).
But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago".
In their cited https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.
You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention.
Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day.
TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.
We were considering TigerBeetle, but found blockers:
* We use Cloudflare Workers. TigerBeetle client app is not supported. It might work using Cloudflare Containers, but then the reason we use Cloudflare is for the Workers. --> https://github.com/tigerbeetle/tigerbeetle/issues/3177
You should chat to our solutions team, who'd be able to help you solve auth etc. at the logical level with end-to-end encryption, something we typically do for our customers.
Wait, what? A database in 2025 doesn't support any kind of auth? A financial database? WTF?
C'mon folks, the least you can do is put a guide for adding an auth proxy or auth layer on your site.
Particularly since you don't use HTTP (cant easily tell from the docs, I'm assuming), then folks are going to be left wondering: "well how the hell do I add an auth proxy without HTTP" and just put it on the open internet...
> In less than a decade, the world has become at least three orders of magnitude more transactional. And yet the SQL databases we still use to power this are 20-30 years old. Can they hold up?
Errr yes. Without much sweat really.
Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.
The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL.
In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked.
Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true.
Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago.
So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk.
Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s.
I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.
I disagree. If we are talking about distributed systems in which we have N different databases, then distributed transactions are left as an exercise to the reader (that’s why we have things like Sagas).
Within a single machine, yeah, relational dbs still work like a charm.
- Jepsen claimed that FDB has more rigorous testing than they could do.
- New programming language, Flow, for testing.
You probably could solve the same problems with FDB, but TigerBeetle I imagine is more optimized for its use case (I would hope...).
AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
The only reason is Apple. They liked the product that was released in 2013 so much they bought the whole company, and all other FoundationDB users were abandoned and were forced to drop it.
Who would trust a database that can be yanked out of you at any moment? Though a lot of products have license terms like this only a handful were ever discontinued so abruptly. It's under Apache license now but the trust is not coming back.
And a more serious comment, to separate it from the silly one below:
Interesting that they didn't release it with an SQL client, is there no way to make it compatible? Even with extensions to SQL, I imagine it would be great for a lot of use cases.
> AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
I started writing this comment:
> It seems interesting, but considering what it's for, why aren't the hyperscalers using it?
And while writing it I started searching for FoundationDB and found this:
Pretty much agree, I honestly think that TB team should invest some more effort on evangelizing the double-entry model for non-financial scenarios, like managing stocks, booking a show ticket etc. The 1000x API improvement has been done, now people must know how to use it
It's worth noting that the breathless worship of everything about TigerBeetle in this post is likely related to the fact that the author is an investor in TigerBeetle.[0]
Yes, this is by design. SQL is a great general purpose query language for read-heavy variable-length string workloads, but TigerBeetle optimizes for write-heavy transaction processing workloads (essentially debit/credit with fixed-size integers) and specifically with power law contention, which kills SQL row locks.
I spoke about this specific design decision in depth at Systems Distributed this year:
It helps to know what kind of data TigerBeetle handles. The data committed by its transactions is an immutable Transfer of id:128-bit, debit_account_id:128-bit, credit_account_id:128-bit, amount:128-bit, ledger:128-bit, code:16-bit, flags:bitfield, timestamp:64-bit, user_data_128, user_data_64, user_data_32.
Transactions atomically process one or more Transfers, keeping Account balances correct. Accounts are also records, their core fields (debits_posted, credits_posted, etc).
This gives a good idea of what TigerBeetle might be good for and what it might not be. For anything where latency/throughput and accuracy really, really matters, it could be worth the effort to make your problem fit.
>Most teams write code fast. TigerBeetle tries to write code slow. Most teams treat testing as a necessary evil. TigerBeetle is built entirely on Deterministic Simulation Testing (DST). Most teams build their software on top of loads of other software. TigerBeetle has zero dependencies.
I believe that was more like the norm 25+ years ago. Before Google and Facebook brought in the move fast and break things mentality across software industry.
I work at a bank with old old monstrous sql queries.
I thought I can make a use of tigerbeetle to simplify the system. But sadly, I just couldn't figure out how to make it work. Transaction requires lot of business logics but I couldn't convert that to make it work with combination of RDBMS + tigerbeetle.
I wish there were some realworld example that I can get insight using tigerbeetle.
Looking at the Zig vs. Rust angle — an interesting takeaway from the article is that Zig makes it considerably easier to build statically allocated applications than Rust.
For all of Rust’s talk about “safety”, assuming the above assertion is true, than perhaps Zig is a better choice to augment or replace C in embedded, safety-critical software engineering, where static memory allocation is a core principle.
It says: “TigerBeetle uses a single core by design and uses a single leader node to process events. Adding more nodes can therefore increase reliability, but not throughput.”
How does this work with multi-region? Does this mean that regardless of where in the world your users might live, they need to make a request to the leader node for a write to occur?
The way I understand it yes. Latency will be dictated by going to the leader node and replication to enough followers. The CAP theorem dictates that if you want strong consistency the way they offer you have to talk to a quorum of nodes IIRC N/2 + 1. Paxos and Raft also work that way.
Why hasn't anyone mentioned the game they made? It looks fantastic. The concept of letting players try to crash a system to demonstrate its robustness is brilliant.
TigerBeetle looks interesting. A whole networked database using statically allocated memory. I usually use that technique for small keyboard-shortcut triggered applications.
One question in case folks who work there see this:
This is the most technically impressive zig project I've seen so far. Do you have a blog post detailing your perspective on zig? i.e what design decisions of ziglang helped you in massive way, what were the warts, and in general any other thoughts too?
this has such great marketing zingers, I probably didn't know I needed TigerBeetle for something. I don't know yet, but I promise you, I'll find something to use it for.
Just to be a little pedantic for a second, isn't TigerBeetle a database management system (DBMS) whereas a TigerBeetle database would be actual data and metadata managed by TigerBeetle?
[+] [-] blenderob|5 months ago|reply
[+] [-] itunpredictable|5 months ago|reply
[+] [-] sroussey|5 months ago|reply
It’s on said investment company’s website under the tag “Portfolio Spotlight”.
[+] [-] 9rx|5 months ago|reply
Use that precious mind space for understanding basic logic instead.
[+] [-] noir_lord|5 months ago|reply
[+] [-] sgarland|5 months ago|reply
I think the paragraph about multi-node is a bit misleading. Contrary to what cloud native folk will tell you, a single beefy DB, well-tuned and with a connection pooler, can serve a dizzying amount of QPS just fine. At a former employer, during a maintenance period, I once accidentally had all traffic pointed to our single MySQL 8 RDS instance, instead of sharing it between its read replicas. That was somewhere around 80-90K QPS, and it didn’t care at all. It wasn’t even a giant instance - r6i.12xlarge - we just had a decent schema, mostly sane queries, and good tuning on both ProxySQL and MySQL. At peak, that writer and two .8xlarge read replicas handled 120K QPS without blinking.
A DB hosted on a server with node-local NVMe (you know, what used to be normal) will likely hit CPU limits before you saturate its I/O capabilities.
For redundancy, all RDBMS designed for networked activity have some form of failover / hot standby capability.
My other mild criticism is in the discussion on TigerBeetle’s consensus: yes, it seems quite clever and has no other dependencies, but it’s also not trying to deal with large rows. When you can fit 8,190 transactions into a 1 MiB packet that takes a single trip to be delivered, you can probably manage what would be impossible for a traditional RDBMS.
None of this should be taken as belittling their accomplishment; I remain extremely impressed by their product.
[+] [-] justinhj|5 months ago|reply
edit: from the horses mouth is better https://news.ycombinator.com/item?id=45437046
[+] [-] zenmac|5 months ago|reply
[+] [-] parthdesai|5 months ago|reply
Isn't that the point? They're saying to separate out transactions workload from other workloads. They're not saying they'll replace your OLGP db, you remove transactionally important data into another db.
It's something similar that we see with another db: https://turbopuffer.com/
[+] [-] dangoodmanUT|5 months ago|reply
But I've always felt the way they treat normal OLTP (they call OLGP) seems unfair. For example, comparisons using clear sub-optimal interactive SQL transactions for financial workloads, like locking rows rather than using condition checks at commit time, because "that's how OLTP was intended to be used when it was designed ~50(?) years ago".
In their cited https://tigerbeetle.com/#performance the lowest the slider can go is 1% contention. Do you think Stripe has 1% contention directly on an OLTP DB? Definitely not.
You can build systems that _expect contention_, and elegantly handle it at REALLY high throughput. These systems protect the DB from contention, so you can continue to scale. From talking to folks working on these systems, I roughly know the kinds of transactional (financial) throughput of DBs like Stripe's and other systems - they have _many_ more zeros behind them than their performance comparison page proposes they could possibly have at even 0.01% contention.
Their marketing largely ignores this fact, and treats everyone like they just slam the DB with junior engineer-designed interactive transactions. Most developers (I hope) are smarter than that if they're working at a payments company. There's even the title "payments engineer" for the kind of person that's thinks about scaling contention and correctness all day.
TigerBeetle is great, but I find the pattern of being quite misleading about other OLTPs off putting.
[+] [-] wiradikusuma|5 months ago|reply
* We use Cloudflare Workers. TigerBeetle client app is not supported. It might work using Cloudflare Containers, but then the reason we use Cloudflare is for the Workers. --> https://github.com/tigerbeetle/tigerbeetle/issues/3177
* TigerBeetle doesn't support any auth. It means the containing server (e.g. a VPS) must restrict by IP. Problem is, serverless doesn't have fixed IP. --> https://github.com/tigerbeetle/tigerbeetle/issues/3073
[+] [-] vjerancrnjak|5 months ago|reply
* spawning 1000 workers all opening a connection to a db,
* solved by service/proxy in front of db,
* proxy knows how to reach db anyway, let's do private network and not care about auth
[+] [-] cakealert|5 months ago|reply
[+] [-] jorangreef|5 months ago|reply
[+] [-] caust1c|5 months ago|reply
C'mon folks, the least you can do is put a guide for adding an auth proxy or auth layer on your site.
Particularly since you don't use HTTP (cant easily tell from the docs, I'm assuming), then folks are going to be left wondering: "well how the hell do I add an auth proxy without HTTP" and just put it on the open internet...
[+] [-] justinclift|5 months ago|reply
Errr yes. Without much sweat really.
Just because something started ~30 years ago doesn't mean it hasn't updated with the times, and doesn't mean it was built on bad foundations.
[+] [-] jorangreef|5 months ago|reply
Without much sweat for general purpose workloads.
But transaction processing tends to have power law contention that kills SQL row locks (cf. Amdahl’s Law).
We put a contention calculator on our homepage to show the theoretical best case limits and they’re lower than one might think: https://tigerbeetle.com/#general-purpose-databases-have-an-o...
[+] [-] Xelbair|5 months ago|reply
It still holds up basically whole of internet.
in most cases SQL is good enough for 90% of workloads.
[+] [-] yomismoaqui|5 months ago|reply
Sometimes I feel like we software engineers have the worst memory of any engineers.
[+] [-] cmrdporcupine|5 months ago|reply
The relational model has shown itself to be exactly the flexible and powerful model that Codd said it was, even in its relatively-debased from in SQL.
In fact the potential of it as a universal and flexible data model that abstracts away storage still remains to be fully unlocked.
Now as for existing SQL databases, yes, many of them were built on foundational assumptions about the nature of memory and secondary storage that no longer hold true.
Many of us in this industry still have our heads stuck in the past world of spinny spinny magnetic disks on platters with heads and cylinders and grinding noises. Real world hardware has moved on. We have a couple orders of magnitude higher IOPS generally available than we did just 10-15 years ago.
So I'm excited about products like CedarDB and others (Umbra before it, etc) that are being built on a foundation from day one of managing with hybrid in-memory/disk.
Throwing out SQL is not really the recipe for "performance" and the lessons of separating the storage system from the data model is still key, since the 1960s.
I am willing to grant that a specialized transaction/ledger system like TigerBeetle might have its place as an optimization strategy in a specific industry/domain, but we should not make the mistake of characterizing this as a general problem with the relational model and data storage broadly.
[+] [-] dakiol|5 months ago|reply
Within a single machine, yeah, relational dbs still work like a charm.
[+] [-] pkphilip|5 months ago|reply
[+] [-] websiteapi|5 months ago|reply
- Slow code writing.
- DST
- No dependencies
- Distributed by default in prod
- Clock fault tolerance with optimistic locking
- Jepsen claimed that FDB has more rigorous testing than they could do.
- New programming language, Flow, for testing.
You probably could solve the same problems with FDB, but TigerBeetle I imagine is more optimized for its use case (I would hope...).
AFAIK - the only reason FDB isn't massively popular is because no one has bothered to write good layers on top. I do know of a few folks writing a SQS, DynamoDB and SQLite layers.
[+] [-] andrewl-hn|5 months ago|reply
The only reason is Apple. They liked the product that was released in 2013 so much they bought the whole company, and all other FoundationDB users were abandoned and were forced to drop it.
Who would trust a database that can be yanked out of you at any moment? Though a lot of products have license terms like this only a handful were ever discontinued so abruptly. It's under Apache license now but the trust is not coming back.
[+] [-] oblio|5 months ago|reply
Interesting that they didn't release it with an SQL client, is there no way to make it compatible? Even with extensions to SQL, I imagine it would be great for a lot of use cases.
Edit: ah, it's more of a key-value store.
[+] [-] itunpredictable|5 months ago|reply
[+] [-] oblio|5 months ago|reply
I started writing this comment:
> It seems interesting, but considering what it's for, why aren't the hyperscalers using it?
And while writing it I started searching for FoundationDB and found this:
> https://github.com/<<apple>>/foundationdb
Ah, all right :-p
[+] [-] ksec|5 months ago|reply
[+] [-] dangoodmanUT|5 months ago|reply
[+] [-] vrnvu|5 months ago|reply
- single entry point, near-zero deps
- ci locally and tested, one command to runs tests, coverage, lint etc
- property/snapshot/swarm testing, I love writing simulations now and letting the assertions crash
- fast/slow split + everything is deterministic with a seed
- explicit upper bounds + pool of resources. I still dynamically allocate but it makes code simpler to reason about
Thanks to the TB team for the videos and docs they been putting out lately.
[+] [-] jorangreef|5 months ago|reply
[+] [-] flir|5 months ago|reply
Never understood why we turn those off. An assert failing in prod is an assert that I desperately want to know about.
(That "never understood" was rhetorical).
[+] [-] SchwKatze|5 months ago|reply
[+] [-] mtlynch|5 months ago|reply
[0] https://www.amplifypartners.com/portfolio/company
[+] [-] BrentOzar|5 months ago|reply
[+] [-] jorangreef|5 months ago|reply
Yes, this is by design. SQL is a great general purpose query language for read-heavy variable-length string workloads, but TigerBeetle optimizes for write-heavy transaction processing workloads (essentially debit/credit with fixed-size integers) and specifically with power law contention, which kills SQL row locks.
I spoke about this specific design decision in depth at Systems Distributed this year:
1000x - https://m.youtube.com/watch?v=yKgfk8lTQuE
[+] [-] karmakaze|5 months ago|reply
Transactions atomically process one or more Transfers, keeping Account balances correct. Accounts are also records, their core fields (debits_posted, credits_posted, etc).
This gives a good idea of what TigerBeetle might be good for and what it might not be. For anything where latency/throughput and accuracy really, really matters, it could be worth the effort to make your problem fit.
[+] [-] ksec|5 months ago|reply
I believe that was more like the norm 25+ years ago. Before Google and Facebook brought in the move fast and break things mentality across software industry.
I hope TigerBeetle gets more recognition. Worth reading its Jepsen report as well. https://news.ycombinator.com/item?id=44199592
[+] [-] xnzm1001|5 months ago|reply
[+] [-] jorangreef|5 months ago|reply
We typically hear from companies that TigerBeetle is the easier part of their stack. But to be fair, they may have a cleaner architecture.
Did you contact our solutions team for assistance with your business logic modelling? If not, please get in touch! [email protected]
[+] [-] theYipster|5 months ago|reply
For all of Rust’s talk about “safety”, assuming the above assertion is true, than perhaps Zig is a better choice to augment or replace C in embedded, safety-critical software engineering, where static memory allocation is a core principle.
[+] [-] kamranjon|5 months ago|reply
It says: “TigerBeetle uses a single core by design and uses a single leader node to process events. Adding more nodes can therefore increase reliability, but not throughput.”
How does this work with multi-region? Does this mean that regardless of where in the world your users might live, they need to make a request to the leader node for a write to occur?
Is distribution purely for redundancy?
[+] [-] Voultapher|5 months ago|reply
Apparently there is a game about it https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed...
[+] [-] novoreorx|5 months ago|reply
[+] [-] porridgeraisin|5 months ago|reply
One question in case folks who work there see this:
This is the most technically impressive zig project I've seen so far. Do you have a blog post detailing your perspective on zig? i.e what design decisions of ziglang helped you in massive way, what were the warts, and in general any other thoughts too?
[+] [-] gregorvand|5 months ago|reply
https://softwareengineeringdaily.com/2024/09/12/building-a-f...
[+] [-] initramfs|5 months ago|reply
[+] [-] itunpredictable|5 months ago|reply
[+] [-] criddell|5 months ago|reply
[+] [-] jorangreef|5 months ago|reply