[devnull3]

I think there is a wide spectrum between small-homelab and google scale.

I was surprised how far sqlite goes with some sharding on modern SSDs for those in-between scale services/saas

[fatal94]

What you're doing is fine for a homelab, or learning. But barring any very specific reason other than just not liking Kafka, its bad. The second that pattern needs to be fanned out to support even 50+ producers/consumers, the overhead and complexity needed to manage already-solved problems becomes a very bad design choice.

Kafka already solves this problem and gives me message durability, near infinite scale out, sharding, delivery guarantees, etc out of the box. I do not care to develop, reshard databases or production-alize this myself.

[CyberDildonics]

sqlite can do 40,000 transactions per second, that's going to be a lot more than 'homelab' (home lab).

Not everything needs to be big and complicated.

[NewJazz]

Some people don't and won't need 50+ producers/consumers for a long while, if ever. Rewriting the code at that point may be less costly than operating Kafka in the interim. Kafka is also has a higher potential for failure than sqlite.