[TexanFeller]

Ofc I wouldn't us it for extremely high scale event processing, but it's great default for a message/task queue for 90% of business apps. If you're processing under a few 100m events/tasks per day with less than ~10k concurrent processes dequeuing from it it's what I'd default to.

I work on apps that use such a PG based queue system and it provides indispensable features for us we couldn't achieve easily/cleanly with a normal queue system such as being able to dynamically adjust the priority/order of tasks being processed and easily query/report on the content of the queue. We have many other interesting features built into it that are more specific to our needs as well that I'm more hesitant to describe in detail here.

[j45]

Very few things dna start at an extremely high scale event processing.

There’s also an order of magnitude higher events when doing event based work in processing.

This seems like a perfectly reasonable starting and gateway points that can have things organized for when the time comes.

Most things don’t scale that big.

[gytisgreitai]

So perhaps don’t use kafka at all? E.g. Adyen used postgresql [1] as a queue until the outgrew. In this case it seems there are a lot of things that can go south in case of major issue on the event pipeline. Unless the throughput is low.. but then why kafka?

[1] https://www.adyen.com/knowledge-hub/design-to-duty-adyen-arc...