[Jupe]

Isn't it just a matter of having each consumer use their own offset? I mean if the queue table is sequentially or time-indexed, the consumer just provides a smaller/earlier key to accomplish the offset? (Maybe I'm missing something here?)

[altcognito]

Correct, offsets and sharding aren't magic. And partitions in Kafka are user defined, just like they would be for postgresql.

[cortesoft]

Kafka allows you to have a consumer group… you can have multiple workers processing messages in parallel, and if they all use the same group id, the messages will be sharded across all the workers using that key… so each message will only be handled by one worker using that key, and every message will be given to exactly one worker (with all the usual caveats of guaranteed-processed-exactly-once queues). Other consumers can use different group keys and they will also get every single message exactly once.

So if you want an individual offset, then yes, the consumer could just maintain their own… however, if you want a group’s offset, you have to do something else.

[jimbokun]

Yes.

Is a queuing system baked into Postgres? Or there client libraries that make it look like one?

And do these abstractions allow for arbitrarily moving the offset for each consumer independently?

If you're writing your own queuing system using pg for persistence obviously you can architect it however you want.