[zknill]

Assuming the traditional stateless routing of requests, say round robin from load balancers; how do you make sure the returning UI client ends up on the same backend server replica that's hosting the conversation?

Or is it that all your tokens go through a DB anyway?

It's fairly easy to keep an agent alive when a client goes away. It's a lot harder to attach the client back to that agents output when the client returns, without stuffing every token though the database.

[viraptor]

You normally need to do that anyway. The specific backend host may have been destroyed in the meantime so you have to recover the context. And it's not like they're huge after compression.