[joshhart]

Hi, I run the model serving team at Databricks. Usually you run regex filters, LLAMA Guard, etc on chunks at a time so you are still streaming but it's in batches of tokens rather than single tokens at a time. Hope that helps!

You could of course use us and get that out of the box if you have access to Databricks.

[lordswork]

But ultimately, it's an unsolved problem in the field. Every single LLM has been jailbroken.

[accrual]

Has o1 been jailbroken? My understanding is o1 is unique in that one model creates the initial output (chain of thought) then another model prepares the first response for viewing. Seems like that would be a fairly good way to prevent jailbreaks, but I haven't investigated myself.