top | item 46247009

(no title)

flakiness | 2 months ago

Oh it's good old tokenization vs for-LLM tokenizations like sentence piece or tiktoken. We shouldn't forget there are non-ML simple things like this one which doesn't ask you to buy more GPUs.

discuss

order

jamesgresql|2 months ago

Haha, I like “good old tokenization”