[matrix2596]

is is possible for your tokenizer to give different tokenization ever then openai tokenizer? i am asking because there are multiple ways to tokenize the same string?? sry if i am mistaken

[matthewolfe]

Should be the same. Both use Byte-Pair Encoding (BPE) as underlying algo.