[alexchamberlain]

Thanks, that's really interesting. Do they correct for spelling mistakes or internationalised spellings? For example, does `colour` and `color` end up in the same token stream?

[lyu07282]

No it just looks at exact character sequences, try it out yourself here: https://platform.openai.com/tokenizer