Show HN: Shimmytok – Pure Rust GGUF tokenizer (no C++, no extra files)

[MKuykendall]

I got tired of needing llama.cpp bindings or separate tokenizer.json files just to tokenize text, so I wrote a pure Rust library that reads the tokenizer directly from your GGUF model file, the same output as llama.cpp, zero C++ in the dependency tree.

[holg]

Great work, sounds very promising, i did do some AI with burn and candle and always needed to create/convert the own models for this purpose, this could really help, thanks

[MKuykendall]

Thanks! Rust needed it