The tokenizer is not really a good demonstration of the differences between these styles. A more representative comparison would be the later stages that build, traverse, and manipulate tree and graph data structures.
I think a reasonable comparison would have to be DoD Rust parser vs current Rust parser. Comparing across languages isn't very useful, because Zig has very different syntax rules, and doesn't provide diagnostics near the same level as Rust does. The Rust compiler (and also its parser) spends an incredible amount of effort on diagnostics, to the point of actually trying to parse syntax from other languages (e.g. Python), just to warn people not to use Python syntax in Rust. Not to mention that it needs to deal with decl and proc macros, intertwine that with name resolution, etc. etc. This all of course hurts parsing performance quite a lot, and IMO would make it both much harder to write the whole thing in DoD, and also the DoD performance benefits would be not so big, because of all the heterogeneous functionality the Rust frontend does. Those are of course deliberate decisions of Rust that favor other things than compilation performance.
AndyKelley|8 months ago
https://github.com/rust-lang/rust/tree/1.87.0/compiler/rustc... (main logic seems to be in expr.rs)
vs
https://github.com/ziglang/zig/blob/0.14.1/lib/std/zig/Parse...
Again, for those who wish to form their own opinions.
kobzol|8 months ago