[cantpostcanceld]

I don't know if this is related but I work on a project with a language parser and so a tokenizer and one issue we have is the parser/tokenier was not designed to work on incomplete code so it's not useful for auto complete.

That property of how good the design is for helping write and debug (useful error messages) seems like yet another metric that could be used

I also don't know if tokenizers can be divorced from parsers. Some languages have constructs that require context that comes from the parser

[conartist6]

CSTML and BABLR are designed to close this gap completely for people in your position.

We invented a new kind of zero just so that you could write arbitrary parsers and deal with things that are missing