top | item 39005692 (no title) goodside | 2 years ago No, in both tokenizers Unicode tag-block code points like these are converted into bytes (two tokens per character), which is a fallback for code points uncommon enough to not warrant a dedicated token. discuss order hn newest No comments yet.
No comments yet.