In this case it’s not tokenization. I wrote the text preprocessing code that deals with spacing these numbers. This is good feedback. It’s optimized for US-style 10-digit phone numbers, and it should be more flexible than that. For example, if I was reading a US phone number such as (123) 456-7890 over the phone and wanted to make sure it was heard correctly, I’d say “123”, “456”, “78”, “90”. But a 9-digit phone number should be spaced as you said.
No comments yet.