[bigzyg33k]

advanced voice mode operates on audio tokens directly, it doesn't transcribe them into "text tokens" as an intermediate step like the original version of voice mode did.

[cubefox]

But they behave just like models which use text tokens internally, which is also pointed out at the end of the above article.

[bigzyg33k]

we don't know if that's due to inherent limitations of the tokenisation of audio, or a byproduct of reinforcement learning. In my own usage, I noticed a significant degradation in capabilities over time from when they initially released advanced voice mode. The model used to be able to sing, whisper, imitate sounds and tone just fine, but I imagine this was not intended and has subsequently been stunted via reinforcement learning.

I don't find the articles argument that this is due to tokenisation convincing.

[sbrother]

right, but either whatever audio tokenization it's doing doesn't seem to encode pitch, or there's ~nothing where pitch is relevant in the training set.

[oezi]

Absolutely correct! My simple test is if it can tell American and British English Tomato and Potato apart. So far it can't.

[fragmede]

Which "it" are you referring to? There are models that can.