[keymasta]

Tried my personal benchmark on the gpt-oss:20b: What is the second mode of Phyrgian Dominant?

My first impression is that this model thinks for a _long_ time. It proposes ideas and then says, "no wait, it's actually..." and then starts the same process again. It will go in loops examining different ideas as it struggles to understand the basic process for calculating notes. It seems to struggle with the septatonic note -> Set notation (semitone positions), as many humans do. As I write this it's been going at about 3tok/s for about 25 minutes. If it finishes while I type this up I will post the final answer.

I did glance at its thinking output just now and I noticed this excerpt where it finally got really close to the answer, giving the right name (despite using the wrong numbers in the set notation, which should be: 0,3,4,6,7,9,10:

  Check "Lydian #2": 0,2,3,5,7,9,10. Not ours.

The correct answers as given by my music theory tool [0], which uses traditional algorithms, in terms of names would be: Mela Kosalam, Lydian ♯2, Raga Kuksumakaram/Kusumakaram, Bycrian.

Its notes are: 1 ♯2 3 ♯4 5 6 7

I find looking up lesser known changes and asking for a mode is a good experiment. First I can see if an LLM has developed a way to reason about numbers geometrically as is the case with music.

And by posting about it, I can test how fast AIs might memorize the answer from a random comment on the internet, as I can just use a different change if I find that this post was eventually regurgitated.

After letting ollama run for a while, I'm post what it was thinking about in case anybody's interested. [1]

Also copilot.microsoft.com's wrong answer: [2], and chatgpt.com [3]

I do think that there may be an issue where I did it wrong because after trying the new ollama gui I noticed it's using a context length of 4k tokens, which it might be blowing way past. Another test might be to try the question with a higher context length, but at the same time, it seems like if this question can't be figured out in less time than that, that it will never have enough time...

[0] https://edrihan.neocities.org/changedex (bad UX on mobile! - and in general ;)). won't fix, will make new site soon) [1] https://pastebin.com/wESXHwE1 [2] https://pastebin.com/XHD4ARTF [3] https://pastebin.com/ptMiNbq7