[throwaway63820]

By virtue of increasing randomness, we got the correct answer once ... a monkey at a typewriter will also spit out the correct answer occasionally. Temperature 0 is the correct evaluation.

[scrollop]

So your theory would have it that if you repeated the question at temp 1 it would give the wrong answer more often than the correct answer?

[throwaway63820]

There's no theory.

Just in real life usage, it is extremely uncommon to stochastically query the model and use the most common answer. Using it with temperature 0 is the "best" answer as it uses the most likely tokens in each completion.

[moffkalast]

> Temperature 0 is the correct evaluation.

In theory maybe, but I don't think it is in practice. It feels like each model has its own quasi-optimal temperature and other settings at which it performs vastly better. Sort of like a particle filter that must do random sampling to find the optimal solution.

[scrollop]

Here's a quick analysis of the model vs it's peers:

https://www.youtube.com/watch?v=ReO2CWBpUYk