[sureglymop]

Don't LLMs create the same outputs based on the same inputs if the temperature is 0? Maybe I'm just misunderstanding.

[AlotOfReading]

Unfortunately not. Various implementation details like attention are usually non-deterministic. This is one of the better blog posts I'm aware of:

https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

[8note]

i dont think theres anything that makes it essentiall that llms are non-deterministic though

if you rewrote the math to be all fixed point precision on big ints, i think you would still get the useful LLM results?

if somebody really wanted to make a compiler in an LLM, i dont think that nondetermism is problem

id really imagine an llm compiler being a set of specs, dependency versions, and test definitions to use though, and you'd introduce essential nondetermism by changing a version number, even if the only change was the version name from "experimental" to "lts"

[h3lp]

When they run on the deterministic hardware, yes. When they run on some large, parallel, varying-unpredictable-load-dependent-latency hardware, no.