[XenophileJKO]

Can you predict when solar radiation hits a memory cell or when given server node will die? Not really but you can model the probability of it happening. My point was all the systems we work with have failure modes and non-determinate output at some rate. That rate might be really small.. but at what point does the rate of non-deterministic behavior make something "non-deterministic. A language model can be deterministic in that you can get the same output from the same input if you so desire (again baring systemic failures and mitigating for out of order floating point operations).

I think just philosophically it is interesting because any real system is not fully predictable.. we just choose some arbitrary threshold of accuracy to define it as such.

[bandrami]

Right but for an LLM non-determinism is the success mode, not the failure mode. By design you get different outputs when running the same inputs. Which, again, really contradicts the consensus that was widely held even just a year ago about how to do CI.

[XenophileJKO]

I mean you don't have to. With 0 temp, the only source of non-determinism is really the associative math chaning on accumulation of the floating point multiplication. That could be eleminated if we wanted to buy restricting the ordering.