[hbs18]

> The basic design is non-deterministic

Is it? I thought an LLM was deterministic provided you run the exact same query on exact same hardware at a temperature of 0.

[chmod775]

Not quite then as well, since a lot is typically executed in parallel and the implementation details of most number representations make them sensitive to the order of operations.

Given how much number crunching is at the heart of LLMs, these small differences add up.

[biophysboy]

My understanding is that it selects from a probability distribution. Raising the temperature merely flattens that distribution, Boltzmann factor style