[AlotOfReading]

A related property is whether particular kinds of changes to the inputs have proportionally sized changes to the output. Adding a print statement shouldn't change the behavior of the function it's in (sans I/O), for example. Using calling the same function from two different callsites shouldn't change the behavior either. A new compiler version shouldn't change the observable behavior. Etc.

I think this is the more important property and I'm not sure if it has a well-known name. The article obliquely calls it reliability, but regardless it's the key difference from LLMs. Compilers mostly achieve it, ignoring an endless list of exceptions you learn with experience.

LLMs usually don't, even with 0 temperature and floating point determinism.

[1718627440]

Maybe it should be names with something like "algorithmic stability"? https://en.wikipedia.org/wiki/Stability_theory, https://en.wikipedia.org/wiki/Stable_theory, https://en.wikipedia.org/wiki/Lipschitz_continuity.

[sarchertech]

I’ve been labeling this problem chaotic output and yes it’s a much harder and more insidious problem than determinism.