[burningion]

I think another category of error that Simon skips over that breaks this argument entirely: the hallucination where the model forgets a feature.

Rather than the positive (code compiles), the negative (forgets about a core feature), can be extremely difficult to tell. Worse still, the feature can slightly drift, based upon code that's expected to be outside of the dialogue / context window.

I've had multiple times where the model completely forgot about features in my original piece of code, after it makes a modification. I didn't notice these missing / subtle changes until much later.

[simonw]

That doesn't fit the definition of "hallucination" I was using here, which is a model inventing something that doesn't exist. Definitely a problem to watch out for though - I've had to remind the models to use existing functions before. I see that as an inevitable part of the back and forth with the model while iterating on code.