[the_jends]

Because LLMs works like how system 1 does, with heuristics such as pattern matching, frequency heuristics, etc. The fall back to coding stuff works not because it's doing system 2, it's because there are many instances of system 2 thinking for this particular problem done by humans so that the instances can be found using the heuristics.

[j_maffe]

Yeah, I think the analogy here would be how an experienced programmer would write code if asked to write on the spot given a specific prompt with no time to think.