[danjc]

What's missing here is tool use.

For example, if the llm had a compile tool it would likely have been able to correct syntax errors.

Similarly, visual errors may also have been caught if it were able to run the program and capture screens.

[wbolt]

Exactly! The way in which the LLM is used here is very, very basic and outdated. This experiment should be redone in a proper „agentic” setup where there is a feedback loop between the model and the runtime plus access to documentation / internet. The goal now is not to encapsulate all the knowledge inside single LLM - this is too problematic and costly. LLM is a language model not knowledge database. It allows to interpret and interact with knowledge and text data from multiple sources.