[yareal]

This post mortem is sort of classically underdone. It describes a step that was taken that was in the path of the error, but is not the root cause. The root cause here is not "we copy pasted from chat gpt and it hallucinates", but rather a "our systems allowed this failure to get to production". Which in turn should be met with why? Because we didn't have tests or qa that covers this path. Why?

Keep. Asking. Why.

ChatGPT didn't fail, your system allowed ChatGPT to fail. Answering why is the interesting thing to discuss and blog about.