[gmadsen]

as part of the prompt, have a test suite with test files. Its still fully automated by the LLM but adds confidence

[recursive]

If it's under the umbrella of LLM automation, then I'd also need to verify that the test suite behavior actually matches the "production" behavior.

[gmadsen]

sure, but that is less work. you can also have separate LLM QA prompts that assess test suite behavior to production behavior.

ultimately you are right, the buck needs to stop somewhere, but at least in my experience, the more you add quality/test checks as LLM workflows, the higher the rate of success.