[jordn]

For those curious: Humanloop is a evals platform for building products with LLMs. We think of it as the platform for 'eval-driven development' needed for making AI products/features/experiences that work well

We learned three key things building evaluation tools for AI teams like Duolingo and Gusto:

- Most teams start by tweaking prompts without measuring impact

- Successful products establish clear quality metrics first

- Teams need both engineers and domain experts collaborating on prompts

One detail we cut from the post: the highest-performing teams treat prompts like versioned code, running automated eval suites before any production deployment. This catches most regressions before they reach users.