Build a 10-Case Eval Set for Your AI Agent and Run It on Every Change

Your agent worked in the demo. Then you tweaked the system prompt, swapped the model, or renamed a tool argument, and now you're not sure whether it still answers the refund question correctly. Too often, the first person to notice is a user. You don't need an eval platform to fix that. You need a small set of real test cases and a script that runs them every time something changes. Here's how to build one in an afternoon with plain Python and pytest. Why agents break normal tests A unit test assumes a known answer: given this input, assert that output. Agents break that assumption in three ways: They're non-deterministic. OpenAI's evaluation best practices put it plainly: models sometimes produce different output from the same input, which makes traditional testing insufficient on its own. They're open-ended. Most real requests have better and worse answers, not one correct string. They're multi-step. An agent picks tools, passes arguments and reads results. It can r