Consider this scenario. A practitioner improves an AI tool: new rules, better instructions, a cleaner setup. The new capabilities are tested and verified, and work moves on. Two weeks later, something the tool had been doing correctly for months, a small, routine, critical task, turns out to have quietly stopped working on the day of the upgrade. Nobody noticed because everyone was watching the new feature. The old reliability had no spokesman.
That is the structural trap in anything that can be updated: an upgrade does not add to what worked before. It replaces it. The new version is a different worker with the same name. You would not let a new hire take over a sensitive role simply because their predecessor performed it well. With AI tools it happens at every update, every new instruction added, every new capability layered in.
The fix is what practitioners call an evaluation, or eval: a small audition of real tasks with known-good answers, run from start to finish. Not synthetic benchmarks but actual requests from real work, each paired with a result the operator already knows to be correct. Summarize this document, the one where the obvious summary misses the real point. Draft this reply, the one with a detail that must not appear. Pull these numbers, the ones verifiable against the source. The eval tests for the specific failures that matter in this operator's actual work.
What makes an eval powerful is not running it once. It is keeping it. With a person, an audition is expensive, so it happens rarely. With a machine, re-running the audition is nearly free, which means it can happen every time anything changes: a new version, a new rule, a new tool added to the workflow. Same questions, same known answers, a few minutes of work. If new capability quietly broke old reliability, the eval catches it on day one, not in week three when a real person is affected.
The eval grows the only way that matters: every real failure gets added to it. When the tool makes a mistake that costs something in the real world, that exact failure becomes a new trial question, permanently part of the audition, so no future version can make that mistake quietly again. The rehearsal is how past failures teach future versions.