Testing by hand is not evaluating
Typing four questions into the chat after a change and seeing whether it answers well is a check, not an evaluation.
What makes it an evaluation is that it is always the same set, so results can be compared. Without that you cannot tell whether the change improved anything or just moved the problem.
Why it matters
Because in an AI system any change can make worse things that already worked, and you cannot see that by looking.
Adjusting the prompt to fix one answer and breaking ten others is normal without evaluations, and the worst part is that nobody finds out until a customer says so.
Where the cases come from
From your own conversations.
The fifty most frequent questions, plus the cases that went wrong, plus the limits you want to protect, meaning what it must not answer. Inventing generic cases gives you an evaluation that always passes and protects nothing.
What gets checked
Three things in each case.
- That the answer contains the right fact.
- That it does not contain what it must not, such as an invented figure or a promise.
- That it hands the conversation to a person when it should. It is the most forgotten and the most expensive when it fails.
When they are run
Before every change that reaches customers.
It is the same discipline as tests before a deploy, applied to something non-deterministic, which is why it needs it more.
