An evaluation set pairs representative inputs with explicit success criteria. Use it to compare versions and identify regressions, not to claim universal accuracy.
Define correctness before scoring
For a fictional deadline-extraction assistant, check the date, supporting source and behaviour when no deadline exists. A broad “good answer” rating may hide a consequential date error. Keep dimensions separate until their business implications are understood.
| Family | Example |
|---|---|
| Routine | One clearly stated deadline |
| Ambiguous | Two dates serving different purposes |
| Missing information | No dated commitment |
| Conflicting versions | Old and updated documents |
| Adversarial input | Text attempting to change instructions |
Preserve a fair comparison
Hold back some cases from day-to-day tuning. Version inputs, criteria and the evaluated configuration. For a product serving multiple markets, include differing date formats and language conventions. Record when the business definition of success changes, because historical scores may no longer be comparable.
How large should the set be?
There is no universal count independent of diversity and risk. Start with well-chosen case families and add observed failures. Review ambiguous cases with someone who understands the task. A strong average does not cancel a critical failure, so inspect performance by category before approving a release.
Separate language effects from document difficulty
Include ordinary and difficult cases within each important language group. Otherwise a weak result may reflect document complexity rather than language. Keep the breakdown visible so the team chooses an appropriate correction instead of changing the whole prompt blindly.
Sources and further reading
From this guide to your project
AI ↗
We start with a task and a quality criterion. AI belongs where it meaningfully improves the service.
Tell us about your ideaExamples are educational and do not describe client results. Content prepared with AI assistance; see our editorial policy. Our method ↗