Evals
Systematic tests of model or agent behavior — benchmarks, graded tasks, session scoring — used to measure quality, catch regressions, and compare harness or model changes.
Learn
When to use it
Use evals when informal testing or spot checks are insufficient to ensure model reliability and performance. Evals systematically measure model behavior, enabling tasks like catching regressions, comparing model versions, and validating agent decisions in scenarios such as "does this agent handle user queries as expected?"
Quick example
In OpenAI's ChatGPT, developers need to ensure that new model updates maintain or improve response quality. Evals are configured to run benchmarks and graded tasks after each update, providing scores that highlight any regressions or improvements. Here, ChatGPT includes evals as part of its continuous integration pipeline, ensuring systematic quality checks.
prompt → model → evals → score/report → stop
Ecosystem
Evals integrate with model training and deployment workflows, often alongside monitoring and logging systems. These components work together to ensure comprehensive oversight of model behavior.
┌─ logging ──┐
model → │ evals │ → score/report
└─ monitoring ─┘
Misconceptions
| Misconception | Rebuttal |
|---|---|
| Evals are one-time tests | Evals are ongoing and systematic |
| Evals only measure accuracy | Evals assess various behaviors and tasks |
| Evals replace manual testing | Evals complement manual testing, not replace it |
Trade-offs
- Consistency — requires maintaining up-to-date benchmarks
- Comprehensive coverage — can become complex and resource-intensive
- Actionable insights — may generate overwhelming data without clear interpretation