Evaluation: measuring the behavior you need
A useful score starts with a clear task, a representative test set, and a rule for deciding success.
In this lesson
No single benchmark score captures every capability or risk of a complete assistant.
Start with a decision
An evaluation is useful when it informs a decision: which model to use, whether a change is an improvement, or whether a system is ready for a particular task. Define the target behavior first. “Can it answer questions?” is vague; “Can it answer from this manual and cite the supporting passage?” identifies something testable.
The HELM project, introduced in 2022, argues for multiple scenarios and metrics rather than one universal leaderboard number. Accuracy, calibration, robustness, fairness, toxicity, and efficiency describe different properties. A system can improve one while worsening another.
Design the examples and the scoring rule
A test set should reflect realistic inputs and include difficult boundaries. For a document assistant, include direct lookups, questions requiring two passages, contradictory sources, and unanswerable requests. Keep the evaluation data separate from training or tuning where possible, and document any suspected overlap.
Choose a scoring rule suited to the output. Exact match works for some short labels but can reject an equivalent paraphrase. Human judgments can assess usefulness but require a rubric and checks for disagreement. A model-based judge is another learned system; its decisions should not be treated as independent ground truth without validation.
Compare complete and reproducible setups
Record the model version, prompt construction, tools, retrieval collection, decoding settings, and number of trials. A stochastic system may produce different outcomes on repeated runs. A small score difference on a small sample may be noise, so report uncertainty and inspect the actual changed examples.
Distinguish model evaluation from application evaluation. A model may answer correctly when given the right paragraph while the application retrieves the wrong one. Testing only the final model call misses that failure. End-to-end tests should cover the path from user request through tools and context construction to the displayed answer.
Seek failures as well as averages
Averages can hide concentrated failures. Break results down by language, task type, input length, or other relevant slices. Look at severe errors separately from harmless formatting differences. The cost of a wrong answer depends on the application; an identical accuracy figure can represent very different practical performance.
The 2022 language-model red-teaming study illustrates deliberate searches for harmful failures. Such exploratory testing complements a fixed benchmark: it discovers cases you did not know to include. After finding a failure, turn a representative case into a regression test while keeping a fresh held-out set to avoid merely memorizing the test suite.
A small, honest scorecard
For a fictional 100-question manual test, report correct supported answers, unsupported claims, appropriate “not in the manual” responses, retrieval misses, and latency. These categories show more than a single percentage labeled “AI quality.”
The highest benchmark score identifies the best model for every application.
The best choice depends on the task, system configuration, failure costs, and measurements that matter to users.
Take it with you
- Define success before choosing a metric.
- Keep the complete system and test conditions reproducible.
- Inspect error categories and uncertainty, not just averages.
Check your understanding.
Choose an answer, then see the explanation. You can try again.
Practice only. Your answer is not stored or sent anywhere.
Memorization: when training text comes back
Continue when you’re ready. Marking this lesson as read is optional.
Sources & further reading
Primary references for this lesson. The explanation and examples above are original.
- Holistic Evaluation of Language Models ↗Percy Liang et al. · 2022-11-16 · checked 13 Nov 2026
- Red Teaming Language Models to Reduce Harms ↗Deep Ganguli et al. · 2022-08-23 · checked 13 Nov 2026