Evaluation methods
Choose checks and comparisons that help you answer a specific question about AI behavior.
Start with a task you understand
Choose a task you understand and establish the expected result from requirements or reviewed examples. Build checks around that result, keeping the task inputs separate from any answer key used to grade it.
Check that your checks catch mistakes
Use answers that you already know are correct, valid alternatives, plausible mistakes and empty responses. These control tests help establish whether a grader (the check that judges the answer) accepts and rejects the right things. Planning controls does not execute them or confirm the grader works.
Compare fairly
Keep the task, inputs, success criteria, model and execution setup unchanged except for the thing being compared. Start each run without the previous conversation and preserve every failed, cancelled or missing attempt.
Improve without moving the goalposts
Keep the original eval and propose a new version that addresses a specific problem. If you change a grader, compare both versions on the same answers where supported. Changing the task or environment needs new runs with controlled inputs, because it changes what is being measured.
When improving a skill or agent (hillclimbing), freeze the eval definition,
suite, graders, and non-target inputs as your ruler while testing one focused
edit at a time. After each approved run, check Ruler parity
and decide whether to keep or revert the edit. Once the skill passes every case
in the current eval, /eval improve highlights that the eval is saturated so
you can save a steeper candidate eval alongside the original. Follow the
step-by-step hillclimbing guide.
Stronger evidence
| Method | Purpose |
|---|---|
| Coverage records | List the requirements an eval checks and those it leaves unchecked |
| Testing the checks | Run known good and bad answers to see whether the checks judge them correctly |
| Checking meaning | Use reviewed requirements and check whether the grader’s ratings match them |
| Repeated evaluation | Observe variation across attempts while retaining errors and usage information |
| Comparing revisions | Use the same answers or controlled new runs as appropriate for the change |
| Hillclimbing a skill or agent | Keep the eval fixed while testing one focused asset edit per step, then steepen the eval when all cases pass |
| Choosing a version | Keep originals and decide what to change based on results |
Use a runner that supports the required measurement. No universal score or number of runs proves reliability; explain what your evidence leaves uncertain.