GitHub Copilot Evals
InstallSearch /github/evals

Evaluation methods

Choose checks and comparisons that help you answer a specific question about AI behavior.

Start with a task you understand

Choose a task you understand and establish the expected result from requirements or reviewed examples. Build checks around that result, keeping the task inputs separate from any answer key used to grade it.

Check that your checks catch mistakes

Use answers that you already know are correct, valid alternatives, plausible mistakes and empty responses. These control tests help establish whether a grader (the check that judges the answer) accepts and rejects the right things. Planning controls does not execute them or confirm the grader works.

Compare fairly

Keep the task, inputs, success criteria, model and execution setup unchanged except for the thing being compared. Start each run without the previous conversation and preserve every failed, cancelled or missing attempt.

Improve without moving the goalposts

Keep the original eval and propose a new version that addresses a specific problem. If you change a grader, compare both versions on the same answers where supported. Changing the task or environment needs new runs with controlled inputs, because it changes what is being measured.

When improving a skill or agent (hillclimbing), freeze the eval definition, suite, graders, and non-target inputs as your ruler while testing one focused edit at a time. After each approved run, check Ruler parity and decide whether to keep or revert the edit. Once the skill passes every case in the current eval, /eval improve highlights that the eval is saturated so you can save a steeper candidate eval alongside the original. Follow the step-by-step hillclimbing guide.

Stronger evidence

MethodPurpose
Coverage recordsList the requirements an eval checks and those it leaves unchecked
Testing the checksRun known good and bad answers to see whether the checks judge them correctly
Checking meaningUse reviewed requirements and check whether the grader’s ratings match them
Repeated evaluationObserve variation across attempts while retaining errors and usage information
Comparing revisionsUse the same answers or controlled new runs as appropriate for the change
Hillclimbing a skill or agentKeep the eval fixed while testing one focused asset edit per step, then steepen the eval when all cases pass
Choosing a versionKeep originals and decide what to change based on results

Use a runner that supports the required measurement. No universal score or number of runs proves reliability; explain what your evidence leaves uncertain.