GitHub Copilot Evals
InstallSearch /github/evals

Plan a useful check

Write down the task and the evidence that would convince you it was done correctly.

Plan a case

Copy this worksheet and fill in the facts:

Task:
Input and starting conditions:
Expected result:
Source of that expectation:
Required and forbidden behavior:
Allowed alternatives:
Check to use, or requirement that cannot be checked:
Correct, alternative, wrong and empty examples:

This planning worksheet helps you organize the case; it does not validate an eval file. Keep the expected answer separate from the inputs given to the AI being evaluated.

Choose a check

You need to knowUseful observation
Was an output file created?File existence
Does the answer include required content?A supported text or pattern check
Does a code change work?Executed tests of the code’s behavior
Did data change correctly?An observation of the data after the action

Read Checks & limits for supported types and what they cannot establish.

Interpret a result

ObservationMeaning
AI says “done”A statement, not proof of the required outcome
Required file is confirmed missingA possible failed check
Required file cannot be readThe result is unknown because the file could not be checked
Provider or setup errorThe run could not be completed or checked as intended
Usage was not capturedThe cost or usage is unknown; do not count it as zero

If a check accepts a wrong answer, improve the check. If it correctly catches a mistake, investigate the skill or agent.