GitHub Copilot Evals
InstallSearch /github/evals

Improve an existing eval

Fix checks that miss wrong answers or reject valid ones, without losing the original test.

Choose your app

Find your setup instructions

No project ready? Use a sample.

1

Choose the original

Select the eval and a result showing a mistake its checks miss.

Results view for a sample report made without AI, showing expected output, observed output and a failed check.
A saved example made without AI, not a skill run. Evals Canvas · sample data

If the check correctly caught a wrong answer, investigate the asset instead. Improve changes eval files, not the skill or agent.

2

Propose one change

Ask Improve for a separate version that catches the missed mistake.

Create an eval form bound to the sample release-note skill while workspace trust is off.
Describe the task and expected behavior for this source. Evals Canvas · sample data

A candidate is a proposed new version, not proof of improvement. Preserve the task's actual requirements and include a valid alternative answer.

3

Validate and save

Check and review the new files, then save the candidate separately from the original.

Saving is not execution approval. Resolve partial saves before running.

Keep the original definition and results. Without runner checks, files created by your app are saved but not checked by the runner. Never bypass a failed or partial Evals save.

4

Run the new version

Review the candidate's model, tools and limits; approve its run separately.

Total cost and internal model requests are not capped; the temporary folder is not a sandbox. Stop if a required limit or approval is unavailable.

Evals view showing the sample no-AI eval with Improve with Copilot, Queue Eval and Review & run.
Open a saved eval in Evals to queue or review a run. Evals Canvas · sample data

Retain failed, cancelled and incomplete attempts. A higher score from an easier check is not better AI work.

5

Decide what to keep

Read both reports. Does the revised check measure the task better?

Results view for a sample report made without AI, showing expected output, observed output and a failed check.
A saved example made without AI, not a skill run. Evals Canvas · sample data

Changed checks can make scores incomparable. Where supported, test both checks on the same answers. Do not force incompatible results into a matched comparison.

If the checks changed

Apply both check versions to the same answers where supported. Changing the task or environment needs controlled new runs; reading a saved comparison does not score old answers again.

Hillclimb a skill or agent step by step

When a check catches a real mistake in your skill or agent, keep the eval fixed and improve the asset one step at a time. In Copilot CLI and GitHub Copilot App, you do not need to run /eval create, /eval run, /eval review, and /eval compare manually beforehand. Run /eval improve directly on the skill or agent (or run bare /eval improve and select it from the picker):

/eval improve escalation-brief

Copilot binds the selected asset and guides one bounded hillclimbing step at a time:

StepWhat happensApproval and boundaries
1. Inspect and choose improvement areasReads the selected asset (SKILL.md or .agent.md, references/, helper scripts and unit tests) and calls review_evals to inspect project evals and saved runs. When the request is open-ended, offers source-grounded improvement areas (tightening SKILL.md and references/, expanding eval scenarios, adding deterministic scripts/tests, or fixing failing cases) before editing or running.If no non-holdout diagnostic eval exists yet, or if you choose to expand the eval suite, saving uses draft_eval, check_eval_draft, and save_eval_draft under normal host file permissions without overwriting existing files.
2. Capture the baseline and inspect failuresIf no completed saved run exists for the current asset and eval version, calls run_eval to record the baseline score, then reads the failing cases with review_evals.Running requires your app’s single execution and cost confirmation. Shared review_evals and compare_saved_results return full saved-run evidence for the selected eval, so keep private holdout evals separate from the diagnostic eval.
3. Propose one focused edit and re-runStates one testable change and proposes one focused edit to your SKILL.md or .agent.md while keeping the eval definition, suite, graders, and inputs unchanged, then calls run_eval on the unchanged eval.Review the asset edit under normal host file permissions and approve the post-edit run separately.
4. Verify Ruler parity and choose keep or revertCalls compare_saved_results in descriptive mode to verify rulerParity (verified-ruler-parity) and compare pass counts and token usage (or uses strict mode’s matched-comparison checks when eligible), then stops for your decision.Advisory only: Evals never auto-reverts your files or loops through further edits without your explicit direction.

When all cases already pass

When complete saved-run history shows the latest run for an eval version passed every case, /eval improve includes a saturation hint so you can save a harder candidate eval with additional edge cases or stricter checks while keeping the original eval untouched:

/eval improve evals/escalation-brief.eval.yaml

Compare results · Understand Ruler parity · Check whether your skills help