GitHub Copilot Evals
InstallSearch /github/evals

Compare saved runs

See what changed after a revision without mistaking different tests for better results.

Choose your app

Find your setup instructions
Compare saved runs view before a pair is selected; no new comparison was created.
Select compatible saved runs. Evals Canvas · sample data

Choose your app to see how to start. Start with the first-eval guide for the general steps.

With/without benefit is not proven. Separate pass counts do not prove a skill improved. Read the current limits.

Choose the comparison mode

Keep the original result. Change one thing, approve a rerun, then inspect both saved results. For a skill revision, keep task, inputs, checks, and model unchanged.

Choose Compare saved results to read two existing runs without execution. Side-by-side results shows separate counts, not matched eval cases. View details shows the evidence behind those counts.

Use compatible saved runs

Matched test comparison is the UI label for comparing matching eval cases; here, “test” means an eval case, not a different workflow. It requires matching evidence and one change. Reading results starts no runs or edits. Canvas’s Save comparison action writes the report without another approval prompt; it never reruns either eval. Details.

Using a standalone MCP connection

evals_compare_saved_runs reads two emitted saved-run handles with explicit descriptive or strict mode. Strict mode returns eligible axes before an exact axisKey selects the report. It starts no execution and publishes no receipt. MCP selection and limits.

Advanced: with/without experiments

The built Copilot CLI / GitHub Copilot App /eval compare also offers a separate skill experiment. Explore one skill on one task. Approve preparation and each run separately. Leaving the skill out of the prepared inputs does not prove no copy was available.