GitHub Copilot Evals
InstallSearch /github/evals

Are your skills helping Copilot?

Check whether your instructions still help on the work you actually do.

More instructions do not automatically mean better work. A skill or agent may handle one task well and miss another. As your tasks, source, or model change, use evals to revisit what it contributes.

1. Pick a task that matters

Start with something you repeatedly ask Copilot to do, or a mistake you have already seen. Decide what success means before reading the answer.

For a release-note skill: include completed work and exclude unfinished work. Keep the backlog as input and the expected answer separate from it. See the example or create an eval for your own source.

2. Choose the question

QuestionAvailable pathWhat you learn
Does my skill or agent do the job?Run an eval with independent outcome checks.Whether this attempt met those requirements.
Did my revision help on this task?Keep the original, rerun the same eval after one change, and compare saved results.Which observed failures changed, where matching evidence supports the comparison.
Is this skill contributing?Try one task with the skill included, then left out of the prepared inputs, in a supported app.Something to investigate, not proof that the skill was absent or caused a better answer.

3. Explore one skill on one task

In the packaged Copilot CLI / GitHub Copilot App Plugin, use /eval compare and choose the skill experiment. Select the real skill and a saved single-case eval. If the task is missing, use Create first; return only after all files are saved.

Review the task, checks, model, tools, and both conditions. The task should not tell Copilot to use the skill: that would bias the comparison. Approve preparation and each condition’s run separately.

Read the actual answers and independent checks. Keep failed, cancelled, and incomplete attempts. Missing usage or cost is unknown, not zero.

This exploratory path is for one skill, not an arbitrary agent or tool. Omitting a skill from prepared inputs does not prove that another copy was absent. One pair cannot establish general benefit. If your app lacks the required tools, you get setup guidance only, not a completed experiment. Availability and evidence limits.

Paid runs have no hard total-cost cap, and temporary folders are not a sandbox. Stop if a required limit or approval is unavailable.

4. Decide what to investigate next

  • A wrong answer fails: inspect the skill or agent, review a targeted source change, then separately approve a retest.
  • A wrong answer passes: improve the check before relying on it.
  • Results disagree or evidence is missing: investigate the setup and try more representative cases. Don’t decide whether to keep or remove a skill from this alone.

Keep task, inputs, model, and checks unchanged when testing a source revision. Changing the checks changes the measurement, not necessarily the AI’s work.

Compare a change or improve a check.

What stronger benefit testing would require

Proposed experiment: keep conditions the same, repeat the task, inspect results, and review a decision.
Planning only: this test cannot currently prove that a skill helps. The diagram proposes a stronger test than the steps above. Its verdicts are not supported recommendations to keep or remove a skill. Read the current limits.

Where to inspect your work

Assets view with the sample release-note skill selected and its redacted source open.
Select the file in Assets. Evals Canvas · sample data
Evals view showing the sample no-AI eval with Improve with Copilot, Queue Eval and Review & run.
Open a saved eval in Evals to queue or review a run. Evals Canvas · sample data
Compare saved runs view before a pair is selected; no new comparison was created.
Select compatible saved runs. Evals Canvas · sample data