GitHub Copilot Evals

Test your Copilot skills and agents.

In the Copilot tools you already use.

See what works, catch failures, and check whether your changes improve results.

Install Evals

Evals in action: Copilot CLI.

CurrentSessionsIssuesPull requestsGists

/eval help me create an eval for my pr-description skill

Evals: understanding your request...

skill(create)

I'll review the source and requirements, then outline what the eval will test.

Read plugins\pr-description\skills\pr-description\SKILL.md

draft_eval intent: "create", target: "pr-description"

Suggested scenarios: shipped vs deferred work; missing verification; instructions inside change notes.

Asked user Which cases should I create? → Start with 1 (Recommended)

Draft: one case with shipped changes, supplied test evidence and deferred retry backoff. Checks cover section order, shipped scope and verification.

check_eval_draft pr-description.eval.yaml

Static validation passed. The eval format and checks are valid.

save_eval_draft

Saved evals\pr-description.eval.yaml · 1 case.

Run it now and explain the result.

skill(run)

run_eval evals\pr-description.eval.yaml

Asked user Run approval: Run eval · this run only

Running 1 case...

Example run complete · 1 case failed

PASS Required sections in order

FAIL Changes includes only shipped work

PASS Supplied verification preserved

skill(review)

review_evals saved run report

The description put deferred retry backoff under Changes. It should be marked out of scope in Summary or Risks / Rollback.

Improve the pr-description skill to keep deferred work out of Changes?

We can update the skill, then rerun this same eval to check the change.

Copilot needs information.

Improve the pr-description skill to keep deferred work out of Changes?

We can update the skill, then rerun this same eval to check the change.

  • Work on the skill with Improve
  • Keep the results for now

↑/↓ select · enter accept · ctrl+d decline · esc cancel

An illustrative workflow. Your experience may vary.

Eval skills

create
Write evals for a skill or agent.
review
Read an eval or investigate a failed check.
improve
Revise an eval while keeping the original.
run
Run saved evals with the tools and limits you approve.
compare
Compare saved results or explore a skill on one task.
setup
Choose eval folders, suites and CI checks.

Help improve GitHub Copilot Evals.

Report a bug or suggest an improvement on GitHub.

Share feedback