GitHub Copilot Evals
InstallSearch /github/evals

Evals helper skills

The skills that help you evaluate your own skills, agents, and tools.

The Evals Plugin includes helpers for the work below. They are not the skill or agent you are testing. Install the Plugin or the VS Code extension, which includes it.

HelperCopilot CLI / GitHub Copilot App commandPurpose
Setup/eval setup or /eval initOptionally configure folders and suites.
Create/eval createPropose an eval from real source and requirements.
Review/eval reviewExplain an eval or saved result.
Improve/eval improvePropose a separate eval version; keep the original.
Run/eval runPrepare and separately approve execution.
Compare/eval compareInspect saved results or explore one skill’s contribution.

The built Copilot CLI / GitHub Copilot App extension provides these commands. Its six helper skills are hidden from slash menus but can be requested in natural language. Repository skills alone do not provide native commands or the built save/run tools.

In trusted local desktop VS Code, the extension VSIX bundles the same six skills (setup, create, review, improve, run, and compare) hidden from slash menus so they can be requested in natural language or through Evals sidebar, Command Palette, and Canvas actions. There, GitHub Copilot Evals: Compare opens Canvas’s Compare view; opening it does not approve preparation or execution.

The install identity is eval@github-copilot-evals; the product name is GitHub Copilot Evals.

Ask in your own words

Review this eval. What does it check, and what could it miss?
Do not edit or run it.
Improve this eval so it catches a partial answer.
Keep the original and show the proposed changes.

Improve changes the eval, not the tested asset. Review files before saving and approve runs separately. In Copilot CLI / GitHub Copilot App, describe a goal after /eval or use the action picker’s Other option. Both accept requests spanning multiple evals and actions. Copilot considers the whole request, even when its first word is a command name; structured commands and explicit targets retain their direct selection flow.

Sample agents

Prepare the samples before using these source paths:

AgentSource
Project Maintainer.github\agents\maintainer.agent.md
Release Manager.github\agents\release-manager.agent.md

Prepared skills live under .agents\skills. Keep the sample’s versions together. Markdown source alone does not mean your app can run it.

Run your first eval · Configure a project or CI · Understand comparison limits