GitHub Copilot Evals
InstallSearch /github/evals

Run your first eval

Test one real task and learn from your first saved result.

Choose your app

Find your setup instructions

No project ready? Use a sample.

1

Open your project

Native VS Code controls need the extension. Plugin-only editor setup usesconversation requests and your app's supported save/run tools.

Prepare the optional Project Maintainer sample, or open the project containing your own skill, delegated agent or supported tool source.

Assets view with the sample release-note skill selected and its redacted source open.
Select the file in Assets. Evals Canvas · sample data

The source you test is called an asset in the UI. Choose its actual SKILL.md, agent definition or supported tool configuration. Installing Evals does not discover every installed Plugin as local source. The screenshot shows a prepared sample, not a file already in your project.

2

Describe success

Choose a realistic task, its inputs, the expected result and what must not happen.

For release notes: include completed backlog work and exclude unfinished work. Use requirements, not the AI's answer, to decide what is correct. A check accepts or rejects part of the result. Keep answer keys out of inputs given to the AI.

3

Build or ask Copilot

Turn the task into proposed eval and input files. Read the complete draft before saving.

Copilot drafting can use paid quota. A starter template does not prove that your skill, agent or tool was used correctly.

Create an eval form bound to the sample release-note skill while workspace trust is off.
Describe the task and expected behavior for this source. Evals Canvas · sample data

The no-AI starter does not load your source automatically. Configure the real skill directory, agent file with an explicit delegation request, or supported MCP server. Check both that Copilot used it and that the answer is correct. Review every path and check; note requirements you cannot measure. The screenshot shows an unsubmitted form, not a finished release-note eval.

Check the source-loading configuration.

4

Check and save

Check the proposed files, inspect their complete content and paths, then save them.

Saving does not approve execution. Resolve partial saves using the original request, proposal and recovery details before Run.

Checking files does not run the eval. The Evals save service checks files before saving. Saves use normal app permissions without another Evals confirmation. Run checks execution support separately. With only your app's file tools, files are saved but not checked by the runner. Never bypass a failed or partial Evals save.

Resolve checking and saving problems.

5

Approve the run

Review the saved eval's model, tools, inputs and limits, then approve execution separately.

Paid models can cost money; total cost and internal model requests are not capped. The temporary folder is not a sandbox. Stop if a required limit or approval is unavailable.

Evals view showing the sample no-AI eval with Improve with Copilot, Queue Eval and Review & run.
Open a saved eval in Evals to queue or review a run. Evals Canvas · sample data

Model runs need supported sign-in and model access. Agent and MCP runs need full Copilot CLI, not just the SDK runtime. Use Set up test runner when needed; Chat sign-in alone does not set up the runner. The screenshot shows a saved eval before approval. Stop affects this run only.

6

Read the result

Open your saved result. Compare the expected answer, actual answer and each check.

Results view for a sample report made without AI, showing expected output, observed output and a failed check.
A saved example made without AI, not a skill run. Evals Canvas · sample data

The screenshot shows a saved sample made without AI; it does not show a skill run. If a wrong answer fails, investigate the source. If it passes, investigate the check. Setup errors and missing evidence are not passes. Keep this first result: one pass cannot establish reliability. Your first eval ends here.

Choose a realistic task, draft an eval with checks, validate and review every file, save through normal app permissions, approve a run's costs and effects, then read the answer and check results. Improve the eval and retest if needed.
The steps above, not a recorded run. Open the image for a larger view.

After your first result

Project configuration is optional. Use Init / Setup to choose eval folders and suites.

Change your asset and compare with unchanged task, inputs, model and checks. Review edits; approve retests separately.

Improve a check if it accepted a wrong answer. Set up CI after the eval is useful locally.