GitHub Copilot Evals
InstallSearch /github/evals

Try the samples

Explore Evals with made-up projects instead of your own data.

Samples

Before you start

All samples need Node.js 24+. For native VS Code Evals views, add the extension before opening the sample. The local eval runner (@microsoft/vally-cli) reads the eval definition, runs its checks and returns results. In VS Code, choose GitHub Copilot Evals: Set up test runner to find or separately approve installing a compatible runner before your first eval run.

The no-AI sample uses mock responses and does not need the full Copilot CLI, Git or model sign-in. To test a real agent, the full Copilot CLI runs that agent for the editor; you also need Git and model access. See the runner requirements. Installing tools does not approve running an eval.

  1. Download the samples ZIP from releases.
  2. Extract it and open the project-maintainer folder. Keep its hidden files.
  3. Open project-maintainer in your chosen app: start Copilot CLI from that folder, open it in VS Code, or select it for your GitHub Copilot App session’s Canvas. It contains a fictional backlog, skills, agents and evals to explore.

Opening the folder does not sign in, start a model or run an eval. Once it is ready, you can explore it with Create or Review.

Create your first eval with this sample follows the interface you selected on this site. See the examples overview for what the sample covers.

Check the sample tools without AI

From the project-maintainer folder:

node .\helpers\smoke.mjs

This smoke test checks the tools, not how a model uses them.

Try the deliberately flawed version

Use node .\helpers\smoke.mjs --profile mixed to inspect the deliberate negative-control formatter without AI. Keep expected answers unchanged. Review .mcp.json before enabling; tools can change sample data.

Build from source

Use the build instructions. Prebuilt samples do not need a source build.

If it fails

If the destination already exists, choose a new folder rather than overwrite it. Keep hidden files when extracting the archive, and prepare the sample again on another machine so its paths remain correct. The archive’s START-HERE.md has the instructions for that version.