Prerequisites
Open the project you want to test in your Copilot agent. Recommended setup for running evals:
Where do you use Copilot?
Select your Copilot agent to show its installation steps.
In VS Code
Use local desktop VS Code 1.138.0 or later with a trusted local project. This guide covers full authoring; bundled skills are unavailable in remote, web and untrusted workspaces.
Need VS Code? Install VS Code.
- Open GitHub Copilot Evals in the Visual Studio Marketplace, or search the Extensions view for
GitHub.copilot-evals. Verify publisher GitHub. - Choose Install and open your project.
- Open the Command Palette with Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS), then paste and run:
Confirm that Canvas shows the intended folder.GitHub Copilot Evals: Open Evals Canvas
Using the Agents window? Continue with the VS Code Agents setup.
In VS Code Agents
These steps apply to Copilot sessions in the local Agents window.
New to Agents? Set up the Agents window.
- Follow the VS Code installation steps in regular VS Code's default profile, then return here.
- Open Settings with Ctrl+, (Windows/Linux) or Cmd+, (macOS). Search for
chat.canvases.enabledand enable it. - Open the Command Palette with Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS), then run:
Open your project in that window.Chat: Open Agents window - Start a fresh session. Enter
/eval helpto check that Evals commands loaded, then/eval canvasto open GitHub Copilot Evals native Canvas. Confirm that it shows your project.
This route uses the VS Code extension's bundled Plugin. Do not enable a second standalone Evals Plugin in the same Agents session.
In Copilot CLI
Need the CLI? Install Copilot CLI.
- Open a terminal and run these commands one at a time. Stop if a command fails:
copilot plugin marketplace add github/evalscopilot plugin install eval@github-copilot-evals - Check that the source is
github/evalsandevalfromgithub-copilot-evalsis installed and enabled:copilot plugin marketplace list --json copilot plugin list --json - Start a fresh Copilot conversation in your project to use the skills.
Optional: for native /eval commands or Canvas, enable CLI-wide experimental mode with /experimental. Start a fresh conversation, then use /eval help or /eval canvas and confirm the project. The skills work without experimental mode.
Installing from an existing CLI conversation
Send these commands one at a time at Copilot's prompt, then start a fresh conversation in your project:
/plugin marketplace add github/evals/plugin install eval@github-copilot-evalsIn GitHub Copilot App
Need the app? Install GitHub Copilot App.
- Open Customize > Browse. You can filter to Plugins to narrow the list.
- Open the Add dropdown at the top right, then choose Manage marketplaces.
- In the Source field, enter:
Choose Add.github/evals - Confirm that github-copilot-evals appears in the popup's marketplace list, then close the popup.
- Back in Browse, search for
github-copilot-evals. Find eval from that marketplace. If it isn't installed, choose Install. - Confirm the toggle for eval is on. Start a fresh session in your project.
- At the Copilot prompt, enter
/eval help, then/eval canvasto open GitHub Copilot Evals native Canvas. Confirm the selected project.
For installed-plugin management, see the official App customization guide.
In Visual Studio
Use Visual Studio 2022 17.14+ or Visual Studio 2026 18.0+ with Edge WebView2 and a local solution or folder. The VS Code VSIX is not a Visual Studio installer.
- Use a supplied
GitHub.githubCopilotEvalsVisualStudiocandidate. Verify its SHA-256 checksum and install the Visual Studio VSIX:VSIXInstaller.exe "C:\path\to\github-copilot-evals-visualstudio-<version>.vsix" - Restart Visual Studio and open your local solution or folder.
- Open View > Other Windows > GitHub Copilot Evals, or enter
/eval-canvasin Copilot Chat. Confirm the selected folder.
Save or revert unsaved editor changes before checking, saving or running evals. Opening or installing never approves a run.
These steps cover a supplied candidate, not confirmation of public Marketplace availability.
Continue with your first eval in Visual Studio.
Try it
Create your first eval
Choose your Copilot agent above to see an example.
VS Code
In VS Code Copilot Chat, send:
Help me create evals for my skillsChoose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.
VS Code Agents
In the Agents window's Copilot conversation, send:
Help me create evals for my skillsChoose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.
Copilot CLI
At Copilot CLI's interactive prompt, send:
Help me create evals for my skillsChoose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.
GitHub Copilot App
In your GitHub Copilot App project conversation, send:
Help me create evals for my skillsChoose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.
Run your first eval
After saving, ask Copilot:
Run this saved eval and explain the result.Copilot chat and eval runs may use credits. Each run needs one approval of its costs, tools and execution effects. Evals does not enforce a total token or credit cap.