GitHub Copilot Evals

Get started with Evals.

Install Evals in your Copilot agent, then try it with a project skill.

Prerequisites

Open the project you want to test in your Copilot agent. Recommended setup for running evals:

Where do you use Copilot?

In VS Code

Use local desktop VS Code 1.138.0 or later with a trusted local project. This guide covers full authoring; bundled skills are unavailable in remote, web and untrusted workspaces.

Need VS Code? Install VS Code.

  1. Open GitHub Copilot Evals in the Visual Studio Marketplace, or search the Extensions view for GitHub.copilot-evals. Verify publisher GitHub.
  2. Choose Install and open your project.
  3. Open the Command Palette with Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS), then paste and run:
    GitHub Copilot Evals: Open Evals Canvas
    Confirm that Canvas shows the intended folder.

Using the Agents window? Continue with the VS Code Agents setup.

In VS Code Agents

These steps apply to Copilot sessions in the local Agents window.

New to Agents? Set up the Agents window.

  1. Follow the VS Code installation steps in regular VS Code's default profile, then return here.
  2. Open Settings with Ctrl+, (Windows/Linux) or Cmd+, (macOS). Search for chat.canvases.enabled and enable it.
  3. Open the Command Palette with Ctrl+Shift+P (Windows/Linux) or Cmd+Shift+P (macOS), then run:
    Chat: Open Agents window
    Open your project in that window.
  4. Start a fresh session. Enter /eval help to check that Evals commands loaded, then /eval canvas to open GitHub Copilot Evals native Canvas. Confirm that it shows your project.

This route uses the VS Code extension's bundled Plugin. Do not enable a second standalone Evals Plugin in the same Agents session.

In Copilot CLI

Need the CLI? Install Copilot CLI.

  1. Open a terminal and run these commands one at a time. Stop if a command fails:
    copilot plugin marketplace add github/evals
    copilot plugin install eval@github-copilot-evals
  2. Check that the source is github/evals and eval from github-copilot-evals is installed and enabled:
    copilot plugin marketplace list --json
    copilot plugin list --json
  3. Start a fresh Copilot conversation in your project to use the skills.

Optional: for native /eval commands or Canvas, enable CLI-wide experimental mode with /experimental. Start a fresh conversation, then use /eval help or /eval canvas and confirm the project. The skills work without experimental mode.

Installing from an existing CLI conversation

Send these commands one at a time at Copilot's prompt, then start a fresh conversation in your project:

/plugin marketplace add github/evals
/plugin install eval@github-copilot-evals
Updating Evals

To update github-copilot-evals, run these commands one at a time. Stop if a command fails:

copilot plugin marketplace update github-copilot-evals
copilot plugin update eval@github-copilot-evals

Start a fresh Copilot conversation afterward.

In GitHub Copilot App

Need the app? Install GitHub Copilot App.

  1. Open Customize > Browse. You can filter to Plugins to narrow the list.
  2. Open the Add dropdown at the top right, then choose Manage marketplaces.
  3. In the Source field, enter:
    github/evals
    Choose Add.
  4. Confirm that github-copilot-evals appears in the popup's marketplace list, then close the popup.
  5. Back in Browse, search for github-copilot-evals. Find eval from that marketplace. If it isn't installed, choose Install.
  6. Confirm the toggle for eval is on. Start a fresh session in your project.
  7. At the Copilot prompt, enter /eval help, then /eval canvas to open GitHub Copilot Evals native Canvas. Confirm the selected project.

For installed-plugin management, see the official App customization guide.

In Visual Studio

Use Visual Studio 2022 17.14+ or Visual Studio 2026 18.0+ with Edge WebView2 and a local solution or folder. The VS Code VSIX is not a Visual Studio installer.

  1. Use a supplied GitHub.githubCopilotEvalsVisualStudio candidate. Verify its SHA-256 checksum and install the Visual Studio VSIX:
    VSIXInstaller.exe "C:\path\to\github-copilot-evals-visualstudio-<version>.vsix"
  2. Restart Visual Studio and open your local solution or folder.
  3. Open View > Other Windows > GitHub Copilot Evals, or enter /eval-canvas in Copilot Chat. Confirm the selected folder.

Save or revert unsaved editor changes before checking, saving or running evals. Opening or installing never approves a run.

These steps cover a supplied candidate, not confirmation of public Marketplace availability.

Continue with your first eval in Visual Studio.

Try it

Create your first eval

VS Code

In VS Code Copilot Chat, send:

Help me create evals for my skills

Choose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.

VS Code Agents

In the Agents window's Copilot conversation, send:

Help me create evals for my skills

Choose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.

Copilot CLI

At Copilot CLI's interactive prompt, send:

Help me create evals for my skills

Choose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.

GitHub Copilot App

In your GitHub Copilot App project conversation, send:

Help me create evals for my skills

Choose a skill and scenarios with Copilot. Review the proposed cases and checks, then save the eval.

Run your first eval

After saving, ask Copilot:

Run this saved eval and explain the result.

Copilot chat and eval runs may use credits. Each run needs one approval of its costs, tools and execution effects. Evals does not enforce a total token or credit cap.