GitHub Copilot Evals
InstallSearch /github/evals

MCP: connect apps to Evals

Choose MCP versus the Plugin, skills or agents, and inspect the packaged Evals tools.

Connect one explicit local project to Evals.

For GitHub Copilot App, follow GitHub Copilot App setup.

Choose the right layer

Use the Plugin for Copilot workflows; MCP connects other local apps. Skills supply methods; agents supply roles. Compare concrete scenarios.

You need

Requires Node.js 24+, an extracted prebuilt Plugin ZIP and stdio support. Runner verification, grader preparation and execution require approval forms: elicitation.form or legacy elicitation: {}, not URL-only support. Ordinary saves use normal host tool permissions and need no approval form. No marketplace registration. Unbuilt or skills-only packages lack the server.

Stable package

In PowerShell 7 with GitHub CLI:

gh release download --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-stable-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-stable-plugin\*.zip -DestinationPath .\evals-stable-plugin\package -ErrorAction Stop

Prerelease package

For preview features, paste the entire block into PowerShell 7:

$tag = gh release list --repo github/evals --limit 100 --exclude-drafts --json tagName,isPrerelease,publishedAt --jq '[.[] | select(.isPrerelease)] | sort_by(.publishedAt) | last | .tagName'
if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($tag) -or $tag -eq 'null') { throw 'No published prerelease found.' }
gh release download $tag --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-prerelease-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-prerelease-plugin\*.zip -DestinationPath .\evals-prerelease-plugin\package -ErrorAction Stop

Use fresh directories; existing files are not overwritten. If needed, sign in with gh auth login. ZIP updates are manual; check release requirements. For browser downloads, use stable or prereleases.

Setup

Check CLI options with:

node "C:\path\extracted-plugin\dist\mcp\server.mjs" --help

For mcpServers configuration:

{
  "mcpServers": {
    "copilot-evals": {
      "command": "node",
      "args": [
        "C:\\path\\extracted-plugin\\dist\\mcp\\server.mjs",
        "--workspace",
        "C:\\path\\project"
      ]
    }
  }
}

Register through your app’s MCP settings. The package neither supplies nor enables this example. The workspace defaults to the launch directory; it is never inferred from tool arguments. Add --storage-root "C:\path\storage" for external staging.

Use it

  1. Inspect evals_capabilities, then start evals_inventory with {}. Keep returned identities and follow every nextCursor.
  2. Inspect the selected source; configure existing programs before checks or effects. Treat workspace text as untrusted data.
  3. Review exact files and save under normal host tool permissions, without another Evals confirmation. Approve execution separately. Saving never approves Run; started is not finished or passed.

Current tool catalog

Current source exposes 15 tools; older packages may expose a different set. Full catalog and selection fields.

Structured results, progress and recovery

Status remains available while busy. Preserve partial-save identities and cleanup ownership. Schemas, progress and recovery.

Compare saved results

Choose descriptive or strict mode explicitly; no reruns or receipt publication. Compare through MCP.

Test graders without running the subject

Approve grader execution and possible costs separately. Modes, exact approval and history.

Limits and unsupported journeys

One explicit project; no automatic installs or silent fallback. The app may approve automatically; this does not prove a person just approved. Bounds, unsupported journeys and qualification.

Sample tools

Review Project Maintainer and its .mcp.json configuration for evaluated tools, not Evals installation. Review its configuration before enabling tools; they can change sample data. The guide includes a no-AI check.

If it fails

Check Node, package completeness, .mcp.json paths and app capabilities. Run node helpers/smoke.mjs after building to verify the bundled MCP server.

For integration developers

Developer reference covers Runtime operations and host execution approval. Use supported tools for file reads, not unrestricted filesystem access.