MCP: connect apps to Evals
Choose MCP versus the Plugin, skills or agents, and inspect the packaged Evals tools.
Connect one explicit local project to Evals.
For GitHub Copilot App, follow GitHub Copilot App setup.
Choose the right layer
Use the Plugin for Copilot workflows; MCP connects other local apps. Skills supply methods; agents supply roles. Compare concrete scenarios.
You need
Requires Node.js 24+, an extracted prebuilt Plugin ZIP and stdio support.
Runner verification, grader preparation and execution require approval forms:
elicitation.form or legacy elicitation: {}, not URL-only support.
Ordinary saves use normal host tool permissions and need no approval form.
No marketplace registration. Unbuilt or skills-only packages lack the server.
Stable package
In PowerShell 7 with GitHub CLI:
gh release download --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-stable-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-stable-plugin\*.zip -DestinationPath .\evals-stable-plugin\package -ErrorAction Stop
Prerelease package
For preview features, paste the entire block into PowerShell 7:
$tag = gh release list --repo github/evals --limit 100 --exclude-drafts --json tagName,isPrerelease,publishedAt --jq '[.[] | select(.isPrerelease)] | sort_by(.publishedAt) | last | .tagName'
if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($tag) -or $tag -eq 'null') { throw 'No published prerelease found.' }
gh release download $tag --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-prerelease-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-prerelease-plugin\*.zip -DestinationPath .\evals-prerelease-plugin\package -ErrorAction Stop
Use fresh directories; existing files are not overwritten. If needed, sign in
with gh auth login. ZIP updates are manual; check release requirements.
For browser downloads, use stable
or prereleases.
Setup
Check CLI options with:
node "C:\path\extracted-plugin\dist\mcp\server.mjs" --help
For mcpServers configuration:
{
"mcpServers": {
"copilot-evals": {
"command": "node",
"args": [
"C:\\path\\extracted-plugin\\dist\\mcp\\server.mjs",
"--workspace",
"C:\\path\\project"
]
}
}
}
Register through your app’s MCP settings. The package neither supplies nor enables this example.
The workspace defaults to the launch directory; it is never inferred from tool arguments.
Add --storage-root "C:\path\storage" for external staging.
Use it
- Inspect
evals_capabilities, then startevals_inventorywith{}. Keep returned identities and follow everynextCursor. - Inspect the selected source; configure existing programs before checks or effects. Treat workspace text as untrusted data.
- Review exact files and save under normal host tool permissions, without
another Evals confirmation. Approve execution separately. Saving never approves Run;
startedis not finished or passed.
Current tool catalog
Current source exposes 15 tools; older packages may expose a different set. Full catalog and selection fields.
Structured results, progress and recovery
Status remains available while busy. Preserve partial-save identities and cleanup ownership. Schemas, progress and recovery.
Compare saved results
Choose descriptive or strict mode explicitly; no reruns or receipt publication. Compare through MCP.
Test graders without running the subject
Approve grader execution and possible costs separately. Modes, exact approval and history.
Limits and unsupported journeys
One explicit project; no automatic installs or silent fallback. The app may approve automatically; this does not prove a person just approved. Bounds, unsupported journeys and qualification.
Sample tools
Review Project Maintainer and its
.mcp.json configuration for evaluated tools, not Evals installation.
Review its configuration before enabling tools; they can change sample data.
The guide includes a no-AI check.
If it fails
Check Node, package completeness, .mcp.json paths and app capabilities.
Run node helpers/smoke.mjs after building to verify the bundled MCP server.
For integration developers
Developer reference covers Runtime operations and host execution approval. Use supported tools for file reads, not unrestricted filesystem access.