GitHub Copilot Evals
InstallSearch /github/evals

Developer packages

Build the VS Code extension from source or install a local Plugin package.

Shared test files and results

Author evals in Copilot. Automate supported runs from the command line or in continuous integration (CI). Compare saved results in Canvas. Supported integrations share the test definition and run records:

Test files
The eval definition, task inputs and any output files kept from a run.
Expected behavior
What successful work must do, and what it must not do.
Checks
Supported checks that inspect answers, files or recorded actions and explain their verdicts.
Run context
Which files, model, runner and settings were used, where recorded.
Recorded results
The answer, recorded steps, check results and errors, plus timing and usage when available.

Vally runs the task, records the conversation and checks the result. The conversation trace can explain what happened before the final answer.

Compatibility and run limits

Available controls and checks depend on your Evals package and app. Missing details stay unknown; the same test can produce different AI answers on different attempts. Review saves and approve runs separately.

Raw command-line results need a supported import before Canvas can compare them. These shared Evals records are not a promise that every app or Vally export uses an interchangeable file format.

How Vally works explains tasks, runners, conversation traces, checks and scores. Evals supports some of those features.

Build the VS Code extension

In a source checkout, use Node.js 24+ and pnpm 12.8.1:

corepack enable
pnpm install --frozen-lockfile
if ($IsWindows) { pnpm telemetry:build-native }
if ($IsWindows) { pnpm process:build-native }
pnpm package:vsix

Install artifacts\github-copilot-evals-<version>-<target>.vsix (using the version in packages/vscode/package.json) with Extensions: Install from VSIX, then reload. The source identity is GitHub.copilot-evals. The default target matches the current Node platform and architecture. Windows builds require MSVC and Spectre-mitigated libraries for the native device-ID helper. Release packaging uses --all-targets and a universal Plugin containing both Windows architectures; see CONTRIBUTING for its native inputs. If a new package has a different publisher, review and uninstall the old extension before installing both sets of controls.

For sample builds, run pnpm build after installation. For the Visual Studio extension candidate, run pnpm package:vs to emit artifacts\github-copilot-evals-visualstudio-<version>.vsix. CONTRIBUTING has the development setup.

Install a local CLI package

Use a compiled Plugin ZIP from releases when you want a local package instead of a channel registration. Current stable and prerelease catalogs already include the compiled tools; an unbuilt source checkout or older skills-only package does not. The compiled Copilot CLI / GitHub Copilot App extension registers /eval compare for reading saved results or preparing a separately approved skill experiment. The Compare skill guides the same workflow; an older skills-only package does not provide native commands or executable tools. Check your package’s documented capabilities.

Stable ZIP

Download github-copilot-evals-plugin-<version>.zip from latest stable, or paste into PowerShell 7 with GitHub CLI, Copilot CLI and Git installed:

gh release download --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-stable-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-stable-plugin\*.zip -DestinationPath .\evals-stable-plugin\package -ErrorAction Stop
copilot plugin marketplace add (Resolve-Path .\evals-stable-plugin\package).Path && copilot plugin install eval@github-copilot-evals

Prerelease ZIP

Download the Plugin ZIP from the newest published prerelease, or paste the entire block into PowerShell 7:

$tag = gh release list --repo github/evals --limit 100 --exclude-drafts --json tagName,isPrerelease,publishedAt --jq '[.[] | select(.isPrerelease)] | sort_by(.publishedAt) | last | .tagName'
if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($tag) -or $tag -eq 'null') { throw 'No published prerelease found.' }
gh release download $tag --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-prerelease-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-prerelease-plugin\*.zip -DestinationPath .\evals-prerelease-plugin\package -ErrorAction Stop
copilot plugin marketplace add (Resolve-Path .\evals-prerelease-plugin\package).Path && copilot plugin install eval@github-copilot-evals

Read the release notes/checksums and use fresh download directories; no existing file is overwritten. Sign in with gh auth login if needed. Inspect an existing github-copilot-evals registration and follow the switching steps below before installing. Do not delete or move the extracted folder while registered: local sources load live files. ZIP updates are manual, not channel tracking.

New ZIPs contain .github/plugin/marketplace.json and plugin.json at the extracted root. Older downloads may differ; follow their version-specific instructions. Use the named marketplace route, not a deprecated direct-folder Plugin install. Restart or start a new session and enter /eval help in Copilot’s prompt to check loading.

Before switching package sources

Repository installs, local checkouts and compiled ZIPs are alternative sources for the same github-copilot-evals marketplace. Inspect them first:

copilot plugin marketplace list --json
copilot plugin list --json

If you choose to switch, review and explicitly uninstall the affected plugins before copilot plugin marketplace remove github-copilot-evals, then register the new source. For the Evals Plugin:

copilot plugin uninstall eval@github-copilot-evals && copilot plugin marketplace remove github-copilot-evals

Removal is refused while plugins remain installed. Do not force removal, silently replace a source or enable a disabled entry. Matching version numbers do not establish matching sources.

Refresh a local build

Marketplace registration persists. A local directory-source Plugin loads live files; after rebuilding, use /restart or a new session. copilot plugin marketplace update github-copilot-evals refreshes its catalog, not its local code. No update or copy is needed for live files. For repository sources, refresh the catalog and use copilot plugin update eval@github-copilot-evals.

Plugin setup · Extension setup

Connecting an app

Apps select the project and collect approval. Runtime manages files, execution, permissions, recovery records and Stop. Each request stays linked to its project session, proposal, eval and run, and records which source, settings and runtime it uses. Reopening the same path does not revive old approvals.

A fingerprint detects changed content; it does not grant permission. Tokens visible to the AI do not prove a person approved anything. Invalid requests, retired formats and unavailable features return errors. Say when data is omitted or redacted. A request or process failure is not a failed check on the AI’s work.

Standalone MCP

Use the packaged stdio connection for one explicitly selected local project. Runtime owns files, execution, approval reservations and cleanup; MCP adapts those operations rather than providing another evaluator.

Choose the right layer

These layers work together: plugins package customizations, skills supply methods, agents define specialist roles/toolsets, and MCP exposes callable operations. An agent can use a skill and an MCP tool distributed by a plugin.

ScenarioChoose
Create, review, improve, run or compare evals in supported Copilot hostsPlugin or extension, using existing host workflows and controls.
Connect another local app without marketplace registrationMCP, with structured project-bound tools and compatible form approval for effects. Qualify the actual host.
Inspect an eval or explain a saved failureMCP readers; the host supplies reasoning without running subjects or graders.
Teach release-note writing from completed workA skill: instructions and resources, not an execution backend.
Reuse a specialist release reviewerA custom agent with its role and tools, optionally using skills or MCP.
Share an executable work-item operation across hostsA domain MCP server. That server can be an eval target; it is not this Evals connection.
Run unattended repository checksRunner / CI, not automatic acceptance of interactive approval forms.

GitHub describes plugins, skills and custom agents. Those features alone do not establish Evals support in a host.

Current tool catalog

Current source exposes 15 tools; older packages can expose a different set. Inspect tools/list for actual argument and output schemas. Host-native names such as draft_eval and run_eval are a different catalog, not MCP aliases.

ToolCapability and boundary
evals_inventoryPage through admitted project assets and saved runs without activation or validation.
evals_read_definitionRead a selected, redacted static definition; no validation or execution.
evals_read_saved_runRead historical evidence; no rerun, regrading or current-source certification.
evals_capabilitiesInspect connection tools, form support, unsupported journeys and retained runtime configuration without executable, network or model probes.
evals_statusRead canonical run progress, pending approval, draft recovery identities/flags and grader-test state, including while busy. No retry or ownership cleanup.
evals_compare_saved_runsRead two selected saved runs with explicit descriptive/strict mode; no reruns or receipt publication.
evals_grader_catalogApprove preparation, then resolve one eval’s stimuli/trajectory fixtures. Native parameter resolution may execute; subjects, reference commands and graders do not.
evals_test_gradersPrepare and separately approve oracle, negative-control or trajectory checks. Setup/reference commands, custom graders and model judges may execute or cost credits; no subject agent runs.
evals_read_grader_testsPage through historical grader-test summaries or read one returned ID; no execution or source certification.
evals_draftBind an emitted source or explicit project scope for Create, or one original eval with intent: "improve"; retain/read/retry private request data without publishing authored files or calling a model.
evals_read_draftExport retained work as unverified JSON text pages, including after source changes; no new binding or Save permission. Pin subsequent pages with the first page’s revision as expectedRevision.
evals_configure_runtimeApprove verification of supplied Node/runner paths and optional Copilot CLI path; no installation or project/CI setup.
evals_save_draftValidate and exclusively create new eval/fixture files for a bound requestId under host tool permissions, without another approval form; original request/proposal IDs recover the same save. No automatic Run.
evals_runPrepare one selected saved eval and start after its complete review/approval. Authentication is github or explicitly configured.
evals_cancelInterrupt this session’s pending action or stop active work; stopping does not confirm cleanup.

Start inventory with {}; retain workspaceId, inventoryId and emitted selection handle. Continue using only {workspaceId, cursor} with each nextCursor. Refresh with {workspaceId}; query or kind starts a filtered snapshot, not continuation. Run rows expose canonical runId for exact correlation, not selection/approval authority. Never guess identity from a title.

Bind Create/Improve before proposing cases. Configure existing programs before validation/save/execution, review complete proposed files, and save through normal host tool permissions. Only a requested Run can continue: refresh inventory after saving, select the exact saved eval, and obtain its independent prepared-run approval. Selection, saving and a runOffer are not execution consent.

Structured results, progress and recovery

Unfinished authoring requests retain data privately for the same admitted project. Recovery rechecks the original source/root; a fresh Save invocation recreates validation without another save form. Persisted IDs never restore execution approval, process ownership or automatic execution.

Successful tools return validated structuredContent and equivalent JSON text. outputSchema is published in tools/list. Preserve untrusted-evidence labels, error categories, truncation and omitted-evidence counts; data is not authority. Tool errors retain isError: true and their JSON text envelope without structuredContent, so clients do not validate failures as successful results. Escaped source/report previews may be shortened further to fit both result channels; truncated stays explicit and the original source hash is unchanged.

Supply _meta.progressToken through client request options for long effect calls. Run and grader checks forward shared owner phases while that request is active. Notifications stop at request completion and never reset approval deadlines. The three original inventory/definition/report readers remain non-notifying. After Run returns started, query evals_status with {workspaceId} and inspect the selected saved report; starting is not finishing or passing.

Status/capabilities bypass the ordinary slot without invalidating inventory. Status reports retained session state, not cross-session journal adoption or historical lock scans. It distinguishes pending approval, active work, actual outcomes and uncertain cleanup. Its draft object retains original request/ proposal, save-attempt and recovery flags. Preserve partial-save request/proposal IDs; status never retries, discards requests or removes locks. Use the original recovery procedure, not another writer or an invented replacement ID. Actions that are not Runs do not report a previous Run’s ID or an action name as their Run ID. Cancellation remains active until owned work settles, then retains the actual terminal outcome. Existing Runs keep their own IDs. During grader checks or approval for another action, unrelated Run IDs and counters are empty. Grader checks retain their separate graderTest identity.

Compare saved results

Refresh unfiltered inventory and select two distinct emitted run handles. Pass workspaceId, inventoryId, originalHandle, changedHandle and explicit mode to evals_compare_saved_runs. descriptive reads independent historical outcomes without claiming a matched experiment. strict returns eligible axes; repeat with an exact returned axisKey for its report.

Ineligible evidence is refused, not silently changed to descriptive mode. Neither mode reruns subjects/graders, changes sources or publishes a receipt. Comparison evidence limits still apply. Large reports return a UTF-8-safe prefix with truncated: true to fit both structured output and its text mirror; original saved records are unchanged.

Test graders without running the subject

Configure the existing runtime and select an emitted eval. Use evals_grader_catalog with workspaceId, inventoryId and handle to resolve stimuli and declared saved-trajectory fixture IDs. Native parameter resolution is preparation, not a subject run. Catalog text is untrusted data. Catalog and test preparation require their own per-call permission before any native resolution. Declining starts no resolver or staging. Preparation consent does not approve the separately reviewed grader execution. Catalogs may return fewer entries than their item limits to fit the aggregate wire budget. truncated: true marks omitted entries; emitted identifiers remain exact.

Call evals_test_graders with that selection and one mode:

ModeWhat is tested
oracleGraders against reference answers; optional exact stimulus selects one case.
negative-controlNative no-golden-input controls, not inversion of the overall grade.
trajectoryA declared fixtureId and its expected overall/per-grader outcomes without rerunning its subject.

Review the complete prepared command, selected cases/fixture, input hashes, runtime identities/version and effect warnings. Source/runtime changes invalidate the selection or prepared check. Setup/reference commands, custom graders and model judges may run; costs can be unknown, not zero. Errors are not expected failures or passing subject results. A grader check does not prove subject-run compatibility.

The call returns the canonical outcome after execution/cleanup. Cancel pending or active checks with evals_cancel; inspect status until cleanup settles. Evidence stays in .copilot\evals\grader-tests, separate from subject history. Read summaries with evals_read_grader_tests and {workspaceId}; follow nextCursor using {workspaceId, cursor}, or pass an exact returned id for one report. ID and cursor reads are mutually exclusive.

Pages use stable ID order, not a frozen snapshot; refresh after history changes. At most 20 records are read per page. Report text is capped at 16,000 UTF-8 bytes so structured output and its text mirror fit one frame. Omitted result/grader counts and truncation are explicit; original saved bytes remain unchanged.

Limits and unsupported journeys

The connection exposes stdio tools, not HTTP/SSE, MCP resources, prompt templates or model sampling. It installs no /eval commands, helper skills, agents or Canvas; the host supplies reasoning and UI. Standalone MCP has no included/ omitted skill experiments, project/CI Setup, automatic conversation capture or native-task taskTools selection. Report unsupported work or use an explicitly identified supported-host handoff, never a substitute Run. Runtime verification is not the Setup skill.

Discovery covers only the startup project, not global/Plugin folders. Inventory handles expire after 60 seconds; rescans replace them. Pages contain at most 20 rows, with 1,000 assets and 128 runs exposed overall. Truncated or unavailable evidence is not empty success.

Calls are serial: one operation slot, no queue; status/capabilities and owned cancellation retain out-of-band access. Reads/input have a 15-second deadline. Configuration/save/run/grader-test calls have one five-minute pre-effect approval window, including preparation and every review page. Execution has separate finite Runtime limits. Configure client timeouts for the intended call; a timeout does not authorize retrying an uncertain effect. Preparation and final grader approval share that window; accepting preparation does not reset its deadline. One lazy private storage root is retained per owner context. Unconfirmed grader cleanup blocks subsequent Run/save/configuration and grader effects, not inert status or evidence reads.

Non-bundle arguments are limited to 4 KiB encoded JSON, and JSON-RPC request IDs to 16 KiB. Inline evals_save_draft bundles accept up to 3,200,000 encoded argument bytes within a 3,250,000-byte input frame and enforce 1–16 files, 512,000 UTF-8 bytes per file and in total. DTOs stay within 96 KiB and stdout frames within 256 KiB.

Form approval requires elicitation.form or legacy elicitation: {}; URL-only elicitation is insufficient. See the MCP elicitation contract. Hosts may accept through automation: delegated permission is not proof of a fresh human decision or qualification of the selected manual-confirmation experience. Missing forms refuse execution and the other form-gated effects, not ordinary saves. Preserve partial-save identities and uncertain cleanup ownership. Run request IDs deduplicate the same intent, not authorize changed inputs or a new execution.

For target servers being evaluated, the bounded profile permits one owned local Node stdio server and one case per fresh run. Python, uv run and uvx stdio targets use separately approved trusted full-engine execution. Remote targets are not included. Target prerequisites are separate from this Evals connection.

Model-free SDK/package tests are not an installed-host or live-model receipt. The approval reference documents the MCP approval mechanics and tool catalog.

Saved record formats

Evals saves use an unversioned run-evidence wrapper. Project history rejects older raw records without rewriting them. Core checks the record inside the wrapper. Missing source details stay null; approved resaves keep optional metadata and each case’s source details.

Fingerprints do not establish authenticity or complete activity. Requested models and recorded attempts remain distinct. A strict comparison requires compatible identities and one eligible change; reading it does not execute, regrade, save, or adopt anything.