Developer packages
Build the VS Code extension from source or install a local Plugin package.
Build the VS Code extension
In a source checkout, use Node.js 24+ and pnpm 12.8.1:
corepack enable
pnpm install --frozen-lockfile
if ($IsWindows) { pnpm telemetry:build-native }
if ($IsWindows) { pnpm process:build-native }
pnpm package:vsix
Install artifacts\github-copilot-evals-<version>-<target>.vsix (using the version in
packages/vscode/package.json) with Extensions: Install from VSIX,
then reload. The source identity is GitHub.copilot-evals.
The default target matches the current Node platform and architecture.
Windows builds require MSVC and Spectre-mitigated libraries for the native
device-ID helper. Release packaging uses --all-targets and a universal Plugin
containing both Windows architectures; see CONTRIBUTING for its native inputs.
If a new package has a different publisher, review and uninstall the old
extension before installing both sets of controls.
For sample builds, run pnpm build after installation.
For the Visual Studio extension candidate, run pnpm package:vs to emit
artifacts\github-copilot-evals-visualstudio-<version>.vsix.
CONTRIBUTING
has the development setup.
Install a local CLI package
Use a compiled Plugin ZIP from releases
when you want a local package instead of a channel registration.
Current stable and prerelease catalogs already include
the compiled tools; an unbuilt source checkout or older skills-only package does not.
The compiled Copilot CLI / GitHub Copilot App extension registers /eval compare for reading saved
results or preparing a separately approved skill experiment. The Compare
skill guides the same workflow; an older skills-only package does not provide native
commands or executable tools. Check your package’s documented capabilities.
Stable ZIP
Download github-copilot-evals-plugin-<version>.zip from
latest stable, or paste into
PowerShell 7 with GitHub CLI, Copilot CLI and Git installed:
gh release download --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-stable-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-stable-plugin\*.zip -DestinationPath .\evals-stable-plugin\package -ErrorAction Stop
copilot plugin marketplace add (Resolve-Path .\evals-stable-plugin\package).Path && copilot plugin install eval@github-copilot-evals
Prerelease ZIP
Download the Plugin ZIP from the newest published prerelease, or paste the entire block into PowerShell 7:
$tag = gh release list --repo github/evals --limit 100 --exclude-drafts --json tagName,isPrerelease,publishedAt --jq '[.[] | select(.isPrerelease)] | sort_by(.publishedAt) | last | .tagName'
if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($tag) -or $tag -eq 'null') { throw 'No published prerelease found.' }
gh release download $tag --repo github/evals --pattern "github-copilot-evals-plugin-*.zip" --dir evals-prerelease-plugin
if ($LASTEXITCODE -ne 0) { throw 'Plugin download failed.' }
Expand-Archive -Path .\evals-prerelease-plugin\*.zip -DestinationPath .\evals-prerelease-plugin\package -ErrorAction Stop
copilot plugin marketplace add (Resolve-Path .\evals-prerelease-plugin\package).Path && copilot plugin install eval@github-copilot-evals
Read the release notes/checksums and use fresh download
directories; no existing file is overwritten. Sign in with gh auth login
if needed. Inspect an existing github-copilot-evals registration and follow the
switching steps below before installing.
Do not delete or move the extracted folder while registered: local sources
load live files. ZIP updates are manual, not channel tracking.
New ZIPs contain .github/plugin/marketplace.json and plugin.json at the
extracted root. Older downloads may differ; follow their version-specific
instructions. Use the named marketplace route, not a deprecated direct-folder
Plugin install. Restart or start a new session and enter /eval help in
Copilot’s prompt to check loading.
Before switching package sources
Repository installs, local checkouts and compiled ZIPs are alternative sources
for the same github-copilot-evals marketplace. Inspect them first:
copilot plugin marketplace list --json
copilot plugin list --json
If you choose to switch, review and explicitly uninstall the affected plugins
before copilot plugin marketplace remove github-copilot-evals, then register the new source.
For the Evals Plugin:
copilot plugin uninstall eval@github-copilot-evals && copilot plugin marketplace remove github-copilot-evals
Removal is refused while plugins remain installed. Do not force removal, silently replace a source or enable a disabled entry. Matching version numbers do not establish matching sources.
Refresh a local build
Marketplace registration persists. A local directory-source Plugin loads live
files; after rebuilding, use /restart or a new session.
copilot plugin marketplace update github-copilot-evals refreshes its catalog, not its local
code. No update or copy is needed for live files.
For repository sources, refresh the catalog and use
copilot plugin update eval@github-copilot-evals.
Plugin setup · Extension setup
Connecting an app
Apps select the project and collect approval. Runtime manages files, execution, permissions, recovery records and Stop. Each request stays linked to its project session, proposal, eval and run, and records which source, settings and runtime it uses. Reopening the same path does not revive old approvals.
A fingerprint detects changed content; it does not grant permission. Tokens visible to the AI do not prove a person approved anything. Invalid requests, retired formats and unavailable features return errors. Say when data is omitted or redacted. A request or process failure is not a failed check on the AI’s work.
Standalone MCP
Use the packaged stdio connection for one explicitly selected local project. Runtime owns files, execution, approval reservations and cleanup; MCP adapts those operations rather than providing another evaluator.
Choose the right layer
These layers work together: plugins package customizations, skills supply methods, agents define specialist roles/toolsets, and MCP exposes callable operations. An agent can use a skill and an MCP tool distributed by a plugin.
| Scenario | Choose |
|---|---|
| Create, review, improve, run or compare evals in supported Copilot hosts | Plugin or extension, using existing host workflows and controls. |
| Connect another local app without marketplace registration | MCP, with structured project-bound tools and compatible form approval for effects. Qualify the actual host. |
| Inspect an eval or explain a saved failure | MCP readers; the host supplies reasoning without running subjects or graders. |
| Teach release-note writing from completed work | A skill: instructions and resources, not an execution backend. |
| Reuse a specialist release reviewer | A custom agent with its role and tools, optionally using skills or MCP. |
| Share an executable work-item operation across hosts | A domain MCP server. That server can be an eval target; it is not this Evals connection. |
| Run unattended repository checks | Runner / CI, not automatic acceptance of interactive approval forms. |
GitHub describes plugins, skills and custom agents. Those features alone do not establish Evals support in a host.
Current tool catalog
Current source exposes 15 tools; older packages can expose a different set. Inspect
tools/list for actual argument and output schemas. Host-native names such as
draft_eval and run_eval are a different catalog, not MCP aliases.
| Tool | Capability and boundary |
|---|---|
evals_inventory | Page through admitted project assets and saved runs without activation or validation. |
evals_read_definition | Read a selected, redacted static definition; no validation or execution. |
evals_read_saved_run | Read historical evidence; no rerun, regrading or current-source certification. |
evals_capabilities | Inspect connection tools, form support, unsupported journeys and retained runtime configuration without executable, network or model probes. |
evals_status | Read canonical run progress, pending approval, draft recovery identities/flags and grader-test state, including while busy. No retry or ownership cleanup. |
evals_compare_saved_runs | Read two selected saved runs with explicit descriptive/strict mode; no reruns or receipt publication. |
evals_grader_catalog | Approve preparation, then resolve one eval’s stimuli/trajectory fixtures. Native parameter resolution may execute; subjects, reference commands and graders do not. |
evals_test_graders | Prepare and separately approve oracle, negative-control or trajectory checks. Setup/reference commands, custom graders and model judges may execute or cost credits; no subject agent runs. |
evals_read_grader_tests | Page through historical grader-test summaries or read one returned ID; no execution or source certification. |
evals_draft | Bind an emitted source or explicit project scope for Create, or one original eval with intent: "improve"; retain/read/retry private request data without publishing authored files or calling a model. |
evals_read_draft | Export retained work as unverified JSON text pages, including after source changes; no new binding or Save permission. Pin subsequent pages with the first page’s revision as expectedRevision. |
evals_configure_runtime | Approve verification of supplied Node/runner paths and optional Copilot CLI path; no installation or project/CI setup. |
evals_save_draft | Validate and exclusively create new eval/fixture files for a bound requestId under host tool permissions, without another approval form; original request/proposal IDs recover the same save. No automatic Run. |
evals_run | Prepare one selected saved eval and start after its complete review/approval. Authentication is github or explicitly configured. |
evals_cancel | Interrupt this session’s pending action or stop active work; stopping does not confirm cleanup. |
Start inventory with {}; retain workspaceId, inventoryId and emitted
selection handle. Continue using only {workspaceId, cursor} with each
nextCursor. Refresh with {workspaceId}; query or kind starts a filtered
snapshot, not continuation. Run rows expose canonical runId for exact
correlation, not selection/approval authority. Never guess identity from a title.
Bind Create/Improve before proposing cases. Configure existing programs before
validation/save/execution, review complete proposed files, and save through
normal host tool permissions. Only a requested Run can continue: refresh inventory after saving,
select the exact saved eval, and obtain its independent prepared-run approval.
Selection, saving and a runOffer are not execution consent.
Structured results, progress and recovery
Unfinished authoring requests retain data privately for the same admitted project. Recovery rechecks the original source/root; a fresh Save invocation recreates validation without another save form. Persisted IDs never restore execution approval, process ownership or automatic execution.
Successful tools return validated structuredContent and equivalent JSON text.
outputSchema is published in tools/list. Preserve untrusted-evidence labels,
error categories, truncation and omitted-evidence counts; data is not authority.
Tool errors retain isError: true and their JSON text envelope without
structuredContent, so clients do not validate failures as successful results.
Escaped source/report previews may be shortened further to fit both result
channels; truncated stays explicit and the original source hash is unchanged.
Supply _meta.progressToken through client request options for long effect calls.
Run and grader checks forward shared owner phases while that request is active.
Notifications stop at request completion and never reset approval deadlines.
The three original inventory/definition/report readers remain non-notifying.
After Run returns started, query evals_status with {workspaceId} and inspect
the selected saved report; starting is not finishing or passing.
Status/capabilities bypass the ordinary slot without invalidating inventory.
Status reports retained session state, not cross-session journal adoption or
historical lock scans. It distinguishes pending approval, active work, actual
outcomes and uncertain cleanup. Its draft object retains original request/
proposal, save-attempt and recovery flags. Preserve partial-save request/proposal IDs;
status never retries, discards requests or removes locks. Use the original
recovery procedure, not another writer or an invented replacement ID.
Actions that are not Runs do not report a previous Run’s ID or an action name as
their Run ID. Cancellation remains active until owned work settles, then retains
the actual terminal outcome. Existing Runs keep their own IDs.
During grader checks or approval for another action, unrelated Run IDs and
counters are empty. Grader checks retain their separate graderTest identity.
Compare saved results
Refresh unfiltered inventory and select two distinct emitted run handles.
Pass workspaceId, inventoryId, originalHandle, changedHandle and explicit
mode to evals_compare_saved_runs. descriptive reads independent historical
outcomes without claiming a matched experiment. strict returns eligible
axes; repeat with an exact returned axisKey for its report.
Ineligible evidence is refused, not silently changed to descriptive mode.
Neither mode reruns subjects/graders, changes sources or publishes a receipt.
Comparison evidence limits still apply.
Large reports return a UTF-8-safe prefix with truncated: true to fit both
structured output and its text mirror; original saved records are unchanged.
Test graders without running the subject
Configure the existing runtime and select an emitted eval. Use
evals_grader_catalog with workspaceId, inventoryId and handle to resolve
stimuli and declared saved-trajectory fixture IDs. Native parameter resolution
is preparation, not a subject run. Catalog text is untrusted data.
Catalog and test preparation require their own per-call permission before any
native resolution. Declining starts no resolver or staging. Preparation consent
does not approve the separately reviewed grader execution.
Catalogs may return fewer entries than their item limits to fit the aggregate
wire budget. truncated: true marks omitted entries; emitted identifiers remain exact.
Call evals_test_graders with that selection and one mode:
| Mode | What is tested |
|---|---|
oracle | Graders against reference answers; optional exact stimulus selects one case. |
negative-control | Native no-golden-input controls, not inversion of the overall grade. |
trajectory | A declared fixtureId and its expected overall/per-grader outcomes without rerunning its subject. |
Review the complete prepared command, selected cases/fixture, input hashes, runtime identities/version and effect warnings. Source/runtime changes invalidate the selection or prepared check. Setup/reference commands, custom graders and model judges may run; costs can be unknown, not zero. Errors are not expected failures or passing subject results. A grader check does not prove subject-run compatibility.
The call returns the canonical outcome after execution/cleanup. Cancel pending
or active checks with evals_cancel; inspect status until cleanup settles.
Evidence stays in .copilot\evals\grader-tests, separate from subject history.
Read summaries with evals_read_grader_tests and {workspaceId}; follow
nextCursor using {workspaceId, cursor}, or pass an exact returned id for
one report. ID and cursor reads are mutually exclusive.
Pages use stable ID order, not a frozen snapshot; refresh after history changes. At most 20 records are read per page. Report text is capped at 16,000 UTF-8 bytes so structured output and its text mirror fit one frame. Omitted result/grader counts and truncation are explicit; original saved bytes remain unchanged.
Limits and unsupported journeys
The connection exposes stdio tools, not HTTP/SSE, MCP resources, prompt templates
or model sampling. It installs no /eval commands, helper skills, agents or
Canvas; the host supplies reasoning and UI. Standalone MCP has no included/
omitted skill experiments, project/CI Setup, automatic conversation capture or
native-task taskTools selection. Report unsupported work or use an explicitly
identified supported-host handoff, never a substitute Run. Runtime verification
is not the Setup skill.
Discovery covers only the startup project, not global/Plugin folders. Inventory handles expire after 60 seconds; rescans replace them. Pages contain at most 20 rows, with 1,000 assets and 128 runs exposed overall. Truncated or unavailable evidence is not empty success.
Calls are serial: one operation slot, no queue; status/capabilities and owned cancellation retain out-of-band access. Reads/input have a 15-second deadline. Configuration/save/run/grader-test calls have one five-minute pre-effect approval window, including preparation and every review page. Execution has separate finite Runtime limits. Configure client timeouts for the intended call; a timeout does not authorize retrying an uncertain effect. Preparation and final grader approval share that window; accepting preparation does not reset its deadline. One lazy private storage root is retained per owner context. Unconfirmed grader cleanup blocks subsequent Run/save/configuration and grader effects, not inert status or evidence reads.
Non-bundle arguments are limited to 4 KiB encoded JSON, and JSON-RPC request IDs
to 16 KiB. Inline evals_save_draft bundles accept up to 3,200,000 encoded
argument bytes within a 3,250,000-byte input frame and enforce 1–16 files,
512,000 UTF-8 bytes per file and in total. DTOs stay within 96 KiB and stdout
frames within 256 KiB.
Form approval requires elicitation.form or legacy elicitation: {};
URL-only elicitation is insufficient. See the
MCP elicitation contract.
Hosts may accept through automation: delegated permission is not proof of a
fresh human decision or qualification of the selected manual-confirmation
experience. Missing forms refuse execution and the other form-gated effects,
not ordinary saves. Preserve partial-save
identities and uncertain cleanup ownership. Run request IDs deduplicate the
same intent, not authorize changed inputs or a new execution.
For target servers being evaluated, the bounded profile permits one owned
local Node stdio server and one case per fresh run. Python, uv run and uvx
stdio targets use separately approved trusted full-engine execution. Remote
targets are not included. Target prerequisites
are separate from this Evals connection.
Model-free SDK/package tests are not an installed-host or live-model receipt. The approval reference documents the MCP approval mechanics and tool catalog.
Saved record formats
Evals saves use an unversioned run-evidence wrapper. Project history rejects
older raw records without rewriting them.
Core checks the record inside the wrapper. Missing source details stay null;
approved resaves keep optional metadata and each case’s source details.
Fingerprints do not establish authenticity or complete activity. Requested models and recorded attempts remain distinct. A strict comparison requires compatible identities and one eligible change; reading it does not execute, regrade, save, or adopt anything.