GitHub Copilot Evals
InstallSearch /github/evals

What you can evaluate

Choose the skill, delegated agent or supported tool you want to test.

An asset is the source you test: a skill, delegated agent, or supported tool. Start with a question about what it should do.

Bring source, a task and an expected outcome

Open the project containing the real source; start Copilot CLI there or open that folder in your editor/GitHub Copilot App session. Installing Evals does not make every installed Plugin a discoverable local asset. You need permission to read the source and inputs, a realistic task, and a reason the expected outcome is correct that does not rely on the AI’s answer. No prompt text file is required.

SourceConcrete exampleQuestion to evaluate
Skill entry and resources, normally SKILL.mdRelease-note skill sourceDoes it exclude unfinished work?
Delegated agent definition, such as .github\agents\*.agent.mdRelease Manager agent sourceDoes it complete the delegated task correctly?
Supported local tool/server source, settings and test dataProject Maintainer tool contracts and MCP declarationDoes the required tool call produce the correct data change?

Use your own source first. For fictional data, prepare the optional samples; do not assume raw sample paths are already present in your project.

Finding files is not running them

Discovery lists source without running it. Check that your runner supports it. Create proposes task and input files; execution needs a compatible runner and separate approval. Read your first result before drawing conclusions from a pass.

The vally-bounded runner mode supports real skills, delegated agents and one local Node stdio MCP server managed by Evals, with one case per fresh MCP run so each run has isolated process cleanup and does not reuse server or fixture state across cases. It uses mock or copilot-sdk; mock evidence does not demonstrate model behavior. Custom-agent and MCP execution also need the full Copilot CLI, not just the SDK’s standalone runtime. Model execution requires supported authentication and access; runner setup is separate from Copilot Chat sign-in.

Arbitrary built-in app tools, installed Plugins as runners, remote HTTP/SSE MCP servers and multiple MCP servers in one run are not guaranteed by discovery or supported by execution. Local Python, uv run and uvx stdio MCP servers require the separately approved full-engine profile (vally-full-engine-v1), also with one server and one case per fresh run. MCP automatic task discovery is unavailable; supported manually written MCP evals still need separate approvals. Additional full-engine features require their own supported profile and explicit approval; checks and profiles describes the limits. Keep unsupported requirements visible rather than weakening them.

Make sure the run uses your source

Selecting or reading source during Create does not make the run use it. The no-AI Canvas starter does not load your source automatically: edit its proposed files or ask Copilot before checking and saving. Inspect these settings:

  • Skill: agent_environment.skills must reference the real skill directory, resolving from the final saved eval directory. Retain its declared name without changes for invocation checks; do not substitute a fabricated skill or only paste its instructions into the task.
  • Delegated agent: stage the real profile with agent_environment.files, using src relative to the saved eval and the intended .github/agents/<profile>.agent.md destination. Ask the parent to delegate synchronously to its declared name without changes. This is forced delegation, not natural routing or a root-agent selector; verify child completion and its independent correct outcome.
  • MCP tool: use the actual agent_environment.mcpServers stdio declaration. The bounded Node profile requires the configured Node executable, built in-root JavaScript entry and explicit in-root working directory. Python, uv run and uvx use the separately approved full-engine profile described below. Stage declared dependencies with agent_environment.files; do not invent paths, launch the server or build missing code during authoring. Use one server managed by Evals and one case per fresh run.

Check both that Copilot used the selected skill, agent or tool and that the task’s result is correct. A mock run or an output-only answer check does not prove the selected asset ran. Missing activity records mean unknown, not confirmed non-use. Missing prepared source or a required unsupported check is a visible prerequisite or gap, not permission to replace the real asset with an easier one.

Finding a file does not test how it works

The bounded runner cannot test standalone instruction or prompt files; test them through a supported asset. Keep unsupported requirements visible rather than substituting easier checks. Checks and limits.

Create your first eval after Plugin or extension setup.

Choose the source version

Select the project and file; identical names can refer to different files. A shortened or redacted preview is not the complete source.

Opening is not running

Discovery starts no model or server and saves no files. Unreadable or invalid files remain listed with errors.

Recognized paths and profile limits

Instructions include AGENTS.md, CLAUDE.md, CLAUDE.local.md and GEMINI.md. Agents use formats supported by your app; MCP configuration includes .mcp.json, .github\mcp.json and editor settings. Check execution support separately. The bounded MCP profile permits one fresh local Node stdio server and one case per run.

Python, uv run and uvx local stdio servers use separately approved trusted full-engine execution, not the bounded Node profile. Keep the real MCP name, command, arguments, cwd and application environment. Declare local server files, dependencies and test data in agent_environment.files, require the actual namespaced tool call with literal arguments, and check the task’s independent output or file contents. Availability, completion or an echoed result alone is not a correctness check. Use one server, one case and one trial per fresh run. Server copies managed by Evals keep project-relative file and directory destinations from both root and case environments. Conflicting or duplicate destinations are rejected, not overridden. Per-trial dest_root: assets test files are not copied into the separate server workspace; keep their configuration when drafting and follow the runtime error rather than silently rewriting it. An omitted cwd uses the server workspace root; explicit relative cwd values retain their location relative to the eval file. Literal application environment keys are preserved; credential and runtime-policy overrides are rejected. Selected Python virtual environments retain their launcher location and a separately checked binary/configuration identity.

Python and any selected uv/uvx launcher must already exist on the workspace machine. Preparation checks files and records which inputs will be used; it does not start a server, import packages or install a runtime. The execution preview discloses original and prepared launch details, executable hashes, server effects and prerequisites before a separate approval. Interpreter libraries and server imports are not proven ready by file presence. uv/uvx can resolve/download packages and execute package code during an approved run, using caches managed by Evals with Python runtime downloads disabled. Missing tools or dependencies produce actionable prerequisite or execution errors; authoring does not establish that the runtime is ready. uv project environments use runtime storage managed by Evals. A new uv.lock beside a declared pyproject.toml in the service copy is size-limited and sanitized as text; an existing declared lockfile cannot change during the run. Evals adds and tracks unmanaged-project metadata to stop uv from using undeclared parent projects or modifying their lockfiles. Its effects are disclosed before approval; declared service projects remain unchanged.

Full-engine execution uses Copilot’s standard tool approval, without the bounded mode’s pre-tool checks or an OS/network sandbox. Prepared inputs and separate server folders do not restrict what trusted server code can access. Cancellation stops processes started by Evals but cannot undo remote writes. HTTP/SSE servers, custom providers/executors and general suite orchestration are not included in Python stdio support.

Assets view with the sample release-note skill selected and its redacted source open.
Select the file in Assets. Evals Canvas · sample data