GitHub Copilot Evals
InstallSearch /github/evals

Checks and limits

Understand what each check can tell you before using it to judge a result.

Built-in VS Code checks

In VS Code, open the Evals view, expand Built-in baselines, and use Play on a check. Skill and agent checks then ask you to choose an asset.

CheckHow it worksWhat you approveWhat it can tell youWhat it cannot tell you
Find setup problems and broken file links (start here)Reads supported customization files and local links. No AI, no cost, and your files are not edited.NothingWhether supported skill, agent, tool, prompt and instruction files are written correctly, and whether their local links point to real filesWhether the AI will use the file or do the task well
Does this skill run when expected?Copilot reads the selected skill, writes one realistic request and expected result, then runs it. The request may use paid model quota.Sharing the skill source with the model, then the runWhether the request starts the skill and the task finishesWhether the answer is correct. Each run generates a new request, so runs may test different things
Can this agent complete a task?Copilot reads the selected agent, writes one realistic task and expected result, then runs it. The request may use paid model quota.Sharing the agent source with the model, then the runWhether the agent receives the task and finishes itWhether the result is correct, or whether Copilot would pick this agent on its own. Each run generates a new task
Can this MCP tool complete an action?Automatic checks are currently unavailable. Evals does not start or inspect the server; write an eval using local test data.Nothing; no server is startedNothing from this built-in check. Your eval can test an action on that dataWhether a tool or input is safe, or whether Copilot would pick the tool on its own

Each quick check answers the question in its row. To judge whether an asset works well, inspect the answer and use checks for the task’s requirements. Read and validate files before saving, then approve execution separately. Review is read-only; Improve changes the eval, not the skill being tested.

Match the check to the task

Choose a check based on the result you need to observe. Searching for a word can confirm that it appears in the answer, but it cannot tell you whether the answer is right. Similarly, a recorded tool call does not establish that the tool changed the right data.

CheckWhat it can tell youWhat it cannot tell you
Output text or patternWhether the final answer contains specified textWhether it is factually correct
File existsWhether an output file was createdWhether its contents work
File text or patternWhether a file contains specified textWhether the code behaves correctly
CompletedWhether execution finishedWhether all required work was done
Skill/tool useWhether the recorded activity includes the required skill or tool callWhether the task’s result was correct
Repeated tool outputWhether the answer repeats a captured tool responseWhether the tool changed the right state

If the required outcome cannot be observed, explain that the measurement is incomplete. A keyword cannot replace checking whether code works or whether data changed correctly.

Runner limits

A runner profile lists features and limits for running evals, not what Create or Improve may write or save. The vally-bounded profile has these execution limits:

SettingLimit
Cases in an evalAt most 10
Checks per taskAt most 10
AttemptsOne per task
Task time60 or 120 seconds; mock evals use 60
Evaluating an MCP serverOne task per run, with one newly started local Node server (isolates process cleanup and prevents reusing server/fixture state across cases)

Mock uses a made-up response to check the eval setup rather than a model-generated response. It cannot demonstrate model behavior. Task time limits also do not cap total money spent or the number of internal model requests.

A valid file is only the first step

The Evals save service checks file format and safely captures source before saving. Whether the eval can run is checked separately; lack of runner support does not block an otherwise valid save. Create and Improve keep requested cases, settings and custom fields. With only your app’s file tools, approved files are saved but not checked by the runner.

Run separately checks execution support and asks for approval. Neither lint nor compatibility proves that the AI does the task correctly.

Profile and check names

The bounded profile is vally-bounded. All eval runner (@microsoft/vally-cli) consumers accept versions >=0.19.0 without an upper bound, including qualifying prereleases such as 0.20.0-beta.1 (not 0.19.0-beta.1) and build metadata. The repository lockfile fixes the build/test resolution; each user’s chosen executable and version are recorded separately. Strict definition, command, and result checks still apply.

The evaluator calls its checks graders. Bounded-profile graders are output-contains, output-matches, file-exists, file-contains, file-matches, completed, skill-invocation, tool-calls and assistant-contains-tool-output. Tool identity/argument checks use limited literal string matchers; at most 32 argument matchers are accepted.

Supported executors are copilot-sdk and mock, not mock-executor. Timeouts are 60s/1m or 120s/2m; mock accepts only 60s/1m. Model auto is not accepted. Keep an explicitly requested model or leave selection to the separately approved run.

Prefer agent_environment, never alongside environment. Skills reference directories rather than SKILL.md. Sources and destinations have limited relative-path rules. Grader scope is parent, subagent, or omitted for all agents, not scope: all.

Choose checks your runner can use

Some tasks need more than a literal output check, such as several conversation turns, remote tools, executed code checks or a model that judges meaning. The bounded profile restricts these features. If a required measurement is unsupported, describe the gap rather than change the requirement to get a passing file.

Features outside the bounded profile

Native root-agent/instruction/prompt selection, multi-turn/attachments, dynamic/Git/provider environments, setup commands, custom executors, remote MCP and hidden grading environments are unsupported by the bounded profile. Semantic, command/program, diff, transcript and metrics graders are not accepted by that profile. Limited tool-call evidence does not prove call order/count/results or complete activity capture.

Separately approved trusted full-engine mode

vally-full-engine-v1 supports repeated trials, longer native deadlines, multi-turn prompts, attachments, setup commands, HTTPS Git clones, grader-only files, scalar parameters, native prompt/panel/command graders, and local Python stdio MCP servers managed by Evals. It permits at most 100 total trials, 30 graders per case and native durations up to 24 hours. The original and resolved definition must each fit the 512,000-byte limit.

This mode requires separate approval for the prepared inputs, including when the eval uses a mock response. It has no bounded tool policy or OS/network sandbox; setup and command graders can execute code, and model judges can cost money. Review the disclosed effects, credentials and deadlines before approving. Parameter matrices, project plugins/custom executors, local Git worktrees, custom model providers and remote MCP remain unsupported by this execution profile. Preserve essential requirements rather than deleting them to pass.

Full-engine evidence and limits explain what a completed run does and does not establish.

Create · Evaluation methods