Checks and limits
Understand what each check can tell you before using it to judge a result.
Built-in VS Code checks
In VS Code, open the Evals view, expand Built-in baselines, and use Play on a check. Skill and agent checks then ask you to choose an asset.
| Check | How it works | What you approve | What it can tell you | What it cannot tell you |
|---|---|---|---|---|
| Find setup problems and broken file links (start here) | Reads supported customization files and local links. No AI, no cost, and your files are not edited. | Nothing | Whether supported skill, agent, tool, prompt and instruction files are written correctly, and whether their local links point to real files | Whether the AI will use the file or do the task well |
| Does this skill run when expected? | Copilot reads the selected skill, writes one realistic request and expected result, then runs it. The request may use paid model quota. | Sharing the skill source with the model, then the run | Whether the request starts the skill and the task finishes | Whether the answer is correct. Each run generates a new request, so runs may test different things |
| Can this agent complete a task? | Copilot reads the selected agent, writes one realistic task and expected result, then runs it. The request may use paid model quota. | Sharing the agent source with the model, then the run | Whether the agent receives the task and finishes it | Whether the result is correct, or whether Copilot would pick this agent on its own. Each run generates a new task |
| Can this MCP tool complete an action? | Automatic checks are currently unavailable. Evals does not start or inspect the server; write an eval using local test data. | Nothing; no server is started | Nothing from this built-in check. Your eval can test an action on that data | Whether a tool or input is safe, or whether Copilot would pick the tool on its own |
Each quick check answers the question in its row. To judge whether an asset works well, inspect the answer and use checks for the task’s requirements. Read and validate files before saving, then approve execution separately. Review is read-only; Improve changes the eval, not the skill being tested.
Match the check to the task
Choose a check based on the result you need to observe. Searching for a word can confirm that it appears in the answer, but it cannot tell you whether the answer is right. Similarly, a recorded tool call does not establish that the tool changed the right data.
| Check | What it can tell you | What it cannot tell you |
|---|---|---|
| Output text or pattern | Whether the final answer contains specified text | Whether it is factually correct |
| File exists | Whether an output file was created | Whether its contents work |
| File text or pattern | Whether a file contains specified text | Whether the code behaves correctly |
| Completed | Whether execution finished | Whether all required work was done |
| Skill/tool use | Whether the recorded activity includes the required skill or tool call | Whether the task’s result was correct |
| Repeated tool output | Whether the answer repeats a captured tool response | Whether the tool changed the right state |
If the required outcome cannot be observed, explain that the measurement is incomplete. A keyword cannot replace checking whether code works or whether data changed correctly.
Runner limits
A runner profile lists features and limits for running evals, not what Create or
Improve may write or save. The vally-bounded profile has these execution limits:
| Setting | Limit |
|---|---|
| Cases in an eval | At most 10 |
| Checks per task | At most 10 |
| Attempts | One per task |
| Task time | 60 or 120 seconds; mock evals use 60 |
| Evaluating an MCP server | One task per run, with one newly started local Node server (isolates process cleanup and prevents reusing server/fixture state across cases) |
Mock uses a made-up response to check the eval setup rather than a model-generated response. It cannot demonstrate model behavior. Task time limits also do not cap total money spent or the number of internal model requests.
A valid file is only the first step
The Evals save service checks file format and safely captures source before saving. Whether the eval can run is checked separately; lack of runner support does not block an otherwise valid save. Create and Improve keep requested cases, settings and custom fields. With only your app’s file tools, approved files are saved but not checked by the runner.
Run separately checks execution support and asks for approval. Neither lint nor compatibility proves that the AI does the task correctly.
Profile and check names
The bounded profile is vally-bounded. All eval runner (@microsoft/vally-cli) consumers accept versions
>=0.19.0 without an upper bound, including qualifying prereleases such as
0.20.0-beta.1 (not 0.19.0-beta.1) and build metadata. The repository lockfile
fixes the build/test resolution; each user’s chosen executable and version are
recorded separately. Strict definition, command, and result checks still apply.
The evaluator calls its checks graders. Bounded-profile graders are output-contains, output-matches, file-exists,
file-contains, file-matches, completed, skill-invocation, tool-calls
and assistant-contains-tool-output.
Tool identity/argument checks use limited literal string matchers; at most 32
argument matchers are accepted.
Supported executors are copilot-sdk and mock, not mock-executor.
Timeouts are 60s/1m or 120s/2m; mock accepts only 60s/1m.
Model auto is not accepted. Keep an explicitly requested model or leave
selection to the separately approved run.
Prefer agent_environment, never alongside environment.
Skills reference directories rather than SKILL.md. Sources and destinations
have limited relative-path rules. Grader scope is parent, subagent, or
omitted for all agents, not scope: all.
Choose checks your runner can use
Some tasks need more than a literal output check, such as several conversation turns, remote tools, executed code checks or a model that judges meaning. The bounded profile restricts these features. If a required measurement is unsupported, describe the gap rather than change the requirement to get a passing file.
Features outside the bounded profile
Native root-agent/instruction/prompt selection, multi-turn/attachments, dynamic/Git/provider environments, setup commands, custom executors, remote MCP and hidden grading environments are unsupported by the bounded profile. Semantic, command/program, diff, transcript and metrics graders are not accepted by that profile. Limited tool-call evidence does not prove call order/count/results or complete activity capture.
Separately approved trusted full-engine mode
vally-full-engine-v1 supports repeated trials, longer native deadlines,
multi-turn prompts, attachments, setup commands, HTTPS Git clones, grader-only
files, scalar parameters, native prompt/panel/command graders, and
local Python stdio MCP servers managed by Evals. It permits
at most 100 total trials, 30 graders per case and native durations up to 24 hours.
The original and resolved definition must each fit the 512,000-byte limit.
This mode requires separate approval for the prepared inputs, including when the eval uses a mock response. It has no bounded tool policy or OS/network sandbox; setup and command graders can execute code, and model judges can cost money. Review the disclosed effects, credentials and deadlines before approving. Parameter matrices, project plugins/custom executors, local Git worktrees, custom model providers and remote MCP remain unsupported by this execution profile. Preserve essential requirements rather than deleting them to pass.
Full-engine evidence and limits explain what a completed run does and does not establish.