Understand results
Read what happened, what went wrong and what the eval could not tell you.

What each step tells you
| Step | What you have | What to check next |
|---|---|---|
| Draft | A proposed test | Validate its files. |
| Validation | An accepted file | Check that it measures the task correctly. |
| Save | Created files | Approve execution separately. |
| Plan good/bad examples | Proposed check controls | Run the controls through a supported, separately approved workflow. |
| Run | One attempt | Inspect the answer, checks and errors. |
| Decide | A choice supported by those results | Check whether the findings apply to other tasks. |
Wrong result or missing information?
Read the expected result, actual answer and check result. Expected Search only but received Search and unfinished Export? The unfinished-work check should fail. If it passes, improve that check.
An unreadable result is unknown, not a wrong answer or a pass. Keep setup, provider and grader errors separate from failed checks. Retain cancelled and incomplete attempts.
Check which version was evaluated
Saved results stay tied to the recorded attempt, even after you edit the source. Check the recorded files, settings, runner and model rather than matching names. Missing usage or cost stays unknown, not zero.
Run summaries and cost history
The bounded runner runs one saved eval/model configuration per approval. Counts distinguish eval executions from cases and label unresolved models. Mock evals call no model.
Cost history shows which eval versions, models, units and sources the amounts came from. The latest 20 saved runs are considered; omissions are disclosed. Only verified actual costs from completed runs enter ranges. Missing measurements and unknown eval versions are not zero. The current adapter records no cost. Unreadable history shows a warning, not a cost range; it does not block run approval.
Ranges are recorded amounts, not complete billing, forecasts or spending caps. Historical costs do not guarantee the next run will fall within the range. Changes to the eval, tested asset or product can change the cost, even when the eval version is unchanged.
In Canvas, Review & run prepares the plan, then requests one cost/effects confirmation in the App conversation or VS Code dialog. Accepting starts that exact run; declining starts no eval execution. Full details are available from the host review, without another Canvas approval step. Suite approval uses one compact summary with complete details on demand.
Trusted full-engine runs
The Runtime retains bounded mode for compatible definitions. Evals needing multiple trials, longer deadlines, native prompt/panel judges, setup commands, HTTPS clones or scalar parameters get a separate full-engine review. That review lists the actual trial count, resolved inputs, setup/network effects, judge configuration, credential exposure and deadlines. It requires new full-engine approval; saving or approving a bounded run does not authorize it.
Full-engine mode has no bounded tool policy or OS/network sandbox. An eval using a mock response can still use paid AI judges. Stop ends processes started by Evals but cannot undo remote writes or arbitrary setup effects. Individual trial and judge evidence is retained, while missing token usage and unverified model or remote-source identity stay unknown. Strict matched comparisons are unavailable for this mode.
Side-by-side saved results
In Copilot CLI / GitHub Copilot App, /eval compare selects the pair and mode.
CLI uses native forms and a compact summary; GitHub Copilot App opens the pair in Canvas.
VS Code’s Compare saved runs (no AI calls) action offers strict comparison
when eligible and read-only descriptive comparison (including ruler-parity
checks) when strict comparison is unavailable or not selected.
GitHub Copilot Evals: Compare opens the Canvas workflow instead.
Side-by-side results shows a separate test list and pass count for each run, even when tests differ or the model is unverified. Create → Run (1 test) → Improve → Run (2 tests) can show 1 of 1 and 2 of 2 passed. The counts belong to separate runs. Matching test names alone cannot establish that their tasks and checks match. This view creates no matched comparison report and cannot show that a change improved the result.
Choose View details in CLI or expand Details in GitHub Copilot App for run IDs, source identities, expected/observed results, usage, limitations and matching eligibility. The summary shortens long names and lists without changing saved evidence. Cancelled, failed or running results may be partial. Planned/unrecorded cases stay unknown; skipped/not-evaluated rows are not passes. A recorded model name does not prove which model ran. Per-run usage requires verified saved evidence. Invalid or stale evidence is refused.
Ruler parity in descriptive comparisons
When a model-backed run records an unverified model label (modelIdentity: "unknown"),
strict matched comparison is unavailable because a configured model name does
not prove which model ran. Descriptive Side-by-side results still checks
Ruler parity across the two saved runs:
verified-ruler-parity: the eval definition (evaluation.sourceHash), test suite (suite.hash), graders (judge.hash), non-target inputs, and permission policy stayed identical between both runs, and Evals lists the changed skill or agent customization hashes.ruler-drift: the eval definition, suite, graders, non-target inputs, or permission policy changed between the two runs, so pass counts are not from the same ruler.incomplete-evidence: one or both runs lack comparison metadata, so parity cannot be verified.
Compare fairly
Matched test comparison requires matching evidence and one supported change. Choose the recorded change after matching checks pass; saving a matched comparison report is separate. If refused, explicitly choose Side-by-side results to inspect separate results; there is no automatic fallback.
For a skill change, keep task, inputs, checks and model unchanged. Project and runner identities also need to match. If records cannot be compared, read them separately; do not erase their differences.
| Transition | Meaning |
|---|---|
| Fail to Pass | Fixed failure |
| Fail to Fail | Persistent failure |
| Pass to Fail | Regression |
| Pass to Pass | Still passing |
Treat execution errors separately from these transitions: fixing setup does not show that the answer improved. Keep unknown usage marked as unknown rather than zero, and investigate observed differences before drawing broader conclusions. Reading a comparison starts no run, calls no model, scores no answers again and changes no files.
With/without availability
Verified with/without benefit testing is blocked with the pinned runner. The usefulness guide distinguishes available behavior tests and exploratory observations from that stronger claim.
The exploratory Compare skill can observe one task with a skill included and omitted from prepared inputs through supported CLI tools. This does not prove that the skill was absent or helped. Preparation and each run need separate approval through your app, which may allow delegated approval; this does not guarantee a person approved it. Unknown model identity and usage remain unknown. This path produces no matched comparison report or keep/remove advice. If your app lacks the required tools, only setup guidance is available.
Select a real skill and saved single-case eval with checks for success and requirements that must not be broken; use Create first for new tasks. Read-only preview shows both conditions, requested model, tools, timeout, one attempt per condition and zero retries. Approve preparation/static validation and each condition separately. Check which inputs each run used, its outcome, checks and recorded usage. Decline/cancel retains incomplete evidence; there are no automatic edits or retries. Source support has not been verified in an installed app with a live model.
The @microsoft/vally-cli 0.19.0 profile creates fresh sessions and homes,
but always searches for configuration. It cannot fully list or restrict source
files before the AI’s first turn. Another copy of the skill may still be
available. An instruction to ignore it, a disabled name or a missing activity
record does not prove absence.
Run’s fixed task-tool list is optional; it neither coordinates the two runs nor proves the limits were enforced. Its approval hook has a three-second fallback to Copilot’s standard approval. A claim that the skill caused a better result requires verified absence and permission controls. Until those controls are verified, use this path only for exploratory observations. Model-free policy tests do not verify a live model run.
Saved formats and identity details are documented for integration developers.