GitHub Copilot Evals
InstallSearch /github/evals

Run

Review the files, model and tools before approving execution.

Choose your app

Find your setup instructions
Evals view showing the sample no-AI eval with Improve with Copilot, Queue Eval and Review & run.
Open a saved eval in Evals to queue or review a run. Evals Canvas · sample data

Choose your app to see how to start. Start with the first-eval guide for the general steps.

Execute the selected eval

Choose an eval and inspect its plan. Raw output stays out of Evals history. Timestamped reports go in a unique folder under .vally.yaml’s paths.results or the project-local default.

Review the run summary

Inspect planned effects before approving. Understand summaries and historical costs.

In Canvas, Review & run starts disclosed local checks and prepares the plan, then asks once about costs and execution in the App conversation or VS Code dialog. No extra Canvas checkbox is needed. Native App/CLI /eval run and VS Code native Run also prepare first and ask once after opening the plan review. Standalone MCP requires a host form; larger reviews may span pages.

If preparation fails, the cost-confirmation step is not reached and execution has not started. Diagnose the setup or validation error. Choosing an eval, saving it, or viewing Details never approves execution.

Know the limits

Stop if required limits cannot be enforced. Paid models can cost money; total cost and internal model requests are not capped. The temporary folder is not a sandbox. Evals must not silently retry or switch models. Completed does not mean passed.

Trusted full-engine runs

Full-engine runs need their own review and approval. Setup commands and checks can execute code or spend money even when the eval uses a mock response. Stop cannot undo remote writes, and strict comparison is unavailable. Full-engine details.

Cancel or stop a run

Cancel approval before execution; Stop this run affects only this run. Keep failed and cancelled results; do not start the same run twice.

Set up CI after the eval is useful locally · Understand results