GitHub Copilot Evals
InstallSearch /github/evals

Saving and stopping

Review what will be saved or run, and recover from interruptions without losing your work.

Permission for each action

Opening Evals or reading the project does not start an Evals model call, run or file save. Other extensions and app settings may have their own startup behavior, which is separate from what Evals controls.

Ordinary saves and settings changes use your app’s normal tool permissions or an explicit Canvas Save action. Evals does not ask you to confirm the same write again. Your app may still show its own tool-permission prompt.

Runs require a cost-and-effects confirmation. Saving never grants that consent. Checking a draft and preparing a run can start local tools and create temporary files; those effects must be disclosed. A Canvas Review & run click requests that preparation, not eval execution or installation.

SurfaceSaveRun
Copilot App CanvasReview files and click Save; no extra conversation confirmation.One cost/effects prompt in the conversation after preparation.
VS Code Canvas, including supported Agents windowsThe same Save action, with workspace trust and unsaved-editor checks.One non-modal native confirmation lets you approve or choose View details to open full details in a read-only editor first.
Copilot App/CLI native toolsNormal tool permissions; no additional Evals save form./eval run prepares the selected eval, then requests execution approval.
VS Code native chat/editor actionsNormal tool permissions; no second save modal.Prepares and validates, stages the read-only plan review, then requests one non-modal confirmation with View details before execution.
Standalone MCPNormal tool permissions; save does not require form support.A supported host approval form is required; lengthy reviews can span pages.
GitHub ActionsWriting a workflow does not run it.Workflow triggers and repository policy authorize CI; no desktop prompt appears for each run.

Every supported host prepares the run plan before requesting execution approval. Host automation can accept a request without a new human interaction. Source/protocol checks do not prove installed-host presentation.

Save new files without replacing your work

Review the full files and paths before saving. Evals keeps existing files and saves input files before the eval definitions that need them. It refuses a save if inputs changed, paths conflict or unsafe links redirect the destination.

Files save one at a time, so earlier files may already exist if a later save fails. Saves through Evals keep a recovery record, also called a journal, to explain which files were written and help recover the incomplete result. Without the Evals save service, Create/Improve may use your app’s file tools under normal host permissions. That path checks the saved contents but does not create an Evals journal; partial results list every created and unsuccessful path. It never takes over an Evals save that failed, was cancelled or has an uncertain outcome.

ResultWhat to do
All files savedReview the paths; running still needs approval.
Some failed or conflictedKeep the created files and recovery record. Do not run the incomplete eval.
Retry the same saveKeep the same reviewed content and original save record.
Cleanup is uncertainDiagnose it before restarting; do not delete records blindly.
Content changedReview a new proposal; do not use it to bypass a partial save.

Evals must not overwrite your files, claim ownership of someone else’s files, or recreate a file you deliberately removed.

Checking a draft

Guided runner setup resumes a pending check; cancelling leaves it unvalidated. After a check completes, its result is kept for that request. Before saving, use Retry edited draft, then Check draft, if you edited the draft or fixed a check failure. If the source changed, select it again; do not silently use a different version.

Once saving begins, keep the original request, proposal and recovery details. Do not switch save tools or start a new request to bypass a partial save.

Choose the destination

Create uses a configured eval directory or an explicitly chosen safe project-relative path, including a skill-adjacent location. Multiple configured directories need an explicit choice. With none configured, the default is evals. The destination stays bound to the proposal through retries. Create does not rewrite configuration, move existing evals, or overwrite files.

Changing projects

Switching projects or reopening the same path invalidates old approvals. Late AI replies must not silently apply to the new project. If the app cannot identify the current project, it must ask rather than guess from an old folder.

Stop your eval

Stop affects the run started by Evals rather than unrelated programs, and keeps the errors and results recorded before cancellation. Submitting that same run twice must not create a second paid execution. Evals does not automatically retry a failed run until it passes.

Limits you should know

Supported task, attempt and time limits are enforced, but total cost and internal model requests are not capped. A temporary folder is not an operating-system or network sandbox: it does not stop the AI or its tools accessing other files or the network.

If you require a hard cost, tool-access or isolation limit the runner cannot enforce, it must refuse the run rather than quietly weaken your requirement.

Save and recovery details

Canvas presentation preferences also use the current unversioned format. Retired or invalid preferences prevent that project’s Canvas from opening. In file-backed hosts, close its Canvas panels, preserve a copy of the specific presentation preference file named in the error, move only that file aside, then reopen. In VS Code, preferences are stored in extension globalState under canvas:<domain>; choose Reset presentation preferences on the error notification to clear only that project’s preference entry, then reopen. The Canvas starts with fresh selections; it does not convert preferences on load. Do not move authoring storage or workspace results as part of this preference-only recovery.

Initial authoring allows 1–16 new files, 512,000 UTF-8 bytes per file and 512,000 total bytes. It rejects paths that escape the allowed directory, links, case-colliding paths, unsaved selected inputs, changed source/runtime and existing files it does not own.

Each file is saved with its complete contents without overwriting an existing file. The files do not all save at once: some may be saved before another fails. Saving also cannot guarantee that data survives a power loss. If the filesystem cannot save safely, Evals reports an error instead of overwriting.

After saving starts, a retry uses the same request, proposal and save record. If cleanup cannot be confirmed, Evals keeps control of that save to prevent an unsafe restart.

Private recovery records are separate from temporary draft files. A project’s hash alone cannot prove a save completed or give permission to write. Retries require current-format records on the destination drive. Older records are rejected and kept with their files for manual review; Evals does not reuse, convert or delete them.

Recover draft saves checks only Evals’ designated private recovery folder, not older extension-storage folders. It can finish an interrupted current save after verifying which files it created, or archive recognized current records without changing saved eval files. Unsupported records stay untouched. Retry an interrupted archive using its original record and file-ownership details. Unrelated completed saves and records of unsaved proposals stay in place and do not block recovery. Completed history still needs valid private records of completion and file ownership; matching saved contents or project markers are not enough. Pending saves recheck file ownership before finishing. Records of unsaved proposals do not inspect intended destinations. Before an archive moves, Evals rechecks the folders and records. Changed or added records stay in place for review.

A retired private authoring-v1/<workspace-hash> container blocks all new authoring for that workspace, not just retries. Retained workspace ownership markers under .copilot/evals/results/authoring-<nonce> can also block new authoring when Evals cannot verify their original private recovery record. After confirming no operation is active, explicitly review and preserve the private folder, matching project markers and any created files together before moving old ownership records aside. Do not move a lock that is active or cannot be verified, delete results, create replacement ownership records or use matching content hashes to resume an old request. A fresh request still cannot overwrite existing output files.

Troubleshooting · Run