GitHub Copilot Evals
InstallSearch /github/evals

Troubleshooting

Use the error message to find out what failed and what to do next.

ProblemWhat to do
Missing Plugin skillCheck whether your installed version includes the skill and whether Copilot lists it.
Missing pane or workbenchCheck the extension name, open a local folder, reload and read the Evals output log.
File scan did not finishCheck the reported filesystem or resource error, resolve it and Refresh. Treat the list as incomplete until the scan finishes.
Missing runner (the program that runs tests)Use Set up test runner. Review an installation before accepting it.
Unsure whether Init is requiredIt is optional. /eval init and /eval setup configure folders/suites; Set up test runner installs a runner. Choose the right setup.
/eval init is unavailableCommands need the packaged Copilot CLI / GitHub Copilot App Plugin. With repository skills only, ask Setup in your own words; check which tools your app supports.
Invalid eval fileRead the runner’s error to find the field that needs correcting; this website cannot validate the file for you.
Unsupported checkList the requirement it cannot test. Do not replace it with an easier, unrelated check.
Save conflictInspect the conflicting path and existing file before choosing a new destination.
Partly saved evalKeep created files and the original recovery record; do not run it. Recover the save.
Cancelled run or service errorKeep the attempt and its error information so you can distinguish a run failure from a wrong answer.
Results cannot be comparedCheck whether the task, inputs, checks or settings differ. Do not erase those differences.

Sign-in problems

Editor chat and the evaluator may use different sign-in paths. Even if a program reports the correct version, it may still need authentication or model access. Keep credentials out of eval files, reports and sample data.

Report a bug

Report ordinary bugs in GitHub issues with a small example that reproduces the problem. Include the version, operating system, app and failed step, but remove private code, secrets and account details.

Report suspected security problems through SECURITY.md, not a public issue.

Install & use · Saving and stopping