Troubleshooting
Use the error message to find out what failed and what to do next.
| Problem | What to do |
|---|---|
| Missing Plugin skill | Check whether your installed version includes the skill and whether Copilot lists it. |
| Missing pane or workbench | Check the extension name, open a local folder, reload and read the Evals output log. |
| File scan did not finish | Check the reported filesystem or resource error, resolve it and Refresh. Treat the list as incomplete until the scan finishes. |
| Missing runner (the program that runs tests) | Use Set up test runner. Review an installation before accepting it. |
| Unsure whether Init is required | It is optional. /eval init and /eval setup configure folders/suites; Set up test runner installs a runner. Choose the right setup. |
/eval init is unavailable | Commands need the packaged Copilot CLI / GitHub Copilot App Plugin. With repository skills only, ask Setup in your own words; check which tools your app supports. |
| Invalid eval file | Read the runner’s error to find the field that needs correcting; this website cannot validate the file for you. |
| Unsupported check | List the requirement it cannot test. Do not replace it with an easier, unrelated check. |
| Save conflict | Inspect the conflicting path and existing file before choosing a new destination. |
| Partly saved eval | Keep created files and the original recovery record; do not run it. Recover the save. |
| Cancelled run or service error | Keep the attempt and its error information so you can distinguish a run failure from a wrong answer. |
| Results cannot be compared | Check whether the task, inputs, checks or settings differ. Do not erase those differences. |
Sign-in problems
Editor chat and the evaluator may use different sign-in paths. Even if a program reports the correct version, it may still need authentication or model access. Keep credentials out of eval files, reports and sample data.
Report a bug
Report ordinary bugs in GitHub issues with a small example that reproduces the problem. Include the version, operating system, app and failed step, but remove private code, secrets and account details.
Report suspected security problems through SECURITY.md, not a public issue.