GitHub Copilot Evals
InstallSearch /github/evals

Choose your first task

Start with a job you understand and a mistake you want Copilot to stop making.

Pick one real question

Good starting points are a repeated manual check, a known failure, or a requirement in your skill’s instructions. “Write release notes without announcing unfinished work” is easier to evaluate than “write better notes.”

Write down the input, the expected outcome, and why that outcome is correct. A case is one task and its input. A check judges an observable part of the result. You don’t need a large suite to learn from the first case.

When is an eval useful?

Use an eval when you need to inspect how AI handles a task: which information it uses, what it produces, or whether a change introduces new mistakes. Keep ordinary unit tests for deterministic code behavior. Evals can use supported outcome checks alongside those tests; they do not replace them or human judgment.

Check the check

Try a known good answer, an acceptable alternative, an obvious mistake, and an empty answer. A check that passes all four is not helping you. Keep expected answers out of the inputs given to the AI.

Next questionRead
What does this look like?Understand an eval
Is my skill still useful?Check its contribution
How do I turn my task into a test?Plan a useful check
How do I run it?Your first eval
What can the runner measure?Checks and limits