Choose your first task
Start with a job you understand and a mistake you want Copilot to stop making.
Pick one real question
Good starting points are a repeated manual check, a known failure, or a requirement in your skill’s instructions. “Write release notes without announcing unfinished work” is easier to evaluate than “write better notes.”
Write down the input, the expected outcome, and why that outcome is correct. A case is one task and its input. A check judges an observable part of the result. You don’t need a large suite to learn from the first case.
When is an eval useful?
Use an eval when you need to inspect how AI handles a task: which information it uses, what it produces, or whether a change introduces new mistakes. Keep ordinary unit tests for deterministic code behavior. Evals can use supported outcome checks alongside those tests; they do not replace them or human judgment.
Check the check
Try a known good answer, an acceptable alternative, an obvious mistake, and an empty answer. A check that passes all four is not helping you. Keep expected answers out of the inputs given to the AI.
| Next question | Read |
|---|---|
| What does this look like? | Understand an eval |
| Is my skill still useful? | Check its contribution |
| How do I turn my task into a test? | Plan a useful check |
| How do I run it? | Your first eval |
| What can the runner measure? | Checks and limits |