Understand an eval
A release-note skill announces unfinished work. Here is how an eval catches the mistake.
How an eval catches a mistake
Fictional illustration, not measured results.
Search and Export are hypothetical release features. In this release, done means a feature is ready for people to use and marked Done in the backlog. Search is done; Export is not.
1. Create: define the eval
What we test: Release-note skill
Task: Write release notes for completed features.
Backlog: Search is done. Export is not done.
Expected result: Mention Search. Leave Export out.
What we check: Search is included; unfinished Export is excluded.
Why it matters: Release notes tell people what they can use. Listing unfinished work promises a feature that is not ready.
2. Run: inspect the first answer
Before the change
Initial instructions
Include completed work only.
First run
Run the skill and check its answer.
- Search
- ExportNot done - should not be here
First run checks
- Completed Search is included.
- Pass
- Unfinished Export is included by mistake.
- Fail
The instruction change
Updated instructions
Filter the backlog to completed items before writing. Leave unfinished items out of the notes.
Run again after the instruction change
After the change
Rerun
You revise the skill, then rerun the same eval.
- Search
Rerun checks
- Completed Search is included.
- Pass
- Unfinished Export is left out.
- Pass
3. Compare: read both results
The eval catches Export being listed as finished in the first answer. In the fictional rerun, Search remains, Export is excluded, and both checks pass.
For a real comparison, keep the task, backlog inputs, expected answer, model, other tools and checks unchanged. Keep the original result and approve the rerun separately. A comparison reads saved evidence; it does not rerun the task.
One example cannot establish reliability. Compare saved runs and spot new failures.
Try a sample or your own task
Runnable sample: prepare Project Maintainer
using the Releases ZIP and START-HERE.md.
This provides real skill, backlog and eval files to run locally.
The sample uses completed PM-101 and unfinished PM-102.
Create your first eval to approve a run and inspect its saved
evidence. The Search/Export illustration above is fictional; its answers are not
sample run results.
Your task: select a real asset: a skill, delegated agent or supported tool source. Choose a realistic task and its inputs, then define the expected result before seeing an answer. Ask Copilot to draft an eval and its checks. Review which correct answers the checks accept and which wrong, partial or empty answers they reject. Record unchecked requirements. Evals runs those checks; it does not automatically check everything.
Before saving or running
Review full files. Approve saving and running separately. See the save/run guide.