Plan a useful check
Write down the task and the evidence that would convince you it was done correctly.
Plan a case
Copy this worksheet and fill in the facts:
Task:
Input and starting conditions:
Expected result:
Source of that expectation:
Required and forbidden behavior:
Allowed alternatives:
Check to use, or requirement that cannot be checked:
Correct, alternative, wrong and empty examples:
This planning worksheet helps you organize the case; it does not validate an eval file. Keep the expected answer separate from the inputs given to the AI being evaluated.
Choose a check
| You need to know | Useful observation |
|---|---|
| Was an output file created? | File existence |
| Does the answer include required content? | A supported text or pattern check |
| Does a code change work? | Executed tests of the code’s behavior |
| Did data change correctly? | An observation of the data after the action |
Read Checks & limits for supported types and what they cannot establish.
Interpret a result
| Observation | Meaning |
|---|---|
| AI says “done” | A statement, not proof of the required outcome |
| Required file is confirmed missing | A possible failed check |
| Required file cannot be read | The result is unknown because the file could not be checked |
| Provider or setup error | The run could not be completed or checked as intended |
| Usage was not captured | The cost or usage is unknown; do not count it as zero |
If a check accepts a wrong answer, improve the check. If it correctly catches a mistake, investigate the skill or agent.