❯/eval help me create an eval for my pr-description skill
●Evals: understanding your request...
●skill(create)
●I'll review the source and requirements, then outline what the eval will test.
●Read plugins\pr-description\skills\pr-description\SKILL.md
●draft_eval intent: "create", target: "pr-description"
○Suggested scenarios: shipped vs deferred work; missing verification; instructions inside change notes.
●Asked user Which cases should I create? → Start with 1 (Recommended)
●Draft: one case with shipped changes, supplied test evidence and deferred retry backoff. Checks cover section order, shipped scope and verification.
●check_eval_draft pr-description.eval.yaml
●Static validation passed. The eval format and checks are valid.
●save_eval_draft
●Saved evals\pr-description.eval.yaml · 1 case.
❯Run it now and explain the result.
●skill(run)
●run_eval evals\pr-description.eval.yaml
●Asked user Run approval: Run eval · this run only
●Running 1 case...
Example run complete · 1 case failed
PASS Required sections in order
FAIL Changes includes only shipped work
PASS Supplied verification preserved
●skill(review)
●review_evals saved run report
●The description put deferred retry backoff under Changes. It should be marked out of scope in Summary or Risks / Rollback.
Improve the pr-description skill to keep deferred work out of Changes?
We can update the skill, then rerun this same eval to check the change.
~\Projects\sample-project [main]Run on: This computer ctrl+e change environment
Working · ctrl+c interruptGPT-6.1 Sol Copilot needs information.
Here are three scenarios we could test:
- Shipped versus deferred work.
- No verification evidence supplied.
- Instructions hidden in change notes.
Which cases should I create?
- ❯ Start with 1 (Recommended)
- All three
- Choose specific numbers
↑/↓ select · enter accept · ctrl+d decline · esc cancel
Copilot needs information.
The eval is saved. Run it now and explain the result?
↑/↓ select · enter accept · ctrl+d decline · esc cancel
Copilot needs information.
Project: sample-project
Run 1 eval (1 trial)?
Evals: 1 | Cases: 1 | Total executions: 1 (eval/model runs)
Subject model: gpt-6.1-sol (configured)
Task and grading requests may use tokens/credits; costs are uncapped.
No OS/network sandbox. Tools can change files and access the network; credentials reach child processes.
Writes local run artifacts. Cancellation cannot undo setup or remote writes.
Approve this run only.
- ❯ Run eval
- ❯ Cancel
- Details
↑/↓ select · enter accept · ctrl+d decline · esc cancel
Copilot needs information.
Improve the pr-description skill to keep deferred work out of Changes?
We can update the skill, then rerun this same eval to check the change.
- ❯ Work on the skill with Improve
- Keep the results for now
↑/↓ select · enter accept · ctrl+d decline · esc cancel