Advanced CI configuration
Customize GitHub Actions with separate jobs, narrower permissions and automatic report artifact discovery.
Use CI after an eval works locally. Set up GitHub Actions walks through your first configuration and workflow.
Choose the workflow that fits
| Start with | Choose it when | Tradeoff |
|---|---|---|
| One all-in-one job | Recommended starting point. | Evaluation and reporting share token permissions. |
| Separate lint, run and report jobs | Narrower permissions or custom dependencies. | More jobs; automatic artifact discovery, explicit reporting concurrency. |
Before you start
You need .vally.yaml with a named suite, access to github/evals-action,
and organization policy allowing organization-billed Copilot CLI usage.
Use trusted, reviewed workflows and eval sources: same-repository PRs can
execute code with job permissions. Fork PRs are excluded.
Simple: one Action does the work
A root-project adaptation of the repository workflow:
name: Evals
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
evals:
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
copilot-requests: write
pull-requests: write
concurrency:
group: evals-${{ github.workflow_ref }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: github/evals-action@v0
with:
working-directory: .
suite: smoke
upload-transcripts: false
Replace smoke with your suite. Use github/evals-action@v0, the moving 0.x
pilot tag with discovery; fixed v0.1.0 lacks it. Setup-generated workflows
may use full commit pins.
Keep the concurrency block as shown: the Action requires serialized PR reporting.
Default all mode is unchanged:
- validates inputs and lints the whole project;
- runs the selected suite after successful lint;
- publishes evidence and updates one sticky PR summary.
Manual runs retain summaries/artifacts without commenting. Failed evals still report and fail the job. Zero executed trials is not-run, not a passing evaluation.
Advanced: isolate lint, evaluation and reporting
Keep PR-write permission out of evaluation. Lint/run upload minimal structured
artifacts automatically; one report step discovers them by shared report-id,
which defaults to evals.
No step IDs, job outputs, artifact inputs or conditional fallback report steps
are needed.
Lint/run each need checkout; report does not and runs even after failure.
Lint --success--> Evaluate --evidence--> Report
\------------lint evidence----------> Report
(if evaluation evidence is absent)
Evaluation requires successful lint. The lint-only fallback is based on missing evaluation evidence, not failed evaluation. Reporting never turns failure into a pass.
| Job | Action command | Permissions |
|---|---|---|
| Lint | lint | contents: read |
| Evaluate | run | contents: read, copilot-requests: write |
| Report | report | actions: read, contents: read, pull-requests: write |
This uses evals/.vally.yaml; for root projects, change both execution jobs
to working-directory: ..
name: Evals (separate jobs)
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
lint:
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: github/evals-action@v0
with:
command: lint
working-directory: evals
run:
needs: lint
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
copilot-requests: write
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: github/evals-action@v0
with:
command: run
working-directory: evals
suite: smoke
copilot-token: ${{ github.token }}
max-retries: 0
upload-transcripts: false
report:
needs: [lint, run]
if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
actions: read
contents: read
pull-requests: write
concurrency:
group: evals-${{ github.workflow_ref }}
cancel-in-progress: false
steps:
- uses: github/evals-action@v0
with:
command: report
Keep the concurrency block on the report job. It serializes comment
writers to prevent duplicate comments or stale updates; discovery does not
configure or hide concurrency. Separate lint and run jobs do not need it.
Explicit run requires copilot-token: ${{ github.token }} for workflow-token
model access. max-retries: 0 matches all instead of native retry defaults.
The event guards exclude fork PRs from execution; timeouts bound job duration, not spending or tool permissions.
Discovery, failures and reruns
Discovery uses only the current workflow run and attempt, verifying the API-provided SHA-256 digest and downloaded report’s repository, revision, run, attempt and report identity. Missing digests, expired artifacts and ambiguous matches fail; it never guesses newest evidence or reuses previous attempts.
Run evidence is preferred, with a successful companion lint report included when available. Without run artifacts, lint-only reporting warns and never claims an evaluation pass. No matching artifacts fails reporting. A report-only rerun cannot use earlier artifacts: rerun the producing jobs too.
Custom identities and explicit overrides
For independent suites, use the same custom report-id (such as smoke) on
lint/run/report, with one serialized reporter per identity. Discovery supports legacy
-run / -lint identities, such as evals-run and evals-lint, on artifacts
produced by the new Action version.
Explicit overrides remain supported:
| Inputs | Behavior |
|---|---|
source-artifact-id and source-artifact-digest | Bypass all discovery; select the primary artifact and only explicitly supplied lint. Retains existing permissions; does not newly require actions: read. |
lint-artifact-id and lint-artifact-digest | Override lint. Without primary overrides, run is discovered and needs actions: read. |
Each ID/digest pair must be complete. Explicit inputs still verify the digest and repository/revision/run/attempt identity. Artifacts from older Action versions require explicit IDs/digests. See the Evals Action reference for input details.
Check your workflow
| Job view / report | What to verify |
|---|---|
| Lint fails; Run skipped | Report available lint evidence; inspect logs if absent. |
| Run fails; Report succeeds | Workflow remains failed; publishing is not passing. |
| Report skipped | Check for a same-repository PR. Manual runs retain artifacts; missing evidence fails reporting, not skips it. |
| No trials executed | Not a pass. Check suite, file patterns and tags. |
For example, Lint: passed → Run: failed → Report: passed is a failed workflow with a useful report, not a successful evaluation.
Review privacy, permissions and cost
copilot-requests: write permits model-backed evaluation. pull-requests: write
permits the sticky summary. The all-in-one job combines them for convenience;
the advanced workflow isolates them.
Examples disable transcripts. Before enabling them, review access and retention:
prompts, output, tool calls, code and sensitive data may survive redaction.
The Action itself defaults uploads off. Upload detailed transcripts? in
CI Setup maps to the Action’s with inputs:
| Choice | Workflow YAML |
|---|---|
| Keep existing setting; off for new workflows | Preserves booleans, strings and expressions; writes upload-transcripts: false when absent. |
| Off | Writes upload-transcripts: false. |
| On | Sets upload-transcripts to true. |
Setup labels conditional expressions unresolved; it cannot determine their outcome. Off/On replace them. Review the exact file before saving. Only that workflow’s Evals step is affected.
Disabling uploads keeps structured reports and local logs.
It does not sanitize reports/logs or prevent other steps from uploading data.
To stop uploads, choose Off, review and save, then commit and push.
Previously uploaded artifacts remain subject to their existing retention.
See artifact-retention-days in the Action inputs.
Trials are not API calls or spending caps. Costs and internal requests are not capped; execution is not an OS/network sandbox.
Know what Setup changes
Optionally use Init and Setup for folders/suites and CI. Configuration and workflow saves require separate review. Saving only writes files: no commit, Actions enablement, credentials, policy changes, billing verification or execution. Unsafe updates are refused.
Set up CI step by step · Understand results · Troubleshooting