GitHub Copilot Evals
InstallSearch /github/evals

Advanced CI configuration

Customize GitHub Actions with separate jobs, narrower permissions and automatic report artifact discovery.

Use CI after an eval works locally. Set up GitHub Actions walks through your first configuration and workflow.

Choose the workflow that fits

Start withChoose it whenTradeoff
One all-in-one jobRecommended starting point.Evaluation and reporting share token permissions.
Separate lint, run and report jobsNarrower permissions or custom dependencies.More jobs; automatic artifact discovery, explicit reporting concurrency.

Before you start

You need .vally.yaml with a named suite, access to github/evals-action, and organization policy allowing organization-billed Copilot CLI usage. Use trusted, reviewed workflows and eval sources: same-repository PRs can execute code with job permissions. Fork PRs are excluded.

Simple: one Action does the work

A root-project adaptation of the repository workflow:

name: Evals

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  evals:
    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions:
      contents: read
      copilot-requests: write
      pull-requests: write
    concurrency:
      group: evals-${{ github.workflow_ref }}
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: github/evals-action@v0
        with:
          working-directory: .
          suite: smoke
          upload-transcripts: false

Replace smoke with your suite. Use github/evals-action@v0, the moving 0.x pilot tag with discovery; fixed v0.1.0 lacks it. Setup-generated workflows may use full commit pins.

Keep the concurrency block as shown: the Action requires serialized PR reporting.

Default all mode is unchanged:

  1. validates inputs and lints the whole project;
  2. runs the selected suite after successful lint;
  3. publishes evidence and updates one sticky PR summary.

Manual runs retain summaries/artifacts without commenting. Failed evals still report and fail the job. Zero executed trials is not-run, not a passing evaluation.

Advanced: isolate lint, evaluation and reporting

Keep PR-write permission out of evaluation. Lint/run upload minimal structured artifacts automatically; one report step discovers them by shared report-id, which defaults to evals. No step IDs, job outputs, artifact inputs or conditional fallback report steps are needed.

Lint/run each need checkout; report does not and runs even after failure.

Lint --success--> Evaluate --evidence--> Report
  \------------lint evidence----------> Report
                 (if evaluation evidence is absent)

Evaluation requires successful lint. The lint-only fallback is based on missing evaluation evidence, not failed evaluation. Reporting never turns failure into a pass.

JobAction commandPermissions
Lintlintcontents: read
Evaluateruncontents: read, copilot-requests: write
Reportreportactions: read, contents: read, pull-requests: write

This uses evals/.vally.yaml; for root projects, change both execution jobs to working-directory: ..

name: Evals (separate jobs)

on:
  pull_request:
  workflow_dispatch:

permissions:
  contents: read

jobs:
  lint:
    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 5
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: github/evals-action@v0
        with:
          command: lint
          working-directory: evals

  run:
    needs: lint
    if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 15
    permissions:
      contents: read
      copilot-requests: write
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: github/evals-action@v0
        with:
          command: run
          working-directory: evals
          suite: smoke
          copilot-token: ${{ github.token }}
          max-retries: 0
          upload-transcripts: false

  report:
    needs: [lint, run]
    if: always() && github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    timeout-minutes: 5
    permissions:
      actions: read
      contents: read
      pull-requests: write
    concurrency:
      group: evals-${{ github.workflow_ref }}
      cancel-in-progress: false
    steps:
      - uses: github/evals-action@v0
        with:
          command: report

Keep the concurrency block on the report job. It serializes comment writers to prevent duplicate comments or stale updates; discovery does not configure or hide concurrency. Separate lint and run jobs do not need it.

Explicit run requires copilot-token: ${{ github.token }} for workflow-token model access. max-retries: 0 matches all instead of native retry defaults.

The event guards exclude fork PRs from execution; timeouts bound job duration, not spending or tool permissions.

Discovery, failures and reruns

Discovery uses only the current workflow run and attempt, verifying the API-provided SHA-256 digest and downloaded report’s repository, revision, run, attempt and report identity. Missing digests, expired artifacts and ambiguous matches fail; it never guesses newest evidence or reuses previous attempts.

Run evidence is preferred, with a successful companion lint report included when available. Without run artifacts, lint-only reporting warns and never claims an evaluation pass. No matching artifacts fails reporting. A report-only rerun cannot use earlier artifacts: rerun the producing jobs too.

Custom identities and explicit overrides

For independent suites, use the same custom report-id (such as smoke) on lint/run/report, with one serialized reporter per identity. Discovery supports legacy -run / -lint identities, such as evals-run and evals-lint, on artifacts produced by the new Action version.

Explicit overrides remain supported:

InputsBehavior
source-artifact-id and source-artifact-digestBypass all discovery; select the primary artifact and only explicitly supplied lint. Retains existing permissions; does not newly require actions: read.
lint-artifact-id and lint-artifact-digestOverride lint. Without primary overrides, run is discovered and needs actions: read.

Each ID/digest pair must be complete. Explicit inputs still verify the digest and repository/revision/run/attempt identity. Artifacts from older Action versions require explicit IDs/digests. See the Evals Action reference for input details.

Check your workflow

Job view / reportWhat to verify
Lint fails; Run skippedReport available lint evidence; inspect logs if absent.
Run fails; Report succeedsWorkflow remains failed; publishing is not passing.
Report skippedCheck for a same-repository PR. Manual runs retain artifacts; missing evidence fails reporting, not skips it.
No trials executedNot a pass. Check suite, file patterns and tags.

For example, Lint: passed → Run: failed → Report: passed is a failed workflow with a useful report, not a successful evaluation.

Review privacy, permissions and cost

copilot-requests: write permits model-backed evaluation. pull-requests: write permits the sticky summary. The all-in-one job combines them for convenience; the advanced workflow isolates them.

Examples disable transcripts. Before enabling them, review access and retention: prompts, output, tool calls, code and sensitive data may survive redaction. The Action itself defaults uploads off. Upload detailed transcripts? in CI Setup maps to the Action’s with inputs:

ChoiceWorkflow YAML
Keep existing setting; off for new workflowsPreserves booleans, strings and expressions; writes upload-transcripts: false when absent.
OffWrites upload-transcripts: false.
OnSets upload-transcripts to true.

Setup labels conditional expressions unresolved; it cannot determine their outcome. Off/On replace them. Review the exact file before saving. Only that workflow’s Evals step is affected.

Disabling uploads keeps structured reports and local logs. It does not sanitize reports/logs or prevent other steps from uploading data. To stop uploads, choose Off, review and save, then commit and push. Previously uploaded artifacts remain subject to their existing retention. See artifact-retention-days in the Action inputs.

Trials are not API calls or spending caps. Costs and internal requests are not capped; execution is not an OS/network sandbox.

Know what Setup changes

Optionally use Init and Setup for folders/suites and CI. Configuration and workflow saves require separate review. Saving only writes files: no commit, Actions enablement, credentials, policy changes, billing verification or execution. Unsafe updates are refused.

Set up CI step by step · Understand results · Troubleshooting