Set up GitHub Actions
Catch regressions on pull requests after an eval is useful locally.
Add two files to your repository and tag the evals you want to run. This example uses one Evals Action to lint the project, run a small smoke suite and post the results on your pull request. No local Plugin or separate runner installation is required for this workflow.
You need an eval that works locally. Otherwise, create your first eval.
1. Check repository access
In Settings → Actions → General, check that Actions is enabled and
github/evals-action is allowed.
The Action is private; your repository needs access.
An organization owner must enable Allow use of Copilot CLI billed to the organization. This example uses the built-in job token, not a PAT. Organization usage is billed to the organization; personal-repository usage is billed to its owner’s Copilot seat. Authentication and billing.
Opening or updating a same-repository PR can start a paid run without another approval prompt. Total cost is not hard-capped.
2. Choose the evals to run
You can use /eval init or /eval setup in the built Copilot CLI / GitHub Copilot App Plugin to choose
these folders and the suite; the Setup guide walks
through that optional route. To configure it manually, create a
repository-root .vally.yaml with this content. If it already exists,
merge these settings into it rather than replacing other configuration:
paths:
evals:
- evals
evalFilenames:
- "*.eval.yaml"
suites:
smoke:
description: Small set of evals for pull-request feedback.
filter:
lane: ci
This looks for eval files under evals and defines a named suite called
smoke. Change the eval directory if your files live elsewhere.
In each existing eval you want in the suite, add this top-level tag. Merge it with existing tags; do not replace the rest of the eval:
tags:
lane: ci
For example, your project could now contain:
.vally.yaml
evals\
release-notes.eval.yaml
.github\
workflows\
evals.yml
Keep the eval’s input files and skill, agent or tool sources in the repository too. CI needs the same dependencies as the local eval. The Action installs the eval runner, not your project’s custom dependencies; those need their own reviewed preparation steps.
3. Add the GitHub Actions workflow
Create .github\workflows\evals.yml with the following. If it exists,
review it first or choose another filename.
name: Evals
on:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
evals:
if: github.event_name == 'workflow_dispatch' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
copilot-requests: write
pull-requests: write
concurrency:
group: evals-${{ github.workflow_ref }}
cancel-in-progress: false
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- uses: github/evals-action@v0
with:
working-directory: .
suite: smoke
upload-transcripts: false
suite: smoke must match your configuration. The default Action validates,
lints the whole project, runs the selected suite after lint succeeds, and reports
the result.
Keep the concurrency block as shown: the Action requires it to prevent
overlapping runs from creating duplicate PR reports. Use
github/evals-action@v0, the moving 0.x pilot tag. The default all-in-one
behavior is unchanged; this tag also includes automatic artifact discovery
for separate lint, run and report jobs.
This adapts the smoke job in the repository workflow
with a root project, smoke suite, and transcript uploads disabled.
4. Review before enabling execution
The workflow runs on pull requests from this repository, not forks, and supports manual runs. Only enable it for trusted, reviewed code. The job token combines model access and PR-write permission; eval tools can access that authority. Execution is not an operating-system or network sandbox.
upload-transcripts: false retains structured reports without raw transcripts.
For troubleshooting, enable transcripts only after reviewing who can download them
and retention: they can contain prompts, output, tool calls, code, and sensitive
data. Default retention is seven days, subject to repository policy.
Use Init / Setup if your installed release provides CI; otherwise the manual files above are sufficient. Choose the suite, triggers and uploads; review the complete proposal, including its before/after diff and SHA-256. Setup keeps existing upload settings by default; absent settings use Off, matching the Action. Updating Evals does not rewrite existing workflows. Choose On to opt in or Off to replace a choice. Conditional settings are preserved unless changed; their outcome is unknown. Project configuration and the workflow are reviewed and saved separately. Saving does not commit the file, enable Actions, change billing policy or run an eval.
5. Commit and open a pull request
Review and commit .vally.yaml, the tagged evals and their inputs, and
.github\workflows\evals.yml on your branch. Push the branch and open a
same-repository pull request. Opening or updating that PR can start a paid
eval run; there is no desktop approval prompt in CI.
On the PR, open Checks → Evals to follow the job. Once the workflow is on
your default branch, you can also open Actions → Evals → Run workflow,
select a trusted branch and start it manually. If Run workflow is missing,
check that the default branch contains the file with workflow_dispatch.
6. Find and understand the results
Read the sticky Evals PR comment for the suite, outcomes and evidence links. Open the workflow run’s Summary and Artifacts for the reports. Manual runs have those summaries and artifacts but no PR comment.
Check the intended suite, executed trial count, check outcomes, and evidence links, not just whether the workflow finished:
| Illustrative report | Interpretation |
|---|---|
smoke: 2 trials; 1 failed check; evidence available | Evaluation ran and found a failure. A published report does not make it pass. |
smoke: 0 trials; not-run | No passing evaluation. Check paths, filename patterns, and lane: ci tags. |
| Lint failed; evaluation skipped | Fix the definition before interpreting model behavior. |
These are reading examples, not screenshots or measured results. Missing or malformed evidence is an error.
CI output is not automatically imported into Canvas. Use the hosted reports for this run and retain local saved results for local review and comparison.
Need separate permissions or custom stages? See the advanced lint, run and report jobs. Otherwise, this one-job workflow is all you need.