Initialize and set up your project
Use Init or Setup to choose where evals live and group them into named suites.
Run /eval init inside Copilot to configure your project. Use /eval setup
for changes. They are aliases for the same operation.
Setup is optional. Existing valid projects can use Create, Review or Run without initialization.
Choose the right setup
| You need to | Use |
|---|---|
| Install Evals in your app | Choose an install |
| Choose eval folders and suites | /eval init or /eval setup |
| Install a missing eval runner | Your app’s Set up test runner action |
| Add a GitHub Actions workflow | Set up evals in CI |
Project Setup does not install software. It changes .vally.yaml, not your
skills, agent files, eval definitions or saved results.
1. Open Copilot in the project
Install the built Plugin, open a terminal in your repository and start Copilot:
copilot
Then enter this inside the Copilot conversation, not PowerShell:
/eval init
For later changes, use:
/eval setup
The native commands require the built Copilot CLI / GitHub Copilot App extension. With repository skills only, ask the Setup skill in natural language instead:
Set up this eval project. Use evals for eval files and define a smoke suite
that selects lane: ci tags. Preserve unrelated configuration.
Show the proposed changes before saving. Do not run any evals.
Where available in your installed Canvas, open Settings → Project configuration for the same project settings. Do not assume CLI command syntax works in every editor’s chat. If your app lacks Setup’s tools, use a supported app; do not switch save tools to bypass a refused or failed save.
2. Choose locations and a suite
Setup prefills existing values, detected folders or labeled fallback suggestions.
Edit before reviewing. Choose an eval folder and pattern (evals, *.eval.yaml).
Skill and results folders are optional. Without paths.results, Run writes
timestamped reports to a unique folder under the project-local default.
A suite is a named selection of evals. For example, name a suite smoke and
filter it by lane: ci. Newly tagged evals in the configured locations then join
the suite without maintaining a file list. The name and tag are examples, not
required conventions.
Setup configures the filter; it does not tag or create evals for you. Explicit file selectors are also supported. If you combine file selectors and tags, both must match. Keep a filter specific: an empty filter matches all discovered evals.
3. Review and save
Inspect the complete .vally.yaml proposal, then save through normal host tool
permissions or the Canvas Save button. Evals adds no second save confirmation.
Unrelated settings and comments are preserved. Cancelling before Save leaves the file
unchanged. Resolve unreadable configuration, conflicting edits or unsaved editor
changes before proposing again; do not overwrite them to get past an error.
Saving configuration is not native validation, runner installation or proof that an eval passes. It does not run models or authorize evaluation spending.
4. Create an eval or connect CI
Use Create for a new eval, or follow the CI setup journey for an existing one.
The built Copilot CLI / GitHub Copilot App extension offers CI after saving or confirming the project. Check your installed release’s capabilities. To enter directly:
/eval setup ci
/eval init ci is its alias, not a suite selection.
Choose manual runs, pull requests or both, a workflow filename,
and select a suite from the configured-suite dropdown, even if only one exists.
In GitHub Copilot App or VS Code Canvas, Settings → CI pipeline opens
these choices.
For Upload detailed transcripts?, choose Keep existing setting; off for new workflows, Off or On. Keep preserves an existing opt-in. Review the YAML choices and privacy limits before opting in.
With no suite, save one first. Canvas shows the diff before Save. Native App/CLI forms collect settings and save under host tool permissions, without another save form. Changing a choice requires a new proposal; cancelling the configuration flow before saving creates no workflow. Saving does not commit or run it, request secrets or change billing policy. Pushing can trigger paid runs without desktop approval.
If that guided flow is unavailable, use the manual CI example with separate review, never to bypass a failed Setup save.