Continuous evals in CI

Why it matters

The agent’s configuration steers its behaviour as much as code does. A change to CLAUDE.md, a skill or a hook deserves the same release gate that code gets.

What to practise

Write each eval as a JSON file with the commit to start from, the prompt, the folders it may_change, and the checks that define an acceptable result. Run the suite in a GitHub Actions workflow, nightly and on pull requests that touch the configuration. Each run is claude -p with --allowedTools and --output-format json in a scratch checkout of the old code under the branch’s configuration. Watch why the checker fails any change outside the allowed folders, and why the run must not use --bare.

How it fits

The pass-rate threshold enforced as a merge check is 4.3.4’s pass criterion. Each production incident that becomes a permanent eval is the regression coverage of 4.2.4. Compare the lesson’s example, where trimming one line from CLAUDE.md fails one check, with 4.2.3’s regression layer.