Gate releases with continuous agent evaluation

Learning objective

Design a continuous evaluation loop that detects agent regressions and blocks a release when the candidate does not satisfy explicit acceptance rules.

Core idea

Treat evaluation as a release contract, not as a dashboard that somebody might inspect:

versioned candidate + fixed evaluation contract -> evidence -> gate decision

The evaluation contract identifies the dataset, metrics, evaluator versions, required slices, thresholds, and comparison baseline. The gate consumes the result and produces a decision: promote, block, or request an authorized override.

A regression is a meaningful loss against that contract. It is not merely a lower average score. A candidate can regress by violating one invariant, falling below an absolute quality floor, or performing worse than the approved baseline by more than an allowed margin. This model separates measurement from release authority: an evaluator produces evidence; the delivery pipeline enforces policy.

Build the continuous release loop

Freeze a reproducible contract

Version the representative cases, expected tool behavior, metric definitions, autorater rubric and model, thresholds, and slice definitions. Record the agent artifact under test, including its prompt, model configuration, tools, and retrieval configuration. Keep an approved production version as the baseline. Google Cloud MLOps guidance similarly recommends recording pipeline versions, parameters, artifacts, prior-version pointers, and evaluation metrics for reproducibility and comparison. See MLOps continuous delivery and automation pipelines.

Run the candidate and baseline on the same cases under comparable conditions. This paired design prevents a change in case mix from looking like a product change. Store both aggregate and case-level results. Google Cloud evaluation results expose summary metrics and per-instance results, while pairwise metrics identify whether the candidate, baseline, or neither is preferred. See View and interpret evaluation results.

Because model and autorater outputs can vary, repeat enough runs for the risk involved. Use uncertainty or a review band for borderline results. Do not rerun only failed cases until one passes.

Express the gate as policy

Use separate rules for different kinds of risk:

• Invariants: Require zero known safety-critical or policy-critical violations, such as issuing a refund without confirmation.
• Absolute floors: Require minimum task-success or groundedness scores even if the baseline is worse.
• Relative margins: Limit how far the candidate may fall below the baseline on metrics where a small trade-off is acceptable.
• Slice floors: Apply rules to important segments, such as language, workflow, or risk class, so that a strong majority cannot hide a weak group.
• Evidence checks: Block when mandatory cases, metrics, or evaluator jobs are missing. An evaluation error is not a passing result.

Avoid one weighted score for the whole gate. An average can let better writing compensate for an unauthorized tool call. Do not require every metric to improve: a cheaper or faster candidate can be acceptable when quality stays above its floor and within an agreed regression budget.

Run checks at several cadences

Use a fast, stable subset on each pull request and the full suite before release. Run the contract when prompts, models, tool schemas, orchestration, or retrieval changes. If the contract changes, score both the baseline and candidate with the new contract. ADK supports programmatic evaluation through pytest and automated runs through adk eval; its conformance command fails when behavior differs, so it can act as a continuous integration gate. See ADK agent evaluation.

After deployment, evaluate sampled production traces to detect behavior that the curated suite missed. Gemini Enterprise Agent Platform Online Monitors query sampled traces and logs, evaluate them asynchronously, and export scores to Cloud Monitoring. Their sampling controls evaluation coverage and cost. See Continuous evaluation with online monitors.

Production monitoring closes the loop: confirm failures and add representative cases to the offline suite. It does not replace the pre-release gate. An asynchronous sample detects a regression only after production exposure.

Key distinctions and trade-offs

Threshold versus baseline comparison: An absolute threshold asks whether a candidate is good enough. A relative comparison asks whether it became worse. Use both when a poor baseline must not legitimize poor quality.

Continuous evaluation versus continuous deployment: Continuous evaluation means that changes and production behavior trigger repeatable measurement. It does not mean that every passing build must deploy automatically. High-impact systems can retain approval and canary stages after the automated gate.

Broad coverage versus pipeline speed: Pull-request suites should give rapid feedback. Full suites, repeated runs, and expensive autoraters offer stronger evidence but increase latency and cost. Tier the suites; do not weaken release thresholds to make the pipeline faster.

Offline gate versus online monitor: Offline evaluation provides controlled, repeatable evidence before promotion. Online evaluation covers real traffic and changing inputs after deployment. Google Cloud also supports offline evaluation of historical traces or sessions by using predefined or custom metrics. See Run offline evaluations.

Architecture / decision scenario

A retailer changes the prompt and model for a refund agent. Its illustrative release contract contains these rules:

The pull-request suite passes, so the team builds a candidate artifact. The full release suite reports 94% overall task success, but only 82% for cross-border returns. The overall result exceeds its floor; the critical slice does not. The pipeline blocks promotion and preserves the current version.

After the team corrects the prompt, every gate passes. It promotes the candidate, archives the report, makes that version the new baseline, and enables an online monitor for the same quality dimensions. A later alert on a new policy phrase supplies a new curated case. The team changes the contract through review rather than quietly lowering a threshold to make the build pass.

Both a corrected prompt and a production alert lead back to the full release suite.

Common exam traps

• Treating a higher overall average as proof that no important regression occurred.
• Comparing a candidate and baseline on different cases or evaluator versions.
• Using only a relative comparison, so a candidate passes because it is less bad than an unacceptable baseline.
• Treating a failed or incomplete evaluation job as a passing release gate.
• Using production monitoring as the only gate, although it detects failures after exposure.
• Changing a threshold after seeing a candidate result without reviewing the acceptance policy.

What to remember

• Version the complete evaluation contract and compare candidate with baseline on the same evidence.
• Combine invariants, absolute floors, relative margins, slice rules, and evidence checks without averaging away critical failures.
• Run fast checks on changes, full checks before release, and sampled evaluation after deployment.
• Make the delivery pipeline enforce the result; a dashboard alone is not a gate.
• Feed confirmed production failures back into the curated regression suite.

Knowledge check

References

• Google, ADK agent evaluation
• Google Cloud, View and interpret evaluation results
• Google Cloud, Run offline evaluations
• Google Cloud, Continuous evaluation with online monitors
• Google Cloud Architecture Center, MLOps: Continuous delivery and automation pipelines in machine learning