Skip to lesson

Google CloudProfessional Agentic Architect2026-094.1 Evaluating agents in development and in production

Choose the right agent evaluation mechanism

Completed by the knowledge check

Learning objective

Choose Agent Development Kit (ADK) evalsets, Agent Platform GenAI Evaluation, or custom autoraters according to what an agent evaluation must measure.

Core idea

Choose the evaluation mechanism from the evidence and judgment required, not from the agent’s deployment location:

repeat an ADK interaction -> ADK evalset
score common quality dimensions at scale -> Agent Platform GenAI Evaluation
judge a domain-specific semantic rule -> custom autorater

These choices are not mutually exclusive. An evalset stores cases and expected behavior. An evaluation service runs or scores data and manages results. An autorater supplies a judgment. An ADK evalset can hold a conversation that is scored by Agent Platform, which can also register custom large language model (LLM) and code metrics.

How the choices differ

MechanismChoose it when you must measureEvidence it works withMain trade-off
ADK evalsetRepeatable ADK behavior, including turns, state, expected tools, and a reference responseStored single- or multi-turn sessionsFast, code-adjacent testing, but coupled to ADK’s schema and criteria
Agent Platform GenAI EvaluationCommon quality dimensions across datasets, traces, sessions, versions, or deploymentsCases, generated traces, or production telemetryManaged workflows and analysis, with cloud setup, quota, and cost
Custom autoraterA task-specific qualitative rule not captured by a suitable predefined metricPrompt, response, tool trace, reference, and domain contextFlexible judgment, but bias, variability, cost, and calibration become your responsibility

ADK evalsets for code-adjacent behavior

An ADK evalset contains multiple eval cases. A case can represent a multi-turn session and include initial session state, user turns, expected tool use, intermediate responses, and a reference response. This fits integration-style regression tests such as, “Does this prompt change preserve the required approval-before-refund sequence?” ADK can run an evalset from its web interface, command line, or test code. See the ADK evaluation guide.

The evalset is the test fixture, not the measurement itself. ADK applies configured criteria to the actual run. Its criteria include deterministic trajectory matching, response comparison, rubric-based final-response and tool- use quality, multi-turn task success, safety, and other measures. Some criteria delegate scoring to Agent Platform, as documented in ADK evaluation criteria.

Agent Platform for managed and production-scale evaluation

Choose Agent Platform GenAI Evaluation when the system needs a managed workflow around common measures, centralized results, or evaluation of deployed behavior. Agent Platform provides predefined single-turn metrics for final-response quality, hallucination, tool-use quality, and safety. Its multi-turn metrics cover task success, tool-use quality, and trajectory quality. The platform also supports custom LLM and code metrics in a reusable metric registry. See Manage evaluation metrics.

Agent Platform can evaluate traces or complete sessions from development or production. Online monitors can sample live traces and publish numeric scores. This fits the question, “Did task-success quality decline after deployment?” rather than only, “Does this checked-in ADK case still pass?” The workflow requires the necessary agent data and telemetry. See Run offline evaluations and Continuous evaluation with online monitors.

Custom autoraters for organization-specific judgment

An autorater is an automated judge that converts an agent artifact into a score or verdict. Build one when a predefined metric does not express the real acceptance rule. Examples include whether a medical handoff preserves a hospital’s escalation policy or a support reply uses a product-specific resolution taxonomy.

Define the judge’s input fields, criterion, rating scale, and output format. Validate its decisions against expert-labeled examples before using its score as a release gate. If the rule can be computed exactly—for example, whether a JSON field exists or a prohibited tool was called—prefer code over an LLM judge. Custom semantic judgment is valuable only when deterministic comparison cannot represent acceptable variation. Agent Platform exposes custom LLM metrics for natural-language rubrics and custom code metrics for programmatic checks; the Gen AI evaluation service overview describes both approaches.

Key distinctions and trade-offs

Framework fit versus measurement fit: Using ADK does not mean every score must be produced locally. Use an ADK evalset when its session representation and developer workflow fit. Select the scorer separately.

Predefined versus custom: A predefined metric reduces implementation effort. A custom autorater improves domain fit but must be versioned, tested, and monitored. Do not create a custom judge only to rename an existing metric.

Offline cases versus production traces: Curated cases are reproducible and can contain references. Production traces reveal actual inputs and operational behavior, but often have no golden answer and require privacy-aware telemetry. Choose the data source independently of the scoring method.

Architecture / decision scenario

A retailer has an ADK customer-service agent that looks up orders and issues refunds after checking policy and receiving confirmation.

During development, the team stores multi-turn refund conversations in an ADK evalset. The cases include initial order state and the required sequence get_order -> check_refund_policy -> request_confirmation -> issue_refund. These tests catch prompt or tool-definition changes that alter the workflow.

For release comparison and production monitoring, the team sends representative cases and captured traces to Agent Platform. Predefined metrics score final- response quality, hallucination, tool use, safety, and multi-turn task success. Central reports show results by agent version and expose production trends.

The retailer’s policy also requires the agent to explain why a partial refund was selected without promising an outcome outside the policy. No generic metric fully expresses that rule, so domain experts label examples and the team adds a custom rubric-based autorater. Exact checks still verify confirmation and refund amounts. The architecture uses each mechanism for the evidence it handles best; it does not ask one overall judge score to prove every requirement.

Common exam traps

  • Treating an ADK evalset as a metric rather than a container for evaluation cases.
  • Assuming an ADK agent must use only ADK-native scoring.
  • Choosing Agent Platform only because the agent runs on Google Cloud, without identifying the metric, evidence, or managed workflow required.
  • Building a custom autorater for an exact schema, value, or tool-call check that deterministic code can evaluate more reliably.
  • Using a generic response-quality score for a domain-specific policy rule.
  • Treating predefined, custom, offline, and online as mutually exclusive choices instead of composable dimensions.

What to remember

  • Use ADK evalsets to preserve and replay code-adjacent ADK interactions.
  • Use Agent Platform for managed metrics, centralized analysis, and offline or production-trace evaluation.
  • Use a custom autorater when the required qualitative judgment is truly domain specific.
  • Separate the case container, evaluation workflow, and scoring mechanism.
  • Prefer deterministic code when the acceptance rule can be checked exactly.

Knowledge check

References