Skip to lesson

Google CloudProfessional Agentic Architect2026-094.1 Evaluating agents in development and in production

Define measurable success criteria for agent tools

Completed by the knowledge check

Learning objective

Define measurable success criteria for an agent’s tool selection, tool arguments, tool execution, and task completion. Use these criteria to decide whether an evaluation case passes and whether a release meets its quality bar.

Core idea

Treat success as four gates, not one score:

select the right action -> construct the right call -> verify its effect -> satisfy the user's goal

A fluent final answer can hide a wrong tool, unsafe arguments, or a failed side effect. Conversely, a valid tool call does not prove that the tool ran or that the overall task finished. Evaluate each gate from its nearest observable evidence.

Make every criterion an operational definition:

criterion = population + observable + oracle + comparison rule + threshold

The population says which cases or production events count. The observable is the recorded call, result, state, or response. The oracle defines acceptable behavior. The comparison rule converts evidence into pass or fail. The threshold sets the required result over a stated test set or time window. This turns “uses tools correctly” into a reproducible decision.

Define the four gates

GateCase-level success criterionUseful aggregate measure
Tool selectionThe agent invokes every required tool, invokes only allowed tools, and makes no call when clarification or refusal is required.Required-call recall, selected-call precision, prohibited-call count
ArgumentsThe call satisfies the schema and all case-specific value, identity, authorization, range, and freshness constraints.Schema-valid rate, required-key match, key-and-value match, critical-argument pass rate
ExecutionThe tool runs within its operational limits and the observed domain postcondition is true, without duplicate or forbidden side effects.Tool error rate, postcondition success rate, latency percentile, duplicate side-effect count
Task completionThe user’s requested end state is true, and the final response accurately reports the outcome or unresolved failure.End-to-end completion rate, required-fact coverage, unsupported-success count

Google Cloud’s Gen AI evaluation service exposes separate checks for a valid tool call, the tool name, parameter keys, and parameter key-value pairs. It also evaluates complete trajectories separately from final responses. See the Gen AI evaluation service API and agent evaluation metrics.

For selection, state required, allowed, and forbidden calls. Include cases in which the correct decision is no call. Precision penalizes unnecessary calls; recall penalizes missing required calls. Neither measure alone captures both errors.

For arguments, separate syntactic validity from semantic correctness. A call such as refund(order_id="A17", amount=500) can satisfy its JSON schema yet exceed the authorized amount or target the wrong customer. Critical fields can use exact predicates, while flexible fields can use ranges, sets, or domain-specific comparators.

For execution, inspect the function response and, for consequential actions, verify external state. A transport-level success code or a tool’s ok field is evidence that execution returned normally; it is not always proof of the business postcondition. Agent Platform evaluation data can include intermediate_events with function calls and function responses, which provides evidence across this boundary. See Prepare your evaluation dataset.

For task completion, define the goal independently of the preferred wording or path. An action task usually needs a verified state change. An information task usually needs correct, complete, grounded content. When execution fails, an accurate failure report can pass response criteria but must not count as task completion.

Key distinctions and trade-offs

Per-case invariants versus aggregate targets: “Never invoke delete_account without confirmation” is a hard invariant. “At least 98% of eligible searches finish within three seconds” is an aggregate target. Do not average a safety violation into a passing score. Google Cloud’s service-level objective model similarly separates a measured indicator, a performance goal, and a compliance period. See Concepts in service monitoring.

Strict matching versus valid alternatives: Exact tool names and values are appropriate when one authorized action is required. Set-based or constraint- based checks are better when several tools or argument values are equivalent. ADK supports exact, in-order, and any-order trajectory matching; choose the least strict rule that still protects the invariant. See the ADK evaluation criteria.

Offline reproducibility versus production realism: Fixed tools and state make failures repeatable. Production telemetry reveals latency, permissions, dependency failures, and drift. Use the same gate definitions in both places, but set thresholds for named datasets and time windows rather than claiming one universal percentage.

Architecture / decision scenario

A release agent deploys invoice-service version 3.4.2 to staging and reports when it is healthy. The team defines these criteria before running the tests:

GateMeasurable criterion
SelectionEvery eligible case calls deploy_staging; no case calls deploy_production; ambiguous versions cause clarification and no deployment.
Argumentsservice, version, and environment exactly match approved case data; the change-ticket ID belongs to that release. All critical arguments must pass.
ExecutionThe deployment call returns an operation ID, finishes within five minutes, and a separate status observation reports the requested revision healthy. Duplicate deployment count is zero.
CompletionThe requested revision is healthy in staging, and the response reports the same revision and environment. If health cannot be verified, completion fails and the response states that uncertainty.

The release gate requires zero production calls and zero wrong critical arguments. The team may also require, for example, at least 98% verified completion across a named representative set. These numbers are architecture decisions derived from impact and reliability needs, not defaults supplied by an evaluator.

If the agent selects the right tool but swaps staging for production, tool selection passes while arguments fail. If the call is correct but the rollout stalls, selection and arguments pass while execution and completion fail. If the rollout succeeds but the response names version 3.4.1, execution and task state pass while response accuracy fails. The separate gates locate the defect and prevent one success from hiding another failure.

Common exam traps

  • Treating schema-valid arguments as semantically correct arguments.
  • Counting a proposed or accepted tool call as successful execution without checking its result or postcondition.
  • Treating a correct final statement as proof that the requested state change occurred.
  • Using only recall for tool selection and missing unnecessary or forbidden calls.
  • Averaging a prohibited action with ordinary successes instead of making it a hard release gate.
  • Requiring one exact trajectory when multiple paths satisfy the same safety and outcome constraints.

What to remember

  • Evaluate selection, arguments, execution, and task completion as separate gates.
  • Define each criterion with a population, observable, oracle, comparison rule, and threshold.
  • Verify domain postconditions; a successful function return may be insufficient.
  • Keep safety and authorization invariants as hard gates.
  • Set aggregate thresholds for explicit datasets or time windows and report important slices separately.

Knowledge check

References