Distinguish agent evaluation layers

Learning objective

Distinguish response-quality, retrieval-quality, trajectory, and tool-invocation evaluation. Given an agent failure, identify which layer provides the most direct evidence.

Core idea

Evaluate the nearest observable artifact to the behavior you want to test:

retrieved evidence -> individual tool calls -> ordered trajectory -> final response

These are four evaluation lenses, not four mutually exclusive agent stages. A retrieval operation can appear as a tool call, and a response grader can inspect tool results for grounding. The distinction is the primary object being judged. Use separate scores so that one strong layer does not hide another layer’s failure.

Google Cloud similarly separates final-response evaluation from trajectory evaluation, while its managed metrics distinguish tool-use quality from multi-turn trajectory quality.

The four evaluation layers

Response-quality evaluation judges what the user receives. Depending on the task, it can compare with a trusted reference or apply a rubric when several answers are acceptable. A fluent answer is not necessarily correct, grounded, safe, or complete, so “quality” must be decomposed into criteria. Google Cloud’s managed response metrics include reference-based and rubric-based approaches.

Retrieval-quality evaluation stops before generation and inspects the candidate evidence. Precision asks how much retrieved material is relevant; recall asks how much known relevant material was retrieved. NDCG also rewards placing more relevant results higher. Gemini Enterprise’s search-quality evaluation compares retrieved documents or pages with target results and reports these measures at several cutoffs.

Tool-invocation evaluation zooms in on a call. It checks whether a call was needed, whether the agent selected the correct tool, and whether arguments have the required names, values, types, and policy constraints. It must also detect a call that should not occur. Google’s Gen AI evaluation service API exposes checks for a valid tool call, the correct tool name, parameter names, and parameter names and values.

Trajectory evaluation zooms out to the path. Google Cloud defines a trajectory as the sequence of tool calls used to reach a final response and supports exact, in-order, any-order, precision, and recall comparisons. For longer conversations, its trajectory-quality rubric also considers causal validity, efficiency, state and data flow, and recovery from errors. A trajectory can therefore fail even when every call is individually well formed.

Key distinctions and trade-offs

Retrieval quality versus grounded response: Grounding checks whether answer claims are supported by the context supplied to the model. It does not prove that the retriever found the best or required source. An answer can faithfully summarize an irrelevant or outdated passage. Conversely, retrieval can return the right passage while generation ignores or misstates it. Evaluate both sides of the retrieval–generation boundary.

Tool invocation versus trajectory: A call-level check can approve restart_service(service="checkout"), but only the trajectory shows whether the agent obtained authorization before the restart. Per-call correctness does not establish correct order, efficient routing, use of observations, or error recovery.

Trajectory versus response: A correct final answer can hide unsafe, expensive, or unnecessary actions. A valid trajectory can still end in a poor summary. Outcome-only evaluation is cheaper and less coupled to implementation; path evaluation gives better safety and diagnostic evidence but needs richer traces and expectations.

Strict versus flexible path comparison: Exact matching is deterministic and useful when order or side effects are contractual. It is brittle when several safe paths are equivalent. In-order or constraint-based checks tolerate harmless extra steps; rubric-based trajectory assessment can judge efficiency and recovery but adds model-judge cost and variability. Match strictness to the architecture invariant.

Architecture / decision scenario

A field-service agent answers pump-alarm questions. It retrieves approved manual passages, reads telemetry, runs a diagnostic, and schedules a technician only after the reset fails and the customer confirms a time.

For “Pump P-410 reports E27; help me fix it,” the team evaluates one trace at four layers:

1. Retrieval: Did the top results contain the approved P-410 E27 procedure, and did they rank it above manuals for other models?
2. Tool invocation: Did read_telemetry use the correct asset ID? Was schedule_visit absent before the customer supplied a time?
3. Trajectory: Did the agent retrieve the procedure, verify the model, read telemetry, attempt the permitted reset, observe failure, request a time, and only then schedule? Independent read-only checks need not have one fixed order.
4. Response: Did the final answer accurately state the alarm cause, actions taken, remaining risk, and appointment status without unsupported claims?

Suppose the answer accurately summarizes an obsolete manual. Response grounding can pass because the answer follows its supplied context, while retrieval quality fails because the approved procedure was missed. Suppose the agent schedules the correct technician before customer confirmation. The individual call can be schema-valid, while the trajectory fails the required ordering constraint. Separate results identify the component to repair.

Common exam traps

• Treating a helpful final answer as proof that retrieval and tool behavior were correct.
• Calling grounding a retrieval metric; grounding evaluates answer support against supplied context, not whether the right context was retrieved.
• Checking only a tool name and overlooking arguments, policy, or a forbidden invocation.
• Assuming individually correct calls form a correct trajectory.
• Requiring exact trajectory match when several safe orders satisfy the task.
• Combining all four layers into one score and losing diagnostic information.

What to remember

• Judge the nearest artifact: answer, retrieved evidence, individual call, or execution path.
• Evaluate retrieval and grounded generation separately because either side can fail alone.
• Use tool-invocation checks for selection and arguments; use trajectory checks for order, data flow, efficiency, and recovery.
• Apply strict path matching only when the exact path is an invariant.
• Keep layer scores separate, then use them together for an end-to-end release decision.

Knowledge check

References

• Google Cloud, Evaluate Gen AI agents
• Google Cloud, Details for managed rubric-based metrics
• Google Cloud, Evaluate search quality
• Google Cloud, Gen AI evaluation service API
• Google, Agent Development Kit: Why evaluate agents