Decompose end-to-end agent latency

Learning objective

Decompose the end-to-end latency of an agent turn into model, retrieval, orchestration, and tool latency. This lesson excludes loop budgets and failure ownership.

Core idea

Treat latency as a timeline with ownership, not as four dashboard numbers to add together. Start with one end-to-end trace, assign each timed operation to a component, and find the request’s critical path: the dependent sequence that determines when the response can finish.

For a fully sequential turn, end-to-end latency is approximately the sum of its model, retrieval, orchestration, and tool intervals. Agent workflows often run operations in parallel, however. Parallel span durations overlap, so only the slowest required branch determines the join time. Optimizing work that is not on the critical path might reduce resource use without making the response faster.

Google Cloud defines a trace as one end-to-end operation and a span as one finite operation with a start time, end time, and parent relationship. This structure provides the evidence needed to reconstruct the timeline. See Traces and spans.

How it works

1. Define the measured boundary. Start the root span when the agent service accepts the request and end it when the response completes. For a streamed response, also record the first-output event so the same trace provides TTFT and TTLT. If client-perceived latency is the service-level indicator, measure from the client so network and interface delay are not hidden.

2. Create spans at component boundaries. Propagate the trace context through model, retrieval, tool, and remote-agent calls. Without propagation, a remote operation appears as an unrelated trace and leaves an unexplained gap. Cloud Trace uses the trace ID and parent span ID to reconstruct distributed execution; see Trace context.

3. Assign each operation once. Apply one documented ownership rule to every span:

   • Model latency covers each generative model request as observed by its caller, including provider wait and response generation. Record every model call, not only the final answer call.
   • Retrieval latency covers the pipeline whose purpose is selecting grounding context, such as query transformation or embedding, index search, metadata filtering, reranking, and fetching the selected content.
   • Tool latency covers non-retrieval external capabilities, such as an order lookup, payment API, code execution, or remote Model Context Protocol (MCP) server. Include retries in the operation that caused them.
   • Orchestration latency covers controller work between dependency calls, such as loading state, routing, validating arguments, scheduling branches, coordinating handoffs, joining results, and assembling the response.

4. Calculate two views. Component work time is the union of the category’s intervals, or a sum of selected non-overlapping spans. It can exceed wall-clock time across categories because work overlaps. Critical-path contribution is the non-overlapping time that a category adds to completion. Critical-path contributions, plus any explicitly reported uninstrumented gap, reconcile to the root span. Do not add a parent span’s inclusive duration to its child spans.

5. Aggregate comparable turns. Group traces by request class, outcome, model and prompt version, tool path, region, and streaming mode. Compare median and tail percentiles for both end-to-end and component latency. Google SRE guidance recommends distributions because averages hide slow tails; see Service level objectives.

The Agent Platform trace view exposes a span graph and individual model, API, and tool operations. It can therefore support both the execution-path and timing views of a turn. See View agent traces.

Key distinctions and trade-offs

Time to first token versus time to last token: Time to first token (TTFT) measures when a streaming response begins. Time to last token (TTLT) measures model completion. Streaming can improve perceived responsiveness without reducing total completion time. Google Cloud documents both measures in Consumption options.

Retrieval versus tool latency: Classify by the operation’s role, not by the technology it uses. A vector query that selects grounding passages is retrieval. A database query that checks an order before a refund action is a tool call. This rule keeps trends comparable when both operations use the same database or API infrastructure.

Span duration versus contribution: A 500 ms retrieval span and an 800 ms tool span running together consume 1,300 ms of component work but add only 800 ms before their join. Both measurements are useful: work time supports capacity and cost analysis, while critical-path contribution directs latency work.

Per-trace values versus percentiles: Do not estimate p95 end-to-end latency by adding p95 model, retrieval, orchestration, and tool latencies. Those values can come from different requests, and their intervals can overlap. Calculate end-to-end percentiles from root spans and use the corresponding slow traces to identify which component dominated them.

More spans improve attribution but increase telemetry volume and can expose sensitive inputs. Prefer stable, low-cardinality attributes such as operation, model, tool, route, attempt, and outcome. Keep prompt text, credentials, and raw business data out of latency attributes.

Architecture / decision scenario

A commerce agent answers a delivery question. Its trace contains 210 ms of orchestration, two sequential model calls totaling 900 ms, a 420 ms retrieval call, and a 700 ms order-system tool call. Retrieval and the order lookup run in parallel after the first model call. The final model call waits for both.

The component work totals 2,230 ms, but the root span is 1,810 ms. The critical path contains 210 ms of orchestration, 900 ms of model work, and the 700 ms tool branch. The retrieval branch finishes 280 ms before the tool and therefore adds no incremental time at that join.

The team should not begin by tuning retrieval, even though it is 420 ms of work. The order tool is the active bottleneck. They first inspect its network wait, service execution, and retry spans. If they reduce the tool below 420 ms, retrieval becomes the next critical branch. They also inspect slow traces by route: a model-heavy percentile for simple policy questions and a tool-heavy percentile for order questions require different changes.

Common exam traps

• Adding all child-span durations and calling the result end-to-end latency.
• Timing only the final model call when one turn contains planning and follow-up model calls.
• Treating TTFT as total completion latency or assuming streaming reduces TTLT.
• Labeling every database or search API call as retrieval regardless of its purpose in the workflow.
• Optimizing the longest non-critical parallel branch instead of the branch that controls the join.
• Adding component p95 values to predict the end-to-end p95.
• Hiding missing instrumentation inside “orchestration” instead of reporting an unattributed gap.

What to remember

• Define one user-relevant boundary before comparing latency.
• Assign every operation once to model, retrieval, tool, or orchestration.
• Use the critical path for wall-clock latency and component work time for total effort.
• Record every call and preserve parent-child context across services.
• Measure end-to-end percentiles directly, then inspect their component spans.

Knowledge check

References

• Google Cloud, Traces and spans
• Google Cloud, Trace context
• Google Cloud, View agent traces
• Google Cloud, Consumption options
• Google SRE, Service level objectives
• OpenTelemetry, Tracing API