Google CloudProfessional Agentic Architect2026-094.2 Deploying and scaling production workloads

Diagnose failures with traces and logs

Completed by the knowledge check

Learning objective

Use correlated traces and logs to distinguish agent logic failures, tool failures, and infrastructure failures. This lesson focuses on classifying the cause of one failed run, not on measuring aggregate reliability or optimizing latency.

Core idea

Use the trace to locate the failure and logs to explain it. Then classify the first causal divergence from the expected execution path, not the last place where an error appeared.

A trace represents one end-to-end operation. Its spans represent individual operations and preserve parent-child relationships. Span attributes, status, and timing show what ran and in what order. Correlated structured logs add details such as validated arguments, exceptions, retries, and runtime events. Google Cloud can associate log entries with a trace and span through the trace, spanId, and traceSampled fields. See Link log entries with traces.

The visible error location is only a starting point. For example, a tool span can fail because the agent invented an invalid argument. The tool correctly rejected the request, so the originating fault is agent logic.

How it works

Start with the failed outcome. Record what the user requested and the expected completion condition. A technically successful response can still fail the task.
Open the end-to-end trace. Follow the root agent invocation through model or planning spans, tool-execution spans, downstream remote procedure calls, and the final response. Google Cloud’s Agent Runtime trace view exposes the span graph, inputs and outputs, and metadata attributes. See Set up tracing.
Find the earliest meaningful divergence. Compare the observed sequence, decisions, arguments, results, and termination reason with the expected path. Do not stop at the root span: it can report an error propagated from a child operation.
Pivot to logs by trace and span ID. Inspect the log records produced by the component that owns the suspect span. Prefer structured fields over message-text searches so you can compare failure class, attempt number, tool name, and runtime revision consistently.
Classify and confirm. Check neighboring traces. One repeated bad decision suggests logic. A valid request that fails inside one tool suggests the tool. Simultaneous failures across unrelated operations suggest shared infrastructure.

OpenTelemetry defines span status as Unset, Ok, or Error; Unset is the default. Therefore, Unset does not prove success. The specification also says that error meaning depends on operation context. Use status as evidence, not as the diagnosis. See the OpenTelemetry tracing API and error-recording conventions.

Key distinctions and trade-offs

The following table separates the three failure classes by origin:

Failure class	First causal divergence	Typical trace and log evidence	Primary repair area
Agent logic	The agent makes an unsuitable decision although its dependencies operate as specified.	Wrong or missing tool selection, invalid arguments generated before validation, ignored tool output, repeated loop, or premature stop. Spans can all be `Ok` or `Unset` while the task outcome is wrong.	Instructions, routing, state handling, tool descriptions, output validation, and evaluations
Tool	A valid tool request reaches the tool, but the tool cannot perform its contract or returns an invalid result.	An `execute_tool` span ends in error; the tool log records an exception, unexpected domain error, or response-schema violation for valid input.	Tool implementation, adapter, or published contract
Infrastructure	The runtime or transport prevents otherwise valid agent or tool work from executing reliably.	Missing or abruptly ended child spans, deadline or connection failures, process restarts, resource exhaustion, or a correlated surge across unrelated agents and tools.	Runtime capacity, deployment, networking, service health, or platform configuration

This classification is about root cause, not ownership or retry policy. A tool timeout can originate in shared networking. Inspect the request and component logs before assigning the class.

Instrument the boundaries you need to distinguish. At minimum, propagate trace context across agent, tool, and remote-service calls. Log stable identifiers, operation names, outcome codes, attempt numbers, and termination reasons. OpenTelemetry’s developing generative AI conventions define agent and tool-operation spans, including execute_tool and error.type attribute. Because these conventions are still in development, use them as a portable vocabulary but do not assume every framework emits identical fields. See the OpenTelemetry generative AI span conventions.

More detail improves diagnosis but increases telemetry cost and data risk. Google Cloud Agent Runtime telemetry excludes prompts and responses unless you enable content capture, and its documentation requires appropriate user consent and data-handling policies when you do. Prefer redacted values, hashes, schema-validation results, or secure references when raw content is not necessary. Sampling is another trade-off: Google Cloud notes that a log entry can exist even when its trace was not captured, so absence of a trace is not proof that an operation never occurred.

Architecture / decision scenario

A retail support agent receives: “Where is the refund for order 7812?” The trace shows an agent span, a successful planning span, and a failed execute_tool get_refund span. The tool log says INVALID_ARGUMENT: the request contained customer_email where the published schema requires order_id. Valid synthetic requests succeed, and runtime health is normal.

Although the error appears in the tool span, the first divergence is the agent-generated argument. Classify this as an agent logic failure. Improve the tool description and argument validation, and evaluate similar prompts. Retrying the same call does not address the cause.

Compare two later incidents. If the agent sends a valid order_id and the tool throws an exception while reading its database response, classify the incident as a tool failure. If several unrelated tools begin timing out while runtime logs show container restarts and memory exhaustion, classify the shared cause as an infrastructure failure. The same user-visible symptom can therefore require three different repairs.

Common exam traps

Treating the span with Error status as the root cause without inspecting its parent decision, input, and child logs.
Assuming Ok, Unset, or HTTP 200 means the user’s task succeeded.
Calling every failed tool invocation a tool failure, even when the agent generated invalid input.
Calling every timeout infrastructure failure without checking whether the tool itself exceeded its contract or deadline.
Assuming a missing span proves an operation did not run; sampling or broken context propagation can also make a trace incomplete.
Capturing raw prompts, tool arguments, credentials, or personal data without a defined need and protection policy.

What to remember

Use traces for execution shape and logs for component-specific detail.
Classify the earliest causal divergence, not the final propagated symptom.
A failed tool span can originate from agent logic or infrastructure.
Span status is evidence; task outcome and operation context determine meaning.
Propagate trace context and record structured, privacy-aware boundary events.

Knowledge check

References

Google Cloud, Set up tracing
Google Cloud, Set up logging
Google Cloud, Link log entries with traces
Google Cloud, Traces and spans
OpenTelemetry, Tracing API
OpenTelemetry, Recording errors
OpenTelemetry, Semantic conventions for generative AI spans