AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications
Trace tools across agent handoffs
Completed by the knowledge check
Learning objective
Trace tool-call patterns and multi-agent coordination to detect retries, failures, and anomalous usage.
Core idea
An agent’s final answer hides the execution pattern that produced it. A successful answer might follow repeated tool failures, unnecessary delegation, or duplicate business operations. Monitor the sequence of observable calls and results, not only the final response or the agent’s explanation.
For Amazon Bedrock Agents, enable execution tracing and connect its events to application and tool execution records. For other agent implementations, define an equivalent application trace contract; the Bedrock Agents response schema does not automatically instrument every framework. See 2.4.9 for cross-service trace correlation and 3.4.2 for explaining selected agent events. Here the decision is whether an observed pattern represents legitimate work, recovery, or anomalous execution.
Count logical operations separately from physical attempts. Without that distinction, three retries can look like three user requests, and repeated delegation can disappear inside one successful request total.
How it works
Capture the managed agent’s execution path
Collect the InvokeAgent execution events using the trace-enablement contract in 3.4.2, then analyze patterns across turns and attempts.
The documented TracePart includes agentId, agentAliasId, agentVersion, sessionId, and callerChain. For multi-agent collaboration, it also describes collaboratorName. Use the agent and version fields to identify the component that emitted an event. Use callerChain to identify the forwarding path to that component.
sessionId identifies the conversation; using the same value continues that conversation. It is not a unique identifier for every turn or every tool attempt. Add an application request identifier and per-attempt identifiers to distinguish execution within a session. Keep the service identifiers as separate fields so their documented meaning survives the join.
Use orchestration invocationInput and observation for pattern analysis; see 3.4.2 for the distinction between observable execution and rationale.
Reconstruct tools and delegation
The InvocationInput API structure uses invocationType to distinguish ACTION_GROUP, KNOWLEDGE_BASE, FINISH, ACTION_GROUP_CODE_INTERPRETER, and AGENT_COLLABORATOR. Its actionGroupInvocationInput identifies a proposed action-group invocation, while agentCollaboratorInvocationInput identifies a collaborator invocation.
Normalize the action-group operation fields described in 3.4.2 into a tool-operation identity rather than grouping every tool under the agent name.
For multi-agent collaboration, retain agentCollaboratorName and agentCollaboratorAliasArn from collaborator invocation input and corresponding output. Combine those with callerChain and the emitting agentVersion to reconstruct supervisor-to-specialist handoffs. Do not assume wall-clock event order alone identifies parentage when branches overlap.
Pair each proposed invocation with its observation and the backend’s execution record. The trace’s traceId identifies a trace step; it is not an application idempotency key. Record the backend operation identifier and confirmed result when a tool changes business state. A returned text observation can describe success without establishing a durable business commit.
Separate retries from repeated decisions
Define a logical operation by the request, tool operation, validated arguments, and business intent. Give each physical attempt a distinct identifier. Record why another attempt occurred: transport retry, tool error recovery, a new agent decision, or another collaborator’s independent work.
Detect repeated calls to the same operation with equivalent arguments and no intervening useful result. Also detect cycles such as supervisor to specialist to supervisor to the same specialist. Neither pattern is automatically a defect: a polling tool can legitimately repeat, and clarification can require another handoff. Compare with the operation’s approved execution contract and shared attempt budget; see 2.1.7 and 2.4.5 for enforcement and retry policy. In such a cycle, the supervisor hands the request back to the same specialist.
Start with built-in runtime metrics in AWS/Bedrock/Agents. InvocationCount measures API requests; ModelInvocationCount measures requests the agent made to the model. Compare their counts over matched periods and dimensions to detect changes in model calls per agent request. An increase suggests extra orchestration work, but does not by itself prove a retry loop.
Use TotalTime, InvocationThrottles, InvocationClientErrors, and InvocationServerErrors to separate slower execution from request-level throttles and errors. Model-level counterparts are ModelInvocationThrottles, ModelInvocationClientErrors, and ModelInvocationServerErrors. AWS documents dimension sets Operation, Operation, ModelId, and Operation, AgentAliasArn, ModelId, with metric-specific availability. Use the per-alias set for the alias baseline; do not divide metrics from different populations.
Publish application-defined counters through CloudWatch PutMetricData, using Namespace and MetricData. Suggested custom names include ToolAttempts, ToolFailures, RepeatedToolCalls, and DelegationHops; none is a built-in Bedrock metric. Define the denominator for each ratio and separate attempt-level from request-level results. Keep individual request identifiers in trace records rather than creating a metric dimension for each request.
Classify failures and anomalous usage
Use Observation.type: REPROMPT to detect repeated prompting, and inspect repromptResponse.source: ACTION_GROUP, KNOWLEDGE_BASE, or PARSER identifies the output that triggered it. This is a recovery signal, not proof that the SDK retried a network request. Correlate it with subsequent invocations before counting repeated tool attempts.
A FailureTrace can include failureReason, failureCode, traceId, and metadata. Record those fields when present. Metadata.clientRequestId identifies the downstream invocation; totalTimeMs measures an individual model, knowledge-base, guardrail, collaborator, or code-interpreter invocation. In contrast, operationTotalTimeMs is set only for the final response and measures the whole agent execution.
Use downstream identifiers and individual durations to locate the failed or repeated stage. A failed model step, malformed tool arguments, downstream refusal, and an application acceptance failure need different diagnoses. Missing observations remain unknown outcomes until an authoritative record resolves them.
Compare patterns within similar request classes and agent versions. Investigate unexpected tool names, sharp increases in attempts per logical operation, deeper delegation paths, and repeated argument changes after denials. These are application-defined anomaly signals. A larger total call count might instead reflect increased legitimate traffic.
Preserve enough safe fields to explain the pattern: operation identity, timestamps, status, selected error classification, argument fingerprint, and correlation identifiers. Remove secrets before exporting your application trace. Masking does not replace collection minimization; see 3.2.9 for sanitization controls.
Key distinctions and trade-offs
Proposed action versus completed operation: invocationInput describes what the agent invokes. Backend confirmation establishes the business effect.
Conversation versus attempt: sessionId groups a conversation. Your request and attempt identifiers separate turns and executions inside it.
Recovery versus anomaly: Repetition needs a reason and a bound. A retry that eventually succeeds can still reveal an expensive or unsafe pattern.
Trace detail versus exposure: Full inputs can help diagnose argument errors but can contain private data. Selected operational fields often explain call patterns with less exposure.
Architecture / decision scenario
A supervisor agent delegates refund requests to a billing specialist and policy questions to a policy specialist. Final answers still pass review, but tool usage rises after a release.
The team first observes more ModelInvocationCount per InvocationCount in the matched alias baseline, then collects InvokeAgent traces and groups custom ToolAttempts by request class and release. callerChain and collaborator alias fields show that refund requests bounce between supervisor and billing specialist. Repeated actionGroupInvocationInput records propose the same refund operation; backend records show one confirmed refund and later rejected duplicates.
Investigators use REPROMPT and its ACTION_GROUP source, downstream clientRequestId, and backend records to distinguish the original successful operation, a transport retry with uncertain acknowledgment, and subsequent agent-generated repeats. They restore the approved execution budget and business idempotency handling, then verify the trace pattern on representative requests. The incident is not classified as three customer refunds merely because three tool attempts appear.
Common exam traps
- Treating
enableTraceas automatic durable storage of every event. - Using
sessionIdas a unique per-turn or per-attempt key. - Treating rationale text as proof of tool completion.
- Counting repeated calls without distinguishing retries from agent decisions.
- Using
traceIdas a business idempotency key. - Assuming a successful final answer means every intermediate action succeeded.
What to remember
- Reconstruct calls from invocation input, observations, and backend confirmation.
- Preserve collaborator identity, forwarding paths, and agent versions.
- Measure attempts, logical operations, and request outcomes separately.
- Investigate repeated patterns against an approved execution contract.
Knowledge check
A supervisor agent delegates to two specialists whose work overlaps in time, so their trace events interleave in one session. Which evidence attributes each event to the delegation path that produced it?
sessionId shared by the supervisor and both specialiststraceId attached to each individual trace stepcallerChain with the emitting agent and collaborator identitycallerChain records the forwarding path to the component that emitted the event, and the agent version and collaborator fields identify that component. Overlapping branches make timestamp order unreliable, a session groups the whole conversation, and a traceId names one step, not its parent.After a release, the per-alias AWS/Bedrock/Agents metrics show ModelInvocationCount rising faster than InvocationCount over matched periods. What does this establish?
A trace shows an action-group invocation, then an observation of type REPROMPT with source PARSER, then the same action-group invocation again. How should the second invocation be classified?
REPROMPT is the agent’s own recovery signal, and its source names the output that triggered it. It is a physical attempt within the same logical operation, distinct from a network retry the SDK makes and from a new request by the user.A refund operation appears three times in a request’s trace. Backend records show one refund posted and two rejected as duplicates, and the final answer passed review. What should the incident report record?