Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications

Connect dashboards to individual decision paths

Completed by the knowledge check

Learning objective

Design correlated operational, FM interaction, retrieval, agent, and business dashboards and traces.

Core idea

A dashboard identifies a pattern; a correlated trace explains a request that contributed to it. A foundation model (FM) application needs both because healthy infrastructure can still produce an unsupported answer, and a successful tool call can still fail the business task.

Design five views around one application-owned request identifier: operational health, model interactions, retrieval evidence, agent execution, and business outcomes. Preserve the relationship between a user request and each downstream attempt. The application identifier, Bedrock request identifier, agent session identifier, and distributed trace identifier serve different purposes. Record their mapping instead of assuming they are interchangeable.

Correlation is an instrumentation contract you implement. Placing several widgets on a CloudWatch dashboard does not create causal relationships between their data.

How it works

Establish the evidence contract

At ingress, assign an opaque request identifier and record the release configuration: application version, prompt version, selected model, retrieval configuration, and agent alias where applicable. Give each model or tool attempt a separate attempt identifier and parent relationship. Include timestamps and the final outcome so a retry does not look like a second successful customer interaction.

Amazon Bedrock invocation log records contain requestId, operation, modelId, and identity.arn. Optional requestMetadata carries caller-supplied tags. The Converse API accepts requestMetadata as key-value pairs for filtering invocation logs. Put approved correlation and configuration labels there for calls you control; do not assume an agent automatically propagates your labels to every internal invocation. See 4.3.2 for payload logging.

For distributed timing, AWS X-Ray joins instrumented services through a shared trace ID. Segments describe service work, and subsegments describe downstream calls or selected internal work. Instrument retrieval, model invocation, and tool execution at their application boundaries. A client-side subsegment provides your application’s view of a call; it does not expose every internal step of an uninstrumented managed service.

Keep request identifiers in protected logs and trace records. For custom CloudWatch metrics, use bounded dimensions such as environment, feature, and release cohort. Publishing every request identifier as a dimension creates a separate series for every request and makes the aggregate view harder to operate.

Build five complementary views

Give each view one question to answer:

ViewEvidenceDecision it supports
OperationalRequest duration, errors, dependency timing, and Bedrock InvocationThrottlesIdentify the failing or saturated boundary.
FM interactionInputTokenCount, OutputTokenCount, InvocationLatency, model identity, and prompt configurationLocate expensive or slow invocation patterns.
RetrievalMeasured retrieval duration and selected Retrieve.retrievalResults fieldsInvestigate missing, stale, or unsuitable evidence.
AgentSelected trace events, action requests, observations, and failuresExplain orchestration loops and tool outcomes.
BusinessApplication-defined completion, escalation, and reviewed acceptanceDetermine whether the workflow delivered its intended result.

Bedrock runtime metrics use the AWS/Bedrock namespace. They describe runtime inference, not the entire customer journey; see 4.3.3 for their accounting boundaries. Publish application measurements through PutMetricData under your own namespace, with explicit metric names and definitions. A metric named BusinessCompleted is a proposed application counter, not a built-in Bedrock metric.

For retrieval, Retrieve returns retrievalResults with content, location, metadata, and score. Record approved source identifiers and revisions from metadata together with the selected results. Measure duration around the retrieval call. A similarity score helps explain selection, but it does not establish factual support for a generated claim. See 1.5.7 for ranking thresholds and 3.1.6 for evidence verification.

For Bedrock Agents, set InvokeAgent.enableTrace to true when trace collection is appropriate. Returned trace data includes agentId, agentAliasId, agentVersion, and sessionId. OrchestrationTrace describes orchestration inputs and outputs; FailureTrace describes a failed step. Select the events needed for diagnosis and map them to your application request. Agent trace data can include prompts and rationale, so apply the sanitization boundaries from 3.2.9. See 3.4.2 for interpreting execution evidence.

Use managed views and preserve application context

Start with CloudWatch generative AI observability where it covers your workload. Its pre-built Model Invocations dashboard presents invocation volume, token usage, latency, errors, and throttling, plus a curated input/output log table. Enable Bedrock model invocation logging and send the logs to CloudWatch for invocation content; see 4.3.2.

For curated Amazon Bedrock AgentCore agent views, enable Observability in AgentCore. CloudWatch also provides end-to-end prompt tracing across knowledge bases, tools, and models; send structured third-party model traces through the AWS Distro for OpenTelemetry (ADOT) SDK. These managed views complement the selected Bedrock Agents trace events described here; do not treat the two agent mechanisms as interchangeable.

Extend the managed views with custom retrieval-evidence and business-outcome panels for your application contract. Pre-built telemetry does not define which source revision was authorized or which transaction counts as completed.

Make the dashboard navigable

Use a common time window and consistent cohort labels across all five views. CloudWatch PutDashboard accepts a DashboardBody definition and replaces the entire existing dashboard body on update. Version the complete definition so a partial update does not silently remove another view.

CloudWatch dashboards are global within an account, while the underlying evidence still has its own account and Region context. Make those contexts visible. An operator must know whether a model panel and a retrieval panel describe the same workload deployment.

Design a drill-down from an aggregate symptom to a protected request record, then to its attempts, source selections, and business result. Test that path with a failed request. If an operator cannot reconstruct it without guessing across timestamps, the correlation contract is incomplete.

Key distinctions and trade-offs

Metrics versus logs versus traces: Metrics summarize numerical behavior, logs retain selected events and attributes, and traces preserve relationships and duration across instrumented work. No one layer replaces the other two.

Model success versus business completion: An accepted inference call can generate a useless answer. A tool observation can show an HTTP success without proving that a refund or reservation completed. Publish business completion only after the application’s authoritative confirmation.

Agent trace versus distributed trace: Bedrock agent traces expose orchestration evidence. X-Ray traces expose instrumented service relationships. Correlate them explicitly rather than treating a Bedrock traceId as the X-Ray trace ID.

More detailed evidence improves diagnosis but increases storage, access, and review obligations. Keep a broad aggregate view and a narrower protected evidence view. Show the symptom in aggregate panels without requiring every viewer to read customer prompts.

Architecture / decision scenario

A support agent reports rising response time and falling resolution rate. Its operational panel shows no increase in model errors, but the FM panel shows more input tokens. The retrieval panel records more selected passages after a configuration release.

Open a correlated request. Application subsegments show retrieval completing quickly, while selected OrchestrationTrace events show repeated tool calls after an ambiguous result. The business record shows escalation rather than confirmed resolution. Investigate retrieval context growth and the tool-result contract before classifying the incident as Bedrock unavailability.

The team restores the reviewed configuration and checks all five views: dependency duration, token usage, selected sources, agent attempt count, and confirmed resolution. Use the shared request mapping to verify that the same failure path improved.

Common exam traps

  • Treating a dashboard containing several services as automatically correlated.
  • Using a session ID as the unique identifier of every request and attempt.
  • Equating low model error rates with accurate answers or completed business tasks.
  • Assuming X-Ray exposes all internal Bedrock agent orchestration.
  • Publishing prompts or request identifiers as routine metric dimensions.
  • Interpreting retrieval similarity as proof that an answer is supported.

What to remember

  • Map application requests to model attempts, retrieval evidence, agent events, and business outcomes.
  • Use metrics to find patterns and protected logs and traces to explain examples.
  • Instrument service boundaries and preserve release configuration.
  • Define business success independently of inference and tool-call success.
  • Verify the drill-down path with a real failure before relying on the dashboard.

Knowledge check

References