AWSAWS Certified Generative AI Developer - Professional2026-102.5 Implement application integration patterns and development tools
Diagnose integration failures from correlated runtime evidence
Completed by the knowledge check
Learning objective
Diagnose GenAI integration using CloudWatch Logs Insights, AWS X-Ray traces of FM calls, and Q Developer error analysis
Core idea
A failed GenAI response can originate before inference, during a foundation model call, or after the model returns. Logs reveal recorded outcomes across a population; traces reveal the timing and path of individual requests; assisted error analysis suggests explanations and corrective steps. Use these views together without confusing a plausible explanation with observed evidence.
For an engineer diagnosing an integration, the first decision is which boundary failed. Did request validation reject the input? Did an inference call fail? Did the application reject the returned content? Did delivery disconnect after generation? A generic “generation failed” record erases the distinction you need to repair the system.
The correlation design belongs to 2.4.9. This lesson uses that design during an incident: select an affected cohort, reconstruct representative executions, and test a proposed correction. Do not change the prompt merely because the symptom appears in a model-powered application. Operational failures and answer-quality failures require different evidence.
How it works
Find the affected cohort with Logs Insights
CloudWatch Logs Insights query language provides fields to select data, filter to restrict events, parse to extract values, stats to aggregate, and sort and limit to inspect ordered results. Multiple commands are separated by a pipe. AWS recommends selecting only necessary log groups and the narrowest useful time range to avoid excessive query charges.
Build queries around structured application records. Application-defined fields include an attempt identifier, stage, release identity, elapsed time, and accepted or rejected outcome. Keep these distinct from the fields Bedrock supplies automatically in its own invocation records. Record enough to distinguish an attempted call from a completed call and an accepted response from a rejected response. Keep prompt contents out of routine incident records unless the investigation requires them and the access policy permits them.
Bedrock model invocation logging delivers JSON events for supported calls to a configured CloudWatch Logs log group, including automatic requestId, operation, modelId, identity.arn, input.inputTokenCount, and output.outputTokenCount fields. The caller supplies only the optional requestMetadata tags; carry a correlation key there to join invocation records to application outcomes. See 1.6.4 for enabling logging with PutModelInvocationLoggingConfiguration; logging is disabled by default, and the documented capture scope is supported calls through bedrock-runtime, not other endpoints such as bedrock-mantle.
Use the invocation log to answer which model target and principal account for the calls and token consumption. The AWS page demonstrates stats aggregation over token fields grouped by identity.arn; your application logs answer the separate question of which results were accepted or delivered. Do not interpret an invocation record alone as a successful user transaction, and use the automatic fields rather than reconstructing them from application messages.
Distinguish failed attempts from failed requests. Repeated attempts can inflate error counts while recovery reduces terminal failures. Compare attempt counts, completed requests, and accepted results separately before interpreting the incident.
CloudWatch’s Generate query translates plain English into a query and explains it line by line. Access can come from a documented managed policy or cloudwatch:GenerateQuery. Inspect the generated fields and filters before running it. For some Regions, this feature makes cross-Region calls; verify that processing boundary before submitting sensitive prompts.
Select traces that reveal downstream failures
Use the correlation and instrumentation design from 2.4.9 to join an affected request to its X-Ray trace; that Bite covers caller-side spans, annotation mapping, sampling, and the OpenTelemetry instrumentation choice. During diagnosis, choose a filter that selects the failing boundary rather than assuming every failure is visible at the root request.
X-Ray Boolean keywords distinguish HTTP outcomes: error selects 4XX responses, throttle selects 429, and fault selects 5XX. They are not interchangeable labels for every kind of model or application failure. A content rejection after a successful inference call might have none of these downstream HTTP outcomes; use the application’s outcome records to select that cohort.
Timing filters select slow requests without requiring an error status. responsetime > 5 finds response times greater than five seconds; duration >= 5 AND duration <= 8 selects total duration between five and eight seconds, including downstream calls. These example thresholds are investigative choices, not AWS service-level guarantees. Set your incident threshold from the latency regression you observed.
The important distinction is between a trace-level result and a handled downstream error. AWS states that if an error occurs downstream but the application handles it and does not return it to the user, searching for error does not find it. A recovery branch can therefore make the user-facing response successful while concealing the failing call from a root-level error search.
Use service() to filter downstream service nodes and edge() to filter a connection between nodes. Select the observed service or edge names from the trace map and apply the appropriate outcome filter there. These expressions locate traces whose downstream work needs investigation even when the root response is successful. They do not create missing instrumentation or prove that every attempt has been recorded.
Inspect representative traces from that cohort. Compare the downstream call intervals with the full request duration and the retry records in application logs. Repeated failed calls followed by recovery imply a different repair from one slow call or a slow post-processing stage. Verify the span boundaries before assigning the entire response time to model inference.
Use Q Developer within its documented scope
In the AWS Management Console, Q Developer diagnoses common errors for EC2, ECS, S3, Lambda, and Step Functions, plus IAM permission errors across console pages and limited errors elsewhere. For eligible errors, Diagnose with Amazon Q supplies an explanation and corrective steps.
This is not a universal FM failure analyzer. A Bedrock runtime exception or low-quality answer does not imply that console diagnosis supports it. For an eligible error, compare Q’s proposed explanation with the observed configuration and permission boundary.
Q does not maintain a history of previous error-diagnosing sessions. Record the incident evidence and proposed correction in your own operational record. Review a suggested permission change for scope rather than granting broad access to make the symptom disappear. After changing the identified configuration or code, reproduce the failing request and inspect both logs and traces. The disappearance of a console warning alone does not prove that the complete integration works.
Key distinctions and trade-offs
Logs versus traces: Logs support population analysis over recorded events. Traces connect instrumented intervals in representative executions. A trace is not a substitute for the request denominator used to calculate failure rate.
Model failure versus application rejection: An inference call can complete successfully while the application rejects its content. Preserve both outcomes so an incident does not turn a content-policy decision into a service-availability alarm.
Assisted diagnosis versus verification: Q and generated queries reduce investigation effort. The selected records, configuration, and reproduced behavior establish whether the explanation is correct.
Architecture / decision scenario
A support API shows increased latency after a release. Logs Insights filters the release’s records and aggregates outcomes by stage. The team discovers that many log entries are retry attempts, while terminal failures affect fewer user requests. It joins invocation-log requestMetadata to application records and compares modelId and token consumption for affected calls.
A trace-level error search misses handled downstream failures, so the team filters the observed inference service or edge and finds repeated FM-call intervals followed by application recovery. One separate failure originates in a Lambda-related console configuration error that exposes Diagnose with Amazon Q. Q suggests corrective steps; the engineer checks those steps against the actual configuration and records the analysis because the diagnosis session has no retained history.
The team corrects the configuration and repairs the excessive application retry behavior under the policy from 2.4.5. It reruns affected requests, compares request-level failure rates, and inspects call timing in correlated traces. A separate set of content rejections remains unchanged and is sent to quality evaluation rather than folded into the operational fix.
Common exam traps
- Counting retry log entries as independent failed user requests.
- Using trace-level
errorto find a downstream failure that recovery hid from the root response. - Using
faultfor a 429 throttling response instead ofthrottle. - Assuming Diagnose with Amazon Q supports every Bedrock runtime failure.
- Treating an invocation-log record as proof that the application accepted and delivered the answer.
What to remember
- Identify the failing stage before choosing a correction.
- Use Logs Insights for cohorts and X-Ray for correlated request paths.
- Separate attempts, terminal failures, and application content rejections.
- Apply Q Developer error diagnosis only within its documented console scope.
- Verify repairs with reproduced requests and operational evidence.
Knowledge check
After a release, an error dashboard counts 900 failed Bedrock calls in one hour, while support reports that about 60 customers saw a failed request. What should the investigation establish first?
identity.arn , to find the principal causing failureserror , as the count of failed requestsAn FM call returns a 5XX, the application recovers through a fallback, and the user receives a successful response. An X-Ray search for error finds nothing. Which filter selects these traces?
fault , applied to the whole tracethrottle , applied to the inference nodeservice() or edge() on the inference call, with faultresponsetime > 5 , applied to the whole traceservice() and edge() apply a filter to downstream nodes or connections, and fault matches 5XX responses; throttle matches only 429, and a latency filter selects slow traces whatever their outcome.A Lambda function’s console page shows an IAM permission error, and users separately report low-quality answers from a Bedrock model. Where does Diagnose with Amazon Q apply?
Bedrock model invocation logging is enabled. The team wants to know which model and principal produced the responses that the application later rejected. How should it join the invocation records to the application’s outcomes?
output.outputTokenCount as the rejected onesrequestMetadata and record the same key with each outcomerequestMetadata is the only invocation-log field the caller supplies, so it is where a join key goes. An invocation record shows that a call happened, not whether the application accepted or delivered its result, and timestamps or token counts cannot identify a specific request.