AWSAWS Certified Generative AI Developer - Professional2026-105.2 Troubleshoot GenAI applications
Trace template confusion through response contracts
Completed by the knowledge check
Learning objective
Diagnose template confusion with CloudWatch Logs and AWS X-Ray, and catch output-format drift with schema tests and response validators.
Core idea
Template confusion is a mismatch between the instructions you approved and the request the application actually assembled. It can result from selecting the wrong prompt version, binding a variable to the wrong value, appending contradictory messages, or applying instructions intended for a different response contract. Diagnose the assembled invocation rather than assuming the template file is what the model received.
Output-format drift is a related but separate failure. The response can express the intended answer while breaking the consumer’s schema, or satisfy the schema while expressing the wrong business decision. You need evidence from prompt assembly, invocation, decoding, and validation to locate the boundary that failed.
CloudWatch Logs provides request evidence and searchable outcomes. X-Ray provides the application’s execution path and timing where you instrument it. Neither determines semantic correctness by itself. Schema tests and response validators supply explicit acceptance decisions that logs and traces can then explain.
How it works
Correlate the actual invocation
Amazon Bedrock model invocation logging is disabled by default. For supported operations, including Converse and ConverseStream, you can configure CloudWatch Logs, Amazon S3, or both as destinations in the same account and Region. The log record includes requestId, operation, modelId, input.inputBodyJson, and output.outputBodyJson. Configure and protect these records according to the logging lifecycle in 4.3.2.
Invocation logging is regional configuration, not a per-template application log. Add application records for prompt selection and variable binding, and correlate them with the actual Bedrock request. Converse.requestMetadata supplies caller-defined tags that appear in invocation logs. Use non-sensitive identifiers for the application request, prompt revision, and schema revision; these names are your conventions, not fields Bedrock invents automatically.
Do not conclude that content was absent merely because it is absent from a CloudWatch event. Invocation log events include JSON input and output bodies up to 100 KB. Larger bodies and binary data use separate S3 objects when large-data delivery is configured. Verify the relevant evidence destination before diagnosing a missing instruction.
For Prompt management invocations, Converse.modelId can identify a prompt-version ARN and promptVariables provides variable values. Additional messages append to that prompt. You cannot also supply additionalModelRequestFields, inferenceConfig, system, or toolConfig in this request; define them through Prompt management. See 1.6.2 for versioning. During diagnosis, check the version-qualified ARN and the bound variables, not merely the prompt’s friendly name.
Use CloudWatch Logs Insights to find the failing request with filter, inspect identifiers with fields, and compare failure counts by application-recorded prompt and schema revisions with stats. The grouping works only if your application records those fields consistently. A missing field is an instrumentation gap, not evidence that every request used the same revision.
For example, supply prompt_revision and schema_revision as application-defined requestMetadata keys. In the invocation log group, count calls by that pair:
stats count(*) by requestMetadata.prompt_revision,
requestMetadata.schema_revisionThis query measures invocation distribution, not validation failures. Compare it with validator outcomes in the application log group using the shared request identifier. A spike in one revision pair narrows the investigation; inspect successful and failing requests from that pair to distinguish a universal template defect from a specific variable or branch. Keep the query time window aligned with the rollout so unrelated traffic does not dilute the comparison.
Trace the assembly and validation path
Instrument prompt selection, variable substitution, the model call, decoding, and validation as distinguishable operations. X-Ray segments describe request work; subsegments provide granular timing and downstream-call details. A model call that completes successfully can therefore sit inside an application trace whose later validator fails.
Use the annotation and metadata conventions in 2.4.9 to connect the selected template branch to the failing request. Preserve a protected diagnostic reference rather than putting raw customer text in searchable trace attributes.
Tracing is subject to sampling, so the absence of a trace does not prove that an invocation never occurred. Correlated application logs provide another evidence path. Also scope what a trace can show: instrumenting the client-side model call does not reveal the model’s internal reasoning or automatically reconstruct your prompt assembler.
The X-Ray SDKs and daemon are in maintenance mode; instrument added stages with OpenTelemetry as described in 2.4.9. See 2.5.9 for general integration diagnosis; the specific question here is which template or validation stage changed behavior.
Test the consumer’s response contract
Create schema fixtures for valid responses and for each meaningful invalid shape: missing required fields, unexpected fields, wrong types, invalid enum values, and incomplete output. Test the validator directly so a permissive or incorrectly configured validator does not turn production drift into silent acceptance. These tests are application-owned; CloudWatch and X-Ray do not supply them automatically.
For supported models and inference configurations, Bedrock structured outputs can constrain JSON generation. In Converse, configure outputConfig.textFormat with type: json_schema and the schema structure. Confirm support for the chosen model and inference profile. Numerical minimum and maximum constraints are unsupported, so enforce the scenario’s amount limit in your validator. If you set additionalProperties, its value must be false. See 3.1.4 for the supported schema subset and invalid-schema handling.
This distinction changes the repair. If a contract requires an amount within a business limit, enforce that limit in the application validator rather than inserting an unsupported constraint into the Bedrock schema. Structured outputs supplies shape; a business validator checks source identifiers, permitted decisions, numeric bounds, and cross-field consistency against trusted application context.
Decode the response through the selected API’s response contract and inspect stopReason. A max_tokens outcome differs from a normal end_turn, while tool_use requires the tool path. Do not treat every returned message as the final consumer object. Keep incomplete or rejected responses out of downstream business actions. See 5.2.2 for error classification and 1.6.1 for enforcement boundaries.
Key distinctions and trade-offs
Template source versus assembled request: The approved source says what should happen. Invocation and assembly evidence establish what was submitted.
Logs versus traces: Logs preserve searchable events and selected payload evidence. Traces locate instrumented work along a sampled execution path.
Schema compliance versus business validity: Types and required fields do not establish that a decision is authorized or supported by evidence.
Runtime validation versus schema tests: The runtime validator gates each response. Tests demonstrate that the validator rejects the defects you intend it to catch.
Full payload capture can accelerate diagnosis but increases the amount of sensitive evidence you retain. Record enough correlation and revision information to find a protected reproduction, and follow the existing logging policy rather than enabling indiscriminate capture for every incident.
Architecture / decision scenario
A claims assistant begins returning a field named decision_text where the consumer requires decision. Some responses also approve amounts beyond the policy limit.
The team groups validation failures in CloudWatch Logs Insights by recorded prompt and schema revision. Failures cluster around one rollout. Correlated invocation evidence shows a prompt-version ARN paired with an appended message that requests the obsolete field. X-Ray confirms the request passes through a fallback template branch before the model call; validation fails after inference completes.
The team removes the contradictory assembly path and adds a fixture that rejects decision_text. It verifies structured-output support for its chosen model configuration and applies the accepted schema. A separate business validator rejects amounts beyond the trusted policy limit, because a valid number is not an authorized amount. The regression run includes both the obsolete-field case and the excessive-amount case, then checks other fallback paths. Restored field names alone would leave the consequential defect unresolved.
Common exam traps
- Examining the saved template without checking the assembled request and variables.
- Assuming invocation logging is enabled or available for every inference endpoint.
- Treating an unsampled request as evidence that execution never occurred.
- Searching X-Ray metadata as if it were an indexed annotation.
- Sending unsupported JSON Schema constraints to Bedrock structured outputs.
- Treating valid JSON as proof of business correctness or completed generation.
What to remember
- Correlate application revisions with the invocation that actually ran.
- Use logs for evidence and traces for instrumented execution boundaries.
- Distinguish the X-Ray service from its SDK and daemon lifecycle.
- Test validators with both accepted and rejected fixtures.
- Enforce business invariants even when structured output supplies valid shape.
Knowledge check
The approved prompt template asks for a decision field, but some production responses return decision_text. Which evidence locates where the obsolete field entered the request?
requestMetadata identifiers — finds the path that reintroduced the obsolete field.A long instruction is missing from the input.inputBodyJson of a CloudWatch invocation log event. What should the engineer check before concluding that the request omitted it?
A failing request has correlated invocation and application logs but no X-Ray trace. How should the missing trace be interpreted?
A claims response satisfies the structured-output schema but approves an amount above the policy limit. Where should the limit be enforced?
maximum constraint added to the Bedrock JSON schemaminimum and maximum , and schema-valid shape is not business authorization. The application validator checks numeric bounds and permitted decisions before any downstream action; a later log query is too late, and an instruction is not enforcement.After the decision_text incident, the team discovers that its runtime validator had accepted the obsolete field without complaint. What shows that the validator now catches this drift?