AWSAWS Certified Generative AI Developer - Professional2026-105.2 Troubleshoot GenAI applications

Isolate output regressions with controlled cases

Completed by the knowledge check

Learning objective

Debug unexpected model outputs with controlled prompt cases, version comparisons, and systematic refinement.

Core idea

An unexpected answer is an observation, not a diagnosis. The model might have received ambiguous instructions, incomplete evidence, a different prompt revision, altered sampling settings, or a correct tool response that the application interpreted incorrectly. Start with a specific expected behavior and the complete failing request.

The diagnostic goal is to identify the smallest controlled change that explains the failure. A prompt that produces one pleasing answer is not necessarily repaired. Preserve the original failing case, compare reproducible configurations, and test the proposed repair against neighboring cases that could regress. See 1.6.6 for prompt regression contracts and 5.1.5 for controlled deployment experiments.

How it works

State an observable failure

Replace “the model behaves strangely” with a checkable statement: it omits the refund exception, emits an unrecognized category, follows instructions embedded in source text, or refuses an answerable request. Specify the authoritative evidence, acceptable result, and response boundary where the failure appears.

Save a sanitized diagnostic case containing the question, required facts, assembled context, model identifier, prompt revision, supported inference settings, and observed output. Include the application’s acceptance rule. If the UI displays the wrong answer but the model response is correct, the fault belongs to rendering or response processing rather than prompt wording.

Use a compact case family rather than only the failure. Include the original request, a shorter version with the same meaning, a valid contrasting case, and an unanswerable case. For a missing exception, include both a policy with that exception and a policy without it. This detects repairs that teach the model to repeat the exception regardless of evidence.

Freeze the invocation contract

For direct Amazon Bedrock Converse calls, control modelId, system, messages, inferenceConfig, and supported model-specific settings in additionalModelRequestFields. Fix the supplied tool and guardrail configuration when those affect the test. Changing several of these at once prevents a clear causal comparison.

The common inference configuration includes temperature, topP, maxTokens, and stopSequences. AWS describes lower temperature as making higher-probability options more likely, but that description is not a guarantee of identical repeated outputs. Hold supported sampling settings constant during diagnosis and repeat consequential cases to assess variability. See 4.2.6 for sampling experiments.

Inspect Converse.stopReason before labeling a response a semantic regression. max_tokens can explain an unfinished artifact; stop_sequence can explain an unexpected cutoff. guardrail_intervened and content_filtered identify distinct termination signals that should not be treated as evidence that the prompt lacks task knowledge. Guardrail enforcement is covered in 3.1.2.

For tool-driven cases, distinguish a tool_use response from a final answer. A request for a tool is not an ordinary completed text response. Keep the observed tool result fixed when investigating the model’s interpretation of it; otherwise a changing external response can masquerade as model variability.

Compare frozen versions

Amazon Bedrock Prompt management distinguishes an editable draft from a version that snapshots the prompt and its configuration. Use version-qualified prompt references to reproduce a comparison; testing a draft that continues to change weakens the evidence.

In the console, Prompt management Compare displays two versions’ JSON definitions side by side and highlights fields present in only one. Fill Test variables and choose Run prompt to compare outputs. While refining a draft, Compare variants supports up to three variants; use the same test-variable values and choose Run for each. These tools expose differences, while you design and interpret the controlled comparison.

When invoking a managed prompt with Converse, put its version ARN in modelId and supply variable values through promptVariables. Any supplied messages are appended after the prompt’s defined messages. Preserve those appended messages in the case identity so a version comparison does not accidentally compare different conversations.

Managed prompt invocations cannot include additionalModelRequestFields, inferenceConfig, system, or toolConfig in the runtime request. Define those through Prompt management. To compare a changed setting, compare appropriately configured prompt versions instead of adding an unsupported runtime override.

Run a small comparison matrix. First compare the previous and failing prompt versions against the same model and evidence. Then, if the model changed, compare the relevant model configurations with prompt content held constant. A prompt-only regression and a model-specific interaction can require different repairs. Do not describe the matrix as a service-provided automatic root-cause detector; you design and interpret the comparison.

Refine a measured cause

Classify failures before editing. Missing evidence calls for retrieval or assembly diagnosis, not stronger instructions to invent an answer. Ambiguous task wording calls for a clearer decision rule. A schema mismatch calls for response-contract validation. Instruction following from untrusted source text calls for the trust-boundary controls in 3.1.8.

Write one hypothesis and one targeted change. For example: “The exception is labeled as background, so the prompt does not require it in eligibility decisions.” Move the exception into the explicit decision criteria and rerun the unchanged case family. Then verify that an unanswerable case still abstains and that a policy without the exception does not acquire it.

Record the version pair, cases, observed failures, and acceptance outcomes. A useful refinement is one whose measured benefit survives repeats and does not break required boundaries. If the evidence is inconclusive, retain the failing case and gather more observations rather than promoting a favorable sample.

Key distinctions and trade-offs

Controlled case versus production replay: A controlled case isolates a suspected cause. A representative replay checks whether the repair remains useful under real variation. Use both, in that order.

Text equality versus behavioral equality: A paraphrased answer can satisfy the same evidence and schema requirements. Exact-string matching can overstate regressions; acceptance rules that ignore decisive facts can understate them.

Prompt version versus entire request identity: A frozen template does not freeze its variable values, appended messages, external evidence, or application processing. Preserve those dependencies as well.

Local repair versus generalization: A narrowly tailored instruction can fix one failure while damaging legitimate neighboring cases. Case families expose that trade-off before promotion.

Architecture / decision scenario

A warranty assistant starts approving claims outside the policy’s exception window. Both its prompt and selected model changed during a release.

The engineer saves the failing claim, the exact policy passage, and the expected rejection. The controlled family adds an eligible claim, an expired claim without the exception, and a claim whose purchase date is absent. Termination inspection rules out an output cutoff.

The engineer compares prompt versions with the same model and variable values, then compares the model configurations with the same prompt. The failure follows the prompt revision: it moved the exception into an optional summary. A targeted revision requires the decision to account for that clause and to request the missing date when necessary.

Repeated comparisons show the expired claim rejected, the eligible claim accepted, and the missing-date case requesting clarification. The team retains the whole family for regression testing and applies the release gates from 1.6.3. It does not promote the revision merely because the original failing answer now looks correct.

Common exam traps

Editing a live draft and assuming its identifier preserves the tested configuration.
Changing the prompt, model, and temperature together and claiming a cause.
Treating lower temperature as a deterministic-output guarantee.
Ignoring variable values or appended messages in a managed prompt comparison.
Adding prohibited runtime settings to a managed prompt invocation.
Fixing absent evidence by instructing the model to answer more confidently.

What to remember

Define the observable failure and preserve the complete case identity.
Inspect termination before investigating semantic behavior.
Compare frozen prompt and model configurations one variable at a time.
Refine a stated hypothesis and retest contrasting and unanswerable cases.
Promote measured behavior, not a single attractive answer.