Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications

Diagnose failures with controlled output comparisons

Completed by the knowledge check

Learning objective

Diagnose model failures using golden datasets, output comparisons, and safe execution traces.

Core idea

A bad answer is an observed application outcome, not a diagnosis of defective model weights. Missing evidence, a changed prompt, invalid tool results, truncated generation, and an incorrect acceptance rule can all produce an apparent model failure. Use a fixed set of reviewed cases to reproduce the symptom, compare outputs under controlled conditions, and inspect safe execution evidence to locate the failing stage.

A golden dataset is your application’s reviewed diagnostic contract. Amazon Bedrock evaluation jobs provide concrete scoring mechanisms, but AWS does not decide which business cases represent your production obligations. See 1.6.6 for prompt regression contracts and 4.3.4 for publishing quality measurements. This lesson uses evaluation evidence to explain a failure and decide which component to repair.

Keep diagnosis separate from execution. Replaying a request that sends money, updates a customer record, or deletes data must not repeat the production side effect.

How it works

Build a versioned diagnostic dataset

Select reviewed cases covering routine requests, ambiguous requests, unavailable evidence, tool failures, and consequential actions. For each case, preserve the input, expected evidence, acceptable outcome, and failure classification. An acceptable outcome can be a bounded clarification or abstention rather than a generated answer.

Add representative cases from incidents after removing private data and reviewing expected behavior. Maintain stable case identifiers and version the dataset. Keep a held-out set so repeated repair against known cases does not become the only evidence of correctness. These are application design practices, not automatic Bedrock dataset curation.

For Bedrock model-as-a-judge evaluation, custom prompt datasets use JSON Lines and the .jsonl extension in S3. Each line is a valid JSON object, and a job supports up to 1,000 prompts. The schema includes prompt, optional referenceResponse, and optional category. Use categories to expose diagnostic slices rather than hiding every failure inside one overall score.

A supplied referenceResponse contributes to Builtin.Correctness and Builtin.Completeness. Those metrics also support evaluation without a reference response, so the mere existence of a correctness score does not establish comparison with approved ground truth. Record whether the job used references and review their quality.

Compare outputs without changing the experiment

Run the baseline and candidate against the same cases and source snapshot. Record model target, prompt revision, generation settings, retrieval configuration, tool contract, and evaluator identity. Change one suspected dependency at a time when isolating cause. A repair experiment that changes both the model and the evidence cannot tell you which change corrected the answer.

Compare outputs by case identifier. Use exact comparisons for deterministic structure, required fields, and explicit business constraints. Use reviewed semantic criteria for prose whose wording can vary. Repeated generations help distinguish a consistent failure from output variability; a single improved answer is weak evidence of a reliable repair.

Bedrock model-as-a-judge evaluation can score responses you generated yourself. Include modelResponses with response and modelIdentifier; Bedrock skips model invocation and evaluates supplied response data. That mode supports only one response per prompt and one unique modelIdentifier per evaluation job. Compare baseline and candidate through separate jobs with matched cases rather than inserting two model responses into a dataset that requires one.

Keep the evaluator configuration and rubric fixed across those jobs. Bedrock returns scores and explanations, with the full evaluation report in the selected S3 bucket. Inspect the per-case report and compare reviewed failures, not only a console histogram. A judge explanation is assessment evidence; it does not replace domain review or establish that an operational action completed.

Isolate retrieval, reasoning, and execution

Record Converse stopReason for each golden case before semantic scoring; see 2.5.1 for the generation-outcome contract. Distinguish max_tokens, guardrail_intervened, content_filtered, malformed_model_output, malformed_tool_use, and model_context_window_exceeded from an ordinary end_turn. Preserve stop_sequence and tool_use as distinct outcomes too: tool_use requests tool execution, so a non-end_turn value is not automatically a failure. Check delivery and application acceptance separately before attributing a score decline to model quality.

For a retrieval-augmented generation (RAG) application, inspect the evidence supplied to generation. Bedrock retrieve-only RAG evaluation jobs can evaluate Knowledge Bases or supplied retrieval response data. Use that boundary to test whether expected evidence was retrieved before changing the generator. If the relevant policy never reached the prompt, a model swap does not repair the missing source.

For Bedrock Agents, set InvokeAgent.enableTrace to true and collect the returned execution events. The trace includes model invocation details and orchestration invocationInput and observation objects. A failureTrace supplies failureReason and traceId. Correlate those events with the application’s case and attempt identifiers; see 4.3.7 for tool-pattern analysis.

Locate the first observable contract violation. Examples include a missing retrieved reference, a proposed tool operation with invalid arguments, a failed tool response, or a rejected final answer despite successful intermediate steps. An agent rationale can describe its intended interpretation, but it is not proof of evidence support or backend completion.

Preserve safe diagnostic evidence

Build a selected execution view containing operation names, configuration references, timestamps, statuses, case identifiers, approved source references, and sanitized argument summaries. Treat prompts, retrieved text, tool results, and rationale as potentially sensitive. Do not export a complete trace simply because it is useful to an investigator.

Sanitize application trace fields before persistence or export using the controls in 3.2.9; masking is not deletion or minimization.

For replay, use an application test harness with approved fixtures or isolated tool adapters. Preserve the recorded observation as input when you want to compare only the generation stage. If you intentionally test live retrieval, label that experiment separately because its evidence can change. Restrict any required raw payload review to the approved investigation path rather than copying it into the golden dataset.

Key distinctions and trade-offs

Golden case versus production transcript: A golden case has a reviewed expected outcome. A transcript records what happened and might contain unsafe or incorrect behavior.

Output comparison versus root cause: A score difference locates a symptom. Controlled dependency changes and execution evidence support attribution.

Generator evaluation versus retrieval evaluation: Model scoring evaluates responses. Retrieve-only RAG evaluation isolates evidence retrieval.

Trace masking versus safe replay: Masking limits exposure of recorded content. A replay harness must independently prevent business side effects.

Architecture / decision scenario

A claims assistant starts rejecting eligible customers after a prompt release. The team reproduces the issue with reviewed golden cases, including eligible, ineligible, and missing-evidence cases.

Paired outputs show lower correctness for one category. Direct Converse golden runs first record stopReason, separating malformed_tool_use from tool-use requests and normal completion; a structurally valid call can still contain a business-invalid identifier. Separate supplied-response evaluation jobs hold the judge and references constant. Retrieve-only checks show that both releases received the approved policy, narrowing the hypothesis to generation or later execution.

A selected agent trace reveals that the candidate proposed a claim-lookup tool with a malformed identifier. The backend rejected it, and the final answer interpreted the failure as customer ineligibility. Investigators inspect sanitized argument summaries and failureReason where emitted, without copying raw customer records or private rationale into the dataset.

The team repairs the tool-argument contract, replays with isolated fixtures, and compares all categories against the baseline. It also verifies the missing-evidence cases still abstain. It promotes the repair only after business acceptance checks and per-case review support the result.

Common exam traps

  • Blaming the model before checking retrieval, tools, and transport.
  • Treating optional reference responses as automatically present ground truth.
  • Supplying two responses per prompt to a judge job that supports one.
  • Changing the evaluator or evidence while claiming a controlled comparison.
  • Replaying production tool side effects during diagnosis.
  • Treating masked traces as deleted or harmless raw data.

What to remember

  • Diagnose against versioned cases with reviewed expected outcomes.
  • Compare matched outputs while controlling dependency and evaluator changes.
  • Use retrieval evaluation and execution traces to locate the failing stage.
  • Sanitize diagnostic evidence and isolate replay from business side effects.

Knowledge check

References