Prove fixes against representative regression cases

Learning objective

Prove troubleshooting fixes using representative golden cases and regression datasets before release.

Core idea

An incident reproduction proves that a defect exists. A successful replay after a change proves that the change can repair that reproduction. Neither result alone proves that the candidate is ready for release. You must also show that the repair preserves the behaviors, evidence boundaries, and operating constraints the application already supports.

A golden case is a reviewed example with an explicit expected outcome. A regression dataset combines such cases across the supported workload, including previous failures and cases that should remain successful. The word “golden” describes review quality, not an expectation that every valid answer must match one sentence exactly.

Use three evidence layers: the original failure no longer occurs, representative neighboring behaviors remain acceptable, and the repaired path meets operational requirements. See 5.1.2 for dataset construction and 4.3.9 for controlled diagnosis. This lesson turns those mechanisms into a pre-release decision about a specific troubleshooting fix.

How it works

Freeze the failure and its expected outcome

Preserve the original input, trusted caller context, source revision, relevant configuration, and observable failure. Describe the expected result before evaluating the candidate. A reviewed expectation might require a particular policy exception, correct abstention when evidence is absent, rejection of an invalid response, or completion within an application deadline.

Record enough configuration to reproduce the path: model selection, prompt revision, retrieval destination, embedding contract, response schema, and validator version where they affect the incident. Application-maintained case identifiers connect these records to the results. They are your test harness conventions, not a built-in Bedrock golden-case resource.

First run the baseline and confirm that the preserved case still exposes the defect. If it does not, investigate changed sources, routing, or test context before claiming the candidate fixed it. A test that no longer exercises the failed boundary cannot establish a repair, even if every run returns a plausible answer. Record a non-reproducing baseline as inconclusive, not repaired. For a freshness incident, for example, first establish whether a source refresh already changed the baseline’s evidence; otherwise you might attribute the disappearance of a stale answer to an unrelated prompt change.

Then replay the candidate with the intended repair isolated. Where output varies, use repeated runs and report the number of attempts and failures. Do not claim that a deterministic schema check makes the generated business answer deterministic. The validation rule can be exact while the upstream language generation varies.

Expand around the repair’s impact

Add neighboring cases that the repair could damage. A chunking repair needs questions requiring adjacent clauses and questions needing a precise isolated fact. An embedding repair needs evidence that ingest and query configurations agree across the intended corpus. A template repair needs every relevant binding and fallback path. A parser repair needs invalid as well as valid response fixtures. Choose these cases from the dependency that changed, then explain what each protects. A larger chunk might preserve an exception while diluting an exact product-code match; a stricter validator might reject a legitimate optional-field response. Include both sides of that boundary rather than adding unrelated easy questions to raise the pass rate.

Include negative expectations. No matching authorized evidence can require abstention; an invalid object can require rejection; an incomplete generation can require a terminal failure outcome. Measure those as successful policy decisions when they meet the contract. Maximizing the count of returned answers would reward a system that ignores these boundaries.

Keep a broader holdout set beyond the cases used to develop the repair. Include representative languages, document types, request lengths, and risk categories from the actual application. “Representative” is a property you justify from workload coverage, not a guarantee supplied by an evaluation service. Rare consequential cases may need explicit coverage even when their production volume is small.

Run the real application and evaluate its results

A complete regression run must execute the changed application path. Invoking a foundation model with a flat question does not test your prompt selection, access filters, context assembler, decoder, or validator. Use your harness to capture the final result and the relevant intermediate evidence, then apply deterministic checks and reviewed quality rubrics.

Evaluate responses captured from the changed application path with Amazon Bedrock’s bring-your-own-response judge mode; see 5.1.2 and 4.3.9 for dataset layout and separate baseline and candidate jobs. Preserve the complete application configuration in the experiment records, and use the disagreement handling in 5.1.4 when a judge’s score conflicts with the case’s reviewed evidence.

For retrieval fixes, add retrieve-only evaluation or direct passage checks. Bedrock’s retrieve-only dataset uses conversationTurns rather than the flat model-evaluation layout. Its referenceResponses describes the expected end-to-end answer, not expected retrieved chunks. In bring-your-own-results mode, referenceContexts can describe expected passages for custom metrics, but the built-in metrics do not use it. Keep exact source-revision and permission assertions in your harness instead of assuming those fields establish them automatically.

Turn results into a release gate

Use the CreateEvaluationJob and GetEvaluationJob lifecycle and acceptance-gate distinction in 5.1.7 to collect evidence before deciding whether this repair can proceed.

Set the release rules before reading candidate scores. Require the incident cases to pass, forbid specified consequential failures, and bound acceptable changes in representative quality and operational measurements. Your delivery process must implement this gate: Bedrock evaluation does not automatically know your business thresholds or approve your release.

Compare paired case outcomes, not only two overall averages. A better aggregate can hide a regression in the exact risk slice the repair must preserve. Separate quality checks from schema, authorization, freshness, and latency checks so a high judge score cannot compensate for a violated hard boundary. Classify each paired case as still passing, repaired, newly failing, or still failing. Inspect every newly failing case against the changed component and retain the associated intermediate evidence. Report incident-repair results separately from preserved-behavior results: a candidate that fixes all reproductions but introduces one forbidden failure must remain blocked.

Store the candidate configuration, dataset identity, evaluator configuration, per-case results, and approval decision together. If a test fails, keep the candidate out of release and investigate the failing boundary. Avoid repeatedly changing the gold labels until the candidate passes; revise a label only through a documented review of the expectation itself.

A candidate reaches release only from a reproducing baseline, and only when its quality checks and its hard-boundary checks both pass.

Key distinctions and trade-offs

Reproduction versus regression coverage: The reproduction targets one known cause. The regression set checks the repair’s wider consequences.

Exact checks versus semantic judgments: Schema and permission checks have explicit acceptance rules. Natural-language equivalence requires a suitable rubric and reviewed evidence.

Evaluation completion versus promotion: A successful evaluation job produces evidence. Your release gate decides whether that evidence is acceptable.

Coverage versus run cost: Larger sets improve coverage but consume time and inference resources. Use a focused development set during diagnosis and the agreed broader set before release, rather than deleting difficult cases to shorten the gate.

Architecture / decision scenario

A support assistant has a stale refund policy, a chunk missing an exception, and a template branch producing an obsolete field. The proposed repair updates ingestion, rebuilds the data source, and changes prompt assembly.

The team preserves three incident cases with distinct checks: the retrieved revision must be approved, the exception must appear in supporting evidence, and the final object must pass the consumer schema. It then runs normal refund questions, denied-access requests, unanswered questions, and long-input cases through both complete application configurations.

Separate judge jobs evaluate captured baseline and candidate responses on aligned records. The candidate improves average answer quality but one access-restricted case returns evidence it should never receive. The permission assertion blocks promotion despite the score increase. After repairing that boundary, the team repeats the affected cases and the agreed regression set, verifies representative timing, and records the release decision against the exact tested configuration.

Common exam traps

• Declaring success after one corrected incident response.
• Testing a direct model call when the fix changes application orchestration.
• Treating a reference answer as a mandatory byte-for-byte response.
• Mixing baseline and candidate responses into an unsupported judge-job layout.
• Assuming retrieve-only referenceResponses labels expected chunks.
• Allowing improved average quality to offset authorization or schema failures.

What to remember

• Confirm the baseline reproduction before attributing success to a fix.
• Test the changed path and its neighboring failure boundaries.
• Use Bedrock evaluation as evidence alongside application-owned assertions.
• Inspect aligned per-case and category results before promotion.
• Release the configuration that passed the agreed gates, with traceable evidence.

Knowledge check

References

• AWS, Create a prompt dataset for a model evaluation job that uses a model as judge
• AWS, Create a prompt dataset for retrieve-only RAG evaluation jobs
• AWS, CreateEvaluationJob
• AWS, GetEvaluationJob
• AWS, Review model evaluation job reports and metrics in Amazon Bedrock