Test prompts against explicit acceptance contracts

Learning objective

Test prompt templates with expected output, boundary inputs, adversarial cases, and regression suites.

Core idea

A successful example proves that one invocation produced an acceptable response. A prompt test suite asks whether a specific template and its surrounding controls repeatedly satisfy the application contract. For an architect, the decision is which failures block release and which quality changes justify further investigation.

Define expected behavior before generating responses. An expected output can be an exact label, a required fact, a schema, a supported conclusion, or a correct refusal. Free-form text usually needs a rubric rather than byte-for-byte equality. Keep hard requirements separate from graded preferences: a fluent response that exposes unauthorized information must not pass because its writing score is high.

Test the complete invocation configuration. The template, variable values, model, inference configuration, context, and enforcement controls jointly determine the result. Reuse the versioning and release mechanisms in 1.6.2 and 1.6.3; this Bite defines the evidence those releases need.

How it works

Establish a reproducible invocation

Amazon Bedrock Prompt management provides an interactive test window. You enter values in Test variables and choose Run, but those test values are temporary and are not saved with the prompt. Store the test cases independently so that another engineer can reproduce them.

For automated invocation, Converse accepts a managed prompt ARN in modelId and variable substitutions in promptVariables. Record the version-qualified ARN with each run. When using a managed prompt with Converse, you cannot supply additionalModelRequestFields, inferenceConfig, system, or toolConfig in the request. Those restrictions matter to test parity: a harness must exercise the saved configuration rather than silently use a different configuration. The request can append messages after the template’s messages, so record that additional context too.

Separate four case families

Use explicit expectations for each family:

An omitted required variable and an ambiguous business request are different boundaries. The former tests invocation or input validation; the latter tests model behavior. A transport failure is also distinct from an appropriate refusal. Store the observed category so the suite does not treat all missing answers as equivalent.

For length boundaries, use the model-specific token counting and context budgeting covered in 1.3.9. Do not use character count as a substitute for tokens. Include inputs that approach the permitted budget and cases your application rejects before inference; test both the model path and the rejection path.

Adversarial cases should include malicious text inside otherwise relevant evidence, requests to disclose private instructions, and attempts to force prohibited output through an unusual format. Evaluate the complete safety path from 1.6.1. A prompt’s apparent resistance is not proof that independently enforced controls work, and a blocked response does not prove the prompt interpreted the task correctly.

Match scoring to the requirement

Use deterministic checks for machine-readable contracts: parse the response, verify required fields, compare labels, and validate business invariants. Bedrock structured outputs can constrain supported models through outputConfig.textFormat in Converse; that controls response shape, not factual correctness. Continue checking whether the values are supported and whether the correct action was selected.

For semantic quality, use reviewed rubrics and representative references. An Amazon Bedrock model evaluation job using a judge model accepts a custom JSONL prompt dataset in S3, with up to 1,000 prompts per job. Each line supplies prompt, an optional referenceResponse used when calculating Builtin.Correctness and Builtin.Completeness, and an optional category that produces evaluation scores reported by category. Assign categories to the expected-output, boundary, adversarial, and regression families so their scores remain separately visible.

The optimizer’s evaluator, see 1.6.7, is not your independent release suite. Keep security failures and application-level assertions visible even when an evaluation score improves; the release-blocking policy belongs to your application.

Preserve an independent regression suite

Separate examples used to refine a prompt from examples used to decide release. AWS explicitly recommends held-out data after optimization. Otherwise, repeated edits can fit the visible cases while degrading unseen requests.

Retain the baseline outputs, candidate outputs, case identifiers, grader configuration, and failure reasons. Repeat selected cases to expose variability instead of treating a single favorable generation as stable behavior. Compare by case family and risk class, not only by the overall average. A rare authorization failure can be hidden by hundreds of improved ordinary answers.

Key distinctions and trade-offs

Exact output versus acceptable output: Exact matching suits labels and deterministic fields. A rubric suits paraphrases and summaries, but introduces judgment variability. Test the grader on obviously good and bad outputs before trusting its scores.

Boundary handling versus robustness: Correctly rejecting an invalid payload demonstrates validation. Correctly handling an awkward but valid request demonstrates prompt behavior. Require both where the application accepts both classes.

Optimization evaluation versus release evidence: An evaluator guides prompt improvement. A regression suite decides whether to ship that improvement. Independent data and hard failure gates preserve that distinction.

A large suite costs more to run, while a small suite leaves important behavior unobserved. Maintain a compact release-blocking set for consequential failures and a broader quality set for representative traffic. These are application policies, not automatic Prompt management features.

Architecture / decision scenario

A support assistant returns a product category and a short explanation. The team saves its baseline prompt version and runs approved questions, empty descriptions, conflicting product identifiers, and documents containing hostile instructions. Historical cases include a misclassified accessory and an explanation that invented warranty coverage.

The harness invokes the managed prompt with Converse, its version ARN, and recorded promptVariables. Deterministic checks score the category and response contract; a reviewed rubric scores explanation accuracy. The candidate improves ordinary summaries but invents warranty terms for a boundary case. The team rejects the release despite the improved average.

After correcting the instruction, the team reruns the held-out suite and repeated adversarial cases. Only a candidate that passes the hard requirements and the agreed quality comparison enters the approval process in 1.6.3. Every discovered failure becomes a named regression case with an expected outcome.

Common exam traps

• Treating temporary console test variables as a persisted test dataset.
• Testing a draft while production invokes a different prompt version.
• Requiring exact wording when the contract permits equivalent answers.
• Accepting a higher average score despite a hard safety failure.
• Treating schema compliance as proof of correct facts.
• Evaluating only the examples used to optimize the prompt.

What to remember

• Define expected behavior before inspecting generated responses.
• Cover expected, boundary, adversarial, and historical failure cases separately.
• Invoke the same saved configuration that the application uses.
• Combine deterministic assertions with calibrated semantic grading.
• Use held-out regression evidence and explicit release-blocking failures.

Knowledge check

References

• AWS, Test a prompt using Prompt management
• AWS, Converse
• AWS, Create a prompt dataset for a model evaluation job that uses a model as judge
• AWS, Define evaluation methods
• AWS, Structured outputs