Compare prompt fairness with calibrated judges

Learning objective

Compare bias and fairness with Prompt Management/Prompt Flows A/B tests and LLM-as-a-judge calibrated against human ratings.

Core idea

A prompt change is an intervention, and a judge score is a measurement with its own failure modes. To compare fairness, keep the intervention reproducible, evaluate the same meaningful slices, and establish how the judge agrees with human reviewers before using it to make a release decision.

The objective uses “Prompt Flows.” AWS documentation names the workflow service Amazon Bedrock Flows and the reusable prompt feature Prompt management. Use those documented names. Their testing and versioning mechanisms provide experiment artifacts; the application or evaluation harness owns A/B assignment, sample accounting, and fairness acceptance criteria.

A better overall response score can coexist with worse outcomes for a smaller group. Compare both the aggregate and the slices that matter, and keep semantic harms separate from outcome-rate disparities.

How it works

Freeze the candidates

Prompt management supports testing prompts and creating a version as a snapshot for production use. For API testing, AWS documents Converse with the prompt ARN in modelId and variable values in promptVariables. Keep the version-qualified ARN and the supplied variable values in the experiment record. Prompt version mechanics are covered in 1.6.2. Prompt variants provide alternative message, model, or inference configurations. The console’s Compare variants mode lets you run up to three variants side by side with specified test variables; use the same values for a controlled comparison. This interactive comparison helps select candidates, while traffic assignment, slice accounting, and scoring remain harness responsibilities.

For a workflow comparison, use Bedrock Flows versions and aliases. InvokeFlow identifies the flow through flowIdentifier and its alias through flowAliasIdentifier. AWS documents setting enableTrace to TRUE to view node inputs and outputs. A trace can explain where candidates differ, but it does not itself grade fairness.

A working draft changes when you edit the flow. Do not compare an evolving draft against a frozen candidate and call the result a reproducible A/B test. Record the resolved flow version behind each alias. If a prompt, model configuration, retrieval source, or node changes, make that change part of a separately identified candidate.

Choose the comparison design

For an offline paired comparison, run the same evaluation case through both candidates. Keep all conditions you are not testing aligned: model configuration, source snapshot, tool behavior, and input variables. Generation can vary across runs, so use repeated samples where that variability could change the conclusion.

For a live A/B comparison, assign eligible traffic under an application-owned policy and preserve that assignment through the interaction. Account for both attempted and completed cases. If candidate B produces more errors and the evaluator sees only successful responses, the comparison favors B through selection bias.

Separate genuine population variation from experimental variation. A test that sends easy English questions to A and difficult multilingual questions to B cannot identify the prompt’s effect. The dataset and slice contract from 3.4.4 defines what must remain comparable. Record refusals, timeouts, incomplete outputs, and tool failures as explicit outcomes rather than removing them silently.

Use a judge for a defined semantic metric

Bedrock model evaluation with an LLM judge uses a generator model and an evaluator model. The evaluator scores responses for selected metrics and supplies explanations. AWS also supports evaluating supplied inference responses, in which case Bedrock skips generator invocation. That path is useful when evaluating outputs already produced by versioned prompt or flow candidates.

The custom dataset uses JSON Lines in Amazon S3. For supplied responses, the documented fields include prompt, optional referenceResponse, optional category, and modelResponses, whose entries identify modelIdentifier and response. Each job supports one response per prompt and one unique modelIdentifier. Evaluate candidates in separate jobs, give each a stable identifier, and join results through the harness’s paired case records and exact candidate configurations.

The documented limit is up to 1,000 prompts per evaluation job. If a larger study needs multiple jobs, preserve the slice membership and denominators when aggregating. Do not treat a small job’s absence of failures as proof that a rare harm cannot occur.

Builtin.Stereotyping evaluates whether response content contains positive or negative stereotypes. It is a semantic content measure, not a demographic acceptance-rate measure. category supports reported evaluation scores by category; choose categories that make the intended slice comparison visible. Use reviewed outcomes and the defined bias measures from 3.4.4 when the question is whether groups receive comparable useful assistance.

Calibrate against human ratings

For supported model comparisons, Bedrock model evaluation jobs that use human workers provide a managed way to collect ratings. Evaluators belong to a workteam, which you can create as an Amazon Cognito managed workteam in the Bedrock console. A job can rate or compare responses from up to two foundation models and requires at least one Bedrock foundation model. Use those ratings as calibration evidence; the slice-wise agreement analysis remains application-owned.

Select a calibration subset spanning candidates, slices, score ranges, and ambiguous examples. Ask qualified reviewers to apply the same rubric without candidate labels that could influence their ratings. Retain independent ratings and adjudicate material disagreements.

Compare judge ratings with the adjudicated human ratings by slice. Inspect false negatives on harmful content, false positives on benign identity mentions, and disagreement near the release boundary. An evaluator explanation can help find a rubric mismatch, but it does not validate its own score.

Set an application-owned acceptance policy from the error consequences. If the judge repeatedly misses a stereotype in one language, require human review for that slice or change the evaluation approach. Do not correct the reported result by pretending that the judge is calibrated globally when it works only for a subset.

Each candidate’s outputs are judged in a separate job, joined through the paired case records, and compared with adjudicated human ratings by slice.

Freeze the evaluator model, metric configuration, and rubric with the study. After changing any of them, recalibrate against retained human-rated cases before comparing scores with an earlier release. The numeric scale alone does not establish that two evaluator configurations measure the same thing.

Key distinctions and trade-offs

Prompt comparison versus workflow comparison: Prompt candidates isolate instruction changes more closely. Flow candidates can change retrieval, tool use, or branching, so conclusions must cover the entire changed path.

Offline pairing versus live A/B: Pairing controls case differences and supports detailed review. Live assignment measures behavior under real traffic but needs assignment integrity and full outcome accounting.

Stereotyping versus outcome fairness: Content can avoid stereotypes while still giving one group worse help. Equal answer rates can also hide stereotyped wording. Measure each relevant harm separately.

LLM judges scale semantic review but introduce evaluator errors and rubric sensitivity. Human review is slower but provides independent evidence for calibration. Combine broad judged coverage with targeted human review instead of treating either one score or one reviewer as infallible.

Architecture / decision scenario

A support team compares two prompt versions embedded in otherwise equivalent flow versions. Candidate B has a better aggregate quality score.

The harness runs paired cases across language and topic slices, records the prompt ARNs and resolved flow versions, and supplies the captured outputs to Bedrock judge evaluation. Builtin.Stereotyping appears acceptable overall, but human calibration reveals missed stereotypes in one language. Reviewed useful-answer labels also show worse billing outcomes in that slice.

The team does not promote B. It fixes the prompt, retains the failed cases for regression testing, and repeats the comparison with the frozen evaluator configuration. A later live A/B test includes all attempts and failures, so improved fairness is not inferred from selectively completed outputs.

Common exam traps

• Assuming prompt or flow testing supplies experiment assignment automatically.
• Comparing mutable drafts without retaining candidate versions.
• Using an LLM judge without checking human agreement by slice.
• Treating Builtin.Stereotyping as demographic parity.
• Dropping failed generations from the experiment denominator.
• Comparing scores after changing the judge without recalibration.

What to remember

• Use versioned Prompt management and Bedrock Flows artifacts.
• Keep A/B assignment and outcome accounting explicit in the harness.
• Evaluate semantic harms and outcome disparities separately.
• Calibrate judge ratings against independent human evidence.
• Promote only when the candidate passes the agreed slice-level gates.

Knowledge check

References

• AWS, Test a prompt using Prompt management

• AWS, Test a flow in Amazon Bedrock

• AWS, Deploy a flow to your application using versions and aliases

• AWS, Evaluate model performance using another LLM as a judge

• AWS, Create a prompt dataset for a model evaluation job that uses a model as judge

• AWS, Use metrics to understand model performance

• AWS, Create a prompt using Prompt management

• AWS, Creating your first model evaluation that uses human workers