AWSAWS Certified Generative AI Developer - Professional2026-103.4 Implement responsible AI principles
Define fairness slices before choosing metrics
Completed by the knowledge check
Learning objective
Choose representative evaluation datasets, fairness slices, and defined bias metrics using SageMaker Clarify.
Core idea
A fairness result is meaningful only relative to a population, an outcome definition, a comparison, and a measure. A balanced test file does not automatically represent production, and a low value for one bias metric does not establish fairness under every definition.
For an architect, choose the harm you need to detect before choosing the metric. Then build a dataset that exposes that harm, with enough examples in the affected slices to support a decision. Clarify can calculate configured bias measures; your organization supplies the population assumptions, labels, slice choices, and acceptable boundaries.
AWS documentation states that Amazon SageMaker Clarify is no longer open to new customers. Existing customers can continue to use it, and AWS does not plan new features. The objective retains its Clarify wording. AWS documents a replacement path: reproduce the standardized bias formulas with pandas and scikit-learn and log results to SageMaker AI MLflow. Bedrock Evaluations and Guardrails replace Clarify’s foundation model evaluation (FMEval) capability, but do not replace predictive tabular/classical ML bias detection. See 3.4.5 for LLM evaluation.
How it works
Define the evaluated population
Begin with the production task: who asks for help, in which languages, on which topics, and under which conditions. Select examples reflecting that operating population, then deliberately include rare but consequential cases. Record which dataset approximates traffic and which dataset stresses a specific harm; they answer different questions.
For a GenAI assistant, possible outcomes include receiving a correct useful answer, experiencing an unjustified refusal, or receiving a stereotyped response. These are evaluation definitions, not automatically generated Clarify labels. Reviewers must apply a documented rubric to convert outputs into consistent outcome labels before a classification-based bias metric can measure them.
Keep reviewed reference outcomes separate from system-predicted or system-produced outcomes. If you label every released response as successful, the metric measures release behavior rather than useful assistance. That can hide a system that serves two groups equally often while giving one group worse answers.
Maintain a held-out dataset for release comparison. Do not tune a prompt repeatedly on the same examples and then report their improvement as independent evidence. Retain dataset version, selection method, label rubric, and exclusions with the result so reviewers can assess what population the conclusion covers.
Define facets and meaningful slices
Clarify calls the sensitive attribute used for a bias comparison a facet. Its analysis configuration uses a facet array, with name_or_index identifying the attribute and value_or_threshold defining the compared group. The configuration’s label_values_or_threshold identifies positive outcomes. These definitions determine how the counts are partitioned.
Choose positive to mean a beneficial outcome in the analysis, and document the choice. For a support assistant, that might mean a correct answer on an answerable request. For a safety evaluation, a successful refusal of prohibited content might be positive. Mixing those definitions in one pooled label makes the result difficult to interpret.
Do not assume fairness requires sending protected attributes to the model. AWS documents that Clarify can analyze facets even if the model was trained without sensitive attributes. For CSV analysis, facet_dataset_uri can identify a separate sensitive-attribute dataset; join it correctly to the evaluated records and restrict access to it.
Look beyond one attribute at a time when the risk warrants it. A language slice can hide a disparity concentrated in a particular topic or access condition. Build separate documented analyses for consequential intersections and report their sample sizes. Clarify’s group_variable serves the conditional demographic disparity metrics; it is not a general promise that one configuration discovers every meaningful intersection automatically.
Select the metric from the question
Clarify distinguishes pre-training data bias metrics from post-training data and model bias metrics. Pre-training metrics do not depend on model predictions. CI, Class Imbalance, measures unequal representation across facets. DPL, Difference in Proportions of Labels, measures differences in observed positive outcomes. These diagnose the dataset, not the quality of generated answers.
Post-training metrics use model outcomes to ask different questions. DPPL, Difference in Positive Proportions in Predicted Labels, compares predicted positive rates between facets. DI, Disparate Impact, compares those rates as a ratio. RD, Recall Difference, compares how effectively the model finds observed positives across the groups.
In the analysis configuration, select these under methods.pre_training_bias.methods or methods.post_training_bias.methods. Name the metric and its orientation in the release evidence. A difference and a ratio do not share a neutral value: equal positive rates produce a difference of zero and a ratio of one. Interpret results under the selected definition rather than applying one threshold to every metric.
For a GenAI workflow, first decide whether a reviewed binary or categorical outcome faithfully captures the harm. Clarify bias metrics are not an automatic semantic judge of free-form text. Stereotyping assessment and prompt comparison belong to 3.4.5; their outputs can support an outcome-labeling process, with human validation.
Key distinctions and trade-offs
Representation versus outcome disparity: CI can reveal missing examples. It cannot establish equal answer quality. Conversely, equal outcome rates on an unrepresentative dataset do not establish production fairness.
Observed versus predicted labels: Observed labels define reviewed reference outcomes. Predicted labels represent the evaluated system’s outcomes. Swapping them changes the meaning of the metric.
Metric neutrality versus acceptable risk: A neutral difference or ratio describes the selected measure. It does not certify the workload as fair or remove other harms.
Oversampling a rare slice improves diagnosis but changes aggregate proportions. Report slice results separately and explain weighting if you estimate production behavior. Fairness definitions can conflict, so approval requires stakeholder judgment about which harms take priority.
Architecture / decision scenario
An existing Clarify customer evaluates a support assistant serving two language groups. Production traffic is uneven, and answerable requests differ by topic.
The team builds a traffic-representative set plus a deliberately balanced diagnostic set. Reviewers label answerability and whether each answer provides correct useful help. Separate facet data associates the permitted language grouping with each evaluated record.
CI exposes representation imbalance in the traffic set. DPPL compares useful-answer rates, while RD checks whether genuinely answerable requests receive useful answers equally across groups. Topic-specific analyses show that the aggregate result hides a problem in billing questions. The team blocks release on that slice and records the dataset limitations rather than declaring fairness from the global average. A new customer can compute the same defined bias measures in an owned pandas/scikit-learn pipeline and retain the per-slice results in SageMaker AI MLflow, following AWS’s documented replacement path.
Common exam traps
- Assuming Clarify is available to new customers.
- Treating a pre-training metric as a measure of generated-answer quality.
- Measuring outcomes before defining positive labels and facet groups.
- Confusing a difference metric’s zero with a ratio metric’s one.
- Assuming removing protected attributes eliminates every disparity.
- Declaring fairness from an aggregate that hides small or underserved slices.
What to remember
- Define population, harm, labels, and facets before computing bias.
- Separate traffic-representative evaluation from targeted stress testing.
- Choose
CI,DPL,DPPL,DI, orRDfor the question each measures. - Validate GenAI outcome labels instead of treating text as ready-made classification evidence.
- Respect Clarify’s new-customer access restriction.
Knowledge check
Reviewers have labeled which support requests were genuinely answerable and which answers gave correct, useful help. The team asks: “Of the answerable requests, does each language group receive useful answers equally often?” Which Clarify metric answers that question?
CI , comparing how many records each group contributesDPL , comparing observed positive labels across groupsDPPL , comparing predicted positive rates across groupsRD , comparing recall for the observed positives in each groupCI and DPL are pre-training metrics about the dataset, and DPPL compares overall positive rates without conditioning on which requests were answerable.A team reuses one alert rule for all its bias metrics: flag any value more than 0.1 away from zero. Both groups receive useful answers at exactly the same rate. What happens with DI?
DI is a ratio of predicted positive proportions, so equal rates give 1, while a difference such as DPPL gives 0. Each metric needs a threshold defined around its own neutral value and orientation.To compare useful-answer rates across groups, a team labels every response the assistant released as a positive outcome. One group receives answers as often as the other but reports worse help. What does the metric actually measure?
A diagnostic set oversamples billing questions from a small language group so that its results support a decision. How should the team report results from this set?
A team that has never used SageMaker Clarify must compute DPPL and DI for a tabular eligibility classifier and keep the results under governance. Which path does AWS document?