Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-103.3 Implement AI governance and compliance mechanisms

Detect governance changes against explicit baselines

Completed by the knowledge check

Learning objective

Detect misuse, quality drift, bias drift, and policy violations against explicit baselines.

Core idea

A change in traffic, a decline in answer quality, a disparity between groups, and a policy violation are different findings. Give each a baseline, a measurement, and a decision owner. A single dashboard labeled model drift cannot tell you whether to investigate an attack, repair retrieval, review fairness, or stop an unauthorized behavior.

For Bedrock workloads, combine explicit safety assessments with model evaluation and application-defined metrics. AWS provides mechanisms to produce evidence, but your organization defines the comparison population, acceptable change, and response. Treat the baseline as a versioned contract rather than whatever production happened to do yesterday.

How it works

Define the comparison before collecting scores

A usable baseline identifies the model and application release, evaluation dataset, scorer or reviewer rubric, approved policy version, traffic population, and time window. Record the numerator and denominator of every rate, the minimum evidence required, and the response to missing observations. Without these details, a metric can change because its definition changed.

Keep two complementary comparisons. A fixed regression set detects changes on the same cases after a release. A reviewed production sample detects changes in actual use, including cases absent from that set. If the live population shifts, retain that fact in the finding instead of attributing every difference to model degradation.

Use application-defined identifiers for the baseline and release in your decision records. Publish numeric observations through CloudWatch PutMetricData, with Namespace, MetricData, and bounded Dimensions that separate the intended populations. Names such as policy-violation rate are your custom metrics, not built-in Bedrock metrics. Avoid dimensions containing personal data or a unique identifier for every request; retain request-level evidence in protected records instead.

Detect misuse through behavior and assessment evidence

Misuse asks whether callers are attempting prohibited behavior, not merely whether a model is producing low-quality answers. Evaluate representative attacks and legitimate requests against the approved guardrail version. ApplyGuardrail accepts guardrailIdentifier, guardrailVersion, source: INPUT | OUTPUT, and content. Its action is NONE or GUARDRAIL_INTERVENED, and assessments explain the policy findings.

Bedrock Guardrails publishes InvocationsIntervened and Invocations in AWS/Bedrock/Guardrails. The GuardrailContentSource dimension separates Input from Output; GuardrailPolicyType separates ContentPolicy, TopicPolicy, WordPolicy, SensitiveInformationPolicy, and ContextualGroundingPolicy, while GuardrailArn and GuardrailVersion identify the assessed policy version. Use input interventions as evidence of attempted misuse only after identifying the policy finding; intervention counts do not measure successful bypasses.

Measure attempted abuse separately from successful bypasses. An increase in input interventions can mean a control is stopping more attacks. It does not establish that harmful responses increased. Conversely, few interventions can mean quiet traffic, insufficient coverage, or a bypass. Review labeled samples of both accepted and intervened requests. See 3.1.10 for detector confusion matrices rather than rebuilding their rules here.

For detailed outputScope: FULL assessment behavior, see 3.1.10; retain only approved evidence fields because assessment detail can contain sensitive content.

Measure answer quality against task evidence

Bedrock model evaluation with an LLM as a judge scores responses and provides explanations. It can generate responses from a selected Bedrock model or evaluate inference responses you supply in the prompt dataset. The full report is delivered to the specified Amazon S3 bucket. This supports evaluation of captured application responses, provided you prepare an appropriate dataset and protect it.

Choose metrics that match the failure. Builtin.Correctness assesses correctness and can consider a reference response. Builtin.Completeness assesses whether all questions are answered. Builtin.Faithfulness checks for information absent from the prompt context. Builtin.Refusal identifies refusals, while Builtin.Harmfulness measures harmful content. A helpful response is not necessarily faithful, and a refusal is not necessarily an error.

Freeze the judge, rubric, and dataset when comparing releases. A change in evaluator can change scores without a change in the generator. Review consequential and disputed cases with human evidence. Judge-based evaluation produces assessment signals, not a guarantee of truth or a continuous monitoring schedule merely because an evaluation job exists.

Separate bias drift from aggregate quality

For generated language, Builtin.Stereotyping evaluates stereotypes in response content. Compare results on an explicitly reviewed set of relevant groups and contexts, and examine false refusals or unequal task success where those affect your use case. Stereotyping is one measure; it does not cover every definition of fairness.

For eligible existing SageMaker Model Monitor customers, SageMaker Clarify bias monitoring compares deployed predictions against chosen bias thresholds. AWS illustrates DPPL with an allowed range. That range is an example, not a universal approval threshold. The service is closed to new customers; see 3.3.7 for its supported workloads and availability restriction.

Keep population change separate from outcome change. If one group’s requests become harder or its reviewed sample becomes too small, report that limitation. Do not interpret an aggregate improvement as proof that every group improved, or silently drop a group whose evidence is incomplete.

Detect violations of the approved policy

A policy violation is behavior that breaks a named rule, even if it appears helpful. Compare Guardrails assessments to the approved policy and distinguish observed content from the action the application actually took. GUARDRAIL_INTERVENED indicates intervention, not that your application handled the returned content correctly.

Record whether the request was blocked, masked, withheld, or released, along with the rule and release version. Authorization or business-rule violations require their own application checks; a clean guardrail assessment does not prove those checks passed. See 3.2.2 for authorization boundaries and 3.1.9 for layered enforcement.

Use PutMetricAlarm to encode the chosen threshold, ComparisonOperator, EvaluationPeriods, and DatapointsToAlarm. Set TreatMissingData deliberately. Missing telemetry is not evidence of compliance: handle it as a distinct monitoring-health condition. The acceptable rate and minimum sample are organizational decisions, not defaults supplied by these APIs.

Key distinctions and trade-offs

Attempted misuse versus successful misuse: Input detection measures attempts. Reviewed released outputs and action records establish bypasses. Combining them hides whether the control worked.

Quality drift versus traffic drift: Changed outcomes on a fixed set suggest release regression. Changed outcomes on a changed population require investigation of both workload and release.

Bias signal versus fairness approval: Stereotyping and prediction-disparity metrics measure particular phenomena. Approval requires the relevant groups, context, and harm definition.

Violation versus detector intervention: A detector can prevent a violation. A clean detector response can also miss one. Preserve independent labels and the actual release decision.

Broad production sampling finds unfamiliar failures but increases privacy and review costs. Fixed sets are reproducible but incomplete. Use both, with enough protected evidence to explain a finding and without retaining unnecessary raw conversations.

Architecture / decision scenario

An employee assistant releases a prompt update. Its team retains a fixed evaluation set and reviews production samples against the same baseline contract. It tracks input interventions as attempted misuse, reviewed prohibited released answers as policy failures, Builtin.Faithfulness and Builtin.Correctness as quality signals, and Builtin.Stereotyping plus reviewed task-success differences as bias evidence.

After release, input interventions rise while reviewed bypasses remain stable. The security owner investigates the traffic rather than declaring a quality regression. Faithfulness falls on unchanged retrieval cases, so the release owner investigates the prompt. One group’s refusal rate rises despite stable aggregate quality, which triggers fairness review. A separate missing-telemetry condition prevents the dashboard from reporting a clean policy interval when the observation pipeline fails.

Common exam traps

  • Treating more blocked attacks as proof of more successful abuse.
  • Comparing scores after changing both the generator and evaluator.
  • Equating low latency or low error counts with correct answers.
  • Treating Builtin.Stereotyping as complete fairness certification.
  • Replacing an approved baseline with degraded production results.
  • Treating absent telemetry as zero policy violations.

What to remember

  • Define a separate baseline and decision for each governance concern.
  • Track misuse attempts, bypasses, and application release actions separately.
  • Freeze comparison datasets and scorer definitions to interpret drift.
  • Measure relevant groups rather than relying on aggregate quality.
  • Use explicit alarm settings and monitor the evidence pipeline itself.

Knowledge check

References