AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications

Publish quality scores with explicit denominators

Completed by the knowledge check

Learning objective

Publish measured prompt effectiveness, hallucination rates, and response-quality scores as custom CloudWatch metrics.

Core idea

CloudWatch stores the numbers you publish; it does not determine whether a prompt is effective or an answer is correct. A useful quality metric requires a measurement contract before a publishing API call.

Define the evaluated population, rubric, evaluator version, scoring scale, denominator, and observation time. Then publish numerical summaries with bounded dimensions. Keep detailed evidence in protected records correlated to the request, not in metric labels.

Three distinct measurements matter: whether the prompt achieves its intended task, whether the answer introduces unsupported claims, and how well the answer satisfies the broader response-quality rubric. Do not collapse them into one score whose meaning changes between releases.

How it works

Define prompt effectiveness against a task

Prompt effectiveness asks whether the prompt causes the required behavior on representative inputs. An application might require a correct decision, a complete answer, and compliance with output instructions. Define acceptance from those requirements rather than response fluency.

Amazon Bedrock judge-based model evaluation provides Builtin.Correctness, Builtin.Completeness, Builtin.FollowingInstructions, and other named metrics. Correctness and completeness can use reference responses supplied in the prompt dataset. These are evaluator measurements, not automatically published CloudWatch application metrics.

Choose which measurements determine your acceptance label. For example, a support-answer rubric might require correct policy selection and complete handling of the customer’s question. Record the prompt version, model configuration, evaluator identity, and rubric version for each evaluation. An apparent prompt improvement is uninterpretable if the judge or reference answers changed at the same time.

Publish proposed application counters such as PromptAccepted and PromptEvaluated through PutMetricData. Their period sums let you calculate accepted divided by evaluated for a fixed cohort. Keep this evaluated acceptance rate distinct from business completion: a good draft can still fail downstream delivery.

Measure hallucinations with an evidence boundary

Define a hallucination label in terms the reviewers can apply: for example, a response containing at least one consequential claim unsupported by the approved evidence. State whether the denominator is responses or claims. Mixing those units makes a rate impossible to interpret.

Bedrock’s Builtin.Faithfulness identifies information in a response that is not present in the prompt context. It is useful evidence about source adherence, but it is not the same as factual correctness in the world. A response can faithfully repeat an incorrect source.

Bedrock Guardrails contextual grounding checks assess grounding against a supplied source and relevance to a supplied query. The documented supported tasks include summarization, paraphrasing, and question answering; conversational question-answering and chatbot use cases are not supported by that check. Use the mechanism only within its documented task boundary.

For a supported check, ApplyGuardrail returns assessments[].contextualGroundingPolicy.filters[] with type, score, threshold, detected, and action. Preserve the check configuration alongside its result. See 3.1.6 for evidence verification and 3.4.3 for calibrated uncertainty signals.

A guardrail detection is a detector outcome. It does not become a reviewed hallucination label simply because a threshold was crossed. Publish detector measurements under separate application names, such as GroundingDetected, and reserve HallucinationResponses for your defined evaluation label. Track evaluator failures and unsupported cases separately rather than assigning them zero hallucinations.

Publish response quality without hiding its scale

Response quality can include correctness, completeness, helpfulness, relevance, and instruction adherence. Bedrock provides named judge metrics including Builtin.Helpfulness and Builtin.Relevance. Choose a rubric that fits the task, and keep the individual components visible even when a composite helps operations.

If your application normalizes a composite to a zero-to-one scale, document that transformation as your own contract. Do not imply that every Bedrock evaluation metric shares that scale. Validate automated judgments against human-reviewed examples, especially near acceptance thresholds.

Publish a proposed ResponseQualityScore metric for individual evaluated responses and a QualityEvaluated count for coverage. The score’s average can summarize a cohort, but its sample count and distribution matter. A high average can conceal a small set of severe errors.

CloudWatch PutMetricData accepts Namespace and MetricData. Each data point can include MetricName, Dimensions, Timestamp, Unit, and Value. Choose an application namespace such as Company/GenAIQuality; AWS advises against custom namespaces beginning with AWS/.

You can also publish through the CloudWatch embedded metric format: CloudWatch extracts custom metrics asynchronously from structured events written to CloudWatch Logs. For a Lambda evaluator, this keeps scores beside diagnostic events. It requires logs:PutLogEvents, without requiring cloudwatch:PutMetricData. Extraction provides at-least-once delivery, so occasional duplicate metric values are possible.

Amazon Bedrock AgentCore Evaluations online evaluation automatically writes results as JSON log entries and publishes scores as CloudWatch metrics in Bedrock-AgentCore/Evaluations. Set outputConfig.cloudWatchConfig.metricsNamespace for an application namespace; it cannot begin with AWS/. These remain evaluator scores, not reviewed hallucination labels.

Use bounded dimensions such as environment, feature, prompt version, and rubric version. Put request IDs and source evidence in the correlated evaluation record instead. CloudWatch permits up to 30 dimensions per metric, but that allowance is not a reason to create a series for every customer or request.

Preserve aggregation and evaluation timing

For a hallucination rate, publish HallucinationResponses and ResponsesEvaluated with matching cohort dimensions and timestamps. Compute the ratio from period sums. Do not average precomputed percentages from unequal batches.

Suppose one batch has 1 hallucination among 10 responses (10%) and another has 45 among 90 (50%). Averaging the two percentages gives 30%; the combined population has 46 hallucinations among 100 responses, or 46%. Count-based aggregation preserves the weighting.

Use an explicit no-data state when no responses were evaluated. Zero means evaluation found no qualifying errors; missing means the measurement is unavailable. Publish separate coverage and evaluator-error counters to distinguish them.

Choose whether dashboards report evaluation time or original request time. Delayed reviews do not measure live quality immediately. PutMetricData accepts timestamps up to two weeks in the past and two hours in the future, and older submitted data can take longer to become queryable. Treat backfilled evaluations as retrospective evidence rather than immediate alarm input.

The API supports Values and Counts for up to 150 distinct values per metric per request, preserving repeated samples and percentile retrieval. Arbitrary statistic sets cannot generally recover percentiles. Choose the representation from the distribution you need to inspect.

Key distinctions and trade-offs

Prompt effectiveness versus response quality: Prompt effectiveness measures achievement of prompt-specific requirements. Response quality measures the chosen rubric for the resulting answer.

Grounding score versus hallucination rate: One is a model-generated evidence signal; the other is an aggregate of labels with a defined population and unit.

Score change versus measurement change: A judge, rubric, or sampling change can move a metric without changing application behavior. Version those contracts and compare like cohorts.

Production samples reflect real traffic but can miss rare failures. A fixed evaluation set improves release comparison but does not represent every production shift. Present both with clear population labels instead of merging them invisibly.

Architecture / decision scenario

A support team compares two prompt versions. It evaluates the same reviewed questions and publishes prompt acceptance counts, response-quality scores, hallucination-response counts, and evaluation coverage under each version and the same rubric.

One prompt has a higher average quality score but more unsupported policy claims. The hallucination metric prevents the team from treating the average as sufficient approval. A separate grounding-detector metric helps investigate those examples without substituting its threshold for the reviewed label.

When the evaluator fails on a subset, the team publishes evaluator errors and lower coverage. It does not count those responses as correct. After review, it backfills results with a documented time policy so the dashboard cannot imply that the evidence was available earlier.

Common exam traps

Expecting Bedrock runtime metrics to include business-defined answer quality.
Publishing a grounding score as a measured hallucination percentage.
Averaging rates from unequal evaluation batches.
Counting unevaluated responses as acceptable.
Comparing scores across changed rubrics or judges without labeling the change.
Using request IDs and raw prompts as metric dimensions.

What to remember

Define the population, rubric, scale, evaluator, and denominator first.
Publish custom metrics with PutMetricData in an application namespace.
Keep prompt acceptance, hallucinations, quality, and coverage distinguishable.
Calculate rates from matching numerator and denominator counts.
Preserve evidence and versioning so a score change can be investigated.