Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-105.1 Implement evaluation systems for GenAI

Report comparisons with decision-ready evidence

Completed by the knowledge check

Learning objective

Report experiment outcomes and model comparisons with stakeholder-facing dashboards. You need a report that lets a decision owner understand which candidate qualifies, what evidence supports the comparison, and which uncertainties remain.

Core idea

A stakeholder dashboard is a decision surface. It should connect a declared question, a controlled comparison, observed outcomes, and an accountable next action. A collection of charts cannot establish that two model scores are comparable or that the higher score satisfies the product’s constraints.

Amazon Bedrock judge-based evaluation supplies scores and explanations in the console and a full report in S3. Amazon CloudWatch supplies dashboard widgets and metric publication APIs. Your reporting pipeline joins that evidence with experiment identity, denominators, operational measurements, and acceptance policy. AWS does not choose the business winner from a dashboard’s appearance.

See 5.1.3 for model selection, 5.1.5 for controlled experiments, and 4.3.1 for operational telemetry. This lesson translates their evidence into a report that stakeholders can use.

How it works

Preserve the comparison contract

Start the dashboard with a text summary naming the decision: retain the baseline, promote a candidate, gather additional evidence, or reject both. Include the candidate and baseline model identities, prompt and application revisions, dataset revision, evaluation method, evaluator identity, test period, and release gates. These are application-owned experiment records, not metadata inferred from the plotted score.

Distinguish controlled benchmark results from live cohort results. A benchmark compares behavior on fixed cases; a production experiment measures outcomes under its actual assignment and traffic conditions. Display the experiment type and sampling contract so a stakeholder does not interpret a selected test dataset as a causal estimate for all production users.

Keep incomparable runs visibly separate. Changing the judge rubric, dataset composition, or score scale changes the measurement. If a comparison must cross that boundary, rerun the baseline under the candidate’s measurement contract or state that the results do not establish a paired improvement.

Convert evidence into interpretable measures

For judge-based model evaluation, Bedrock’s console summary displays a histogram of scores and explanations for the first five prompts in the dataset. The full report is available in the S3 bucket selected for the job. Use the full report for stakeholder aggregation; those five examples are not a representative sample or the entire evaluation.

Create an application-owned summary that distinguishes expected cases, completed assessments, failed assessments, accepted outputs, and critical failures. Retain references to protected detailed evidence. Describe every rate’s denominator. An acceptance rate among scored outputs differs from an acceptance rate among all attempted cases, especially if scoring failures are concentrated in a particular slice.

CloudWatch PutMetricData publishes custom metrics through Namespace and MetricData. Each datum can include MetricName, Value, Unit, Timestamp, and Dimensions. Use an application namespace rather than an AWS/ prefix. Names such as EvaluationAccepted, EvaluationScored, and EvaluationCriticalFailures are examples you define, not built-in Bedrock metrics.

Publish numerator and denominator counts with consistent dimensions and time boundaries. A CloudWatch metric widget can use math expressions to display a rate. Do not average percentages from unequal cohorts and present that as the combined rate; aggregate the corresponding counts first. Preserve zero or absent denominators as unavailable evidence rather than inventing a perfect result.

Use bounded dimensions for model candidate, experiment, and risk slice where needed. Keep per-request identifiers and sensitive prompts in protected detailed records, not as the public vocabulary of a stakeholder chart. A metric should summarize a defined population, while an evidence reference lets an authorized reviewer inspect individual failures.

Build the dashboard around the decision

CloudWatch PutDashboard accepts DashboardName and a JSON DashboardBody containing widget definitions. The dashboard body includes text, metric, and log widgets. Text widgets use properties.markdown; metric widgets display measurements and expressions; log widgets use a CloudWatch Logs Insights query for tabular or other supported views.

Arrange the report in the order the owner makes the decision. Put the experiment question and gate verdict first. Follow with comparable candidate results, critical slices, sample sizes, latency and usage outcomes, and links to protected failure evidence. A small business table often communicates the decision more clearly than several unrelated time series.

Keep quality, latency, token use, and business completion as distinct columns or panels. State which constraints are hard gates and which are trade-offs among eligible candidates. Mark illustrative cost calculations separately from billed spend; a token-count difference is not automatically an equivalent difference in total application cost.

PutDashboard replaces the entire existing dashboard body when updating it. Maintain the intended body as a versioned reporting artifact and inspect DashboardValidationMessages after publication. An empty result indicates successful validation; warnings can mean that some elements do not render even though the dashboard was created. A successful API response alone does not prove that the owner can see a usable report.

Design access with the evidence

Choose dashboard access according to the audience. Email sharing covers a single CloudWatch dashboard with up to five email addresses; each recipient sets a password. Public sharing covers a single dashboard and grants access to anyone with its link. Third-party single sign-on (SSO), integrated with Amazon Cognito through a SAML provider, shares all CloudWatch dashboards in the account; it cannot restrict the stakeholder to one experiment dashboard. Public sharing is an access decision, not a convenient report export. AWS documents that shared-dashboard permissions for cloudwatch:GetMetricData and ec2:DescribeTags cannot be scoped to particular metrics or instances; viewers can query all account metrics and EC2 names and tags allowed by that sharing role.

CloudWatch Logs Insights widgets are not visible to shared-dashboard viewers by default. Additional IAM permissions are needed to expose them. For a report that should show only sanitized aggregates, keep detailed logs behind an independently reviewed access path. Verify the intended stakeholder’s view, including evidence links, rather than reviewing only as the dashboard administrator.

Key distinctions and trade-offs

Summary versus supporting evidence: The dashboard states the verdict and its conditions. Detailed reports retain case-level evidence. A chart without a recoverable denominator or evaluation identity is insufficient for review.

Average quality versus critical-slice acceptance: A broad improvement can coexist with a release-blocking defect. Display critical gates separately so a strong average cannot visually hide them.

Operational trend versus experiment outcome: A time-series change can reflect traffic composition or sampling. A controlled comparison needs its own cohort and measurement contract.

Accessibility versus disclosure: Broader access helps decision owners, but shared-dashboard roles can expose more account telemetry than the visible widgets suggest. Report access and evidence access require separate review.

Architecture / decision scenario

A product owner compares two support models. Candidate B scores better overall and uses fewer tokens, but its high-risk refund slice contains critical failures. Candidate A meets the refund gate and the response-time objective.

The reporting pipeline reads complete Bedrock S3 reports, publishes application counts through PutMetricData, and builds a CloudWatch dashboard with a text verdict, metric comparisons, and a protected failure-evidence reference. The top panel recommends A and explains that B is ineligible despite its aggregate gain. It also shows how many cases were scored and which assessments failed.

The owner uses the sanitized stakeholder view. Detailed prompt and response logs remain restricted, and the team checks sharing-role permissions before providing access. When the evaluator changes, the dashboard labels the measurement boundary and reruns the baseline rather than plotting both score series as a continuous improvement trend.

Common exam traps

  • Selecting the model with the highest aggregate score despite a failed gate.
  • Treating the first five console explanations as the full report.
  • Averaging cohort percentages without preserving denominators.
  • Interpreting token savings as total billed cost savings.
  • Ignoring dashboard validation warnings or unreadable evidence links.
  • Assuming a shared dashboard exposes only its visible widgets.

What to remember

A useful report names the decision, comparison contract, sample sizes, quality gates, and trade-offs. Use full evaluation evidence, publish interpretable custom metrics, and build a validated CloudWatch dashboard. Match access to the audience and keep the acceptance verdict separate from favorable aggregate scores.

Knowledge check

References