AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications

Separate operational alerts from billing anomalies

Completed by the knowledge check

Learning objective

Detect token bursts and quality drift with CloudWatch and cost spikes with AWS Cost Anomaly Detection.

Core idea

Token consumption, response quality, and billed spend need different detectors. A token burst can require an immediate operational investigation. Quality drift requires evaluated responses with a stable measurement contract. A billing anomaly appears only after cost data is processed.

Use CloudWatch for service and application metric signals, and AWS Cost Anomaly Detection for anomalous spend patterns. Connect their investigations through the workload, account, Region, release, and time window. Do not wait for a billing alert to contain an ongoing retry loop, and do not claim that an operational token alarm measures actual billed cost.

An anomaly means behavior differs from an expected pattern. It does not by itself establish a cause, a policy violation, or a safe remediation action.

How it works

Detect token bursts in the operational series

Track AWS/Bedrock InputTokenCount and OutputTokenCount using period sums, alongside accepted Invocations and InvocationThrottles. A burst can come from more requests, larger contexts, longer answers, or repeated attempts. Compare total volume with per-request application measurements before classifying the cause; see 4.3.3.

CloudWatch anomaly detection creates an expected band from a metric’s history. It supports AWS-vended metrics and custom metrics, so you can apply it to a Bedrock token series or a bounded application usage series. The model considers trends and hourly, daily, and weekly patterns.

The detector is specific to a metric and statistic. A token Sum detector measures volume patterns; an average per-attempt series measures request size. They are not interchangeable. Select the statistic from the failure mode you want to detect.

Create the detector with PutAnomalyDetector. Use ANOMALY_DETECTION_BAND to represent the expected range in metric math. PutMetricAlarm uses ThresholdMetricId to identify that band expression, and ComparisonOperator: GreaterThanUpperThreshold detects upper-band breaches. Configure EvaluationPeriods and DatapointsToAlarm for the desired persistence, rather than treating every isolated spike as an incident.

Keep a static policy threshold beside the learned band when the workload has an absolute limit that must not be exceeded. A predictable daily token surge might fall inside the learned band while still exceeding an approved consumption policy. Learning normal behavior does not enforce an organizational ceiling.

Detect drift in measured quality

Publish the quality metrics from 4.3.4 before trying to detect their drift. Use a stable rubric, judge, cohort, and sampling method. Monitor hallucination-response counts together with evaluated-response counts, and response-quality scores together with coverage.

A lower-band anomaly on a quality score can identify a departure from the cohort’s expected quality. An upper-band anomaly on a measured hallucination rate can identify worsening outcomes. CloudWatch supports anomaly detection on supported metric-math expressions, including basic arithmetic, so count-derived rates can be candidates for a detector.

Protect the denominator. A rate based on one evaluated response should not carry the same confidence as a rate based on hundreds. Apply a minimum-sample decision in the evaluation pipeline or investigation workflow and keep coverage visible. When the evaluation pipeline stops, publish its health signal; do not replace missing quality data with zero errors.

Use TreatMissingData deliberately. PutMetricAlarm supports breaching, notBreaching, ignore, and missing. A sparse quality stream and a continuously emitted evaluator-health stream need different treatment. Setting all missing quality data to nonbreaching can conceal evaluator failure.

CloudWatch anomaly detection trains on up to two weeks of history, but it can be enabled without a full two weeks. Use reviewed static thresholds while the baseline is weak. Exclude known unusual periods from training where appropriate; do not automatically exclude every release regression and thereby erase the evidence of change.

A learned baseline also adapts over time. Keep fixed release comparisons and reviewed acceptance thresholds so gradual deterioration does not become acceptable merely because it becomes normal.

Detect spend anomalies after billing processing

AWS Cost Anomaly Detection uses machine learning to detect unusual spend patterns in net unblended cost, after applicable discounts. It uses Cost Explorer data and runs approximately three times a day after billing data is processed.

Cost Explorer data can lag by up to 24 hours, so detecting an anomaly after usage can take up to 24 hours. A monitor can take 24 hours to begin detection after creation. After you subscribe to a service, the detector needs 10 days of historical service usage before anomalies can be detected for that service. These limits make the feature unsuitable as a per-request cost circuit breaker.

Configure a cost monitor to define the spending population: AWS services, linked accounts, cost allocation tags, or cost categories. Create monitors for linked accounts, cost allocation tags, and cost categories in the management account. Attach an alert subscription to define recipients, frequency, and impact thresholds. The monitor detects anomalies; the subscription decides which detections generate notifications.

Individual alerts require an Amazon SNS topic and notify when an anomaly is detected. Daily and weekly summaries use email. An individual alert is faster than a summary after detection, but it does not remove the billing-data delay.

Subscriptions support absolute cost-impact thresholds and percentage-impact thresholds, which can be combined with AND or OR. A relative increase can be large on a tiny baseline; an absolute increase can be significant even when the percentage is modest. Select both according to the financial investigation policy.

Detected anomalies below a subscription threshold still appear in the detected-anomalies view. Raising the threshold suppresses notifications, not detection. For investigation, AWS identifies potential root causes by service, account, Region, and usage type, ranked by dollar impact. A subscription decides only which detections notify; every detection still appears in the detected-anomalies view.

Cost Anomaly Detection excludes most third-party AWS Marketplace products and services, but AWS documents an exception for third-party foundation models on Amazon Bedrock. Do not apply a blanket Marketplace exclusion to those model charges.

Key distinctions and trade-offs

Token anomaly versus spend anomaly: Tokens reveal operational consumption. Billing data reflects the financial accounting boundary. Cache behavior, model selection, and purchase arrangements prevent a token count from being an exact spend ledger.

Quality drift versus traffic drift: Changed traffic can alter the aggregate score without changing performance within a fixed cohort. Investigate cohort composition before blaming a prompt release.

Detection versus notification versus response: A detector produces evidence, a threshold controls alerting, and a runbook determines action. Configure each deliberately.

More sensitive bands catch smaller deviations but produce more investigations. Longer persistence reduces transient noise but delays detection. Calibrate those choices with observed incidents and the time available to contain damage.

Architecture / decision scenario

A support assistant develops an orchestration loop after a release. The CloudWatch token Sum series breaches its upper band, while application attempt counts rise faster than ingress requests. The operational team opens correlated examples and applies the agent-loop bounds from 2.1.1 and execution envelope from 2.1.7.

The quality stream also shows a higher reviewed hallucination rate with adequate evaluation coverage. The team compares the affected release cohort against its fixed acceptance threshold and restores the approved configuration through its release procedure. It does not treat low coverage as improved quality.

After billing data arrives, a Cost Anomaly Detection individual SNS alert identifies elevated spend for the relevant service, account, Region, and usage type. Finance correlates that interval with the operational incident and reviews the impact. The billing alert confirms a financial investigation; it did not provide the timely containment signal.

Common exam traps

Waiting for Cost Anomaly Detection to stop a live token burst.
Treating a learned normal band as an absolute safety or spending limit.
Detecting quality drift without a stable rubric and evaluation denominator.
Assuming every missing quality point means no hallucinations.
Treating alert subscription thresholds as controls over anomaly detection itself.
Assuming all AWS Marketplace charges are excluded, including Bedrock foundation models.

What to remember

Use CloudWatch for operational token and measured-quality signals.
Match the detector to the metric, statistic, cohort, and failure direction.
Preserve static acceptance thresholds and evaluation-health evidence.
Use cost monitors and subscriptions for delayed billing investigations.
Correlate alerts with request evidence before selecting remediation.