Check eligibility before choosing drift checks

Learning objective

Identify SageMaker Model Monitor drift checks for supported endpoints and its restriction for new AWS customers.

Core idea

SageMaker Model Monitor is a production monitoring mechanism with specific data and hosting boundaries. It is not a universal detector for hallucinations, unfairness, or changes in any foundation model response. First determine whether you can use the service and whether it can analyze your workload. Then choose the check whose evidence matches the suspected failure.

AWS states that Amazon SageMaker Model Monitor is no longer open to new customers. Existing customers can continue using it. The restriction concerns Model Monitor; do not extrapolate it to all SageMaker AI hosting or to every separate Clarify capability.

How it works

Check the workload boundary

Model Monitor supports monitoring with real-time endpoints and batch transform jobs, including regularly running batch jobs and scheduled monitoring of asynchronous batch transform jobs. The latter wording describes batch transform scheduling; it does not establish support for the separate SageMaker Asynchronous Inference hosting mode.

For endpoints, AWS documents support for a single hosted model and excludes multi-model endpoints. Inference pipelines are supported, but capture and analysis cover the whole pipeline rather than individual containers. A team cannot use that pipeline-level report to claim it isolated drift inside one container without additional evidence.

The built-in metrics and statistics operate on tabular data. An image classifier can have its tabular label output monitored without having its image input analyzed by these checks. Similarly, raw conversations do not become semantic answer-quality measurements merely because you capture them. A supported endpoint topology and a supported analytical representation are separate prerequisites.

Model Monitor is not the monitoring API for Bedrock-managed inference. For Bedrock application quality and governance measurements, use the evaluation and application evidence described in 3.3.6. Do not select Model Monitor solely because the question mentions a foundation model.

Select among four checks

The four monitoring categories answer different questions:

Data quality does not need an actual correct answer for each prediction. Model quality does. Bias and feature attribution checks measure other aspects of model behavior; neither is a synonym for predictive accuracy. A complete governance plan can require several checks because no one category subsumes the others.

Make the baseline and schedule explicit

Enable data capture so inference inputs and outputs are stored in Amazon S3, then establish the baseline appropriate to the check. For data quality, DefaultModelMonitor.suggest_baseline runs a baseline processing job using the provided dataset. The job writes statistics.json and constraints.json to the selected output_s3_uri.

Statistics describe the baseline population. Constraints define the checks applied to later data. Review suggested constraints before treating them as production policy: an unusual training sample can suggest a threshold that your organization should not accept. Preserve the approved baseline alongside the model release so a finding remains interpretable after a later release changes the expected distribution.

A monitoring schedule defines collection and analysis rather than making each inference request wait for a verdict. A downstream report can reveal drift after affected predictions have already been returned. Use request-time enforcement for decisions that cannot tolerate that delay.

The data-quality violations report contains violations, including feature_name, constraint_check_type, and description. Documented types include data_type_check, completeness_check, missing_column_check, extra_column_check, categorical_values_check, and baseline_drift_check. The distribution check compares distance against monitoring_config.comparison_threshold; a distribution violation is evidence to investigate, not proof that the model is inaccurate.

Supply the evidence each check requires

Model quality monitoring joins captured predictions with actual labels stored in S3. It computes metrics according to the machine learning problem type; AWS gives mean square error (mse) as a regression example. Plan for delayed labels and their association with predictions. Without the actual labels, you have captured outputs, not a measured accuracy comparison.

SageMaker Clarify bias monitoring can evaluate a selected metric against an allowed range. AWS illustrates DPPL with a range of (-0.1, 0.1). Those numbers illustrate configuration; they do not prescribe acceptable bias for every use case. Your baseline must specify the relevant groups, metric, and risk-based threshold.

Feature attribution monitoring compares how feature contributions change. AWS describes Normalized Discounted Cumulative Gain (NDCG) for comparing feature-attribution rankings while accounting for attribution importance. A changed ranking warrants investigation of the data and model behavior. A stable ranking does not prove that predictions are correct, equitable, or policy-compliant.

Check that observations remain available

Data Capture stops capturing requests at high disk usage to avoid affecting inference. AWS recommends keeping disk utilization below 75% so capture continues. This is an operational recommendation, not an assurance that every request below that level is captured.

DescribeMonitoringSchedule returns MonitoringScheduleStatus and LastMonitoringExecutionSummary. The summary’s MonitoringExecutionStatus is Pending | Completed | CompletedWithViolations | InProgress | Failed | Stopping | Stopped. Distinguish Failed, an unsuccessful execution, from CompletedWithViolations, a completed check that found violations; neither means a clean result. Check execution and capture health before interpreting drift, then route the evidence to the owner for investigation.

Key distinctions and trade-offs

Eligibility versus capability: Existing-customer eligibility does not make an unsupported endpoint or non-tabular input supported. An eligible team must still check topology and data.

Distribution drift versus performance drift: A feature distribution can change while predictions remain useful. Predictive accuracy can deteriorate without a conspicuous input-distribution change. Use the check that directly measures the claim.

Bias versus explanation: Bias monitoring measures a chosen disparity. Feature attribution describes contributing features. An explanation signal cannot establish fairness by itself.

Scheduled detection versus inline enforcement: Model Monitor analyzes captured data. It does not replace an authorization or safety gate before releasing an individual prediction.

Existing customers can retain an established monitoring investment, subject to its boundaries. For customers without access, AWS’s availability-change page describes open-source SageMaker AI monitoring solutions with CloudWatch and governance dashboards. Treat those as solutions you must deploy and operate, not as an automatic continuation of the managed Model Monitor configuration.

Architecture / decision scenario

An existing Model Monitor customer runs a single-model real-time eligibility classifier. A source change introduces missing fields and changes one group’s input distribution. Data-quality monitoring flags completeness and baseline drift. Delayed outcome labels then reveal weaker predictive performance, while the approved bias metric also leaves its allowed range. Feature attribution shows that a formerly minor feature has become more influential.

The owner preserves all four findings instead of treating the first alert as the full diagnosis. The team repairs the source transformation, checks the affected predictions, and re-evaluates the release before changing any baseline. It also verifies that disk pressure did not interrupt capture during the incident.

A second team has no Model Monitor access and proposes a multi-model endpoint serving free-text answers. It rejects this monitoring choice on customer eligibility, endpoint topology, and analytical representation. It implements application-level answer evaluation and selects an operated monitoring solution rather than promising that tabular drift checks detect hallucinations.

Common exam traps

• Selecting Model Monitor for a new customer without checking its restriction.
• Assuming multi-model endpoints are supported because single-model endpoints are.
• Treating captured free text as a built-in semantic correctness check.
• Calling input distribution drift proof of reduced predictive accuracy.
• Measuring model quality without supplying actual labels.
• Treating a scheduled finding as a pre-release safety decision.

What to remember

• Model Monitor remains usable by existing customers and is closed to new customers.
• Check endpoint topology and tabular representation independently.
• Data quality, model quality, bias, and feature attribution answer different questions.
• Preserve baseline statistics, constraints, labels, and monitoring reports.
• Monitor capture health and investigate violations before changing the baseline.

Knowledge check

References

• AWS, Data and model quality monitoring with Amazon SageMaker Model Monitor
• AWS, Amazon SageMaker Model Monitor availability change
• AWS, Create a Baseline
• AWS, DescribeMonitoringSchedule
• AWS, MonitoringExecutionSummary
• AWS, Schema for Violations
• AWS, Model quality
• AWS, Bias drift for models in production
• AWS, Feature attribution drift for models in production