Replicate the assets recovery actually needs

Learning objective

Replicate model-dependent Amazon S3 assets across Regions using S3 Cross-Region Replication to meet recovery objectives. Make the secondary Region able to load a coherent approved release rather than merely contain some copied files.

Core idea

S3 Cross-Region Replication (CRR) asynchronously replicates eligible objects between buckets in different Regions. It is an asset-delivery mechanism, not a complete regional recovery system. A secondary model deployment also needs compatible configuration, permissions, encryption access, runtime resources, and verified readiness.

Start with a release inventory and recovery objectives. The recovery point objective (RPO) defines tolerable missing or stale data; the recovery time objective (RTO) defines tolerable restoration delay. Replication lag contributes to RPO, while loading and validating a usable model deployment contributes to RTO.

How it works

Select a coherent asset set

Inventory the Amazon S3 objects required to restore the approved serving path. Depending on the design, these can include custom model weights, LoRA adapters, tokenizer files, packaging metadata, and release evidence. Managed Bedrock foundation-model weights are not application-owned S3 objects that CRR makes available in another Region.

Bind the assets to the same release identity used in 1.2.7. A base-model artifact from one release and an adapter from another can both be readable while being incompatible. Store a release inventory that identifies the required objects and their reviewed content identities so the recovery check can verify the complete set.

Separate recoverable runtime assets from optional historical data. A large training archive can have a different recovery urgency from the adapter needed to serve the current task. Replication rule scope should match those needs rather than replicate every object without regard to usefulness or permitted location.

A release inventory is your application mechanism for consistency. CRR does not publish all related objects as one atomic deployment. Gate the secondary release on the presence and validation of every required asset rather than treating the first replicated file as evidence that the deployment is ready.

Configure replication prerequisites

Both source and destination buckets must have S3 Versioning enabled. Configure a replication rule on the source bucket, with the intended object scope and destination. Give Amazon S3 a replication role with the permissions needed to read the eligible source objects and replicate them to the destination.

Use a rule filter to scope the asset set. Prefixes or tags can distinguish active release assets from unrelated data. Review that filter whenever packaging locations change; a perfectly healthy rule can omit a new adapter stored outside its selected prefix.

Live replication covers eligible objects created after replication is configured. Use S3 Batch Replication for existing objects that need backfilling. A source bucket full of approved weights does not become a ready secondary merely because you add a live replication rule.

Test with the serving identity as well as the replication identity. The role that writes replicas is not the role that loads the model. Verify that the secondary runtime can read the destination objects under the intended account and bucket permissions. Storage success without serving access cannot meet the recovery objective.

Make encryption work in the destination

For SSE-KMS or DSSE-KMS source objects, configure the documented encrypted-object replication settings rather than assuming ordinary replication configuration is sufficient. SourceSelectionCriteria.SseKmsEncryptedObjects selects those objects, and Destination.EncryptionConfiguration.ReplicaKmsKeyID identifies the destination encryption key.

The replication role needs kms:Decrypt for the source key and kms:Encrypt for the destination key, with the additional permissions required by the chosen encryption configuration. The secondary runtime also needs permission to decrypt the assets it loads. A valid destination object name does not demonstrate that either identity can perform its required operation.

Use a destination-Region key as required by the replication configuration. S3 treats multi-Region KMS keys as single-Region keys and does not use their multi-Region features, so they do not remove this requirement. Check the actual permissions and key identity before the recovery drill. Do not postpone that validation until the source Region is unavailable, when the missing dependency is hardest to repair.

Measure lag and individual failures

Enable S3 Replication metrics for the rule. The documented measures include Bytes Pending Replication, Operations Pending Replication, Replication Latency, and Operations Failed Replication. Observe the active-release scope rather than accepting an aggregate dashboard that conceals a missing consequential artifact.

Replication Latency describes how far the destination is behind for the rule; it is not the time to start a model endpoint. Use the failed-operations signal with the OperationFailedReplication event to identify object failures and their reasons. Pending and failed work require different operational responses.

S3 Replication Time Control (S3 RTC) provides a documented replication-time objective: 99.9 percent of uploaded objects within 15 minutes, with associated service-level terms and constraints. It does not promise zero lag or guarantee that every object arrives before a failover. Evaluate those terms against your workload rather than declaring your application RPO satisfied by the feature name.

RTC also provides OperationMissedThreshold and OperationReplicatedAfterThreshold events. Use them to detect assets that exceed the threshold and to determine when they subsequently arrive. Metrics and events provide monitoring; the release-readiness decision still checks the actual required assets.

Handle deletion and activation deliberately

Deletion behavior is not a simple mirror. With a replication configuration that uses the Filter element, delete markers are not replicated by default; supported non-tag-based rules can enable delete-marker replication. Deleting a specific source object version does not delete that version from the destination.

Choose retention and cleanup separately for the replica. Retaining a previous approved release can support rollback, but keeping obsolete assets indefinitely can violate the application’s retention policy. Source deletion alone is not evidence that every regional copy has been removed.

Activate a secondary release only after its assets are complete, readable, compatible, and tested. Then combine that readiness with the API recovery policy from 1.2.10. Replication reduces the work remaining during an incident; it does not provision endpoint capacity or switch client traffic on its own. Existing objects and new writes reach the destination by different paths, and the secondary release is activated only after the recovery check verifies the complete set.

Key distinctions and trade-offs

Replication versus deployment: CRR copies eligible objects. The deployment process creates and validates the serving resources that use them.

Replication lag versus recovery time: Lag measures asset freshness. Recovery time includes selecting the release, loading resources, validating the task, and restoring service.

Live replication versus backfill: Live rules cover eligible subsequent writes. Existing assets require a deliberate Batch Replication path.

RTC adds monitoring and replication-time terms at additional cost. A stricter application RPO might still require withholding a newly published release until its critical replica set is verified. Match the mechanism to the freshness requirement rather than assuming asynchronous copying is instantaneous.

Architecture / decision scenario

A SageMaker AI service loads a base artifact and several domain adapters from S3. Its secondary Region has compute capacity, but the approved artifacts existed before the team enabled CRR.

You backfill those assets with S3 Batch Replication, then use live rules for subsequent releases. Encrypted-object configuration and destination KMS permissions let the secondary runtime load them. A release inventory checks the complete compatible set before marking the secondary ready.

A later adapter fails replication. The failed-operations signal and event identify the affected object, and the secondary retains the preceding complete approved release. The API recovery plan does not activate a mixture of the new adapter metadata and missing weights merely because most objects arrived.

Common exam traps

• Assuming CRR provisions model endpoints or replicates managed Bedrock models.
• Expecting a live rule to backfill every existing artifact.
• Omitting Versioning or encrypted-object replication permissions.
• Treating replication latency as endpoint recovery time.
• Interpreting RTC as a zero-loss guarantee for every object.
• Assuming source version deletion removes all destination copies.

What to remember

• Replicate a versioned, compatible release set tied to recovery objectives.
• Enable Versioning, configure the rule, and authorize replication and serving identities.
• Backfill existing assets separately from live replication.
• Use replication metrics and events, then verify individual release readiness.
• Combine asset recovery with deployment and traffic recovery mechanisms.

Knowledge check

References

• AWS, Requirements and considerations for replication
• AWS, What does Amazon S3 replicate?
• AWS, Replicating encrypted objects (SSE-S3, SSE-KMS, DSSE-KMS, SSE-C)
• AWS, Using S3 Replication metrics
• AWS, Meeting compliance requirements with S3 Replication Time Control