Bound failover before adding fallback models

Learning objective

Design bounded application-level model failover with fallback models, circuit breakers, graceful degradation, and no routing loops. Preserve the task contract when the preferred model cannot complete a request.

Core idea

Fallback is an application policy that selects another approved execution path after a defined failure. It is not permission to keep trying models until one returns text. Every attempt must remain inside the request’s deadline, cost budget, capability requirements, authorization, and processing-location boundary.

Represent fallback as a finite ordered plan with a terminal outcome. Keep SDK retry, application model replacement, and Bedrock cross-Region routing separate. Each can add attempts or change execution conditions; together they can amplify load unless one request-level policy bounds the whole operation.

How it works

Classify the failure before switching

Bedrock’s Converse API documents distinct errors including ThrottlingException, ServiceUnavailableException, ModelTimeoutException, AccessDeniedException, and ValidationException. ModelNotReadyException is a transient not-ready condition with HTTP status 429. Use the structured error identity rather than searching message text for a generic failure word.

Transient capacity or service failures can justify a bounded retry or approved fallback. Authorization failures require correcting the grant or refusing the request. Invalid request bodies require correcting the caller or adapter. Changing models to work around a policy denial can violate the boundary the failure was enforcing.

A successful HTTP response can also contain an outcome that is not a usable answer. Inspect stopReason and your output validator. A response stopped by a guardrail is not an invitation to call an unguarded model. An output that reached its response limit might need a bounded task-specific recovery, but not an endless sequence of larger responses.

Distinguish an operational error from an intended abstention. If evidence cannot support the requested answer, another model’s willingness to answer does not make the answer valid. Carry the same evidence and safety contract into any replacement path.

Allocate one request budget

Configure SDK retries through retry_mode and max_attempts, which counts the initial request plus retries; see 2.4.5 for jittered backoff, retry limits, and transient-error handling.

For example, a plan with two model targets and three SDK attempts per target can issue up to six underlying calls. That is an application example, not a service recommendation. A request deadline can make even that bound too large. Allocate time for validation and the final response rather than spending the entire budget inside model calls.

Track the total number of attempts, visited targets, elapsed time, and remaining budget in request state. Select the next target only if it is unvisited and can still complete within the allowed envelope. Never let model A’s fallback select B while B’s independent fallback selects A for the same request. Only a transient failure advances the plan, and only to an unvisited target while budget remains; every other failure ends without switching models.

Own the policy at one orchestration boundary. If an adapter retries, an API layer retries, and a model router retries without shared accounting, a nominally small limit becomes a large multiplication. The SDK handles documented service retries; the application owns cross-target selection and the total deadline.

Prevent repeated calls to a failing target

A circuit breaker is application state that temporarily stops sending ordinary traffic to a target with sustained eligible failures. AWS does not provide a Converse parameter that creates an application cross-model circuit breaker. Implement the closed, open, and limited-probe behavior in your router, with explicit thresholds and recovery rules. See 2.1.7 for circuit breakers on agent execution.

Scope the breaker to the actual failure domain: model target, Region or inference profile, and relevant operation. One malformed request should not open a global breaker for every tenant. A quota failure for one constrained route should not automatically disable an unrelated model path.

Use Bedrock CloudWatch metrics such as InvocationThrottles and InvocationServerErrors to observe service behavior. Add application measurements for fallback selections, exhausted budgets, and breaker transitions because those are decisions your router makes. Preserve the request and configuration identities so you can explain a final result without exposing sensitive content.

Permit only bounded recovery probes after the open interval. Releasing every queued request simultaneously can recreate the overload. A successful probe is a reason to restore traffic gradually under your policy, not evidence that all future requests will succeed.

Define safe degradation

Evaluate each fallback for the task before production. A replacement must fit the input modality, context budget, response contract, and processing boundary; see 1.2.1 and 1.2.4. Do not assume another model in the same service has compatible behavior.

Graceful degradation can be a simpler approved task, a delayed result, an evidence-only view, or a handoff. State the changed behavior visibly. If the only safe terminal result is unavailable, return that result. Preserving availability of text generation is less valuable than preserving the meaning and safety of the task.

For streamed output, define whether fallback is permitted before any response is exposed. Once the user has received part of a model’s answer, silently appending another model’s continuation can produce an incoherent or misleading result. Prefer a visible interruption and a separately identified retry path.

Model fallback does not make external actions safe to repeat. If a workflow can already have executed a tool action, confirm its state before regenerating or repeating the action. This lesson owns model selection recovery; see 2.1.11 for retry-safe tool calls and idempotency controls.

Key distinctions and trade-offs

Retry versus replacement: Retry repeats an operation against its target. Replacement chooses another approved target and can change behavior.

Circuit breaker versus rate limiter: A breaker reacts to target health and blocks calls during failure. A rate limiter bounds admitted demand even when a target is healthy.

Cross-Region routing versus cross-model fallback: An inference profile routes the selected model’s compute. Your application decides whether a different model is acceptable.

A longer fallback chain can increase completion opportunities but also increase latency, cost, and correlated overload. Keep the shortest plan that meets the recovery contract, and measure terminal failures as well as successful recovery.

Architecture / decision scenario

A support drafting API receives a burst of throttles from its preferred model. Its first design retries three times in the SDK, then tries a second model whose adapter sends failures back to the first.

You replace the loop with one router-owned plan: primary, one approved fallback, then an unavailable or handoff outcome. Both targets retain the same evidence and safety controls. The router enforces a shared deadline and visited-target set, while a breaker temporarily removes the failing primary from ordinary traffic.

A guardrail intervention terminates the response without changing targets. A service-unavailable error can advance the plan while budget remains. The resulting policy improves recovery without making forbidden content or an endless routing loop the price of availability.

Common exam traps

• Treating every error as eligible for another model.
• Bypassing a guardrail intervention through an unguarded fallback.
• Multiplying SDK retries by an unbounded application chain.
• Allowing mutually recursive target fallback rules.
• Concatenating a replacement answer onto an exposed partial stream.
• Assuming model fallback provides tool-action idempotency.

What to remember

• Classify errors and response outcomes before selecting recovery.
• Bound total attempts, visited targets, time, and cost together.
• Implement application circuit breakers at the relevant failure boundary.
• Preapprove fallback capability, quality, and processing location.
• End with a defined safe outcome when recovery cannot meet the contract.

Knowledge check

References

• AWS, Converse
• AWS, Retry behavior
• AWS, Monitor bedrock-runtime inference using CloudWatch metrics
• AWS, Route model inference requests across AWS Regions with cross-Region inference