Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-102.4 Implement FM API integrations

Bound SDK retries before switching models

Completed by the knowledge check

Learning objective

Handle SDK throttling and transient errors using jittered exponential backoff, retry limits, and safe model fallback.

Core idea

Retry a failed transport attempt only when another attempt can plausibly succeed within the application’s budget. A retry policy needs an error classification, randomized delay, attempt limit, and deadline. A model fallback adds a second inference path, so it must share those limits rather than restart the budget.

AWS SDKs supply retry mechanisms, but their configuration and behavior must be explicit in the deployed client. An application loop around an SDK that already retries can multiply requests during the exact overload it is trying to escape.

This lesson focuses on the SDK-to-application boundary. See 1.2.5 for the broader failover design, including circuit breakers and terminal outcomes. Here the deciding question is how many attempts the deployed integration actually sends before returning a result or failure.

How it works

Classify the failure before retrying

For Bedrock Converse and InvokeModel, ThrottlingException returns HTTP 429 and indicates that the request exceeded account quotas. ServiceUnavailableException returns 503, while InternalServerException returns 500. These belong in the integration’s capacity or transient-failure handling, rather than a prompt rewrite.

ValidationException returns 400, AccessDeniedException returns 403, and ResourceNotFoundException returns 404. Repeating an unchanged invalid request or unauthorized target does not repair it. Surface the configuration or input defect to the appropriate owner instead of hiding it behind a different model.

ModelTimeoutException returns 408. Distinguish it from a caller-side socket timeout: Bedrock reports that model processing exceeded its timeout, while a lost client response can leave the service’s outcome uncertain. Your application should decide whether another inference attempt fits the remaining deadline and whether repeated processing is acceptable.

ModelNotReadyException also returns 429, and the Bedrock API reference states that the SDK automatically retries that operation up to five times. Do not assume every 429 is the same error or that a remembered generic default describes every Bedrock exception. Inspect the SDK configuration and attempt evidence when accounting for the actual request budget.

Configure the SDK’s retry policy

Use the SDK’s supported settings for standard retries and an explicit total attempt count: AWS_RETRY_MODE=standard and AWS_MAX_ATTEMPTS, or the corresponding client configuration. In the cross-SDK setting, three maximum attempts means one initial request and at most two retries. Language-specific configuration APIs differ; do not infer their meaning from an identically named option in another SDK.

AWS documents a 2026 retry behavior that requires AWS_NEW_RETRIES_2026=true until it becomes the default. Without that opt-in, the SDK uses the pre-2026 behavior, whose timing and defaults differ. Verify the installed SDK’s support before relying on the documented 2026 timing. Setting standard alone does not establish that timing.

For the opt-in behavior, the documented full-jitter delay is:

delay = random(0, 1) × min(20,000 ms, base_delay × 2^retry)

The first retry uses exponent zero. The documented base delay is 50 ms for transient errors and 1,000 ms for throttling, with each delay capped at 20 seconds. Randomization spreads simultaneous retry attempts instead of synchronizing callers into another burst.

Standard mode also uses a retry-quota token bucket that can stop retries before the configured maximum. Adaptive mode adds client-side rate limiting and can delay the initial request. Its limiter is shared by every request on that SDK client instance. Use it only when that shared throttling behavior fits the workload; a client spanning unrelated tenants or model paths can slow healthy requests because one path is throttled.

Coordinate attempts with the application deadline

Assign one retry owner for ordinary transport failures. If the SDK owns those retries, the application should observe its final outcome before considering another path. If an application layer needs custom retry scheduling, explicitly account for or disable underlying retries rather than nesting independent loops.

Suppose the SDK allows three total attempts and an outer loop calls it three times. The combination can produce nine attempts before a fallback begins. An SQS redelivery adds another layer; see 2.4.2. Record a shared job budget across deliveries when the outcome must remain bounded across worker restarts.

An attempt limit is not a wall-clock deadline. Include request duration, connection waits, retry delay, response validation, and fallback work in the remaining time. Before scheduling another application attempt, require enough time for the next operation and result handling. If not, return the application’s timeout or deferred-work outcome rather than starting work that cannot finish for the caller.

Use Bedrock CloudWatch InvocationThrottles and InvocationServerErrors to distinguish capacity pressure from service failures. AWS notes that the observed throttle count depends on SDK retry settings. Join these signals with application attempt counts and terminal outcomes; low returned error rates can hide expensive internal retries.

Apply fallback without changing the safety contract

After eligible retries fail, consider only a preapproved fallback whose model interface, capability, permissions, and processing location satisfy the request. Use the target’s supported modelId and adapter. A successful call to another target does not establish that its answer satisfies the original result contract.

Keep validation and safety outcomes intact. For Converse, stopReason: "guardrail_intervened" is a response outcome, not a transport failure to bypass with another model. Revalidate a fallback result through the same application checks described in 2.4.3. Do not weaken those checks to raise the success rate.

A retry can generate different content and repeat inference cost. If partial output has reached the caller, do not silently append a fallback continuation. Bedrock streaming can report modelStreamErrorException after progress; terminate that response as incomplete and start a separately identified response only if the application protocol permits it. See 2.4.4 for stream completion handling.

Key distinctions and trade-offs

Retry versus fallback: A retry repeats the same path after an eligible failure. Fallback invokes a different approved path. Both consume the same application’s time and attempt budget.

SDK quota versus service capacity: The SDK retry quota limits retry traffic from the client. It does not increase Bedrock capacity or guarantee that another attempt is admitted.

Attempts versus retries: A total-attempt setting includes the initial request. Counting only the retries understates traffic and produces incorrect nested-loop budgets.

More retries can recover short disruptions, but add latency, traffic, and uncertain repeated inference. Adaptive limiting can protect a concentrated workload while harming unrelated requests that share a client. Choose from measured error patterns and explicit isolation boundaries.

Architecture / decision scenario

A support API gives each request an application deadline and uses a configured SDK client for one approved Bedrock path. The deployment records whether 2026 retry behavior is enabled, the standard retry mode, and the total attempt setting.

When ThrottlingException occurs, the SDK performs bounded jittered retries. The application receives either a result or the final failure, logs the attempts, and checks remaining time. It does not launch a second outer retry loop. A ValidationException is surfaced as a defect; it does not enter the capacity-recovery path.

If an eligible transient failure exhausts retries and enough time remains, the application invokes one approved fallback using its compatible adapter. It validates the result against the same contract and records which path produced it. If the primary had already streamed partial text, the client receives an incomplete outcome instead of a stitched answer. CloudWatch throttle and server-error signals help the team decide whether repeated fallback reflects a capacity problem that retry tuning cannot solve.

Common exam traps

  • Counting three maximum attempts as three retries plus the initial call.
  • Nesting SDK, application, and queue retries without a shared budget.
  • Retrying invalid requests or permission failures unchanged.
  • Assuming standard alone enables the documented 2026 backoff timing.
  • Sharing an adaptive client across paths that require independent latency.
  • Treating a safety intervention or partial stream as permission to bypass the primary contract.

What to remember

Classify errors, configure the deployed SDK explicitly, and keep attempts inside one deadline. Jitter reduces synchronized retry bursts; limits prevent amplification. Approved fallback preserves the original safety and result contract and spends the remaining budget rather than creating a fresh one.

Knowledge check

References