Classify inference failures before retrying them

Learning objective

Distinguish FM authorization, throttling, timeout, request-validation, and response-parsing errors.

Core idea

A foundation model (FM) request can fail before inference, during processing, or after a successful response reaches your application. Those boundaries require different repairs. Retrying an invalid payload preserves the defect; expanding permissions does not fix a parser that expects another model’s response structure.

Classify the failure using the operation, exception name, HTTP status, request configuration, and the stage that produced it. Keep service exceptions separate from application decoding and business validation. The response status alone is insufficient: two errors can share a status but describe different conditions, and an HTTP 200 response can contain output your application must reject.

How it works

Identify the service error

Amazon Bedrock’s Converse reference defines the following error classes. These mappings are specific to that operation; inspect the reference for other inference interfaces before assuming identical behavior.

Capture the exception before translating it into a generic application error. A message such as “AI unavailable” is useful for a user interface but too broad for diagnosis. Store a protected correlation record that identifies the attempt and failure stage; logging and sanitization are covered in 4.3.6 and 3.2.9.

Diagnose authorization without changing the prompt

Converse requires bedrock:InvokeModel, even though the operation is named Converse. For a managed prompt invocation, also check bedrock:RenderPrompt on the prompt resource. Inspect the caller used by the failing application, not the developer identity that succeeds in an interactive test. Compare the resolved model resource and deployment Region with the authorized route.

AWS’s troubleshooting guidance for AccessDeniedException directs you to check IAM permissions and whether temporary security credentials have expired. A third-party model’s HTTP 403 can also reflect an AWS Marketplace agreement failure or pending subscription, including MPAgreementBeingCreated; inspect the error before treating it as a policy defect. Do not respond by granting broad inference access before identifying the denied action and resource. Permission to count tokens or retrieve documents does not establish permission to invoke the chosen model. See 3.2.2 for the authorization layers and 1.2.4 for inference-profile boundaries.

A controlled authorization test uses the same caller and route with a minimal valid request. If it fails with the same exception, changing wording, temperature, or retrieved evidence cannot remove that permission failure. Repair the access configuration and repeat the test before investigating output quality.

Separate throttling from timeouts

ThrottlingException means the request exceeded account quotas. Check demand across all application workers and retries, rather than considering one client in isolation. AWS distinguishes HTTP 503 service unavailability and HTTP 529 overloaded_error model-capacity failures from HTTP 429 account throttling; a quota increase is not a diagnosis for every capacity-related error.

A service ModelTimeoutException is also different from an application or SDK deadline expiring before a service response arrives. Record which component’s timer fired and how much request time remained. A client timeout does not provide the same evidence as an explicit Bedrock exception, and it does not establish that model processing never began.

Also check the network path. Bedrock troubleshooting describes a 350-second idle TCP timeout on paths through NAT Gateways, interface VPC endpoints, or Network Load Balancers. Idle connections can be dropped, causing client resets or timeouts without a Bedrock exception. Enable Boto3 Config(tcp_keepalive=True) and configure the OS keep-alive interval below that idle timeout; enabling the SDK setting alone is insufficient. Repair connection configuration rather than relying on retries or quota increases.

For transient recovery, AWS recommends exponential backoff with jitter. Apply the shared attempt and deadline budgets described in 1.2.5. Count SDK attempts alongside application attempts, and avoid retry layers that multiply work. This lesson classifies the fault; it does not replace the bounded failover policy.

Validate the request before parsing the response

ValidationException identifies an input-contract failure. Check the exact operation and payload submitted after serialization. In InvokeModel, the request contentType must be application/json, and the body follows the selected model’s native schema. A payload copied from another model can be valid JSON while violating that schema. See 1.3.7.

Converse provides a common envelope, but model support still matters. A concrete validation trap is invoking a Prompt management prompt while also supplying inferenceConfig, system, toolConfig, or additionalModelRequestFields. The Converse documentation prohibits those combinations; define the configuration through Prompt management instead.

Response parsing occurs after the transport or service response. InvokeModel returns a response body and contentType; decode that body using the selected model’s response contract. Converse returns an output union whose message content must be interpreted as content blocks, rather than assumed to be one plain string. A decoder that expects an unrelated native response field can fail after inference succeeded.

Then inspect termination and model-output signals. Converse.stopReason includes max_tokens, malformed_model_output, and malformed_tool_use. These are response signals, not synonyms for ValidationException. Finally, validate the generated artifact against the application’s schema and business rules. Successful decoding does not prove a generated decision is acceptable; see 3.1.4.

Bedrock returns the service exceptions, while a local deadline, response decoding, and artifact validation fail inside your application.

Key distinctions and trade-offs

Service exception versus local failure: A local JSON decoder exception does not become a Bedrock server error because it happened after a Bedrock call. Preserve the service result and the local stage separately.

Same status versus same cause: HTTP 429 can identify ThrottlingException or ModelNotReadyException in Converse. The exception name carries the deciding distinction.

Recovery versus repair: A bounded retry can recover a transient condition. Authorization, unsupported fields, and incompatible response decoders need configuration or code repair before unchanged requests are useful.

Diagnosis detail versus data exposure: Keep enough configuration and sanitized response evidence to reproduce the defect. Do not publish complete prompts and responses merely to improve debugging convenience.

Architecture / decision scenario

A support application reports that every model failure is “rate limited.” The engineer reconstructs five failed attempts using their recorded stage and exception.

The first is AccessDeniedException: the production caller lacks the required model invocation grant. The second is ThrottlingException after a burst of duplicated retry attempts. The third is ModelTimeoutException, while a separate local deadline test produces no service exception. The fourth is rejected as a request-contract violation because a managed prompt request includes prohibited runtime configuration. The fifth receives HTTP 200 but its native-response decoder expects a field belonging to a different model.

The team repairs the permission and managed-prompt configuration, reduces duplicate attempts under the existing retry budget, and gives service and local deadlines separate labels. It replaces the incorrect decoder and verifies its selected model contract. Controlled replays prove each repaired boundary without changing prompt content to mask unrelated defects.

Common exam traps

• Granting a guessed bedrock:Converse action instead of checking bedrock:InvokeModel.
• Treating every HTTP 429 as ThrottlingException.
• Interpreting a client timeout as proof of a service-side model timeout.
• Retrying unchanged invalid requests or denied requests as transient failures.
• Treating valid JSON as proof of a valid model request or acceptable generated artifact.
• Calling an application parser exception a model authorization failure.

What to remember

• Classify by operation, exception, status, and failure stage together.
• Repair authorization and request contracts before replaying them.
• Distinguish quotas, service availability, model readiness, and deadlines.
• Parse the selected interface’s response before validating its business meaning.
• Keep all recovery attempts inside the existing shared budget.

Knowledge check

References

• AWS, Converse
• AWS, Troubleshooting Amazon Bedrock API Error Codes
• AWS, InvokeModel
• AWS, Inference using Converse API