Protect inference with server-side request throttles

Learning objective

Enforce server-side request rates with API Gateway throttling and usage plans independently of client retry policies.

Core idea

A client retry policy governs what one caller does after a failure. A server-side throttle governs admission across callers, including clients that ignore your retry guidance. You need both boundaries when a foundation model (FM) sits behind an API: bounded retries prevent amplification, while API Gateway throttling reduces the rate of requests reaching the integration.

The word enforce needs a qualification. AWS documents API Gateway throttling and usage-plan quotas as best-effort targets, not guaranteed request ceilings. These controls help protect capacity; they do not prove that every request above a contractual limit is rejected or that inference spending stays below a fixed amount. AWS recommends AWS Budgets to monitor costs and AWS WAF to manage API requests; REST APIs support WAF integration. Neither recommendation makes usage-plan targets hard ceilings. Retain separate application admission and accounting for strict model-aware budgets. See 2.3.10 for gateway-level policy.

How it works

Separate rate, burst, and quota

API Gateway uses a token bucket. The request rate replenishes tokens, and the burst setting controls bucket capacity. A request consumes a token. When traffic exceeds the applicable targets, clients can receive 429 Too Many Requests.

In a REST API usage plan, ThrottleSettings.rateLimit is the target rate in requests per second, and ThrottleSettings.burstLimit is the burst target. A sustained rate and a brief surge are different operating conditions. Increasing burst capacity accommodates a surge but can expose the integration to more simultaneous work; it does not increase downstream model capacity.

A usage-plan quota addresses a different timescale. QuotaSettings.limit sets a target number of requests, while QuotaSettings.period takes DAY, WEEK, or MONTH. A monthly quota cannot stop an intense burst early in the month. A requests-per-second target does not allocate a monthly request allowance. Use the controls together when both conditions matter.

These are request counts, not token budgets. Two accepted API requests can have very different prompt sizes, output lengths, and costs. If inference capacity is the bottleneck, use request throttling alongside model-aware application admission. For example, reject or defer a request whose allowed output budget would exceed your workload’s remaining capacity reservation. That reservation is your application rule, not a capability provided by a usage-plan quota.

Apply limits at the correct scope

For REST APIs, AWS documents this order for throttling settings:

1. Per-client or per-method targets configured for a stage in a usage plan.
2. Per-method targets configured for an API stage.
3. Account-level throttling for the Region.
4. AWS Regional throttling.

A per-client usage-plan target cannot exceed the account-level limits for the Region, and account limits cannot exceed AWS Regional throttling limits. Account-level capacity is shared across APIs in that account and Region, so unrelated traffic can affect an inference API. Set method-specific targets when operations have different backend costs, rather than assigning one broad target to every method.

Usage plans associate API keys with deployed stages. Throttling and quota usage for each key aggregate across the stages within the plan. A key can belong to multiple plans, but only one usage plan for each API stage. If a customer uses one key across several stages in a plan, those requests share the plan’s allowance; the key does not receive a fresh allowance for each stage.

API keys identify clients for usage management. They are not authentication or authorization credentials. Use an IAM role, Lambda authorizer, or Amazon Cognito user pool for access control. Do not let possession of a usage-plan key substitute for a verified caller identity or model entitlement.

Keep server policy independent of retries

Configure the admission targets even when every supported SDK client uses bounded exponential backoff. A client might use a different SDK, run many processes, or deliberately ignore the policy. None of those behaviors should decide whether the server has a request-rate boundary.

Conversely, a server 429 is not permission for unlimited retries. The caller should follow the bounded attempt and deadline policy described in 2.4.5. Each resubmission creates more admission traffic. A synchronized retry surge can therefore sustain throttling even after the original burst ends.

Distinguish a gateway rejection from a backend failure. An API Gateway throttle can reject work before inference, whereas a request that reaches the integration can encounter a separate model-side throttle. Preserve that origin in your application error contract and logs; do not classify every 429 as evidence that the foundation model ran out of capacity.

Key distinctions and trade-offs

The following boundaries answer different questions:

API type matters. API Gateway REST APIs support API keys and per-client usage throttling. HTTP APIs support stage and route throttling but do not provide the REST API usage-plan capabilities. Choose the API type from the required control, rather than assuming that all API Gateway products expose identical policy surfaces.

Architecture / decision scenario

A document assistant serves partner applications through a REST API. Short summaries and long document analyses share the account, but analyses consume substantially more inference capacity. Partners use different client libraries, and some retry aggressively.

The team assigns method-level throttling targets for each operation and a usage plan to each partner’s API key. As illustrative workload settings, one partner receives a target of five requests per second, a burst target of ten, and a monthly request allowance. These values come from load testing; they are not AWS default quotas. The team also configures stage-level targets for the shared integration independently of partner retry behavior.

An authorizer verifies partner identity separately. The application evaluates document size and the requested output budget before calling the model. A strict contractual spending cap uses application accounting because the usage-plan quota is best effort. Partners retain bounded retry policies so rejections do not become another sustained burst.

When callers report 429 responses, operations first checks whether API Gateway rejected admission or the integration reported a backend throttle. It changes the relevant boundary only after identifying the bottleneck. Raising a partner target while the shared account or model path is constrained would not resolve that failure.

Common exam traps

• Selecting SDK backoff as the server-side request-rate control.
• Treating a usage-plan quota as a guaranteed spending ceiling.
• Using an API key as proof of authentication or authorization.
• Assuming a per-client usage-plan target can exceed the account-level limit.
• Confusing a burst target with sustained throughput or model concurrency.
• Choosing HTTP APIs when the requirement explicitly needs REST usage plans.

What to remember

Protect admission independently of caller behavior. Set rate, burst, and longer-period quotas according to their different scopes. Usage plans manage API-key clients, while authorization remains separate. API Gateway targets are best effort, and request counts do not measure inference tokens or cost. Keep retries bounded and identify the rejecting layer before changing capacity.

Knowledge check

References

• AWS, Throttle requests to your REST APIs.
• AWS, Usage plans and API keys for REST APIs in API Gateway.
• AWS, Throttle requests to your HTTP APIs.
• AWS, Choose between REST APIs and HTTP APIs.
• AWS, ThrottleSettings.
• AWS, QuotaSettings.