Control model traffic before retrying throttles

Learning objective

Manage model throughput limits, TPM/RPM quotas, 429 responses, and retry/backoff policies.

Core idea

For an architect managing multi-agent model traffic, a successful retry is only one part of throughput control. Several agents can exhaust a shared deployment even when each agent sends few requests. Control the combined demand, identify the limiting dimension, and spend retries inside one deadline.

Azure OpenAI in Microsoft Foundry Models distinguishes tokens per minute (TPM) from requests per minute (RPM). Token-heavy requests can exhaust TPM; many short requests can exhaust RPM. HTTP 429 is evidence that the service refused admission, not evidence that the model needs a different prompt or that the application should retry immediately.

See 2.4.9 for execution admission and 2.3.8 for tool recovery. This lesson concerns model deployment limits and the model client’s retry policy.

How it works

Locate the quota pool and deployment allocation

Azure OpenAI quota is assigned to a subscription by region, model, and deployment type. Assigning TPM to a deployment draws from that pool and sets its inference rate limits. Creating another resource in the same pool does not create another independent subscription quota allocation.

RPM is proportional to the TPM allocation, but the ratio varies by model. The quota documentation gives o3 one capacity unit of 1 RPM and 1,000 TPM; o3-mini gets 1 RPM and 10,000 TPM per unit. These documented allocation ratios are not promised throughput for arbitrary requests. Do not apply one model family’s ratio to another, or assume TPM and RPM can be tuned independently.

In the Foundry portal, Manage > Quota separates Token per minute allocations for standard deployments from Provisioned throughput unit allocations for provisioned deployments. Reallocate unused quota or request an increase when the sustained workload exceeds the allocation. A requested increase is not an admission guarantee. The documentation allows up to 15 minutes for allocation changes or quota requests to propagate, so verify the effective allocation before judging a load test.

Understand what admission counts

For the standard TPM calculation, Azure OpenAI estimates the maximum tokens processed when a request arrives. The documented estimate includes prompt text and count, max_tokens, and best_of. It differs from the token count computed for billing after generation.

This means an output allowance of 4,000 tokens can consume more admission capacity than a 200-token allowance even if both requests generate about 200 tokens. Set supported output limits to what the task needs. The quota page names max_tokens and best_of; do not assume every model or API accepts those parameters. Apply the equivalent supported output control for the deployed interface, rather than sending an unsupported parameter.

The TPM counter resets each minute. RPM enforcement expects evenly distributed traffic and evaluates short periods, typically 1 or 10 seconds. The documentation’s example is a 600-RPM deployment evaluated over 1-second intervals: more than 10 requests in a second can trigger throttling even if the minute total is below 600.

A concurrency cap therefore cannot establish compliance by itself. Fast calls can send many requests per second, and slow calls with large token allowances can exhaust TPM at low concurrency. Combine bounded concurrency with request pacing and a shared token admission estimate. Provisioned throughput units use a different utilization-based admission calculation; do not apply the standard counter explanation to them.

Use service evidence before changing capacity

Azure OpenAI returns x-ratelimit-limit-requests, x-ratelimit-limit-tokens, x-ratelimit-remaining-requests, and x-ratelimit-remaining-tokens. Record the deployment and the limiting dimension when diagnosing throttling. The remaining values are observations, not reservations for every concurrent caller.

Standard pay-as-you-go deployments share a resource pool, and capacity pressure can temporarily reduce a deployment’s effective rate limit without changing its configured quota. Compare x-ratelimit-limit-tokens with the configured TPM allocation to detect that adjustment. Follow retry-after-ms backoff; for workloads requiring consistent throughput, consider Provisioned Throughput rather than assuming a quota increase repairs shared-pool pressure.

For a 429 response, retry-after-ms gives the recommended wait in milliseconds. Use it when present. The reset headers, x-ratelimit-reset-requests and x-ratelimit-reset-tokens, provide additional rate-limit evidence; keep their interpretation tied to the API’s documented headers rather than assuming they are interchangeable with retry-after-ms.

In Azure Monitor, inspect Azure OpenAI Requests (AzureOpenAIRequests) for 429 responses on pay-as-you-go deployments. For PTU-managed deployments, Provisioned-managed Utilization V2 (AzureOpenAIProvisionedManagedUtilizationV2) helps identify capacity pressure. Join those signals with application attempts, waits, and outcomes. Low final error counts can hide many retries, and low billed tokens can coexist with a high admission estimate.

Give retries one owner

Microsoft documents built-in exponential-backoff retries in the Azure OpenAI Python openai client, with two retries by default. The SDK automatically respects retry-after headers and uses exponential backoff with jitter. AzureOpenAI(..., max_retries=...) sets the client policy; client.with_options(max_retries=...) overrides it for a request. Two retries means up to three attempts, including the initial request.

If an application scheduler owns custom retry timing, set max_retries=0 on the underlying client and implement the bounded policy there. If the SDK owns retries, do not add an independent loop that assumes each client call is one attempt. Three outer calls to a client allowing two retries can send nine attempts.

Honor the service wait advice. Without it, use exponential backoff with random jitter and a maximum retry count. Check remaining deadline before scheduling another attempt. Unsuccessful requests still count toward the per-minute rate limit, so tight loops amplify the overload they are intended to repair.

Backoff handles temporary pressure; sustained excess demand needs pacing, reduced request budgets, reallocation, or capacity changes. A retry policy cannot enlarge the quota pool. Keep the shared admission policy active during retries so recovering requests do not starve first attempts indefinitely.

Key distinctions and trade-offs

Quota versus allocation: Subscription quota bounds what you can assign within its region, model, and deployment-type scope. Deployment allocation bounds that deployment’s traffic.

Admission estimate versus billed usage: Admission budgets the estimated maximum request, while billing reflects processing. They answer different questions.

Concurrency versus rate: Concurrency counts simultaneous work; RPM and TPM measure traffic over time. Each needs its own control.

Retry count versus total attempts: max_retries excludes the initial attempt. Every layer must account for what the next layer can send.

Architecture / decision scenario

A research workflow starts six specialists per request against one standard Azure OpenAI deployment. Returned answers are short, but every specialist reserves a large output allowance. The service returns 429 responses although billed-token charts appear low.

The team records the rate-limit headers and finds token capacity is nearly exhausted. It reduces the supported output allowance to the validated result contract and paces combined specialist traffic. A second test exposes an RPM burst when several workflows start together, so admission spreads requests across time rather than raising concurrency.

The Python client retains one explicit max_retries policy. The SDK owns retries, including respect for service retry-after headers and backoff with jitter. The application records attempts and deadline consumption without wrapping the client in another retry loop or adding a second service-wait layer. Persistent saturation leads to a quota-allocation review; short bursts lead to pacing changes. The team verifies valid completion rate and latency after the change instead of treating a lower visible 429 count as sufficient.

Common exam traps

• Equating billed tokens with the admission estimate.
• Assuming configured TPM is always the effective limit.
• Assuming a minute total below RPM rules out burst throttling.
• Treating quota as independently available to every agent or resource.
• Applying one model’s TPM/RPM ratio to all models.
• Counting two retries as two total attempts.
• Nesting retries or immediately resending unsuccessful requests.

What to remember

Identify the shared deployment, quota scope, and limiting dimension. Pace requests and budget estimated tokens. Read the headers, bound retries with backoff and a deadline, and change capacity when demand persistently exceeds allocation.

Knowledge check

References

• Microsoft, Manage Azure OpenAI in Microsoft Foundry Models quota
• Microsoft, Quotas and limits
• Microsoft, Latency and performance