Admit agent work within shared limits

Learning objective

Control agent spawning, batching, concurrency, execution quotas, and cancellation.

Core idea

Every delegation creates demand on shared execution capacity. Bound how much work may be created, how much may run simultaneously, how quickly requests enter a deployment, and how long they remain useful. These are separate controls: limiting concurrent calls does not limit total spawned tasks or token consumption.

An orchestrator should admit work before allocating execution resources. Give each child a share of the parent’s remaining work and time budget, then preserve that accounting across retries and replanning. Azure OpenAI deployment limits remain shared even when each agent is configured with its own budget.

See 2.4.5 for delegation contracts and bounded replanning, and 2.4.2 for aggregation and partial-failure policy. This lesson places admission and cancellation around those patterns so parallelism remains within operational limits.

How it works

Reserve a budget before spawning

Use the dispatch gate in 2.4.5 for total and active child limits, retries, and replanning. Across orchestrator workers, keep reservations in shared authoritative state keyed by tenant and parent operation, with model-call and token allowances as well as child budgets. This application-owned ledger supports fair admission across workers; select its store and transaction strategy from 1.2.5.

Bound running calls separately from request rate

In Python applications, asyncio.Semaphore controls how many coroutines enter a protected section. Acquire it around the actual model or tool call and release it through an async with block. asyncio.BoundedSemaphore also detects releases that raise the counter above its initial value. These controls apply within the process; coordinate separately across replicas.

A semaphore bounds simultaneous calls, not requests per minute (RPM). Fast calls can exceed RPM even with low concurrency. Long prompts can exceed tokens per minute (TPM) at a modest call rate. Combine execution slots with request admission and a shared gateway token policy against the target deployment.

Azure OpenAI quota is assigned per subscription, region, model, and deployment type. TPM allocated to a deployment maps to its TPM limit and a model-dependent RPM limit. Do not assume a universal RPM-to-TPM conversion or separate capacity merely because calls originate from different agents.

Microsoft documents rate-limit token estimation at request arrival, distinct from billed token counts after processing. The estimation includes prompt text and applicable output-limit settings; its quota example names max_tokens and best_of. Use settings supported by your chosen API, rather than copying a Chat Completions parameter into Responses.

For agents routed through one API Management gateway, configure llm-token-limit with a trusted tenant or caller identity in counter-key. Set tokens-per-minute, token-quota over token-quota-period, or both; the period values are Hourly, Daily, Weekly, Monthly, and Yearly. The policy supports classic, v2, self-hosted, and workspace gateways. It shares the counter across policy scopes using the same key, giving orchestrator replicas a common token limit when their calls traverse that gateway.

Set estimate-prompt-tokens="true" to estimate demand before sending prompts; with false, enforcement uses returned usage and can allow a request before detecting overrun. Rate-limit overrun returns HTTP 429; quota overrun returns HTTP 403. The default Retry-After header reports the recommended retry interval in seconds. Counters are independent per gateway, not aggregated across the instance, and concurrent requests can temporarily exceed token limits. Keep strict operation reservations in the application ledger and coordinate capacity across gateways separately.

Observe admission and throttling signals

Read x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens to monitor remaining deployment allowances. On HTTP 429, retry-after-ms provides the documented recommended delay in milliseconds. Put retries back through admission and charge them to the original operation’s allowance instead of spawning replacement agents outside it.

Azure OpenAI can evaluate request arrivals over short intervals, typically one or ten seconds. A minute’s total can remain below RPM while a burst is rejected. Microsoft illustrates that a 600-RPM deployment evaluated over one-second windows can throttle above ten requests in that second. Smooth arrivals; increasing semaphore size does not repair burst throttling.

Batch independent offline work

Use Azure OpenAI Batch API when requests are independent and asynchronous completion fits the task. Create a request file, assign each line a custom_id, upload it through client.files.create(..., purpose="batch"), and submit through client.batches.create(...). The documented completion_window is "24h". Create a GlobalBatch or DataZoneBatch deployment (portal: Global-Batch or Data Zone Batch); each input line’s model must name the same target batch deployment. The standard input-file limits are 200 MB and 100,000 requests per file. Split larger workloads while retaining item identifiers and checking available enqueued-token quota before each submission.

Batch processing uses separate enqueued-token quota, which protects online workload capacity from the same batch requests. Its 24-hour turnaround is a target, not a guaranteed expiry: Microsoft states that jobs taking longer are not expired for that reason. Do not route a live approval or an immediately dependent agent step into a batch merely to reduce online pressure.

Correlate returned results with their custom_id and inspect output_file_id and error_file_id. Treat each item as an individual outcome. A completed job does not mean every input produced an accepted answer. Keep batch admission and pending work in the parent ledger until you reconcile the results.

Propagate cancellation through the owned work

For local Python work, Task.cancel() requests cancellation; it does not guarantee immediate termination. See 0.10 for cleanup and asyncio.CancelledError propagation. asyncio.TaskGroup cancels remaining group tasks when one fails with a non-cancellation exception, so select it only when that failure behavior matches your task policy.

asyncio.wait_for() cancels the awaited task on timeout and waits for cancellation, so elapsed time can exceed the timeout. asyncio.wait(..., timeout=...) returns pending tasks without cancelling them. Explicitly handle those pending tasks; returning a timeout response while they continue executing is not cancellation.

For a provider batch, call client.batches.cancel(batch_id). The job can remain cancelling for up to ten minutes before cancelled, with partial results in its output file. Completed work is returned and charged. Record cancellation requested separately from cancellation confirmed, then reconcile partial outputs.

A local cancellation signal is not proof that a remote side effect stopped. Mark the parent operation terminal, prevent additional admission, propagate supported cancellation to children, and inspect external outcomes before deciding what recovery requires.

Key distinctions and trade-offs

Spawning allowance limits total delegated work. Concurrency limits simultaneous execution. RPM and TPM limit deployment admission. Batch quota limits enqueued offline demand. A deadline limits useful elapsed time. Cancellation ends owned work where the execution mechanism supports it.

Higher concurrency can reduce completion time only while downstream capacity remains available. More queued work can improve utilization but increase wait time and cancellation cleanup. Partition shared budgets by tenant while retaining a deployment-wide capacity ceiling.

Batching trades interactive response for asynchronous throughput. It is suitable for independent evaluations or classifications, not a substitute for a dependency-aware runtime. Preserve the same ownership and accounting even when execution happens outside the orchestrator process.

Architecture / decision scenario

A document-review supervisor receives hundreds of files, spawns specialist analysis, and offers a cancel control. Interactive decisions share a model deployment with ordinary users; nightly classifications can finish asynchronously.

Reserve each specialist against a parent child-and-call ledger before scheduling it. Use process-local semaphores for running calls and request admission for deployment RPM. Apply llm-token-limit at the common gateway with a tenant counter-key, a TPM limit, and a Daily token quota. Retain the parent ledger for strict work reservations and fair queueing.

Move independent nightly classifications to Batch API and reserve their enqueued-token demand separately. Track each file through custom_id. Keep interactive analysis online and use rate-limit headers plus retry-after-ms to delay admitted retries.

When the user cancels, mark the parent terminal, stop admitting children, cancel owned local tasks, and request cancellation of its batches. Await cleanup and reconcile partial batch outputs. Report which work completed and which cancellation remains pending; do not promise that all execution stopped when the button was pressed.

Common exam traps

• Equating a semaphore with an RPM or total-spawn limit.
• Giving retries and replans fresh independent execution budgets.
• Treating a 24-hour batch target as automatic job expiry.
• Assuming asyncio.wait() cancels pending tasks after timeout.
• Discarding completed batch results or charges after requesting cancellation.

What to remember

Reserve work before spawning, separate concurrency from rate admission, and share budgets across agents using one deployment. Batch independent offline requests, propagate cancellation deliberately, and reconcile completed work before declaring an operation stopped.

Knowledge check

References

• Microsoft, Manage Azure OpenAI quota.
• Microsoft, llm-token-limit.
• Microsoft, Azure OpenAI Batch API.
• Python Software Foundation, Synchronization primitives.
• Python Software Foundation, Coroutines and tasks.