Fit retries inside the response deadline

Learning objective

Implement bounded retries with exponential backoff, jitter, and timeouts.

Core idea

A retry is another chance to complete the same operation, paid for with elapsed time and dependency capacity. Bound it in two dimensions: how many attempts can run, and how much time the whole operation can consume. Exponential backoff reduces retry frequency, jitter separates callers’ schedules, and timeouts stop individual attempts from occupying the entire budget.

None of these controls establishes whether repeating a write is safe. Classify the failure first, as in 2.4.1, and reconcile uncertain side effects as in 2.4.4. This lesson focuses on the attempt scheduler: who owns it, what it counts, and when it must stop.

How it works

Choose one retry owner

Inspect every layer between the workflow and the dependency. A tool adapter can retry; an LLM provider configuration can request retries; its SDK can retry; the agent can react to a returned error and call again. Retries at these boundaries can overlap or multiply unless you locate the actual execution owner.

In NeMo Agent Toolkit, inspect llms.<name>.max_retries: the nim provider’s NIMModelConfig lists it as the maximum retries for a request. The openai provider also exposes max_retries and request_timeout, with the latter expressed in seconds. Include these provider settings when locating the retry owner; do not assume their configuration is independent of the underlying SDK’s attempts.

For example, the OpenAI Python client’s README documents max_retries, with two retries by default. The documented default retry categories include connection errors, HTTP 408, 409, 429, and server errors of 500 or greater. max_retries=0 disables its retries. These are Python-client behaviors, not a universal policy for every OpenAI-compatible server or every NVIDIA integration.

If a custom adapter owns scheduling, disable that client’s internal retries and implement the adapter’s classified retry policy. If the SDK owns retries, account for its internal attempts instead of adding another loop for the same failure. Three outer attempts around a client permitting two retries can produce nine physical attempts: three calls multiplied by three attempts per call.

NeMo Agent Toolkit’s ReAct tool_call_max_retries is another distinct control. It bounds tool-call error retries; it does not configure the SDK’s transport policy or establish an end-to-end deadline. See 1.2.2 for agent-loop controls. Keep the transport scheduler beneath that boundary and prevent a returned error from starting an unbudgeted second recovery cycle.

Increase delay and randomize each schedule

For a custom Python adapter, one explicit policy uses a capped exponential interval and samples the actual delay uniformly from zero to that interval. Express that policy as:

retry index k = 0 for the first retry
interval(k) = min(delay_cap, base_delay × 2^k)
delay(k) = random.uniform(0, interval(k))

base_delay and delay_cap here are application policy parameters, not NeMo Agent Toolkit configuration keys. Python’s random.uniform(a, b) supplies the sample, and asyncio.sleep(delay) yields control while the adapter waits. Sample separately for each retry rather than applying one random offset to every caller’s whole schedule.

The exponential interval gives a struggling dependency more recovery time after repeated failures. The cap bounds an individual wait. Randomization spreads a group of callers that failed together across the next interval instead of placing them all at its endpoint. A random delay alone does not reduce the maximum number of attempts, and a delay cap alone does not bound total elapsed time.

Before sleeping, test both the attempt counter and the remaining deadline. After sleeping, test the deadline again before sending. Scheduling delays or other work might have consumed the available time. Stop when there is insufficient time for a useful attempt plus result handling; do not start one merely because the attempt counter has room.

Each retry tests the remaining deadline twice: with the attempt counter before sleeping, and again before sending.

Apply timeouts at the right scope

Use a per-attempt execution timeout and an outer deadline covering attempts, waits, and completion handling. In Python, asyncio.timeout() limits a block and converts its internally triggered cancellation into TimeoutError outside the context. asyncio.timeout_at() accepts an absolute deadline measured using the event loop’s clock. Keep the whole retry scheduler inside that outer scope.

NeMo Agent Toolkit also documents timeout middleware under middleware, with _type: timeout and a positive timeout value in seconds. Attach it through a function’s ordered middleware list. Its timer covers the intercepted call and the remaining middleware chain; placement therefore determines whether retry delays are included. Around a streaming invocation, it covers the entire stream duration rather than resetting for every chunk.

An SDK timeout is a separate layer. The OpenAI Python client exposes timeout and raises APITimeoutError; its README also states that timed-out requests are retried by default. Setting a client timeout without inspecting max_retries can therefore exceed the intended logical-operation budget. Coordinate the client setting with the outer deadline rather than treating one request timeout as the final user response limit.

Timeout expiry limits local waiting; it is not proof that a remote write never happened. Preserve cancellation after cleanup, and direct ambiguous writes to reconciliation. See 1.8.2 for the broader resource-isolation implications of cancellation and execution limits.

Measure attempts and terminal outcomes

Have the adapter emit a record with the logical operation identifier, physical attempt index, classified failure, sampled delay, remaining time, and terminal outcome. These are proposed application record fields. Distinguish attempt exhaustion from deadline exhaustion so the operator can determine which control actually stopped the work.

Verify the policy with injected failures: a recoverable first failure, permanent input failure, dependency timeout, cancellation during backoff, and simultaneous callers. The expected result is a bounded physical request count and bounded local waiting, with no further send after cancellation or deadline expiry. Lower returned error rates alone do not show whether recovery is affordable.

Key distinctions and trade-offs

The four controls answer different questions:

A longer retry budget can improve recovery from brief failures while increasing load during an outage. Circuit-breaker admission, taught in 2.4.3, addresses sustained dependency failure across calls. It should not be replaced with an ever-larger per-request retry count.

Architecture / decision scenario

An insurance analytics assistant has an eight-second response deadline for a read-only policy-statistics lookup. The workflow owner reserves 0.5 seconds for setup and 0.75 seconds for result handling. Each dependency attempt is allowed 1.5 seconds. The selected adapter policy uses a 0.25-second base interval, a one-second delay cap, uniform sampling, and three total attempts. Microsoft recommends exponential backoff with jitter for background work and immediate or regular-interval retries for interactive work. This deployment has 200 concurrent interactive callers sharing the dependency, so the owner chooses jittered exponential backoff to spread their retries, accepting up to 0.75 seconds of waits within the same deadline.

The worst selected waits before those attempts are 0.25 and 0.5 seconds. The planned upper budget is therefore 0.5 + 4.5 + 0.75 + 0.75 = 6.5 seconds, leaving scheduling headroom within the eight-second deadline. The actual outer deadline remains authoritative even if overhead grows beyond that headroom.

A colleague proposes four total attempts to improve success rate. That plan adds a 1.5-second attempt and a wait of up to one second, raising the planned total to nine seconds. The owner rejects it because it violates the same deadline. The adapter’s HTTP client has automatic retries disabled so the three-attempt policy describes physical sends. Permanent request defects fail immediately; an eligible transient failure can consume the remaining attempts. The service operator reviews both final failures and physical request volume before changing the budget.

Common exam traps

• Counting two retries as two total attempts.
• Multiplying agent, adapter, provider, and SDK retry policies without locating their retry owner.
• Adding jitter without an attempt cap or outer deadline.
• Setting only a per-attempt timeout and ignoring backoff time.
• Retrying permanent failures or writes with unknown outcomes unchanged.
• Starting another attempt after cancellation because the counter has room.

What to remember

Choose one retry owner and count physical attempts. Bound attempts, randomize capped exponential delays, and enforce a deadline around the whole scheduler. A retry is eligible only when the operation is safe to repeat and enough time remains to complete useful work.

Knowledge check

References

• Microsoft, Recommendations for handling transient faults
• NVIDIA, Configure the ReAct Agent
• NVIDIA, LLMs
• NVIDIA, Middleware
• OpenAI, OpenAI Python API library
• Python Software Foundation, Coroutines and Tasks
• Python Software Foundation, Generate pseudo-random numbers