Contain failures before they consume capacity

Learning objective

Isolate failures with timeouts, bulkheads, capacity limits, and sandboxed tools.

Core idea

A slow tool can consume every available execution slot without crashing anything. A generated program can exhaust memory even when its caller catches exceptions. Failure isolation therefore needs several controls: time bounds, separate allocations of concurrent work, resource ceilings, and a constrained place to execute untrusted code.

Each control answers a different question. A timeout bounds waiting. A bulkhead reserves capacity for an independent workload. A capacity limit bounds admitted work or resource consumption. A sandbox constrains where code runs. Combining them prevents one dependency’s failure from becoming the whole assistant’s outage.

See 0.7 for basic asynchronous error handling. Here the decision is how to stop a failing tool from taking capacity away from healthy tools and users.

How it works

Bound the wait, including admission

In Python, asyncio.timeout() limits an asynchronous context. When its deadline expires, it cancels the current task and transforms the cancellation into TimeoutError, which you catch outside the context. Put the admission wait and the tool call inside the intended deadline; otherwise a request can spend its entire useful lifetime waiting for a slot before its execution timeout even begins.

Cancellation is cooperative. Python recommends cleanup with try/finally and normally propagating CancelledError after cleanup. Code that suppresses cancellation can interfere with timeout behavior. asyncio.wait_for() also waits for the awaited task to cancel, so total elapsed time can exceed its timeout. Do not read a timeout value as proof that remote work has stopped at that instant.

NeMo Agent Toolkit’s code_execution.timeout defaults to 10 seconds to wait for an execution request. The mcp_client function group, requiring nvidia-nat[mcp], exposes tool_call_timeout, defaulting to 60 seconds, and max_sessions, defaulting to 100 concurrent session clients. Set these bounds for the dependency; a request timeout does not guarantee remote computation has stopped, and a session-client cap is not a universal tool-call quota.

Allocate separate concurrency pools

A bulkhead separates capacity so one workload cannot occupy the allocation reserved for another. In an asynchronous Python tool adapter, separate asyncio.Semaphore instances can limit simultaneous calls to independent dependencies. Semaphore.acquire() waits when the counter reaches zero; async with releases the acquired slot on exit.

Reserve separate semaphores for retrieval and analytics so analytics cannot fill the lookup allocation. A shared semaphore caps total work without reserving capacity for either dependency. The application chooses the pool sizes.

Separate semaphores reserve capacity for each dependency, while a shared semaphore reserves capacity for neither.

A semaphore is an in-process control, and Python’s asyncio synchronization primitives are not thread-safe. Four processes each holding an eight-slot semaphore can admit 32 calls in total. Size the aggregate against backend capacity, or place the constrained tool behind a separately sized worker service. A per-process counter is not a fleet-wide quota.

Distinguish waiting limits from resource limits

A semaphore limits active holders; it does not by itself cap the number of tasks waiting to acquire it. Bound admission waiting with asyncio.timeout() and return the application’s busy outcome when that budget expires. This stops callers from waiting indefinitely; it does not turn unlimited task creation into bounded storage. Keep the admitted request volume within the service’s intended operating range.

For tool workers deployed on Kubernetes, container resources.requests supports scheduling and resources.limits constrains resource use. CPU limits are enforced through throttling. Memory limits are enforced reactively through out-of-memory kills, rather than a smooth reduction in memory use. A memory limit protects the surrounding system but can terminate the worker and lose its unfinished work.

For nat serve async jobs, install nvidia-nat[async_endpoints]. The --dask_worker_memory_limit default is 0, meaning no memory limit. --max_concurrent_jobs defaults to 10 but creates Dask workers rather than directly capping jobs: the page gives actual concurrency as max_running_async_jobs * dask_threads_per_worker. That flag applies only when --scheduler_address is unset; an external scheduler needs its own capacity configuration.

Put a resource-hungry tool in a separate Pod when it needs an independent failure domain. A semaphore inside the assistant process cannot protect that same process from a native crash or memory exhaustion. If the tool reports OOMKilled, Kubernetes documents excessive memory use as the cause: inspect consumption and adjust the workload or its memory allocation. Increasing a caller timeout does not repair that failure.

Execute generated code outside the assistant

The toolkit’s code_execution function sends Python code to an execution server. NVIDIA documents the included Docker-based local_sandbox and a remote Piston server; the server must be running and accepting requests. Configure uri and sandbox_type for the execution service; sandbox_type defaults to "local". An unavailable server is a dependency or deployment problem, not a reason to run the generated code inside the assistant process.

The function also exposes max_output_characters, whose documented default is 1,000. This limits returned text; it is not a memory ceiling for the sandbox. On successful execution, the tool returns what the code prints to standard output, rather than returning files or in-memory objects. Code that calculates a result but never prints it needs an output correction, not a larger response limit.

For a Linux sandbox worker on Kubernetes, the Restricted Pod Security Standard supplies concrete hardening controls, including securityContext.allowPrivilegeEscalation: false, runAsNonRoot: true, dropping ALL capabilities, and a seccompProfile.type of RuntimeDefault or Localhost. Apply the fields at the documented Pod or container level. These settings reduce execution privileges; they do not replace the tool’s deadline or resource allocation.

Key distinctions and trade-offs

Choose each control for the failure it contains:

Isolation can reduce peak utilization because healthy workloads cannot automatically borrow every reserved slot. That is an intentional availability trade-off. Raise a pool only when its dependency can absorb the aggregate load, rather than because waiting callers make the local queue look busy.

Architecture / decision scenario

An insurance analytics assistant handles 40 document lookups and 4 generated-Python analyses per second. Its document-lookup response budget is 2 seconds. Measurements for this scenario show lookups occupying a slot for 0.2 seconds, while analyses occupy a slot for 3 seconds and occasionally stall for 30 seconds.

A proposed 16-slot shared pool looks sufficient for ordinary lookup traffic: 40 multiplied by 0.2 gives about 8 occupied lookup slots on average. But analyses add about 12 occupied slots even before stalls. Raising the shared pool to 64 would admit more work to the same constrained analytics backend and still provide no reserved lookup allocation.

The service owner uses three assistant replicas, each with a four-slot analysis semaphore and a four-slot lookup semaphore. That admits 12 analysis calls and 12 lookups across the deployment, leaving four lookup slots above the measured average occupancy of eight. Calls that cannot enter within the allocated admission budget return a busy outcome. The lookup timeout includes both admission and execution, preserving its two-second budget.

Generated code goes to a separately deployed code_execution service, with an explicitly selected uri, a five-second request timeout, and an output-character limit. The platform owner uses three Linux sandbox worker Pods with container CPU and memory limits and Restricted security settings. Those worker allocations constrain sandbox resources independently of assistant-side counters: after a timeout releases a semaphore slot, the remote analysis may still consume its worker’s allocation. A stalled analysis therefore cannot occupy the lookup semaphore, and a worker memory failure does not kill the assistant process. The application treats a timed-out remote analysis as uncertain rather than promising that execution stopped exactly at five seconds.

Common exam traps

• Assuming a timeout releases remote resources immediately.
• Using one shared concurrency pool and calling it failure isolation.
• Treating a per-process semaphore as a global quota.
• Confusing output-character truncation with sandbox memory protection.
• Assuming async Dask workers have a memory ceiling when --dask_worker_memory_limit remains 0.
• Increasing timeouts to fix OOMKilled or an unavailable execution server.

What to remember

Bound time, reserve independent capacity, constrain resources, and execute generated code in a separate environment. Give each layer a specific mechanism and owner. A sandbox without capacity limits can still exhaust resources; a timeout without bulkheads can still let one dependency occupy the service.

Knowledge check

References

• NVIDIA, Code Execution
• Python Software Foundation, Coroutines and Tasks
• Python Software Foundation, Synchronization Primitives
• Kubernetes, Resource Management for Pods and Containers
• Kubernetes, Pod Security Standards
• NVIDIA, MCP Client
• NVIDIA, API Server Endpoints