NVIDIANVIDIA-Certified Professional: Agentic AI2026-102.4 Implement error handling (retry logic, graceful failure recovery).
Isolate failing dependencies with circuit breakers
Completed by the knowledge check
Learning objective
Apply the circuit-breaker pattern to failing dependencies.
Core idea
A retry asks whether one failed operation might succeed on another attempt. A circuit breaker asks whether the application should admit another call to this dependency at all. When failures persist, repeated attempts occupy connections, delay responses, and compete with useful work. A breaker temporarily rejects calls so the dependency and its callers can recover.
NeMo Agent Toolkit v1.9.0 documents _type: circuit_breaker middleware. Its mechanism is a state machine with per-target state and limited recovery probes. Use that release’s documentation when configuring it: the published 1.8 middleware page does not document this component. See 2.4.2 for per-operation retry budgeting and 1.8.2 for concurrency and capacity isolation.
How it works
Separate admission from execution
A breaker has three states. Configure the transition conditions deliberately rather than treating “open” as another retryable error:
| State | Admission decision | NVIDIA mechanism | Cost or risk |
|---|---|---|---|
CLOSED | Admit ordinary calls and observe failures | failure_threshold counts consecutive downstream exceptions | Calls can still consume their full execution timeout |
OPEN | Reject calls without invoking the downstream function | CircuitBreakerOpenError; wait for cooldown_period | Fast failure or an explicit degraded result |
HALF_OPEN | Admit a probe; reject concurrent calls while it runs | probe_timeout and half_open_success_threshold | Recovery traffic is limited, but useful calls may still be rejected |
In the v1.9.0 implementation, failure_threshold defaults to 3, cooldown_period to 60.0 seconds, and half_open_success_threshold to 1. These values must be greater than zero. probe_timeout defaults to None; when configured, it must also be greater than zero. They are component defaults, not recommended values for every service.
While closed, a successful invocation resets the consecutive-failure count. After the threshold is reached, the breaker opens. After the cooldown, an arriving call can become the half-open probe. Consecutive successful probes close the breaker once the recovery threshold is met. A probe failure or timeout reopens it and restarts the cooldown. Expiry of the cooldown therefore permits a test; it does not prove recovery.
Attach the breaker at the dependency boundary
The toolkit defines components under middleware and attaches an ordered list to a function’s middleware field. Configure the breaker around the tool that calls the unreliable service rather than around an entire agent that also uses healthy tools. The tool remains responsible for translating business rejections into domain results; see 2.4.1 for that classification.
The v1.9.0 implementation counts downstream Exception failures rather than inspecting HTTP status codes. An error hidden inside a returned string looks like successful completion to the breaker. Conversely, repeatedly raising expected business rejections can open it even when the service is healthy. Classify at the adapter boundary so the breaker measures the condition you intend to isolate.
Middleware order changes what counts as an invocation. If custom retry middleware wraps a breaker, each admitted downstream attempt can affect its counters. If a breaker wraps the retry layer, it observes the final result after that layer finishes. Draw the actual invocation order and select the measurement unit: physical dependency attempt or logical tool operation. Do not use the same threshold while silently changing that unit.
Bound probes and ordinary calls separately
probe_timeout limits probe execution; it is not a replacement for an ordinary tool timeout. The toolkit’s _type: timeout middleware bounds intercepted calls, with timeout expressed in seconds. Position it so the work that needs a deadline lies inside its scope. A closed breaker still admits work that can hang unless another layer bounds execution.
Streaming also needs care. The v1.9.0 implementation applies a configured probe timeout to each downstream __anext__ wait during a streaming probe, excluding time spent in the consumer between yields. Use timeout middleware for the whole-stream deadline rather than interpreting probe_timeout as that deadline. Successful early chunks alone do not establish that the stream completed.
Cancellation is separate from dependency failure. The circuit-breaker documentation states that asyncio.CancelledError during half-open probing releases probe status without counting a failure or changing breaker state. This lets another call test recovery without treating caller abandonment as a dependency outage.
Match state scope to the failing resource
The v1.9.0 implementation keeps an in-memory state dictionary on the middleware instance, keyed by context.name. Different target names have separate state. Identically named functions exposed by different components can collide within that scope; the implementation explicitly notes this limitation. Give distinct dependency targets distinct names rather than assuming component identity is part of the key.
This also means separate middleware instances in serving replicas have separate counters and probes. One half-open probe per instance is not one probe across a four-replica deployment. Account for the aggregate recovery load, and make the platform owner responsible for any wider admission policy. A local breaker does not create a distributed coordinator.
Key distinctions and trade-offs
A consecutive-failure threshold differs from a failure count within a time window. Intermittent successes reset the v1.9.0 implementation’s closed-state counter. If the business needs a time-window failure-rate policy, do not describe failure_threshold as implementing it.
A breaker protects capacity by refusing work; it does not repair credentials, add GPU capacity, reverse writes, or validate a fallback answer. Raising the threshold improves tolerance of isolated failures but admits more work during an outage. A longer cooldown reduces probes but delays recovery detection. More required probe successes increase confidence while extending degraded service.
Keep CircuitBreakerOpenError out of an immediate retry loop. Route it to an approved fallback or an explicit unavailable outcome. Neither path should re-enter the same failing target under another tool name merely to bypass its admission decision.
Architecture / decision scenario
A manufacturing research assistant calls a parts-availability service at 80 logical requests per second. During an outage, a proposed three-attempt policy can offer up to 240 dependency attempts per second. The assistant’s other tools remain healthy, and the platform team wants availability lookups to fail promptly while production engineers continue reading maintenance documents.
The workflow owner attaches v1.9.0 circuit-breaker middleware to the availability tool with failure_threshold: 5, cooldown_period: 30.0, half_open_success_threshold: 3, and probe_timeout: 2.0. Ordinary calls also have a two-second execution timeout, attached in the order middleware: ["availability_breaker", "availability_timeout"]. The timeout runs inside the breaker so a timed-out call raises an exception that counts toward the five-failure threshold. These are selected deployment values. The owner rejects retry-only recovery because every arriving request would receive its own retry budget during the same sustained outage.
After five consecutive failures, the breaker opens. After 30 seconds, the next admitted probe tests the dependency. Three consecutive successful probes restore ordinary admission; one failure reopens it. While open, the assistant reports availability as unavailable and can still search maintenance documents. It never invents stock counts from cached prose.
The platform runs four serving replicas, so the operator plans for up to four simultaneous probes from their independent middleware instances. The service owner fixes the underlying outage; the workflow owner owns admission settings and verifies that restored traffic does not immediately retrip the breakers.
Common exam traps
- Treating open-state rejection as permission for immediate retries.
- Assuming a cooldown timer proves that the dependency recovered.
- Reading a consecutive-failure threshold as a rolling error percentage.
- Assuming
probe_timeoutbounds ordinary calls or an entire streaming response. - Expecting one middleware instance’s state to coordinate every serving replica.
- Returning error-shaped strings and expecting an exception-counting breaker to trip.
Where the exam guide differs
The study guide recommends “Circuit Breaker Pattern — Azure Architecture Center.” That reading uses a time-based closed-state failure counter that resets periodically: a threshold is reached by failures within a specified interval. The NeMo Agent Toolkit v1.9.0 component instead counts consecutive exceptions and resets the count on a successful ordinary call.
For an exam item about the general pattern, use the suggested reading’s terms: Closed, Open, and Half-Open; a failure threshold within an interval; and a time-out timer before probing recovery. For a question naming the toolkit component’s failure_threshold, use its consecutive-failure semantics rather than substituting a time-window policy.
What to remember
Use a breaker to decide whether work is admitted, then use execution timeouts and retry budgets inside that decision. Check the release, the counted failure type, the target key, and the replica scope. Restore traffic through successful bounded probes rather than timer expiry alone.
Knowledge check
A NeMo Agent Toolkit v1.9.0 circuit_breaker middleware wraps a parts-availability tool. During an outage, the tool’s adapter catches each connection error and returns the string “Service error: upstream unreachable”, and the breaker stays CLOSED. What should change?
failure_threshold to 1 so a single error trips itcooldown_period so that recovery gets tested soonerAn operator wants the breaker to open when failures dominate a minute even though successes are interleaved. They configure the v1.9.0 component with failure_threshold: 30 because about 30 calls fail per minute during these incidents. Why can the breaker still stay closed?
failure_threshold counts consecutive failures, and a successful ordinary call resets it, so it is not a failure rate within a time window. The cooldown governs how long the breaker stays open, the component counts exceptions rather than status codes, and its state lives in each middleware instance.The assistant runs four serving replicas, each with its own v1.9.0 breaker instance on the availability tool. The dependency owner wants at most one recovery probe at a time across the deployment. Which approach meets this?
half_open_success_threshold: 1 on every replica’s middlewareprobe_timeout so that each recovery probe ends soonercontext.namecontext.name separates targets within one instance rather than sharing state between replicas.Ordinary availability calls need a two-second execution limit, and a call that times out must count toward the breaker’s failure threshold. Which middleware list on the tool does that?
["availability_breaker", "availability_timeout"]["availability_timeout", "availability_breaker"]["availability_breaker"] with probe_timeout: 2.0TimeoutError where the breaker observes it as a downstream failure. With the timeout outside, its timer also covers the breaker and the expiry surfaces outside it rather than as a failure the breaker records, and probe_timeout limits only half-open probes, leaving ordinary calls unbounded.While the availability breaker is open, users need current stock counts, and the assistant also searches healthy maintenance documents. No approved alternate source can supply a live count. How should the assistant handle availability requests?
CircuitBreakerOpenError to an approved fallback or an explicit unavailable outcome. Retrying or re-entering the same service under another name bypasses that decision, and even labeled stale counts do not meet the live-count requirement.References
- NVIDIA, Middleware, v1.9.0
- NVIDIA, Circuit breaker middleware implementation, v1.9.0
- NVIDIA, Middleware
- Microsoft, Circuit Breaker pattern
- NVIDIA, NVIDIA-Certified Professional: Agentic AI Exam Study Guide