Middleware

Why it matters

Timeouts, caching and input checks belong around a call, not inside every tool. In NeMo Agent Toolkit that is middleware, and its order shapes each failure.

What to practise

Declare components under middleware and attach them to a function or function group by name. Build a custom middleware to follow its four phases: preprocess, call next, postprocess, continue. Configure the built-in timeout middleware with timeout, timeout_message and register_llms for LLM calls, and note that it times a stream as a whole, not per chunk. Set the cache middleware’s enabled_mode and similarity_threshold, and see where the guardrails middleware runs NeMo Guardrails input and output rails at the function boundary. Note that a timeout times only the chain after its own position, and that an is_final=True middleware ends the chain without calling the function.

How it fits

Set the timeout against 2.4.2’s deadline budget and let 2.4.1 decide who sees the TimeoutError. Pair it with 2.4.3’s circuit breaker, which rejects calls to a failing dependency instead of waiting out each timeout.