Scale replicas without copying session ownership

Learning objective

Scale out with stateless serving, externalized state, and lazy tool and skill loading.

Core idea

Stateless serving means a replica does not exclusively own the durable information needed for the next request. The assistant can still have conversation history, workflow checkpoints, and long-term memory; those records live behind shared interfaces rather than only inside one serving process.

Lazy loading addresses a different constraint. It defers expensive tool construction or detailed skill text until needed. That can reduce startup work or prompt size, but it does not externalize conversation state or increase model-serving capacity. A scalable design identifies which resource each change actually saves.

See 1.4.1 for memory categories and 1.6.3 for state graphs. This lesson adds the deployment question: what must remain available when a different replica handles the next request?

How it works

Make serving replicas replaceable

Kubernetes Deployment .spec.replicas controls the desired number of replicated Pods. A Service provides a stable access point for selected Pods. These mechanisms let the deployment replace or add serving processes without clients naming individual Pod addresses.

Continuity requires the next process to load the thread’s records. Increasing .spec.replicas multiplies workers, not private history. Keep dependency configuration consistent across replicas (see 1.8.1), and treat process-local caches as disposable.

For nat serve async jobs, requiring nvidia-nat[async_endpoints], the defaults create a temporary SQLite database and a local Dask cluster. Recall persistent history in 1.1.1: for several replicas, configure the same shared --db_url and --scheduler_address on each. Otherwise a status request for a job_id can reach a replica whose local history never recorded that job.

Externalize execution state separately from memory

For a LangGraph workflow, compile the graph with a checkpointer and pass configurable.thread_id on invocation. The checkpointer uses that identifier to store and retrieve thread checkpoints. A serving replica can then address the same persisted execution state instead of relying on the previous process’s Python objects.

A different replica can handle the next request because it retrieves the thread checkpoints the checkpointer stored.

LangGraph documents InMemorySaver for experimentation and the separately installed langgraph-checkpoint-postgres package for production use, exposing PostgresSaver and AsyncPostgresSaver. Choose the async implementation when executing the graph asynchronously. Configure replicas against the same intended backend; separate in-memory savers remain separate histories even when the thread identifier matches.

This framework can run through NeMo Agent Toolkit’s workflow._type: langgraph_wrapper, with graph identifying the graph definition and nvidia-nat-langchain installed. The wrapper integrates the graph into the toolkit; the graph’s checkpointer still determines its execution persistence. Do not assume that selecting the wrapper creates a durable checkpoint database.

Checkpoint durability also costs time. LangGraph’s durability="sync" writes checkpoints before the next step begins, adding persistence overhead. "async" persists while the next step executes, with a documented risk of missing writes if the process crashes. "exit" persists only when execution exits and cannot recover a mid-run process crash. Choose the mode according to the recovery requirement.

Long-term memories remain a distinct concern. NeMo Agent Toolkit’s top-level memory configuration selects a provider, and its memory subsystem includes Redis through the nvidia-nat-redis plugin. That interface serves memory operations; it should not be mistaken for the LangGraph checkpoint interface. An assistant can recover preference records while still losing its unfinished execution if only the memory backend is durable.

Choose shared or per-user construction

NVIDIA’s custom-function guidance places implementation classes or imports inside the registered construction coroutine, avoiding that work for unused components. Shared functions are built once at startup and reused across users; this import pattern does not defer every configured shared tool until invocation.

For user-isolated instances, @nat.plugin_api.register_per_user_function requires explicit input and output schemas and instantiates the function lazily when that user’s first request arrives. A shared workflow cannot depend on per-user functions; register the containing workflow as per-user too. Recall authenticated user scope in 1.4.2. Lazy creation adds initialization latency to a user’s first request and memory for each active user’s instance.

Inactive instances are retained for per_user_workflow_timeout, default 30 minutes, with cleanup checks every per_user_workflow_cleanup_interval, default 5 minutes. enable_per_user_monitoring: true exposes /monitor/users for resource monitoring. These are serving-process instances, so keep durable user data in shared storage rather than relying on instance retention across replica replacement.

Disclose skills progressively

LangChain Deep Agents loads skills in three levels: every configured skill’s name and description from SKILL.md frontmatter enter the system prompt at startup; the full instructions load when invoked; supporting files in scripts/, references/, and assets/ stay on the backend until the agent reads them. This reduces routine prompt content while retaining discoverable capabilities.

Configure create_deep_agent with backend=... and skills=["./my-project/skills/"]. The default StateBackend stores files in the current thread’s LangGraph state; FilesystemBackend uses disk beneath root_dir; StoreBackend uses a LangGraph store for durable cross-thread files. Select shared state persistence, consistent replica filesystem contents, or a shared store according to the backend. The backend’s name alone does not make all replicas see the same files.

NVIDIA’s langgraph_wrapper documentation uses create_deep_agent as its example agent, connecting this framework mechanism to the toolkit integration. Skills loading remains Deep Agents behavior, rather than a toolkit-specific YAML switch. Measure instruction and resource reads on the task path as well as startup prompt size.

Key distinctions and trade-offs

Use the following comparison to locate each scaling cost:

Replica scaling and deferred loading are complementary, but neither fixes every bottleneck. If inference is saturated, more assistant replicas can increase its waiting traffic. Apply the aggregate capacity controls in 1.8.2 before multiplying callers.

Architecture / decision scenario

A manufacturer runs an employee maintenance assistant on two application replicas. A load test finds that each replica handles 15 requests per second within the response target, while the next shift requires 45. The existing inference and persistence services have measured headroom for 60 requests per second. Ten specialist skills exist, but only 10% of requests need their full procedures.

The tempting proposal is to add a third replica and keep each conversation in the process that first received it. That reaches a nominal 45 requests per second but leaves no application headroom, and replacing a process loses its private conversation state. Eagerly inserting all ten procedures into every prompt also spends context on the other 90% of requests.

The platform owner selects four serving replicas for nominal capacity of 60, using a Deployment and Service. The workflow owner configures a shared AsyncPostgresSaver and preserves configurable.thread_id across turns. For this workflow, the recovery requirement warrants durability="sync"; the team repeats the capacity measurement with that persistence cost included rather than assuming the original 15-per-replica figure remains valid. Long-term equipment preferences use the configured memory provider separately.

For background jobs, the platform owner also shares --db_url and --scheduler_address across replicas. The component owner chooses per-user lazy functions only where isolated instances are needed, budgets their first-request latency and retained memory, and externalizes durable records. The assistant owner configures Deep Agents skills with a StoreBackend connected to the shared store, so every replica can read the same procedures progressively. The team measures first-use latency and completed answers as well as routine prompt size. If the new aggregate exceeds the 60-request downstream envelope, it limits admitted work rather than adding replicas indefinitely.

Common exam traps

• Interpreting stateless serving as an assistant with no memory.
• Assuming replicas or a Service replicate process-local conversation history.
• Persisting long-term memory and forgetting unfinished workflow checkpoints.
• Assuming supporting skill resources load at startup; Deep Agents reads them only as needed.
• Treating temporary SQLite history, a local Dask cluster, or per-user instances as shared replica state.
• Assuming deferred imports remove tool schemas or provide additional inference capacity.

What to remember

Scale the serving layer around shared durable state, and name the state system that owns continuity. Defer unused component work and detailed skill instructions at their respective boundaries. Measure the persistence overhead and first-use latency, then keep aggregate downstream demand within capacity.

Knowledge check

References

• Kubernetes, Deployments
• Kubernetes, Service
• LangChain, Checkpointers
• NVIDIA, Running Existing LangGraph Agents in NVIDIA NeMo Agent Toolkit
• NVIDIA, Memory in NVIDIA NeMo Agent Toolkit
• NVIDIA, Writing Custom Functions
• LangChain, Skills
• NVIDIA, Per-User Functions
• NVIDIA, API Server Endpoints