NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.1 Deploy and orchestrate multi-agent systems at production scale.
Split services by execution and ownership
Completed by the knowledge check
Learning objective
Separate agent runtimes, orchestration services, and model-serving endpoints into deployable services.
Core idea
A specialist agent is a logical role; a deployable service is an operational boundary. Put a role behind its own service when it needs independent release, permissions, resources, or failure handling. Do not turn every prompt into a network hop.
Distinguish three responsibilities: an agent runtime executes a specialist’s reasoning and tools; an orchestration service coordinates the overall request; a model-serving endpoint performs inference. The runtime and orchestrator can use the same framework, but neither should require a private model container merely because it has a separate process. See 1.5.1 for specialist topologies and 1.8.1 for replaceable component interfaces. Here the decision is which component becomes independently deployable and who keeps it operating.
How it works
Distinguish configuration components from services
NeMo Agent Toolkit declares callable components under functions, model clients under llms, and the entry point under workflow. A workflow can select an agent with _type: react_agent, reference tool_names, and select an LLM with llm_name. Those declarations describe composition; their presence in one configuration does not establish separate deployed services.
To expose a workflow as a service, nat serve --config_file CONFIG_PATH serves a FastAPI application for that configuration. CONFIG_PATH is the deployed workflow configuration. A specialist can remain an in-process function when its lifecycle and access requirements match its caller. If you deploy it separately, define the network request and response that its calling adapter handles, rather than assuming an in-process invocation retains its failure behavior over HTTP.
The API server documents /v1/workflow and legacy /generate for non-streaming workflow execution. Its versioned workflow paths are marked experimental because of HTTP human-in-the-loop and OAuth support; the page states one-to-one compatibility with legacy routes for workflows that do not rely on those behaviors. Choose and test the interface deliberately. A reachable route establishes transport, not permission to execute a business action.
Keep model inference behind its own address
For a NIM model client, llms.<name>._type: nim and base_url select the inference integration and target. The agent’s llm_name refers to that configured client. Several agent services can therefore address a shared model endpoint rather than each embedding a model-serving process.
NIM for LLM and VLM documents POST /v1/chat/completions for chat inference. This request is different from asking a workflow service to complete a task: the workflow can select tools and coordinate several inference calls, while the model endpoint receives an inference request. Even when both services expose compatible chat routes, the target address and deployed handler determine what executes.
Make the model endpoint an independently operated deployment when several workflows share its capacity. The model-serving owner controls that deployment’s readiness and capacity; each workflow owner controls its call volume and task behavior. Adding agent replicas increases potential inference demand. It does not, by itself, add model-serving capacity. Several workflows share one model endpoint that the model-serving owner operates.
Give each service a stable address and meaningful health
See 0.8 for the Kubernetes Deployment and Service baseline. Give the orchestrator and separately deployed specialist independent Deployments and Service addresses when their scaling or release requirements differ.
For inference managed by NVIDIA NIM Operator, declare a NIMService custom resource (nimservices.apps.nvidia.com). The inference team sets spec.replicas and GPU allocation under spec.resources.limits using nvidia.com/gpu. Alternatively, spec.scale.enabled: true creates a horizontal pod autoscaler configured through spec.scale.hpa; this mode cannot also define spec.replicas. Choose fixed replicas or autoscaling explicitly rather than letting application replica counts determine inference capacity.
NIM distinguishes GET /v1/health/live from GET /v1/health/ready. Liveness returns 200 when the container is running; readiness returns 200 when the model is loaded and inference is available. Connect these meanings to Kubernetes livenessProbe and readinessProbe, respectively. A failed readiness probe removes the Pod from Service traffic; a failed liveness probe can restart the container.
NIM Operator enables spec.readinessProbe and spec.startupProbe by default as HTTP GET probes to /v1/health/ready. Its startup probe uses a failure threshold of 120 and a ten-second period, documented as allowing up to 20 minutes for startup; all other probes remain disabled until startup succeeds. Adjust spec.startupProbe.probe from measured initialization time instead of repeatedly killing a healthy process before it can load. The agent runtime needs its own health check; a successful model probe does not prove that its tool adapters or workflow configuration work.
Key distinctions and trade-offs
Use the responsibility that must change independently to choose the service boundary:
| Role | Deciding condition | Concrete mechanism | Cost and owner |
|---|---|---|---|
| Agent runtime | A specialist needs separate tool dependencies or operational access | A workflow served with nat serve; its configured functions and llms | Network and serialization cost; specialist team owns execution |
| Orchestration service | End-to-end routing and completion must evolve separately | Its own workflow entry point and service interface | Coordination latency and another deployment; application team owns completion |
| Model-serving endpoint | Multiple workflows share inference resources | NIM base_url and /v1/chat/completions; Operator NIMService with spec.replicas | Shared capacity contention; inference team owns serving |
Separate deployment buys independent releases and resource allocation, but adds network failures and more operational surfaces. Co-location avoids a hop when the same team owns the same lifecycle. Separating model serving is especially valuable when changing a tool adapter should not restart inference, or when a model deployment must serve several applications.
A Service address also does not make orchestration state durable. Replacing a Pod can remove in-memory work even though callers keep using the same address. See 4.1.3 for distributed task recovery; do not infer recovery from replica count.
Architecture / decision scenario
An insurer runs a customer assistant with policy lookup and claim-status specialists. Its proposed design bundles the router, both specialists, and a model server in one deployment. During load tests, the application needs eight CPU-side runtime replicas, while its approved GPU budget supports two model-serving replicas. Claim-status adapters change weekly; the inference team validates model releases monthly.
The tempting proposal is to duplicate the entire bundle eight times. That couples the CPU scaling requirement to eight model replicas, exceeding the two-replica serving budget. The application architect instead separates the orchestration entry point and the claim-status runtime from the shared NIM endpoint. Both callers select the model through their configured llms client and base_url. The policy lookup remains an in-process function because it shares the orchestrator’s release and access requirements.
The platform team gives the orchestration and specialist runtimes independent Deployments and Services. The inference team declares its NIMService with spec.replicas: 2, using the Operator’s readiness and startup probes against /v1/health/ready. A model container that has started but has not loaded its model receives no Service traffic. The application team owns the eight runtime replicas and verifies their aggregate demand against the two model replicas’ measured inference capacity.
The split satisfies the release and GPU constraints, but it creates a shared failure dependency. The application team must bound model calls and return an explicit unavailable outcome when serving capacity is exhausted, using the failure controls in 2.4. The split is justified by independent ownership and resource needs, not by the number of named agents.
Common exam traps
- Treating every entry under
functionsas a separately deployed service. - Equating a specialist role with a required container or model replica.
- Scaling GPU model servers whenever CPU agent replicas increase.
- Treating a running NIM container as ready for inference.
- Using a Service address as proof that in-flight workflow state survives replacement.
- Assuming a chat-compatible route identifies whether the target runs a workflow or only inference.
What to remember
Separate agent execution, end-to-end coordination, and model inference by the resources and ownership they require. Use explicit workflow interfaces, configured model addresses, independent runtime Deployments, an inference-owned NIMService, and readiness checks. Keep specialists together when their lifecycles match; split them when an operational boundary earns the network and deployment cost.
Knowledge check
A support assistant bundles its router, two specialists and a NIM model server in one Deployment. Load tests need ten CPU-side runtime replicas, but the approved GPU budget covers three model-serving replicas. Which change meets both constraints?
base_urlllms client and base_url . Duplicating the bundle or giving each specialist a model container ties model replicas to the runtime count, which the GPU budget rules out. More runtime replicas still raise inference demand, so the team checks that demand against the three replicas’ measured capacity.A team deploys NIM with plain Kubernetes manifests. Right after a rollout, requests routed to new Pods fail while the model is still loading, although the containers are running. The team wants a new Pod to receive Service traffic only once inference is available. Which probe does that?
readinessProbe on /v1/health/readyreadinessProbe on /v1/health/livelivenessProbe on /v1/health/readylivenessProbe on /v1/health/live/v1/health/ready returns 200 only when the model is loaded and inference is available. /v1/health/live returns 200 as soon as the container runs, so it lets traffic in during loading. A liveness probe controls restarts rather than traffic eligibility; repeated failures on the ready route can restart a container that is still loading.A NIMService managed by NIM Operator keeps the Operator’s default probes. After the team moves to a larger model whose measured load time is about 25 minutes, its Pods are restarted repeatedly during startup, and the logs show the model was still loading each time. What should the inference team change?
spec.readinessProbe on the NIMService while it loadsspec.startupProbe.probe for the measured load timespec.replicas on the NIMServicespec.scale with an HPA that sets minReplicas: 2In a claims assistant, the orchestrator calls a document-lookup specialist that is released together with the orchestrator, uses the same credentials and resources, and needs no independent failure handling. A reviewer proposes giving it a separate Deployment because it is a distinct agent role. Which design meets these requirements with the least deployment and network overhead?
nat serve behind its own Deployment and ServiceReferences
- NVIDIA, Workflow Configuration
- NVIDIA, Command Line Interface
- NVIDIA, NVIDIA NeMo Agent Toolkit API Server Endpoints
- NVIDIA, API Reference
- NVIDIA, Managing NIM Services
- Kubernetes, Deployments
- Kubernetes, Service
- Kubernetes, Configure Liveness, Readiness and Startup Probes