NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.4 Scale deployments using containerization (Docker, Kubernetes) with load balancing.

Scale workers against useful serving capacity

Completed by the knowledge check

Learning objective

Autoscale model and agent workers on custom inference metrics, managing cold-start cost and state affinity.

Core idea

Autoscaling changes a desired worker count before it changes usable capacity. A model replica still needs hardware, model initialization, readiness, and useful cache state. An agent replica might instead be waiting on that same model. Scaling both tiers from the same rising latency can consume more resources without increasing completed work.

Choose the pool, pressure signal, and readiness delay together. Kubernetes HorizontalPodAutoscaler (HPA) supports custom metrics; NVIDIA Dynamo Planner adds inference-specific model scaling. Both need warm-capacity and worker-removal planning. See 4.4.1 for GPU admission and 4.4.2 for routing.

How it works

Connect a metric to the correct pool

An autoscaling/v2 HPA uses spec.scaleTargetRef to identify its target and spec.metrics to describe its scaling inputs. CPU and memory come through the resource metrics API. A per-Pod custom metric uses custom.metrics.k8s.io; an external signal uses external.metrics.k8s.io. The metrics pipeline needs a suitable adapter API server. Exposing a Prometheus endpoint alone does not make that metric available to HPA.

For a per-Pod queue signal, use a Pods metric with a target of type AverageValue. Give the adapter’s published metric an explicit name and unit in the deployment specification. For example, a team can publish inference_queue_requests as its adapter mapping of Dynamo’s worker-side dynamo_request_queue, a gauge of requests queued in Dynamo but not yet in the engine. The adapter name is team-defined. Worker metrics require DYN_SYSTEM_PORT; exposure is disabled by default, and the Kubernetes operator typically sets 9090.

In the simplified HPA calculation, desired replicas equal the ceiling of current replicas multiplied by observed metric divided by target metric. Three replicas averaging 16 queued requests against a target of 8 suggest six replicas. That arithmetic assumes usable measurements; HPA also accounts for missing metrics, readiness, tolerance, and configured bounds. Six desired replicas are not six ready replicas.

Scale agents separately with an application metric such as agent_pending_runs, counting runnable work awaiting an agent slot. Expose it through the custom or external metrics API. Exclude runs blocked on a saturated model: more callers cannot relieve that bottleneck. The workflow owner defines inclusion rules; the platform owner verifies metric mapping and HPA behavior.

Use inference-aware control where it earns its cost

Dynamo Planner changes prefill and decode engine replica counts using traffic signals and engine performance models. Its PlannerConfig goes under spec.features.planner when configured through a DynamoGraphDeploymentRequest. This is a controller for the model-serving pool, distinct from an HPA for application workers.

Choose optimization_target deliberately. throughput uses static queue-depth and KV-utilization thresholds without SLA targets or profiling. load exposes user-defined thresholds; latency uses aggressive low-latency thresholds. sla uses the engine-query performance model to target ttft_ms and itl_ms, representing time to first token and inter-token latency. For the first three targets, the documented TTFT and ITL fields are ignored. Setting a latency number without choosing sla therefore does not configure an SLA-driven controller.

Set the minimum footprint and resource ceiling explicitly. min_endpoint sets the minimum in agg mode and applies to both prefill and decode in disagg mode; prefill_min_endpoint and decode_min_endpoint override the corresponding roles and must each be at least 1 when specified. max_gpu_budget caps the total GPU allocation the Planner may request. The Planner rejects a minimum footprint exceeding max_gpu_budget at startup. A GPU budget cannot make an unavailable GPU schedulable.

First set advisory: true to report recommendations without scaling. Compare dynamo_planner_predicted_num_prefill_replicas and dynamo_planner_predicted_num_decode_replicas with readiness and achieved latency. In sla mode, insufficient bootstrap data can produce model_not_ready until native estimates or enough observed Forward Pass Metrics are available. Distinguish an unready performance model from a stable recommendation.

Budget initialization and avoid replica churn

Measure decision-to-ready time, including scheduling and model loading. Existing replicas carry traffic during initialization. Reactive scaling can arrive after a short burst while still adding cost; retain a warm minimum when startup exceeds the burst duration.

Reduce loading time before buying a larger warm floor. On Dynamo vLLM workers, --load-format runai_streamer enables concurrent weight reads; on Lustre, stripe model files across storage targets as well. NVIDIA documents these as complementary fixes for loader access and filesystem bandwidth. runai_streamer is a vLLM load format; SGLang and TensorRT-LLM benefit from filesystem striping without that flag. Measure the resulting initialization time for your deployment rather than assuming the documented case study’s speedup.

With HPA, configure spec.minReplicas, spec.maxReplicas, and spec.behavior.scaleDown.stabilizationWindowSeconds. A stabilization window prevents immediate removal after a brief decrease; scaling policies constrain the rate of change. Keeping replicas warm costs idle GPU time but avoids repeatedly paying startup and cache-warming costs. Compare ready capacity and useful completed work, rather than reporting success from desired replica count alone.

Preserve affinity without making it durable state

Dynamo’s --router-session-affinity-ttl-secs enables session-affinity synchronization across router replicas, but the operations guide describes it as best effort. Each replica owns an idle timer and validates targets against its local worker membership. Dropped updates and long requests can desynchronize those timers. Strict affinity requires ingress stickiness or authoritative storage; affinity is not a durable workflow checkpoint.

Removing a model worker can discard useful cache locality. Removing an agent worker can interrupt application execution, so use the replica-state design in 1.8.3 and persisted-progress recovery in 4.1.3 rather than treating a sticky route as persistence. For ordinary Kubernetes termination, implement graceful handling in the application and budget terminationGracePeriodSeconds; a preStop hook, when used, consumes that same grace period. The kubelet forcibly terminates remaining processes after the grace period. A longer timer only helps when the worker actually stops admitting work and completes or durably records its in-flight operations.

Key distinctions and trade-offs

Choose the control mechanism by what it can measure and change:

Mechanism	Best fit	Required input	Main cost or limitation
HPA custom metric	Application or model pool with a measured pressure target	Adapter-backed `Pods`, object, or external metric	Metric mapping and target calibration belong to the team
Planner threshold target	Queue/KV pressure drives model capacity	`optimization_target: throughput` or explicit thresholds	TTFT/ITL target fields do not control these modes
Planner SLA target	Prefill/decode capacity must meet latency targets	`optimization_target: sla` and a ready performance model	Model bootstrap and workload fit affect predictions
Warm minimum plus slow scale-down	Startup or cache recovery is costly	Minimum replicas/endpoints and removal policy	Idle resource cost buys faster useful capacity

Assign one scaling authority per replica count. An application HPA and model Planner can control different pools without competing for the same count.

Architecture / decision scenario

A media company’s meeting-summary workflow faces a scheduled morning surge. Scenario measurements show model startup takes 180 seconds, while the surge lasts 120 seconds. Four ready model replicas sustain 40 summaries per minute; demand reaches 60. Agent CPU remains at 25%, and traces show agent runs waiting on inference.

Adding agent workers on that wait time is the tempting alternative. It increases callers competing for the same serving capacity. The workflow owner instead leaves agent scaling tied to runnable agent backlog and gives the model-serving owner the inference-pressure decision.

The serving owner first evaluates Planner in advisory mode, then sets mode: agg and selects sla with scenario targets of ttft_ms: 800 and itl_ms: 60, min_endpoint: 6, and max_gpu_budget: 8 on a fixed eight-GPU cluster. The unnamed model configuration is validated at one GPU per endpoint. These are workload choices, not product defaults. The team keeps six endpoints permanently warm: two extra GPU allocations above the four-replica baseline buy immediate surge capacity. During scale-down from any additional replicas, the team drains in-flight summaries and accepts cache loss only after durable run state is safe. Session affinity improves locality but does not justify deleting the only copy of a run’s progress.

Common exam traps

Assuming Prometheus exposition alone supplies HPA custom metrics.
Scaling agent callers because the downstream model queue grows.
Configuring TTFT/ITL targets while leaving a threshold optimization mode selected.
Counting desired replicas as ready model capacity.
Expecting cold scale-out to absorb a burst shorter than initialization.
Treating affinity or a termination timer as durable workflow state.

What to remember

Scale each pool from pressure it can relieve. Connect custom metrics through the Kubernetes metrics APIs, or use Planner with an explicit optimization target. Budget startup before the burst, bound allocation, and remove workers through graceful execution handling rather than relying on cache locality or sticky sessions.