NVIDIANVIDIA-Certified Professional: Agentic AI2026-104.5 Optimize deployment costs while ensuring high availability.
Size replicas against measured latency targets
Completed by the knowledge check
Learning objective
Size GPU capacity from target users, latency SLOs, and throughput.
Core idea
A user count is a demand input, not a GPU specification. You need the rate of user tasks, the inference calls each task produces, their token lengths, and the latency the application must deliver. Then measure how much of that workload one serving replica can serve within its service level objectives (SLOs).
Distinguish three quantities: offered requests per second, completed requests per second, and completed requests that satisfy the latency constraints. NVIDIA AIPerf calls the third quantity goodput. A configuration that increases total throughput while pushing most requests outside their deadlines is not additional usable capacity.
This lesson turns demand into a capacity estimate. See 4.5.2 for reducing serving cost and 4.5.3 for the extra capacity needed during failures. Keep those additions visible rather than hiding them inside an unexplained utilization percentage.
How it works
Translate users into a representative load
Start with a workload sheet owned by the application team: active users during the design peak, tasks per user per minute, inference calls per task, input and output token distributions, and burst duration. An agent that performs three sequential calls has different latency behavior from one that launches three calls together, even when both consume the same request count.
For an illustrative average, 600 active users submitting one task every 120 seconds produce five tasks per second. At three model calls per task, the model tier receives 15 requests per second. This calculation estimates demand; it does not establish the concurrency, GPU count, or response time of the deployment.
Preserve long-context and long-output cases in the load. Averaging a short classification and a lengthy report into one artificial prompt hides the requests most likely to exhaust memory or occupy decode capacity. Record tool waits separately: adding GPUs cannot repair an external search service that consumes most of the end-to-end deadline.
Establish a deployable unit
Use NIM’s list-model-profiles for the exact image being evaluated. A model profile includes backend, precision, tensor parallelism (TP), and pipeline parallelism (PP). Set NIM_MODEL_PROFILE to the returned profile ID or complete description when you need the benchmark and deployment to select the same configuration. See 3.4.2 for choosing precision; here the profile determines the unit whose capacity you measure.
NIM distinguishes Compatible, Low memory, and Incompatible profiles. Low memory means weights fit but the full context length exceeds available memory; the listing suggests a reduced --max-model-len. Incompatible means even the weights do not fit, for which higher tensor parallelism or a quantized precision can be alternatives. Reducing context is acceptable only if it still accommodates the application’s required inputs and outputs.
A multi-GPU profile is one serving replica, not several independently available servers. Count its entire TP/PP allocation when converting replica counts into GPUs. A successful container startup establishes neither representative-load memory safety nor sufficient throughput.
Estimate capacity from the whole workflow
NeMo Agent Toolkit’s nat sizing calc estimates cluster size from workflow performance at different concurrency levels. It requires profiler support (nvidia-nat[profiler]) and assumes an isolated GPU cluster; public endpoint variability can make its estimates inaccurate.
Collect measurements with --config_file, --calc_output_dir, a comma-separated --concurrencies range, and --num_passes. NVIDIA recommends at least ten concurrency values for a robust slope-based fit. The output separates P95 LLM Latency from P95 WF Runtime, both in seconds: the workflow runtime includes the orchestration and tool waits that endpoint measurements miss.
Reuse those measurements with --offline_mode, the same --calc_output_dir, --test_gpu_count, --target_workflow_runtime, and --target_users. The test count is the GPUs used during measurement, not the requested cluster size; the runtime target is p95 seconds. An optional --target_llm_latency adds a p95 model-latency target. The JSON output reports gpu_estimate_by_wf_runtime and gpu_estimate_by_llm_latency when their targets apply.
Inspect failed passes before accepting an estimate. A concurrency flagged workflow_interrupted: true is excluded, so an estimate does not demonstrate success at that load. Tool waits also remain part of workflow runtime: a fitted GPU estimate is a planning input that needs a deployment-level load test, not a promise that GPUs accelerate every workflow step.
Find the useful endpoint operating point
Use AIPerf to measure the model tier with the production prompt and output mix; see 4.3.1 for how AIPerf counts good requests. Inspect goodput and good_request_fraction, then retain a sustainable offered rate whose latency attainment and errors satisfy the requirement. This is the per-replica capacity used in the demand calculation.
Pair that endpoint rate with the calculator’s workflow estimate. Neither alone establishes both the model tier’s capacity and the complete task’s response time. Repeat the test at the proposed replica count to expose shared bottlenecks and uneven distribution.
The per-replica capacity and the workflow estimate come from separate measurements that meet in the test at the proposed replica count.
Key distinctions and trade-offs
The sizing inputs answer different questions:
| Input or measurement | What it establishes | What it does not establish |
|---|---|---|
| Target active users and task frequency | Offered business demand | Model calls or GPU count without workflow fan-out |
| Token-length distributions | Representative prefill and decode work | A universal requests-per-GPU rate |
| NIM profile compatibility | A feasible model configuration on the hardware | Required latency under concurrent traffic |
nat sizing calc | Workflow-level GPU estimates against p95 runtime and user targets | Guaranteed capacity when failed concurrency passes are excluded |
| Request throughput | Completed request rate | Compliance with latency constraints |
Goodput and good_request_fraction | Useful completion rate and attainment | End-to-end agent quality or tool-service capacity |
More concurrency can keep hardware busy but also raises contention and memory demand. More GPUs per replica can make a model deployable, yet consume the budget that could otherwise fund additional replicas. Compare whole configurations under the same workload; a larger allocation is not automatically a better capacity purchase.
Architecture / decision scenario
A pharmaceutical research assistant serves 600 active scientists at peak. Each submits one evidence-summary task every two minutes, and the measured workflow averages three inference calls per task: 15 model requests per second. The application team budgets eight seconds at p95 for a complete task, including retrieval; its model-tier test requires at least 99% of requests to satisfy the assigned inference latency constraints.
In this hypothetical benchmark, a compatible candidate uses two GPUs per replica. At five offered requests per second it achieves 99.5% attainment. At eight it completes seven requests per second but only 80% meet the constraints. The tempting proposal buys two replicas on the strength of the higher throughput figure. That leaves each replica handling about 7.5 requests per second, near the measured failing region.
The platform lead instead starts from three replicas at the measured five-request operating point: six GPUs for steady-state load. The performance engineer profiles the workflow on those six GPUs with nat sizing calc, representative trajectories, and at least ten concurrency values. The team also specifies a peak target of 40 simultaneously executing workflows, distinct from the 600 active scientists who submit intermittently. Offline estimation uses --test_gpu_count 6 --target_workflow_runtime 8 --target_users 40. The engineer compares the workflow estimate with the endpoint-based six-GPU baseline, then tests the proposed deployment against both targets; three sequential calls and retrieval must fit the eight-second workflow budget. This is a starting capacity estimate, not the availability allocation; see 4.5.3 for failure headroom.
If procurement offers only four GPUs, the owner must change a measured input: reduce model-call demand, select another quality-qualified configuration, or revise the service target with the product owner. Relabeling the eight-request run as capacity does not resolve the shortage.
Common exam traps
- Dividing registered users by GPUs without estimating active task rate and inference fan-out.
- Using output tokens per second as though every request has the same output length.
- Accepting high throughput when
good_request_fractionmisses the target. - Counting two GPUs in one model replica as two independent serving replicas.
- Treating profile compatibility or a short-prompt benchmark as a production capacity guarantee.
What to remember
Translate users into a representative inference workload, fix the NIM profile, and measure a sustainable rate with explicit latency attainment. Convert replicas into the profile’s full GPU allocation, verify the combined deployment, and add failure headroom as a separate decision.
Knowledge check
A NIM profile uses two GPUs per replica (TP 2). With the production prompt mix, one replica sustains 5 requests per second with 99.5% of requests meeting the latency constraints; at 8 offered requests per second it completes 7 per second, and 80% meet them. Peak demand is 14 requests per second, and at least 99% must meet the constraints. How many GPUs does the steady-state estimate need, before any failure headroom?
An engineer has profiled a workflow with nat sizing calc against a NIM deployment on 6 GPUs. In production, 600 scientists submit tasks intermittently, with a peak of 40 workflows executing at once, and procurement has proposed a 16-GPU cluster. Which arguments should the offline estimate use?
--test_gpu_count 16 --target_users 40--test_gpu_count 6 --target_users 600--test_gpu_count 16 --target_users 600--test_gpu_count 6 --target_users 40--test_gpu_count is the number of GPUs used during the profiling run, not the cluster being sized; the calculator extrapolates from it. The estimate scales the concurrency the run could support within the runtime target up to --target_users , so the target is the peak of simultaneously executing workflows rather than everyone who submits a task during the day.The sizing calculator’s offline estimate meets the workflow-runtime target, but its Alerts column flags the passes at concurrencies 48 and 64 with workflow_interrupted: true. Peak production concurrency is expected to be 60. What should the engineer conclude about the estimate?
workflow_interrupted: true is excluded from the GPU estimate. The estimate therefore rests on the retained concurrency measurements; the failures at 48 and 64 need investigating, and the proposed deployment needs a load test at peak before the figure is accepted.A research workflow misses its 8-second p95 workflow-runtime target. nat sizing calc reports a P95 LLM Latency of 1.5 seconds and a P95 WF Runtime of 11 seconds, and traces show an external search service taking about 8 seconds of each run. Which change addresses the miss?
--target_llm_latency so the estimate follows model latencyOn the evaluated GPUs, list-model-profiles marks a TP-1 profile Low memory and suggests --max-model-len=8192. It marks a TP-2 profile of the same model and precision Compatible. The model supports 32,768 tokens of context, and the application’s requests need up to 20,000 tokens of input and output combined. Which deployment fits?
--max-model-len=8192--max-model-len caps input plus output per request, here below the 20,000 tokens the application requires. A successful startup proves neither memory safety under representative load nor capacity. The Compatible TP-2 profile fits, and sizing then counts both of its GPUs per replica.