Find saturation with realistic workflow traffic

Learning objective

Run load tests with realistic end-to-end traffic mixes and concurrency sweeps, analyzing saturation and tail latency.

Core idea

A load test establishes capacity for a particular workload, arrival process, and success definition. Repeating one short prompt against a model endpoint answers a narrower question than sending the production mix through retrieval, agent decisions, tools, and response delivery.

Use NVIDIA AIPerf to characterize the serving endpoint under controlled traffic. Pair that experiment with a live workflow test when the service includes external dependencies. Preserve the distinction in the results: replayed model requests measure serving behavior, while live agent requests expose the work generated by the workflow itself.

See 4.3.1 for metric definitions. Here the decision is which load level the complete service can sustain, and whether the test actually applied that load.

How it works

Define the workload before increasing traffic

Create a workload specification containing task proportions, input and output length distributions, conversation depth, shared-prefix frequency, tool-call counts, and arrival timing. Include ordinary requests and the expensive minority: long retrieved documents, multi-turn conversations, and requests that invoke several dependencies. Preserve correlations, such as longer prompts accompanying deeper investigations, rather than independently sampling every dimension.

AIPerf supports custom data through --input-file and --custom-dataset-type. Its single_turn and multi_turn formats represent individual exchanges and conversations. The input should reproduce the data your endpoint receives, including the relevant conversation context. If you benchmark only a model endpoint, explicitly identify the workflow stages outside the measurement.

AIPerf’s documented Exgentic loaders replay recorded chat-call snapshots, preserving recorded messages, tool definitions, and request controls. They do not execute tools or add live responses to subsequent requests. This is useful for serving a representative agent-generated LLM workload, but it does not test a tool server, dynamic tool outcomes, or the next decision made from those outcomes.

For a workflow-level sweep, use NeMo Agent Toolkit nat sizing calc from nvidia-nat[profiler] with --config_file, a separate --calc_output_dir, --concurrencies 1,2,4,8,16,32, and --num_passes 2. Its dataset needs representative inputs, not ground truth; --endpoint targets a running workflow server. It reports p95 workflow runtime beside p95 LLM latency. Failed passes carry workflow_interrupted (!W in the summary) and are excluded from GPU estimation. This is concurrency-driven testing; retain an application arrival-rate driver for independent demand and use general.telemetry.tracing for diagnosis (4.3.3). GPU-count extrapolation belongs to 4.5.1.

Separate concurrency sweeps from arrival-rate tests

With --concurrency alone, AIPerf issues replacement work when requests finish. This completion-driven test shows how throughput and latency change as in-flight work increases. Use a series such as 1, 8, 16, 32, and 64, with otherwise identical data and server configuration. The CLI accepts comma-separated concurrency values for a sweep.

That mode also reduces its request issue rate when responses slow. It cannot by itself establish behavior when independent users keep arriving at the original rate. Follow it with --request-rate and an explicit --arrival-pattern: constant schedules regular intervals, while poisson uses exponential inter-arrival intervals. For clustered demand, use gamma with --arrival-smoothness greater than zero and below 1.0; 1.0 is Poisson-like, and values above 1.0 make arrivals smoother. This setting applies only to gamma, and --arrival-pattern applies with --request-rate.

To preserve recorded production arrival shapes and request mixes, use --fixed-schedule with a timestamped dataset such as --custom-dataset-type mooncake_trace and --input-file. Fixed scheduling takes priority over other scheduling options.

With either fixed scheduling or request-rate scheduling, concurrency is a ceiling that can block scheduled dispatch. Compare scheduled traffic with achieved traffic. A run configured for 20 requests/second that only dispatches 12 is not proof that the server sustained 20. Record whether the generator’s concurrency ceiling, client resources, or server response time constrained the run.

A concurrency sweep issues replacement work when requests finish, while an arrival-rate test sends scheduled traffic that the concurrency ceiling can block.

Control warmup, duration, and repeated trials

AIPerf provides --warmup-request-count and --warmup-duration. Declare the warmup policy and distinguish warm steady-state results from a separate cold-start experiment. Repeated identical prefixes can change the workload’s reuse characteristics; preserve the intended prefix distribution rather than accidentally warming every measured prompt.

Set --benchmark-duration or an explicit --request-count rather than relying on a small incidental dataset. For duration-based tests, --benchmark-grace-period defaults to 30 seconds after issuance stops; inf waits indefinitely for all responses. Include the configured grace period and unfinished work in the run record; an early cutoff can remove precisely the slow responses you need to understand.

Use --random-seed to reproduce seed-controlled dataset content and sampling decisions. NVIDIA explicitly distinguishes that reproducibility from performance determinism: async execution, network timing, worker assignment, and server responses still vary. Repeat candidate operating points and retain the spread rather than declaring a marginal improvement from one run.

Locate the knee and inspect the tail

For each load level, retain achieved request rate, throughput, goodput, error counts, and p50, p95, and p99 latency by task class. Include a run record with the configuration, workload revision, measurement window, and sample count. A percentile from a handful of expensive requests is a weak estimate; extend that slice’s measurement rather than treating the aggregate sample count as its own.

The saturation knee is the region where additional offered work produces little extra completion throughput while queueing, tail latency, or failures increase. A good mean does not override a failed tail target. Examine TTFT and complete workflow latency separately: fast token delivery after a slow retrieval stage still leaves the user waiting.

Reduce load to the preceding acceptable point and rerun it. If endpoint results remain stable while workflow latency degrades, investigate the dependency path in 4.3.3 before changing model-serving settings. Choose an operating point with measured allowance for demand variation, rather than treating the last successful sample as guaranteed production capacity.

Key distinctions and trade-offs

Choose each experiment for the question it resolves:

Architecture / decision scenario

A manufacturing engineering assistant must finish 95% of requests within 6 seconds at a peak of 12 user requests/second. Its workload is 70% document questions, 20% maintenance-history lookups, and 10% investigations that call three tools. The performance owner has a fixed on-premises serving allocation.

An illustrative endpoint concurrency sweep plateaus between 32 and 64 in-flight requests. Increasing to 64 raises p99 TTFT from 700 ms to 2.1 seconds with little throughput gain. The tempting proposal is to publish the endpoint’s throughput as application capacity and raise concurrency for the rest.

Instead, the owner tests the live 70/20/10 mix at 8, 12, and 16 arrivals/second. At 12, achieved arrival rate remains 12, but the investigation slice reaches p95 11 seconds because the maintenance tool queues. A test containing only short document prompts had reported p95 3 seconds. The endpoint baseline and live trace identify different limits.

The team rejects the 12/second readiness claim, repairs the tool bottleneck, and repeats the same workload and arrival process. It retains the concurrency sweep to protect inference from saturation. Both the endpoint and the complete service must pass their timing requirements; a fast synthetic mean cannot substitute for the missing workflow result.

Common exam traps

• Testing only average prompt and output lengths while omitting expensive task classes.
• Treating recorded tool-call messages as actual tool execution.
• Assuming completion-driven concurrency reproduces independent user arrivals.
• Reporting configured request rate without checking achieved dispatch rate.
• Comparing warmed and cold runs without identifying the difference.
• Claiming repeatable latency merely because the dataset uses a fixed random seed.

What to remember

Define the workload, sweep concurrency to find the knee, and test independently arriving demand through the complete workflow. Inspect tails and failures by task class. Preserve the run settings and repeat candidate points before declaring sustainable capacity.

Knowledge check

References

• NVIDIA, AIPerf CLI options
• NVIDIA, AIPerf timing modes reference
• NVIDIA, AIPerf benchmark datasets
• NVIDIA, Random number generation and reproducibility
• NVIDIA, Observe workflows
• NVIDIA, Size a GPU cluster with NVIDIA NeMo Agent Toolkit