Measure serving capacity against user deadlines

Learning objective

Measure LLM serving metrics: TTFT, ITL, throughput, goodput, and concurrency.

Core idea

Serving capacity is the amount of work a deployment completes while meeting the user’s timing requirements. A configuration can generate more tokens per second and still deliver fewer acceptable responses because requests wait longer or streams become too slow.

NVIDIA AIPerf separates client-observed request metrics from aggregate benchmark metrics. Use time to first token (TTFT) for initial responsiveness, inter-token latency (ITL) for generation pace, throughput for total completed work, and goodput for requests satisfying explicit service level objectives (SLOs). Concurrency describes the load at which you measured them. It is neither a rate nor a latency target.

These are serving measurements. Answer correctness and complete agent-task latency need separate evaluation; see 3.2.3. A fast model endpoint can sit inside a slow workflow.

How it works

Measure the first token and subsequent generation

AIPerf’s time_to_first_token measures the interval from request start to the first non-empty streamed content response. It includes network transit, queueing, prompt processing, and generation of the first token or token chunk. A rising TTFT identifies a longer initial wait without proving which server phase caused it. TTFT includes reasoning tokens; time_to_first_output_token (TTFO) measures the first non-reasoning output token, so use TTFO when the deadline concerns the visible answer. For models without reasoning, TTFO and TTFT are equivalent.

Enable --streaming on a supported endpoint to measure streaming TTFT and ITL. An early protocol message without content is not the first generated token. Keep the client location and transport path comparable across runs: a client-side interval includes the network, so moving the generator changes the experiment.

AIPerf’s default inter_token_latency divides the post-first-content generation interval by output_sequence_length - 1. A response with 101 output tokens and a 2-second post-first-content interval has an average ITL of 20 ms under that convention. Its reciprocal is 50 tokens/second/user during generation. Neither value includes the initial wait.

ITL requires at least two output tokens. Streaming chunks can contain several tokens, so chunk-arrival intervals and token-normalized ITL measure different things. AIPerf reports inter-chunk latency separately. A reasonable request-average ITL can hide an individual pause in a stream; inspect inter-chunk timing when users report stalls.

For servers bundling multiple tokens into the first content chunk, AIPerf supports --per-chunk-usage with --use-server-token-count, a chat endpoint, and --streaming. This uses reported first-chunk tokens in the divisor. It requires server support for stream_options.continuous_usage_stats; do not assume every OpenAI-compatible endpoint supports that extension. Preserve the counting convention when comparing configurations.

State which throughput you mean

request_throughput is completed request count divided by benchmark duration, in requests/second. output_token_throughput is aggregate output tokens divided by benchmark duration, in tokens/second. total_token_throughput includes both input and output tokens. They answer different capacity questions.

AIPerf’s output_token_throughput_per_user is the reciprocal of ITL. It excludes TTFT and describes one response’s generation pace, whereas aggregate output throughput spans concurrent requests and the benchmark interval. A server producing 2,000 tokens/second in aggregate does not promise each user that generation rate.

Record input and output sequence lengths alongside rates. Longer outputs can raise token throughput while reducing the number of completed requests. Shortening answers can raise request throughput while failing the task. Compare the same workload and output requirements rather than optimizing a denominator in isolation.

Count the requests that satisfy every SLO

Configure AIPerf --goodput using metric tags and thresholds in each metric’s display unit. For example, --goodput "time_to_first_token:800 inter_token_latency:40" expresses an 800 ms first-token threshold and a 40 ms ITL threshold. These are illustrative application targets, not NVIDIA service guarantees.

good_request_count counts requests meeting every configured constraint. goodput divides that count by benchmark duration. A request meeting TTFT but missing ITL is excluded; separate percentile summaries cannot establish that the same requests meet both conditions.

Also inspect good_request_fraction: the good-request count divided by request_count + error_request_count. Errors remain in this denominator, preventing a deployment that drops requests from appearing healthy because its surviving responses are fast. AIPerf writes this fraction to exported results even though it is hidden from the console table. Timing goodput does not establish factual correctness, authorized tool use, or task completion.

Key distinctions and trade-offs

Use these measurements together when choosing an operating point:

Read the client results alongside NVIDIA NIM’s /v1/metrics endpoint. NIM passes through native backend Prometheus metrics, including request latency, throughput, queue depth, and token counts. AIPerf describes the caller’s experience; server metrics show the load held inside the serving system and help investigate queueing when TTFT rises.

With --concurrency alone, AIPerf replaces completed requests to maintain the chosen in-flight level. Raising it can improve aggregate utilization while increasing waiting and slowing individual streams. The useful point is where acceptable work meets demand with room for variation. See 4.3.2 for the load experiment that locates it.

Architecture / decision scenario

An insurer’s customer assistant has a fixed two-GPU serving allocation. The serving owner compares two configurations using identical anonymized prompts and streaming output requirements. The application target requires TTFT below 800 ms and ITL below 40 ms for at least 95% of attempted requests.

In an illustrative 100-second measurement, configuration A at concurrency 32 completes 1,000 requests and generates 100,000 output tokens. Of those requests, 960 meet both thresholds, with no errors: request throughput is 10/second, output throughput is 1,000 tokens/second, goodput is 9.6/second, and the good-request fraction is 96%.

Configuration B at concurrency 64 completes 1,100 requests and generates 121,000 tokens, but only 880 requests meet both thresholds. With no errors, its goodput is 8.8/second and attainment is 80%, despite higher request and token throughput. Its longer first-token waits and slower streams explain the failed requests.

The tempting choice is B because the token-throughput dashboard improves. The serving owner selects A for this workload: it meets attainment and supplies more timely responses within the same GPU allocation. If demand exceeds A’s demonstrated goodput, the team must change capacity or the workload and repeat the measurement; raising concurrency alone has already failed the user requirement.

Common exam traps

• Treating TTFT as a direct measurement of GPU prefill execution time or the wait for a non-reasoning answer.
• Reading average ITL as the worst pause between streamed chunks.
• Comparing per-user decode throughput with aggregate token throughput.
• Calling concurrency a request-arrival rate.
• Assuming separate passing TTFT and ITL percentiles imply joint SLO attainment.
• Treating timing goodput as a correctness score or ignoring errored attempts.

What to remember

Measure initial wait, generation pace, aggregate work, and SLO-compliant work separately. Keep workload lengths, counting conventions, and concurrency visible. Choose capacity from goodput and attainment rather than the largest token-throughput number.

Knowledge check

References

• NVIDIA, AIPerf metrics reference
• NVIDIA, Benchmark goodput with AIPerf
• NVIDIA, AIPerf CLI options
• NVIDIA, Logging and observability