Profiling with AIPerf

Why it matters

Serving capacity is judged by the latencies a user feels, not by a single throughput figure. AIPerf is NVIDIA’s tool for measuring them against an OpenAI-compatible endpoint.

What to practise

Start Qwen3-0.6B in the vLLM container, then run aiperf profile against /v1/chat/completions with --endpoint-type chat, --streaming, --request-rate 32, --request-count 64 and --output-tokens-mean 256. Read the results table — request latency, time to first token, inter-token latency, output token count and request throughput — by average, p50 and p99, and find the JSON export in the run’s artifacts/ directory. Note that one request rate gives one operating point.

How it fits

TTFT, ITL and throughput are the metrics 4.3.1 separates when choosing an operating point. To find saturation, repeat the run across rates or concurrency levels as 4.3.2 describes, rather than trusting a single measurement.