Size a GPU Cluster With NVIDIA NeMo Agent Toolkit

Why it matters

GPU capacity for an agent is set by the whole workflow’s response time under load, not by one model call. The Agent Toolkit’s sizing calculator turns measured workflow behaviour into a GPU count, which is the number a cost decision starts from.

What to practise

Point the simple calculator’s sizing configuration at an LLM you host, then run nat sizing calc over a range of --concurrencies with --num_passes. Read the per-concurrency table — p95 LLM latency, p95 workflow runtime, total runtime — and the plots. Then rerun in --offline_mode with --test_gpu_count, --target_users and --target_workflow_runtime. The estimate is a slope-based linear fit, so watch how it depends on a wide concurrency range — the exercise recommends at least ten values — and on an LLM hosted on an isolated cluster rather than a public endpoint.

How it fits

This is the workflow-level sizing method of 4.5.1. Compare its estimate with the replica headroom 4.5.3 adds for failures: the calculator sizes steady-state demand, not recovery capacity.