NVIDIANVIDIA-Certified Professional: Agentic AI2026-100 Prerequisites
Budget tokens across the inference phases
Completed by the knowledge check
Learning objective
Explain tokens, context windows, and inference phases (prefill, decode).
Core idea
A token is a unit produced by the model’s tokenizer, rather than a fixed number of characters or one word. A context window bounds the sequence a model deployment can process. Input and generated output spend that capacity together, so a prompt that nearly fills the window leaves little room for an answer.
Inference also has two different kinds of work. Prefill processes the input; decode generates the continuation. Distinguishing them explains why a long document and a long answer can stress the same service differently.
How it works
Tokenization converts text into token IDs. NVIDIA NIM exposes POST /tokenize and POST /detokenize; its API reference states that token IDs are model-specific. Count with the deployed tokenizer rather than transferring a character-to-token estimate between models or languages.
The input includes more than the user’s visible question: instructions, conversation history, retrieved passages, and tool results can all enter the request. Chat templates also affect the model’s input representation. The served context limit can be lower than the model’s advertised maximum. For example, NIM’s GLM-5.2 deployment documentation exposes NIM_MAX_MODEL_LEN to limit served context length and warns that available key-value (KV) cache and request concurrency affect capacity. See 7.2.2 for choosing a serving profile.
For autoregressive generation, prefill processes the prompt and creates the KV cache used by subsequent attention computation. Decode extends the sequence using that cached state. This inference state is distinct from retained conversation memory; see 1.4.1. Generated tokens become part of the context for later generation. Reasoning tokens also need a generation allowance when the selected model emits reasoning before its answer.
Prefill turns the input into the KV cache; decode extends the sequence from that cached state, and its generated tokens become part of the context for later generation.
NVIDIA Dynamo documents separate prefill and decode engines: prefill creates the KV cache, the cache transfers to the decode engine, and decode generates tokens. This separation illustrates the different resource demands; it is not a requirement for every deployment. Prefill is generally computation-bound, while decode is often memory-bound. See 7.4.2 for batching, paged KV cache, and chunked context.
Key distinctions and trade-offs
Window versus useful context: Fitting within the sequence limit establishes capacity, not whether the model uses every passage correctly. Retrieval quality remains a separate decision; see 0.3.
First output versus completion: A long prefill can delay the first generated token. A long decode can extend completion time after output begins. Queueing and other application stages also contribute, so neither measurement isolates model computation by itself. See 4.3.1 for time to first token (TTFT) and inter-token latency (ITL).
Request length versus concurrency: Long sequences retain more KV state. A configuration that accepts one long request still needs workload testing before you assume the same capacity for many simultaneous requests.
Architecture / decision scenario
A bank’s policy assistant serves 40 concurrent employees. Its illustrative served window is 32,000 tokens. A request contains 27,000 tokens of instructions, history, and retrieved text, and reserves 6,000 tokens for generation: the combined 33,000 exceeds the configured limit.
The tempting fix is to raise the limit immediately. The inference owner instead asks whether the existing GPU allocation supports that context at 40-way concurrency. The application owner first reduces retrieved text by 3,000 tokens, yielding a 30,000-token request budget. They then measure first-token and completion latency separately: shrinking input addresses prefill work, while shortening the answer targets decode work. These are scenario figures, not a published NIM capacity guarantee.
Common exam traps
- Equating tokens with words or reusing token IDs across models.
- Reserving the whole context window for input and then adding output.
- Treating KV cache as durable conversation memory.
What to remember
Count the complete request with the right tokenizer, reserve generation capacity, and distinguish the served limit from advertised model capability. Prefill prepares the prompt’s state; decode generates the continuation.
Knowledge check
A NIM deployment serves a 16,000-token context. An assistant’s requests carry 13,500 tokens of instructions, history and retrieved passages, and the answer needs up to 3,000 generated tokens. Without changing the deployment, which change makes the combined token budget fit?
max_tokens until the generation allowance covers the full answer lengthmax_tokens adds to the overflow, the served limit rather than the advertised maximum bounds this deployment, and the KV cache is inference state built from the input, not a place to keep history outside it.Users of a contract assistant report a long wait before the first word appears when they attach long contracts; once output starts, it streams at its usual pace. Queueing and the application’s other stages have been measured and are not the cause. Which change most directly targets the delay?
max_tokens so that decode finishes soonerA team migrates an assistant from one NIM-served model to another. Its prompt-budget code reuses a character-to-token ratio and cached token IDs from the old model. Which practice gives reliable token counts on the new deployment?
/tokenize endpointA single-request test shows that a NIM deployment accepts 60,000-token requests after the team raises NIM_MAX_MODEL_LEN. Production will run 30 such requests at once on the same GPUs. What should the team do before relying on that limit?
References
- NVIDIA, API Reference
- NVIDIA, Get Started with GLM-5.2
- NVIDIA, Disaggregated Serving