NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.4 Tune model parameters (e.g., accuracy, latency-efficiency trade-offs).
Tune temperature, top-p, and output ceilings
Completed by the knowledge check
Learning objective
Tune sampling parameters (temperature, top-p, max tokens) for reliability.
Core idea
Generation controls change which answers a model produces and how much output it can generate. They do not establish that an answer is correct. A model that consistently chooses the wrong tool is consistently wrong; a short response that omits a required field is incomplete even when it arrives quickly.
Treat tuning as a controlled comparison of task success, variation, and completion. Hold the model, prompt, task cases, and execution conditions fixed while you change a control. See 3.1.4 for preserving evaluation runs and 3.4.3 for automating the search. Here the decision is which control addresses the observed failure.
How it works
Separate distribution shape from output length
In the OpenAI-compatible request vocabulary, temperature controls sampling randomness: lower values concentrate choices, while higher values permit more variation. top_p performs nucleus sampling, retaining candidates whose cumulative probability reaches the selected mass. A value of 0.1 refers to 10% probability mass, not 10% of the vocabulary. OpenAI recommends changing temperature or nucleus sampling rather than both at once.
That recommendation is a useful first diagnostic experiment. If both controls change together and tool-selection quality falls, you cannot tell which change mattered. Once individual effects are understood, a deliberately bounded joint search can explore interactions; it is a different experiment with more combinations, not an excuse to leave settings unexplained.
max_tokens limits generated output. It is a ceiling, not an instruction to produce exactly that many tokens, and it is not a time limit. NIM documents POST /v1/chat/completions with model, messages, and max_tokens. Use the interface your endpoint documents instead of copying parameter compatibility assumptions from another provider.
In NeMo Agent Toolkit, the NIM LLM configuration lives under llms.<name> with _type: nim. The optimizer documentation explicitly exposes temperature, top_p, and max_tokens as optimizable NIM fields. This is where a workflow-level experiment changes these values; the deployed NIM endpoint remains the inference boundary.
Measure usable completion, not successful transport
Inspect why generation stopped. In the Chat Completions response vocabulary, finish_reason: "length" indicates exhaustion of the requested token limit. stop means a natural stop or supplied stop sequence; it does not certify factual correctness. tool_calls indicates a tool request, which is a different completion outcome from a final answer.
Count a response as successful only after the task’s checks pass. For structured answers, that includes the schema and required business fields; see 2.1.2. For tool selection, compare the chosen action and arguments with the expected action; see 2.6.1. Sampling controls complement those checks rather than replace them. Whatever the termination reason, a response counts as successful only after the task’s checks pass.
A low output ceiling can make latency and token totals look attractive by cutting off difficult cases. Keep an explicit incomplete-output count beside accuracy. Otherwise the comparison rewards a candidate that stops working before it finishes. Inspect response fields and task results together, because a syntactically complete answer can still omit a required conclusion.
Reasoning models make this distinction especially consequential. NVIDIA’s GLM-5.2 NIM documentation says to inspect choices[0].message.reasoning_content separately from choices[0].message.content. If generation reaches finish_reason: "length" before answering, increase max_tokens or disable thinking for that request. See 3.4.2 for choosing reasoning behavior; here the relevant failure is an output limit that starves the final answer.
Bound the experiment to the task
Start from the selected model’s documented settings and a successful baseline. The toolkit’s nim provider defaults are temperature: 0.0, top_p: 1.0, and max_tokens: 300; its default search spaces are respectively low=0.1, high=0.8, step=0.2, low=0.5, high=1.0, step=0.1, and low=128, high=2176, step=512. A reasoning model can exhaust that 300-token output ceiling before answering, so check completion before adopting the provider default. Test a small temperature range while leaving top_p fixed, or vary nucleus sampling while temperature stays fixed. Then test output ceilings against the longest legitimate task cases, including requests that generate tool arguments or explanations.
Repeat each candidate on the same cases. NeMo Agent Toolkit’s optimizer provides reps_per_param_set for repeated evaluation. Repetitions expose variation that one favorable completion hides. An experiment record should retain the effective llms configuration, response termination reasons, task score, and latency for each run.
Do not select a setting solely because identical prompts produce similar prose. The reliability owner needs stable task outcomes: correct routing, valid arguments, required content, and bounded incomplete results. A consistent writing style is a separate objective. When the prompt lacks necessary information, lowering randomness cannot supply it; repair the context or task definition before treating sampling as the remedy.
Key distinctions and trade-offs
The controls address different failure modes:
| Control | Deciding condition | Mechanism | Cost or failure risk |
|---|---|---|---|
| Temperature | Excessive variation between otherwise equivalent task runs | temperature changes sampling randomness | Lower variation can preserve a systematic wrong choice |
| Nucleus sampling | You need to restrict the sampled probability mass | top_p selects cumulative probability mass | Aggressive restriction can remove useful alternatives |
| Output ceiling | Valid answers are truncated or generation spends too many tokens | max_tokens bounds generated output | Too small omits answers; too large permits more generation work |
Temperature and top-p change selection behavior; the token ceiling changes how long generation may continue. None is an accuracy score. Judge the combination through complete task outcomes, with the same validator across candidates.
Architecture / decision scenario
In this illustrative comparison, a retail returns assistant must select a policy route and emit a complete explanation within a measured p95 of 2.5 seconds. The quality owner requires at least 97% correct routes on a 600-case benchmark and fewer than 1% incomplete answers.
The baseline uses temperature: 0.6, top_p: 1.0, and max_tokens: 512. It scores 96% routing accuracy with 0.5% incomplete responses and p95 at 2.4 seconds. Lowering temperature to 0.2 while retaining the other settings raises measured routing accuracy to 98%, with 0.4% incomplete responses and p95 at 2.3 seconds. A separate nucleus-sampling trial changes only top_p to 0.3 from the baseline: it scores 95%, with 0.5% incomplete responses and p95 at 2.2 seconds; restricting the candidate mass is not the better choice here.
A tempting speed proposal changes only max_tokens to 128 from the baseline. Its p95 reaches 1.7 seconds and routing accuracy remains 96%, but 8% of responses are incomplete. The team rejects it despite the faster headline result. The application owner selects the lower-temperature candidate with the 512-token ceiling, retains response checks, and asks for further model or prompt work if production traffic exceeds the latency target. Changing one control per trial lets the team attribute the gain to temperature rather than to truncation.
Common exam traps
- Equating low temperature with factual accuracy or guaranteed repeatability.
- Reading
top_p: 0.1as retaining 10% of vocabulary entries. - Treating
max_tokensas an exact response length or elapsed-time deadline. - Reporting low latency after silently excluding incomplete responses.
- Assuming a successful HTTP response or
finish_reason: "stop"proves task success.
What to remember
Tune the control that matches the failure: variation, candidate restriction, or insufficient output space. Compare repeated, complete task outcomes with fixed validators. Keep truncation visible so a shorter failure cannot win an accuracy–latency comparison.
Knowledge check
A returns agent chooses the wrong tool on 15% of benchmark cases. At temperature: 0.0, five repetitions of each failing case pick the same wrong tool every time. The prompt describes both the refunds tool and the returns tool but never says which one owns exchanges. Which change addresses the failure?
top_p to concentrate sampling on the likeliest tooltemperature to let repeated runs try other toolsmax_tokens to give tool selection more output roomAn engineer sets top_p: 0.1 for a deployed chat assistant. Which candidates does sampling keep at each step?
0.1 means 10% of the probability mass, not 10% of the vocabulary. A threshold on each token’s own probability is a different rule, and nothing in nucleus sampling selects tokens at random before sampling.A routing assistant must keep incomplete answers below 1%. A trial that changes only max_tokens from 512 to 128 shows p95 latency falling from 2.4 to 1.7 seconds and routing accuracy unchanged at 96%, while 8% of its responses end with finish_reason: "length" and fail required-content checks. What should the team do with this trial?
temperature to shorten answersfinish_reason: "length" means the requested token limit ran out; the failed required-content checks establish that these answers are incomplete. An 8% incomplete rate violates the 1% limit despite the latency gain. Temperature changes sampling rather than the output allowance, and counting only finished answers hides the failures instead of fixing them.A benchmark of structured order updates counts a response as a success when the NIM endpoint answers HTTP 200 and finish_reason is "stop". Each answer must match a schema and include the order ID and the new status. Which rule should count successes?
finish_reason set to "stop""length"stop means a natural stop or a supplied stop sequence, and a successful transport says nothing about content; a syntactically complete answer can still omit a required field. Whatever the termination reason, a response counts as successful only after the task’s checks pass.A workflow calls the GLM-5.2 NIM with thinking enabled and the NeMo Agent Toolkit nim provider’s default max_tokens: 300. Many responses carry text in choices[0].message.reasoning_content, an empty content, and finish_reason: "length". Which change does the NIM documentation give for this failure?
reasoning_content as the answer when content is emptymax_tokens or disable thinking for those requestsmax_tokens to make the reasoning end soonertemperature to shorten the reasoning before answeringmax_tokens or disable thinking for the request. Parsed reasoning is not the delivered answer, a smaller ceiling starves the answer further, and temperature changes sampling rather than how long generation may continue.References
- NVIDIA, API Reference
- NVIDIA, NVIDIA NeMo Agent Toolkit Optimizer
- NVIDIA, Get Started with GLM-5.2
- OpenAI, Create chat completion