NVIDIANVIDIA-Certified Professional: Agentic AI2026-103.4 Tune model parameters (e.g., accuracy, latency-efficiency trade-offs).
Choose deployable results from automated search
Completed by the knowledge check
Learning objective
Run automated model, hyperparameter, and prompt optimization and select operating points on Pareto fronts.
Core idea
Automated optimization searches the choices you expose against the measurements you define. It does not decide what the business can tolerate. A Pareto front identifies candidates where improving one measured objective requires sacrificing another; choosing an operating point still requires explicit quality, latency, and resource requirements.
NeMo Agent Toolkit provides nat optimize for this search. Use the sampling controls from 3.4.1 and the deployment candidates from 3.4.2 as inputs. This lesson owns the search configuration, its computational cost, and the decision from trial results to a deployable candidate.
How it works
Expose only valid candidates
The optimizer is optional: install nvidia-nat[config-optimizer] or nvidia-nat-config-optimizer. Its configuration sits under optimizer in the workflow file. Run nat optimize --config_file CONFIG_PATH, replacing CONFIG_PATH with that file’s path. The ordinary evaluation runtime has its own installation requirements; the evaluation documentation specifies nvidia-nat[eval] for full nat eval execution.
A field becomes tunable through OptimizableField, with a SearchSpace specifying its permitted values. Configuration objects use OptimizableMixin to support optimizable_params. Under a component such as llms.<name>, optimizable_params selects fields and search_space supplies or overrides their search definitions.
Use values for categorical choices such as model_name. Use low, high, and optionally step for numerical fields such as temperature or max_tokens; values is mutually exclusive with low and high. A field marked optimizable without a search space in either its data model or configuration causes an error when the optimizer collects fields. Fix that definition before spending evaluation calls.
The candidate list is also an architectural constraint. Every selected model must be reachable through the configured provider and satisfy the deployment requirements established in 3.4.2. An optimizer cannot turn an unavailable model or unsupported precision into a viable service merely by assigning it a high score.
Use the search method that matches the field
The search types have different mechanisms and costs:
| Search target | Deciding condition | Mechanism | Cost or failure risk |
|---|---|---|---|
| Model selection | Several approved model candidates can perform the task | Categorical SearchSpace.values on model_name | Every candidate needs serving capacity and comparable evaluation |
| Hyperparameters | Numeric inference controls need bounded exploration | optimizer.numeric.enabled, numeric.n_trials, and Optuna | Trials multiply workload executions; broad ranges waste calls |
| Prompts | Instruction wording is the variable to improve | is_prompt, prompt_purpose, and optimizer.prompt.enabled | Genetic search spends mutation and evaluation tokens; useful constraints can be lost |
Numeric optimization uses Optuna. numeric.sampler: "grid" performs exhaustive search and requires explicit values or a stepped range. "bayesian" and None use Optuna’s documented defaults, TPE for single-objective studies and NSGA-II for multi-objective studies. Grid search is attractive for a small complete comparison, but the Cartesian product grows quickly as you add models and controls.
Prompt optimization uses a genetic algorithm. Mark a prompt search space with is_prompt: true and describe its purpose with prompt_purpose. A base prompt can come from SearchSpace.prompt or the field default; supplying neither causes an error. prompt_population_init_function names the mutation function, while prompt_recombination_function optionally combines parents. Bound work with ga_population_size, ga_generations, and ga_parallel_evaluations under optimizer.prompt.
These controls cap different things. Population and generations determine the scale of the prompt search; parallel evaluations control how much work runs concurrently. Raising concurrency is not free acceleration when candidate serving endpoints or evaluator models are already saturated. A staged search can first refine prompts and then fix them while exploring numeric choices, keeping the cause of each improvement inspectable.
Define objectives before inspecting winners
optimizer.eval_metrics associates each objective with an evaluator_name, a direction of maximize or minimize, and a weight. Configure the corresponding evaluators rather than assuming an arbitrary metric name creates an implementation. Keep quality and latency as separately inspectable measurements even when search uses a combined score.
reps_per_param_set repeats workflow execution for each candidate. Increase repetition when candidate scores fluctuate, while recognizing that each repetition consumes workflow and evaluator calls. The optimizer documentation also exposes multi_objective_combination_mode with harmonic, sum, and chebyshev. optimizer.target can stop the search when a trial’s combined, normalized score reaches the specified value. Both the combined score and this early-stop target express scalar preferences; neither enforces an application deadline or a minimum quality threshold.
Retain the evaluation setup described in 3.1.4. After searching, run the selected configuration through nat eval --config_file CONFIG_PATH on reserved task cases and retain config_effective.yml. This verifies what you intend to deploy rather than reporting only the best result on repeatedly searched cases.
Select from the front, then verify deployment
Under optimizer.output_path, inspect trials_dataframe_params.csv and the trial configurations. NVIDIA documents config_numeric_trial_N.yml for selecting a particular numeric trial and optimized_config.yml for the selected result. Prompt search additionally writes optimized_prompts.json, generation checkpoints, and ga_history_prompts.csv.
pareto_front_2d.png is generated for exactly two metrics; parallel-coordinate and pairwise plots support broader comparisons. A candidate is dominated if another candidate is no worse in every compared objective and better in at least one. Remove dominated choices, then apply hard requirements to the remaining candidates. A point on the front can still fail every production constraint that matters to you.
Key distinctions and trade-offs
Search quality depends on the candidate space and evaluators, not only on trial count. More trials can improve the measured result while increasing inference cost and the opportunity to specialize to the searched cases. Preserve the original prompt and trial artifacts so the selected candidate’s behavior remains reviewable.
A scalar winner and a production operating point answer different questions. The first follows configured metric preferences. The second must satisfy service requirements under the actual deployment load. Keep the measured front and the rejected candidates, because a changed deadline might justify selecting another already-tested point. Selection removes dominated choices, applies hard requirements, and keeps the rejected candidates for a changed deadline.
Architecture / decision scenario
A media company’s meeting-analysis assistant needs at least 95% correct action-item attribution and p95 no higher than 3 seconds. It has a search budget of 120 full benchmark passes. The following figures are illustrative measurements.
The ML owner sets optimizer.numeric.n_trials: 20 and optimizer.reps_per_param_set: 3, allowing 60 passes across approved model choices and bounded generation settings. The prompt stage sets optimizer.prompt.ga_population_size: 4 and optimizer.prompt.ga_generations: 4 with the same three repetitions: 16 individual evaluations × 3 gives an allowance of 48 passes, leaving 12 for validation. The initial population counts as generation one, rather than adding another generation to the allowance. Mutation-model calls consume additional tokens outside those pass counts.
The resulting comparisons show candidate A at 96% and 2.7 seconds, B at 98% and 4.1 seconds, C at 94% and 2.0 seconds, and D at 95% and 3.0 seconds. A and D both meet the requirements, but D is dominated by A, so A is the only feasible candidate on the front. B and C are also on the front, but each fails one requirement: B exceeds the latency ceiling, and C misses the quality floor.
A tempting proposal selects B because it has the highest quality score, or expands prompt search until the budget is exhausted. The product owner instead chooses A’s exact saved configuration and prompt for reserved-case and target-load validation. The platform owner checks its serving allocation. Search ends with a candidate that fits the requirement, not with permission to deploy whichever trial has the most flattering scalar score.
Common exam traps
- Assuming
optimizable_paramssupplies a missing search space. - Confusing numeric Optuna trials with genetic prompt generations.
- Treating metric weights as hard service constraints.
- Calling a dominated candidate a Pareto improvement.
- Deploying the search winner without validating its saved configuration.
Where the exam guide differs
The study guide’s suggested readings use “NVIDIA Agent Intelligence Toolkit” and “AIQ Toolkit.” The product documentation calls the workflow and optimization framework NeMo Agent Toolkit and documents nat optimize. Read those guide names as the predecessor naming of the toolkit; select the documented optimizer mechanism rather than inventing a separate AIQ optimization service.
What to remember
Expose valid model, numeric, and prompt choices; declare evaluator directions; and budget repetitions and search size. Use trial artifacts to identify nondominated candidates, then select an operating point that meets hard requirements and survives independent validation.
Knowledge check
A support assistant needs at least 92% resolution accuracy and p95 at or below 2.5 seconds. After nat optimize, the team’s trial measurements are:
- Candidate P: 93%, 2.4 seconds
- Candidate Q: 97%, 3.3 seconds
- Candidate R: 90%, 1.6 seconds
- Candidate S: 92%, 2.5 seconds
Which candidate should go forward to validation?
A custom LLM configuration declares frequency_penalty with OptimizableField(0.0) and no space. The workflow lists it under optimizable_params and adds no search_space entry for it. What happens when nat optimize collects the optimizable fields?
numeric.sampler is "grid"search_space entry is suppliedoptimizable_params selects fields; it does not define what to try. A field marked optimizable without a search space in either its data model or the configuration’s search_space mapping causes an error when fields are collected, and grid search itself needs explicit values or a stepped range. Fix the definition before spending evaluation calls.A team configures optimizer.eval_metrics with accuracy at weight 0.7 and latency at weight 0.3, and sets optimizer.target: 0.9. Production requires p95 at or below 3 seconds. How should that requirement be enforced?
optimizer.target to the normalized value of the limitmulti_objective_combination_mode to chebyshev for the limittarget all act on one combined, normalized score: they express scalar preferences or stop the search, and none enforces an application deadline. Keep latency separately inspectable, remove dominated candidates, then apply the hard limit to what remains.A numeric search sets numeric.sampler: "grid" over three models, eight temperature values and four max_tokens values, with reps_per_param_set: 3: 288 benchmark passes against a budget of 100. Scores fluctuate between runs, so the team keeps three repetitions. Which change fits the budget?
optimizer.target to stop the search earlyga_parallel_evaluations to finish sooner"bayesian" sampler with numeric.n_trials: 33reps_per_param_set to onetarget stops early only if a trial reaches the score, ga_parallel_evaluations sets prompt-search concurrency rather than the amount of work, and one repetition gives up the visibility into fluctuation the team needs.nat optimize finishes, and the team picks the trial with the best combined score on the searched benchmark. Before deployment it needs evidence about the configuration it will actually ship. What should come next?
optimized_config.yml on the strength of its search scorenat optimize with a higher reps_per_param_set valuenat eval on reserved casesnat eval on reserved task cases, and keeping config_effective.yml , verifies what you intend to deploy; reruns or extra repetitions on the same cases only measure the search set again.