NVIDIANVIDIA-Certified Professional: Agentic AI2026-102.1 Engineer prompts and dynamic prompt chains for reliable performance.
Test prompt revisions against model changes
Completed by the knowledge check
Learning objective
Version prompts and regression-test them for consistency across model changes.
Core idea
A prompt revision is a behavior change. The artifact includes more than its main instruction paragraph: examples, tool guidance, template variables, context assembly, and the expected answer shape can all alter the result. Version those parts together with the model configuration against which you tested them.
Consistency means preserving the required behavior, not reproducing every word. A model change can improve average answer quality while breaking a tool-selection rule or a missing-information response. Your prompt regression suite needs to expose those differences before the changed combination reaches users. See 3.1.4 for the broader evaluation harness and 3.2.1 for controlled-comparison design; this lesson focuses on the prompt artifact and its release decision.
How it works
Define the artifact you can restore
Store prompt text, few-shot examples, assembly code, and output schemas in version control. Assign the release a revision that resolves to those exact files. A label such as “production prompt” is insufficient if editing it destroys the prior text. Your deployment record should identify the revision used for each released combination.
NeMo Agent Toolkit loads any string configuration field from a file:// reference, such as workflow.system_prompt: file://../prompts/system_prompt.txt or a function’s description: file://descriptions/my_tool.md. Relative paths resolve from the configuration file’s directory. Use an allowed extension: .txt, .md, .j2, .jinja2, .jinja, .prompt, .tpl, or .template. Loading inserts plain text; it does not render Jinja2 templates. Version these files alongside the configuration and the assembly code from 2.1.3.
Use the base key to inherit a shared configuration and override selected values in each variant. Keep tool and evaluator settings in base.yml; prompt variants override workflow.system_prompt, model variants override llms.<name>.model_name, and combined variants override both. Relative base paths also resolve from the variant configuration’s directory.
The llms configuration identifies a provider through _type and a model through model_name. Its NIM configuration also documents base_url, temperature, and max_tokens. Record the configured values and deployed model-serving identity.
Do not assume every model accepts the same generation controls. NVIDIA documents temperature and top_p as model-gated fields: explicitly setting an unsupported field fails validation. A migration test must first establish a valid target configuration. Lowering temperature does not replace regression testing.
Freeze the cases that reveal prompt failures
Build a prompt-focused fixture set from required behaviors: correct tool selection, source-backed answers, clarification when an input is missing, supported output shape, and appropriate handling of hostile source text. Add an observed production failure as a fixture with an expected outcome, rather than preserving only its original conversation transcript.
Keep the data used in the comparison stable. If retrieval runs against changing documents or tools query live state, a changed answer may reflect changed observations instead of changed prompt behavior. For an isolated prompt comparison, retain permitted source passages and recorded tool outcomes as test fixtures. Keep a separate end-to-end run to test the live integrations.
In NeMo Agent Toolkit, configure eval.general.dataset with its _type and file_path, and select evaluators under eval.evaluators. The documented evaluation command is nat eval --config_file=.... The full configuration-driven runtime requires nvidia-nat[eval]; profiling additionally requires nvidia-nat-profiler. Install the latter when token or latency profiling is part of acceptance.
Compare prompt and model separately
If a release changes both prompt and model, a two-by-two comparison helps locate the cause. Use the same cases and acceptance rules in all four runs:
| Combination | Question answered | Mechanism | Cost or limitation |
|---|---|---|---|
| Baseline prompt, baseline model | What behavior are you preserving? | base.yml with baseline file:// prompt and model | Requires a reproducible baseline |
| Candidate prompt, baseline model | What does the prompt change? | base: base.yml; override workflow.system_prompt with candidate file:// file | Does not measure the target model interaction |
| Baseline prompt, candidate model | What does the model change? | base: base.yml; override llms.<name>.model_name | May expose format or tool-use regressions |
| Candidate prompt, candidate model | Is the proposed release acceptable? | base: base.yml; override both prompt file and model | Can hide offsetting changes if examined alone |
Give each variant its own eval.general.output_dir to retain its outputs. Before evaluating NIM LLM variants, compare requests through POST /v1/chat/completions/render to inspect differences in their chat templates; see 2.1.3. Run repeated generations when behavior varies: nat eval supports --reps, including --reps=5 for five repetitions. More repetitions consume additional inference and judge calls; they help reveal instability but do not eliminate it.
Retain the result and the actual configuration
NeMo Agent Toolkit writes per-sample results to workflow_output.json, including generated_answer and intermediate_steps. It also saves config_original.yml, config_effective.yml, and config_metadata.json. The effective configuration includes command-line overrides, making it more useful for attribution than a configuration filename alone.
There is an important scope limit for remote evaluation. With --endpoint, the saved evaluation configuration does not capture the workflow configuration running on that remote server. Retain that deployed configuration separately. Otherwise you can reproduce the judge setup while unknowingly evaluating a different prompt.
Use evaluator outputs to investigate cases, not merely to rank averages. An evaluator configured as _type: ragas with metric: AnswerAccuracy supplies an answer-quality signal. Schema acceptance and exact domain conditions need their corresponding deterministic checks or a custom evaluator. The toolkit supports registered custom evaluators; it does not infer your business acceptance criteria from the prompt text.
When profiling is enabled, standardized_data_all.csv includes latency and token counts. Compare these with the team’s release limits as well as with quality. During evaluation, HTTP 429 responses can indicate excessive test concurrency; the documented control is eval.general.max_concurrency. Reducing it helps separate a load-induced failed run from a prompt regression without disguising the failed requests as incorrect answers.
Key distinctions and trade-offs
Behavioral consistency versus text equality: Equivalent explanations can pass, while a fluent answer that violates a required constraint fails. Evaluate the property the application depends on.
Isolated regression versus integration testing: Fixed tool results make a prompt comparison attributable. Live tools test operational behavior. Keep both purposes explicit rather than interpreting every difference as a model defect.
Quality versus release eligibility: A mean judge score can rise while a critical slice fails. Acceptance rules must identify which failures block release and who can authorize a different product behavior.
Architecture / decision scenario
A legal-services firm uses a meeting companion to draft internal action lists. Client names must remain inside its approved serving environment, and an action owner may be named only when the transcript assigns one. The prompt owner proposes a model migration plus shorter instructions to reduce latency.
In a scenario suite of 240 recorded meetings, the candidate prompt on the candidate model improves mean answer score from 0.88 to 0.92 and reduces measured p95 latency from 4.8 to 3.6 seconds. It also invents owners on 11 cases, compared with two on the baseline. The release rule permits no invented owners in that critical slice.
A practitioner proposes shipping because the average improved. The release owner instead runs all four combinations with the same transcripts and fixed evaluator configuration, then examines the failing records in workflow_output.json. If shorter instructions cause the failure, the prompt revision is corrected; if the model change causes it, that model requires a different tested prompt or remains unreleased. The serving owner retains the approved endpoint’s deployed identity, and the meeting-product owner maintains the owner-assignment rule.
The remaining two baseline failures also need correction; a baseline is a comparison point, not a declaration of correctness. The team can restore the recorded prompt and model combination while developing a candidate that meets the rule. A faster but ineligible candidate does not become acceptable through a higher mean score.
Common exam traps
- Versioning the main instruction while changing examples or assembly code without a revision.
- Attributing differences to prompts when retrieved data and tool results also changed.
- Comparing only the combined prompt-and-model candidate against the baseline.
- Assuming a remote evaluation’s saved configuration captures the deployed workflow.
- Accepting a better average while a critical behavior regresses.
What to remember
Version the complete prompt artifact and its tested model configuration. Use stable fixtures to identify regressions, repeated runs to expose variation, and retained per-case results to explain failures. Release the combination that meets the required behavior and cost limits, with a recorded combination available for restoration.
Knowledge check
A release changes both the prompt and the model of a meeting-notes agent. On the same 240 recorded meetings, the combined candidate invents an action owner in 11 cases, against two for the baseline, and the release rule permits none. The team must learn whether the prompt or the model causes the regression. Which runs should they add?
--reps=5temperatureA team evaluates a deployed workflow with nat eval --endpoint and keeps every file the run saves. Weeks later, it cannot tell which system prompt the deployed workflow used at the time. What should the team have retained in addition?
config_original.yml saved in the evaluation outputconfig_metadata.json with the --endpoint argumenteval.evaluators settings used for that evaluation run--endpoint , the saved configuration captures the evaluation settings — dataset, evaluators, endpoint URL — but not the workflow configuration running on the remote server, so that configuration has to be retained separately. The original configuration, the run metadata and the evaluator settings all describe the evaluation, which is why the judge setup can be reproduced while the evaluated prompt is unknown.A prompt comparison runs both prompt revisions against the live document index and live order tools. The index was re-ingested between the two runs, and the answers differ on 30 cases. The team wants to attribute differences to the prompt and still test the live integrations. Which setup fits?
--reps=5eval.general.max_concurrency before rerunning the comparisonDuring a nat eval comparison, many samples of the candidate prompt fail with [429] Too Many Requests, and its score falls below the baseline’s. What should the team do before judging the prompt?
eval.general.max_concurrency and rerun the evaluation--reps and compare the averaged scorestemperature and rerun the evaluationeval.general.max_concurrency ; reducing it separates a load-induced failed run from a prompt regression. Counting the failures as wrong answers disguises failed requests as incorrect ones, more repetitions add load, and temperature changes generation rather than the request rate.References
- NVIDIA, Configure the ReAct Agent
- NVIDIA, Workflow Configuration
- NVIDIA, API Reference
- NVIDIA, LLMs
- NVIDIA, Agent Evaluation in NVIDIA NeMo Agent Toolkit