NVIDIANVIDIA-Certified Professional: Agentic AI2026-105.2 Apply reasoning frameworks (chain-of-thought, task decomposition).
Bound voting, critique, and reflection loops
Completed by the knowledge check
Learning objective
Use self-consistency, reflection, and self-critique, and recognize compounding reasoning errors. Choose a check that can expose the likely mistake rather than adding another model call that repeats it.
Core idea
Self-consistency compares answers produced through multiple reasoning attempts. Self-critique evaluates a proposed answer against criteria. Reflection uses feedback about an attempt to change what happens next. The terms overlap: a reflection loop can include self-critique, and its reviewer can use the same model as its generator.
These techniques spend additional inference to improve a particular result. They do not turn agreement into ground truth, and a feedback loop need not update model weights. Reusing lessons across later tasks belongs to 5.5.1; the focus here is whether additional attempts improve the request being answered.
How it works
Compare answers rather than verbal confidence
For self-consistency, generate several candidate solutions without showing each candidate the others’ answers. Extract the final answer into a representation you can compare: a normalized amount and currency, an option identifier, or another task-specific value. Count equivalent answers and apply a selection rule. Voting over whole paragraphs confuses different wording with different conclusions.
NeMo Skills provides an inspectable implementation of answer aggregation in its evaluation code. MathMetrics uses predicted_answer as its default answer key and invokes _compute_majority_at_k. The base implementation excludes missing answers and counts answers to find the most common result. Its majority@k output measures the selected answer across multiple predictions. The documented Nemotron evaluation recipe runs multiple samples and reports majority@8 alongside pass@8.
Those metrics answer different questions. A majority-selected answer is what a vote returns; an oracle-style success measure over several attempts does not establish that a production selector can identify the correct attempt. Do not substitute “one of the samples was right” for “the answer returned to the caller was right.”
For a production assistant, define normalization, minimum usable sample count, ties, and disagreement handling as application policy. NeMo Skills’ benchmark aggregation is an example to inspect, not a claim that Toolkit workflows automatically perform a vote. A conservative policy can reject a tie or insufficient usable answers. More copies of the same mistaken assumption do not make that policy’s winning answer trustworthy.
Turn critique into actionable feedback
A self-critique should identify a violated requirement and the part of the answer that violates it. “Looks correct” offers no repair instruction. For an analytics report, useful feedback identifies the wrong date filter, a missing denominator, or an unsupported statement, then names the check that could settle it.
LangGraph’s evaluator-optimizer pattern provides a concrete loop. In its Python example, llm.with_structured_output(Feedback) produces a grade and feedback, and StateGraph.add_conditional_edges() routes an accepted result to END or a rejected result back to the generator with feedback. You adapt that structured feedback schema to your task’s criteria; its example’s grading labels are not universal correctness categories.
Keep the critique and the acceptance decision distinct. A model-generated critique can propose a repaired formula, but execution against the source data must establish whether the formula works. Store the candidate, critique, repaired result, and check outcome separately so that a revision does not erase which assumption changed. In a custom evaluator-optimizer graph, add application-defined attempt and budget state to its conditional routing; stop when the allocation is exhausted even if the evaluator keeps rejecting.
An accepted result goes to END, and a rejected result goes back to the generator with feedback until the allocation is exhausted.
Ground reflection in something the first attempt missed
NVIDIA RAG Blueprint implements a narrower, grounded reflection loop through ENABLE_REFLECTION. Its context-relevance check can trigger query rewriting, while its response-groundedness check evaluates whether an answer follows the retrieved context. MAX_REFLECTION_LOOP bounds refinement attempts and defaults to 3. The documented CONTEXT_RELEVANCE_THRESHOLD and RESPONSE_GROUNDEDNESS_THRESHOLD operate on a 0–2 score scale and default to 1.
Those are reflection controls, not guarantees that the answer is factually complete. A grounded response can still rely on incomplete context. Setting a higher threshold can add iterations, and each iteration adds latency. The documented implementation does not support response streaming during response-groundedness checks; Helm reflection supports only on-premises deployment. Account for those boundaries before selecting it for a streaming assistant. See 6.1.4 for agentic retrieval; bound the feedback-and-revision loop independently of the retrieval strategy.
Find the first unsupported assumption
Reasoning errors compound when later steps treat an earlier inference as an established fact. An incorrect customer identifier can produce an accurate lookup for the wrong account, followed by correct arithmetic and an irrelevant recommendation. A final prose critique may approve the arithmetic while missing the identity failure.
As an illustrative model, eight independent required steps with 95% correctness each yield about 66% probability that all eight are correct: 0.95^8. These are hypothetical rates, not model measurements. Real failures can be correlated, especially when candidates share a prompt, model, or faulty source. Do not apply the independence calculation as a production reliability estimate.
When candidates agree but conflict with an authoritative result, correct the shared input or failed dependency before resampling. When a critique discovers an unsupported premise, restart the affected dependent reasoning from the verified input, as in 5.2.2. Rewriting only the final sentence preserves the cause of the error.
Key distinctions and trade-offs
Choose the extra inference by what it can test:
| Technique | Deciding condition | Mechanism | Cost and failure mode |
|---|---|---|---|
| Self-consistency | A task has comparable final answers and alternative attempts may disagree | Candidate generation, answer normalization, vote; NeMo Skills majority@k illustrates aggregation | Multiplies inference and can reinforce a shared false premise |
| Self-critique | A candidate can be assessed against explicit criteria | Structured Feedback and an evaluator call | Adds review cost; the critic can share the generator’s blind spot |
| Reflection | Feedback can change an input, query, or answer | Conditional evaluator-optimizer loop; RAG Blueprint MAX_REFLECTION_LOOP | Adds sequential latency and can cycle without useful feedback |
A calculator or source lookup can be a stronger check than another free-form critique when the disputed fact is externally decidable. More inference earns its cost only when it changes a checked outcome. Count all candidate, critic, and revision calls against the request budget from 5.2.1.
Architecture / decision scenario
A retail analytics assistant drafts a weekly margin explanation. Its owner allows 12,000 generated tokens per report and requires exact agreement with the warehouse aggregate. Three independent narrative candidates are proposed for self-consistency. With an illustrative 2,000-token cap each, they reserve 6,000 tokens; one 1,000-token critique and one 2,000-token revision raise the ceiling to 9,000.
The tempting proposal is to return any margin percentage supported by two candidates. In a test case, all three use gross sales where the metric requires net sales, so their agreement preserves the same error. The analytics owner instead verifies the warehouse query and denominator, uses candidate agreement only for the normalized conclusion, and asks the structured critic to check that every causal claim names supporting data. Unsupported claims trigger one bounded reflection pass that retrieves the missing breakdown or removes the claim.
An optional RAG Blueprint groundedness loop could help source-based narrative claims, but its three-attempt default is not permission to spend beyond the remaining 3,000-token allocation. The owner must measure and budget its calls separately and accept its streaming restriction before enabling it. If the warehouse check still fails or the reflection allocation is exhausted, the report remains incomplete; another confident rewrite cannot authorize publication.
Common exam traps
- Counting duplicate answers as independent verification of a shared premise.
- Equating
pass@kwith the accuracy of the answer a voting selector returns. - Asking for critique without criteria or a repairable finding.
- Assuming reflection implies model-weight updates.
- Raising reflection thresholds without accounting for extra sequential latency.
- Repairing final prose while retaining the wrong upstream input.
What to remember
Self-consistency selects among attempts; self-critique assesses a candidate; reflection changes an attempt using feedback. Bound all three, preserve the original acceptance checks, and investigate the first unsupported assumption. Agreement and polished revision remain weaker than a check against the authoritative result.
Knowledge check
A billing assistant generates five independent candidate explanations of a refund and votes over them. The vote rarely finds a majority, although most candidates state the same refund of €42.50 in different words. What should the selector compare?
An offline evaluation of a math assistant reports, in this scenario, pass@8 of 85% and majority@8 of 66%. In production, callers will receive the answer a majority vote selects from eight samples. Evaluation and production use the same sampling and answer normalization, with no tied votes. Which figure estimates how often the answer callers receive is correct?
pass@8 , because a correct sample makes the selected answer correctmajority@8 , because it scores the answer selected from the samplesmajority@k measures the answer a vote returns. pass@k asks whether any attempt was right; it does not establish that a selector can identify that attempt, and a vote picks the most common answer, not the correct one. “One of the samples was right” is not “the returned answer was right”.Three independent candidates for a weekly margin report agree on 18.4%, but the warehouse aggregate, the authoritative source, gives 16.9%. Inspection shows all three used gross sales where the metric requires net sales. What should happen next?
A LangGraph evaluator-optimizer loop routes on the evaluator’s grade alone: add_conditional_edges() sends a rejected report back to its generator. On some requests the evaluator keeps rejecting, and the loop runs until the request times out. Which change bounds the loop?
max_tokens for each new revisionA team runs NVIDIA RAG Blueprint on-premises and wants self-reflection for a chat assistant whose users require streamed responses. Which part of the reflection feature conflicts with that requirement?
MAX_REFLECTION_LOOP default of three refinement attemptsReferences
- NVIDIA, Reproducing NVIDIA-Nemotron-Nano-9B-v2 Evals
- NVIDIA, MathMetrics implementation
- NVIDIA, BaseMetrics majority aggregation
- LangChain, Workflows and agents
- NVIDIA, Enable Self-Reflection for NVIDIA RAG Blueprint