NVIDIANVIDIA-Certified Professional: Agentic AI2026-105.1 Implement memory mechanisms for short- and long-term context retention.
Keep conversations within a token budget
Completed by the knowledge check
Learning objective
Implement conversation buffers, truncation, and rolling summaries.
Core idea
A conversation buffer is the ordered message history available to a conversation. Its persisted size and the amount you send to the model are separate decisions. As established in 1.4.1, saving history does not make the whole history suitable for every inference. Here the implementation question is which messages survive prompt assembly when the conversation outgrows its budget.
Use three strategies deliberately: retain the buffer while it fits, truncate expendable history when exact recent exchanges matter most, and summarize earlier exchanges when their decisions must survive. A rolling summary preserves selected meaning with fewer tokens, but it is another model-generated artifact that can omit or distort a constraint. Neither storage durability nor compression proves that the model receives enough information to continue correctly.
How it works
Bound the inference view
LangGraph represents conversation messages in MessagesState; a checkpointer persists state for configurable.thread_id. That gives successive turns access to the buffer. The model-call node decides which representation enters inference. NVIDIA documents workflow._type: langgraph_wrapper and its graph field for running an existing LangGraph graph through NeMo Agent Toolkit. The graph implementation owns the message-management logic described here.
Start with a token allocation for instructions, tool definitions, retrieved material, conversation history, and generated output. Count the complete assembled request against the selected model’s supported context. A message-count limit alone is weak: one tool result can contain more text than many ordinary turns. NeMo Agent Toolkit’s react_agent and rewoo_agent use max_history, default 15, to bound conversation history by messages, not tokens (see 1.2.2). Token budgeting and rolling summaries belong in the graph you run through langgraph_wrapper. LangGraph’s documented trim_messages utility accepts max_tokens, a token_counter, and strategy="last" to retain a bounded recent suffix.
The documentation’s count_tokens_approximately is an approximate counter. Use a margin or a model-appropriate counter when enforcing a hard request limit; a trim threshold is an application allocation, not a model’s context-window declaration. Keep fixed instructions outside the expendable conversation suffix and account for them separately.
Message boundaries also matter. The documented trimming example uses start_on="human" and end_on=("human", "tool") to constrain the retained history. Preserve valid assistant tool-call and tool-result relationships when choosing cut points. If an oversized tool output consumes the allocation by itself, reduce that output at its producer or select a smaller relevant excerpt before assembling the conversation. Removing unrelated earlier turns cannot make an unbounded result safe.
Distinguish trimming from state deletion
A model-call node can invoke the model with a trimmed local list while leaving state["messages"] intact. This bounds one inference view without claiming that discarded prompt messages disappeared from persistence. It is useful when exact history remains necessary for later inspection or another selection strategy.
To remove messages from active graph state, LangGraph uses RemoveMessage(id=...). That requires a message state key using the add_messages reducer, as MessagesState does. State removal and model-input selection serve different purposes; choose explicitly rather than assume that passing a shorter list modifies stored state. Checkpoint persistence and resumption are addressed in 5.4.2.
Maintain a rolling summary
LangGraph’s documented summarization pattern extends MessagesState with a summary field. A summarize_conversation node reads the previous summary, combines it with the messages being compressed, and generates a replacement summary. Its state update stores summary and uses RemoveMessage to remove the covered messages. Retain a recent suffix verbatim so immediate references and tool exchanges remain available.
The next model call must actually include the saved summary alongside the recent messages. Merely storing state["summary"] does not put it into the model’s input. Keep the summarization prompt explicit about decisions, unresolved questions, identifiers, and corrections that the task needs. Preserve the distinction between a user assertion, a tool result, and an assistant proposal instead of flattening them into equally certain facts.
The separate langmem package provides langmem.short_term.SummarizationNode as a prebuilt alternative to the hand-written node. Set max_tokens for the post-summarization message budget, max_tokens_before_summary for the compression trigger, and max_summary_tokens for the summary’s budget reservation. The latter does not limit generated length by itself: pass model.bind(max_tokens=max_summary_tokens) as the summarizing model to enforce that output limit. The documented graph stores RunningSummary in context and passes summarized_messages to the model-call node.
Bound the summary itself; repeatedly extending it recreates the original token problem. With the hand-written node, commit the summary and covered-message removal together as the graph state update. If summarization fails, keep the covered messages available; pruning first destroys the material needed to retry. This failure-handling rule is application control logic.
The summary and the covered-message removal commit together, and the next model call includes the saved summary alongside the recent messages.
Key distinctions and trade-offs
Choose according to what must survive, rather than whether the history can be stored:
| Strategy | Deciding condition | Mechanism | Cost and failure risk |
|---|---|---|---|
| Full conversation buffer | The entire relevant exchange fits the allocation | MessagesState passed to the model-call node | Exact wording survives, but prompt tokens and inference work grow |
| Recent-history truncation | Earlier turns are expendable for the task | trim_messages, max_tokens, strategy="last" | No summarization call; earlier constraints can disappear abruptly |
| Rolling summary plus suffix | Earlier decisions matter more than their full wording | summary, summarize_conversation, RemoveMessage | Adds model work and possible semantic drift; bounds repeated prompt size |
A summary is suitable for continuity, not for replacing an exact transaction record. When a specific identifier or quoted clause determines the answer, keep or retrieve that exact material. Evaluate compression on continuation tasks that require corrections and distant dependencies, rather than judge whether the summary reads fluently.
Architecture / decision scenario
An automated meeting companion in a pharmaceutical company drafts actions during a 90-minute project meeting. The team’s chosen inference allocation permits 12,000 tokens per request: 2,000 for instructions and tool definitions, 2,000 for retrieved project material, 2,000 reserved for output, and 6,000 for conversation. These are scenario allocations, not product limits. The transcript reaches 18,000 tokens before the meeting ends.
The tempting proposal is to keep the full transcript because the database can store it. That exceeds the 6,000-token conversation allocation. Keeping only the last 6,000 tokens fits, but an action-owner correction from minute 15 is still relevant at minute 80. Pure truncation therefore fails the continuity requirement, even though it meets the size target.
The workflow engineer chooses a rolling summary capped at 1,000 tokens plus up to 5,000 tokens of recent messages, cutting at valid exchange boundaries. While the buffer fits, the graph uses it directly. Near the threshold, summarize_conversation carries forward action owners, corrections, and unresolved approvals, then removes covered messages from active state. The exact transcript remains a separately managed record.
The product owner requires continuation tests in which an earlier owner is corrected and the final action list must use the correction. The platform owner measures compression latency and total prompt tokens. If compression delays live responses, the team moves its trigger earlier and measures again; increasing the model-call allocation without measuring would hide the original capacity constraint.
Common exam traps
- Treating a persistent buffer as an unlimited inference context.
- Treating
max_historyas a token limit. - Applying a message-count cap while ignoring a single large tool result.
- Assuming
trim_messagesautomatically deletes messages from graph state. - Deleting covered messages before a replacement summary is successfully produced.
- Storing a summary without including it in the next model call.
- Assuming repeated summarization preserves exact wording or validates earlier assistant claims.
What to remember
Budget the whole request, then allocate conversation tokens. Retain the buffer while it fits, trim when distant turns are expendable, and use a bounded rolling summary plus recent messages when earlier decisions must survive. In LangGraph, trim_messages selects an inference view; summary and RemoveMessage implement compression in state. Test the continuation behavior that compression can break.
Knowledge check
A LangGraph support agent extends MessagesState with a summary field. After its summarize_conversation node runs, the checkpoint shows a summary that states the refund limit agreed early in the conversation, and the covered messages are gone from state. Yet the agent’s next answers ignore that limit. Which defect fits this evidence?
add_messages reducer that removal needsstate["summary"] does not put it into the model’s input; the next call has to send the summary alongside the recent messages. Here the summary holds the limit and removal took effect, making omission from the model input the defect that fits. Those stored-state observations alone do not establish that the model received the summary.A project-planning assistant has a scenario allocation of 4,000 conversation tokens per request, and long sessions reach 10,000 tokens of history. A change to the approved rollout region, made early in a session, must shape the final plan; its exact wording does not matter. Which history strategy meets both requirements?
strategy="last"A NeMo Agent Toolkit react_agent runs with max_history: 6. Requests still overflow the model’s context whenever a log-search tool returns its full raw output, which by itself is larger than the model’s whole context window. Which change addresses the overflow?
max_history to keep fewer messagesmax_tool_calls so the agent makes fewer tool callsmax_history bounds the history by message count, not tokens, so one oversized tool result passes any message cap. Removing or summarizing unrelated earlier turns cannot make a single result that exceeds the context window fit, and one call is enough to overflow. Reduce the output at its producer, or select a smaller relevant excerpt, before the conversation is assembled.A LangGraph model-call node passes trim_messages(state["messages"], max_tokens=..., strategy="last", ...) to the model. The team now needs older messages removed from the graph’s active state, not only from the model input. The state is MessagesState. What implements that?
max_tokens passed to trim_messagesRemoveMessage(id=...) updates for the older messagesstart_on="human" and end_on to trim_messagestrim_messages selects a bounded inference view and leaves state["messages"] intact, whatever its token limit or boundary settings. Removing messages from state is a separate operation: RemoveMessage(id=...) updates on a key with the add_messages reducer, which MessagesState provides. A checkpointer persists state for a thread; it does not decide what the state holds.A team uses langmem’s SummarizationNode with max_summary_tokens=512 and passes its chat model unchanged as model. Generated summaries regularly run past 1,500 tokens and crowd out the recent messages. What enforces the summary’s length?
max_tokens_before_summary so that summarization starts earliermodel.bind(max_tokens=512) as the summarizing modelmax_tokens to shrink the post-summarization message budgetmax_summary_tokens below the length the summaries should reachmax_summary_tokens reserves room for the summary in the budget calculation; it is not passed to the model that writes the summary. Binding max_tokens to the summarizing model is what limits generated length. max_tokens_before_summary sets the trigger and max_tokens the message budget after summarization — neither bounds the summary itself.References
LangChain, Memory
NVIDIA, Running Existing LangGraph Agents in NVIDIA NeMo Agent Toolkit
NVIDIA, Configure the ReAct Agent
NVIDIA, Configure the ReWOO Agent
LangChain, Short Term Memory API Reference