Compare compacted summaries with authoritative facts

Learning objective

Detect summary drift after repeated context compaction.

Core idea

For an architect maintaining long-running agents, summary drift is a change in meaning introduced as conversation history is compressed into summaries. A summary might omit a condition, change an entity association, or turn a proposed action into an accepted decision. Repeatedly summarizing the reduced history can preserve and amplify that defect.

Detect drift by comparing each compacted representation with authoritative facts from the original record. Comparing only with the preceding summary can approve an error that entered several compactions earlier. Fluent wording and fewer tokens do not establish fidelity.

See 2.2.4 for the compaction design and 3.2.3 for outright removal by a sliding window. Here the diagnostic question is whether the reduced representation still means the same thing.

How it works

Identify the summarization mechanism and scope

Microsoft Agent Framework’s compaction framework is experimental in Python and .NET and applies to agents with in-memory history. The documentation says its local strategies have no effect on service-managed history. Do not assume a local summary policy governs a Foundry Agent’s service-managed conversation.

Python’s SummarizationStrategy uses a separate SupportsChatGetResponse client. It replaces older history with generated summary text, with a default prompt intended to preserve facts, decisions, preferences, and tool outcomes. That intent is not a fidelity guarantee.

The strategy triggers when included non-system message count exceeds target_count + threshold. Its target_count controls retained recent history, subject to group boundaries. It is not a maximum summary token count. The independent max_summary_input_tokens defaults to 8,000 estimated tokens for the summarizer prompt and transcript. Complete groups are selected within that budget; if none fits, summarization is skipped and the history stays unchanged.

.NET’s SummarizationCompactionStrategy uses an IChatClient and inserts a MessageGroupKind.Summary group. Its MinimumPreserved default is four recent non-system groups. These language-specific controls have different names and behavior. Keep them separate when interpreting a trace.

Preserve a baseline that summaries cannot rewrite

Choose the facts that determine correct decisions: entity identifiers, amounts and units, deadlines, commitments, approval status, rejected alternatives, and conditions attached to decisions. Store their source references and accepted values separately from generated summaries under the state ownership design in 1.2.5.

Build a compact evaluation fixture from that record. For example, record that customer C-19 approved a replacement only if its cost stays below a stated amount. A summary saying the replacement is approved without the condition is defective, even if the customer identifier and amount still appear somewhere.

Mark the distinction between authoritative business values and generated explanations. A summary can propose a fact for validation; it must not overwrite an accepted approval record merely because its sentence sounds more decisive. Inside a Python workflow, WorkflowContext.set_state() and get_state() can transfer accepted state to downstream executors. The executor still needs to select and inject the relevant facts; state storage alone does not make them model-visible.

Test multiple compaction generations

Run a controlled long-conversation fixture through several real compaction events. Retain the source fact set, the input selected for each summarization, and each resulting summary, with the strategy configuration and summarizer deployment recorded. Use minimized data rather than collecting unrelated conversation content.

Compare every generation against the same source facts. Check factual presence, exact values, entity relationships, negation, and decision status. Also check unsupported additions. A summary that changes “request approval” into “approval received” fails even if the next answer happens to be correct.

Every summary generation is compared with the same source facts, not with the summary before it.

Use Microsoft Foundry’s builtin.groundedness evaluator as a screen for unsupported additions, mapping the generated summary to response and the authoritative record to context. Use Response Completeness (preview) to screen for omissions, mapping the expected fact coverage to ground_truth and the summary to response. These LLM judges screen meaning; compare exact business fields such as identifiers, amounts, and approval status deterministically. See 3.1.2 and 3.1.6 for evaluator configuration and interpretation.

Separate loss before summarization from transformation during summarization. If an important group never enters the summarizer input because of max_summary_input_tokens, the missing fact starts with selection. If it enters but changes in the generated text, the summarization step introduces the defect. If the summary remains correct but the answer violates it, the downstream model use is the failing stage.

Probe the eventual behavior as well. Ask a later agent to make the decision that depends on the preserved condition and compare it with the authoritative expected outcome. This detects summaries that retain words but no longer provide a usable constraint. See 3.1.3 for response regression gates; the summary comparison and the response check answer different questions.

Change the mechanism that introduced the defect

Python’s SummarizationStrategy(..., prompt=...) accepts a custom summarization prompt; .NET uses summarizationPrompt. Use that control to require explicit preservation of identifiers, conditions, approval status, and unresolved questions. Avoid prompts that request only a smooth narrative when the consumer needs business constraints.

Adjust the selected-history budget when the necessary source groups are excluded. Python accepts tokenizer= for model-specific counting and max_summary_input_tokens=None to remove its summarizer input bound. Removing the bound does not remove the summarizer model’s own context limit, so validate the request budget rather than assuming unlimited input.

Keep authoritative facts available for reconstruction instead of repeatedly trusting an already defective summary. In .NET, registering CompactionProvider through ChatClientAgentOptions.AIContextProviders can persist synthetic summaries through ChatHistoryProvider, making a summary part of the input to a later compaction. Registering on ChatClientBuilder through UseAIContextProviders(...) compacts only in-flight context and leaves stored history unchanged. See 2.2.4 for choosing the registration point; inspect it to establish whether repeated compaction consumes original messages or a prior summary.

The documentation also warns that Python summarization sends tool names, arguments, results, exceptions, call IDs, and approval decisions to the summarizer. Treat that client as a recipient of the same sensitive evidence, not as a harmless text-shortening utility. Changing summarizer models requires both a fidelity comparison and an appropriate data-processing decision.

Key distinctions and trade-offs

Drift versus omission: Truncation removes evidence. Summary drift can remove evidence or alter its meaning while leaving related text present.

Summary fidelity versus answer correctness: A correct summary does not guarantee a correct answer, and a correct answer can hide a defective summary. Measure both stages.

Adjacent comparison versus source comparison: Agreement with the previous summary tests consistency. Agreement with the authoritative record tests whether a repeated error persists.

Shorter context versus sufficient context: Aggressive compression reduces input size but can erase the conditions required for valid decisions. Preserve those conditions explicitly and measure their downstream use.

Architecture / decision scenario

A procurement assistant summarizes a long negotiation several times. The original record states that supplier S-8 offers a discount only for payment after inspection and that purchasing approval remains pending. A later summary describes the discount as unconditional and purchasing as approved.

The team compares each summary generation with the original record. The first summary preserves both conditions; the second changes approval status, and the third loses the inspection requirement. Both facts entered the relevant summarizer inputs, so this is transformation drift rather than window removal.

The team changes the custom summary prompt to preserve conditions and unresolved approvals as explicit fields in the narrative contract. It reconstructs the context from the authoritative negotiation and approval records instead of refining the defective third summary. A multi-generation fixture checks every summary and the final purchase recommendation. Approval remains enforced by the workflow gate described in 2.4.6; generated summary text cannot satisfy that gate.

Common exam traps

• Treating readable summary prose as proof of preserved meaning.
• Comparing only adjacent summaries and inheriting their shared error.
• Confusing target_count with the summarizer input token budget.
• Assuming every old group enters the summarizer input.
• Fixing the final answer while leaving a defective summary in circulation.
• Letting a generated approval sentence replace an approval record.

What to remember

Keep an authoritative baseline and compare every compaction generation with it. Locate whether selection, summarization, or downstream use introduced the defect. Reconstruct from verified facts and test the resulting decisions, not only the summary’s readability.

Knowledge check

References

• Microsoft, Compaction
• Microsoft, Workflow state
• Microsoft, Retrieval-Augmented Generation evaluators