Test memory across identities and sessions

Learning objective

Evaluate memory recall, entity continuity, and cross-session consistency.

Core idea

A correct answer inside one conversation does not prove that an agent remembers anything. The conversation might still contain the original fact. Evaluate memory by separating what the system stores, what it retrieves, and what the receiving agent uses after you start a separate conversation.

For architects evaluating multi-agent systems, the test is continuity of meaning and identity across time. A customer preference must remain associated with the right customer, an order reference must remain associated with the right order, and a correction must change subsequent behavior. Repeating the same sentence is neither necessary nor sufficient.

Microsoft Foundry Agent Service Memory and the Memory Store API are preview features. Use their documented operations as evidence, and keep your memory acceptance criteria explicit. See 2.2.5–2.2.7 for persistence choices, access controls, and lifecycle configuration; this lesson tests their observable consequences.

How it works

Build a sequence rather than isolated questions

Define each evaluation case as a sequence with a stable subject identifier, facts introduced in one conversation, later probes, and expected facts at each checkpoint. Include facts the agent should retain, corrections that supersede them, and information that must not transfer to another subject. Specify the expected outcome before running the sequence.

Test memory recall with both direct and paraphrased probes. If the first conversation states a dietary restriction, a second conversation can ask for a suitable meal without repeating the restriction. Score whether the necessary fact is available and whether the answer respects it. Test both memory access paths: search_memories with scope but without items or previous_search_id returns static user-profile memories; with items set to the latest messages, it retrieves relevant contextual memories.

Test entity continuity with deliberately similar entities. Give one customer two orders, or two customers similar names, then ask about a specific order using its identifier. Record which entity each retrieved fact describes. A plausible shipping date attached to the wrong order is an entity failure even if the date itself was stored correctly.

Test cross-session consistency with a separate conversation that receives no copied transcript. Repeat equivalent probes across conversations and agents authorized to use the same memory. Judge stable facts and decisions rather than identical wording. Include a correction between probes so that the test distinguishes consistency from stubborn repetition of obsolete information.

Observe the write before testing the read

The Memory usage page distinguishes static memories injected at conversation start from contextual memories retrieved per turn. It also documents an internal update_memories call after an agent response, with actual writes debounced by update_delay: the update completes after the configured inactivity period. An immediate follow-up in another conversation can therefore test an unfinished write rather than recall quality.

For a controlled Python API test, use project_client.beta.memory_stores.begin_update_memories() with the store name, explicit scope, and source items. Its documented sample sets update_delay=0 and waits on the returned poller’s .result(). Preserve the resulting memory_operations, including each operation’s kind and memory_item.memory_id, as the write checkpoint. After completion, use project_client.beta.memory_stores.list_memories(name=..., scope=...) and inspect the returned items as the storage checkpoint. Use get_memory(name=..., memory_id=...) to inspect a particular record by ID. Triggering an update is different from observing completion and stored content.

Do not replace a production-delay test with an immediate-write fixture and claim that you measured production timing. Use completed writes to isolate recall accuracy, then run a separate case with the deployed update_delay to measure when the fact becomes usable. Record whether a probe occurred before or after that checkpoint.

Inspect retrieved evidence and scope

The Python project_client.beta.memory_stores.search_memories() operation accepts name, scope, and query items; MemorySearchOptions(max_memories=5) in the documented example bounds the returned count. Inspect the memories collection and each returned memory_item.memory_id and memory_item.content. Five is an example test setting, not a guaranteed recall count or a service maximum.

Compare those records with the expected facts. If the correct fact is absent after a confirmed update, inspect extraction and retrieval. If the correct fact is present but the answer contradicts it, investigate the receiving agent’s use of context. Keep separate scores for required-fact recall, wrong-entity association, and answer consistency; these are application-defined measures, not names of built-in Foundry evaluators.

Scope is part of the experiment. Low-level memory APIs require an explicit scope. The memory search tool can use scope="{{$userId}}", resolving the x-memory-user-id header when present and otherwise the caller’s Microsoft Entra tenant ID and object ID. In a backend test, accidentally omitting the header can make multiple synthetic users resolve to the same caller identity. Validate the test’s resolved subject before interpreting cross-user recall as a model defect.

Exercise changes and negative cases

Include replacement, forgetting, and expiration cases alongside successful recall. The memory search tool applies explicit remember-or-forget requests immediately and returns memory command items in response output. Preserve those results, list the scope with list_memories to confirm the forgotten item is absent, and probe from another conversation. A relevance-ranked search that omits an item does not prove deletion.

For expiration tests, use the configured default_ttl_seconds, and remember that 0 means no expiration. Microsoft documents that updating and consolidating memory resets its last-updated time, and that TTL applies only to stores created after TTL support was introduced. A repeatedly updated fixture cannot demonstrate expiration from the original creation timestamp.

Keep unrelated facts as negative controls. A memory search can return context that is valid for the subject but irrelevant to the question. The evaluation should reject answers that force that fact into the response or mix it with another entity. Memory usefulness includes restraint as well as recall.

Key distinctions and trade-offs

Stored versus recalled versus used: A completed update plus a direct scope listing establishes storage evidence. A search result establishes retrieval evidence. A correct answer establishes use for that case. None substitutes for the other two.

Semantic continuity versus textual identity: Stable meaning permits different wording. Identical wording can preserve an obsolete fact and fail a correction test.

Shared subject versus separate subject: Two authorized agents can legitimately read the same subject’s memory. Two customers sharing a scope invalidate an isolation test. Diagnose scope before adjusting the model.

Broad memory policies can improve apparent recall while accumulating unrelated material. Narrow policies reduce noise but can omit facts a later task needs. Evaluate both required-fact coverage and irrelevant-fact use against the application’s declared retention purpose.

Architecture / decision scenario

A travel service has a planning agent and a booking agent. In conversation A, customer C states a vegetarian preference and identifies trip T1. The fixture waits for a completed memory update, saves its operation evidence, and lists C’s scope to inspect stored facts. Conversation B has no transcript from A and asks the planner for meals on T1. The search checkpoint must retrieve C’s preference, and the answer must honor it.

Customer D has a different preference and a similarly named trip. A direct listing of D’s scope must contain none of C’s items, and a separate-scope probe must not retrieve C’s records. The team also asks about C’s second trip, T2, to ensure that trip-specific dates do not migrate from T1. These cases test entity continuity separately from preference recall.

C then corrects the preference and starts conversation 3 with the booking agent. The team checks the update result, searches the same authorized scope, and compares the booking decision with the corrected expected fact. Missing evidence identifies a memory-path problem; correct evidence followed by an obsolete choice identifies a consumption problem. A final forgetting case checks command output, the direct scope listing, and later probes, rather than accepting the agent’s acknowledgment as proof.

Common exam traps

• Testing a second turn with the original transcript still present and calling it cross-session recall.
• Probing before the debounced update completes and labeling the result a retrieval failure.
• Treating a stored record as proof that the agent retrieved or used it.
• Scoring identical text instead of stable facts and correct entity associations.
• Letting backend test users share the caller’s fallback identity unintentionally.
• Expecting expired memory when consolidation has refreshed its last-updated time.

What to remember

Evaluate a timed sequence with explicit identities and expected facts. Confirm writes, inspect retrieved memory records, and score downstream use independently. Cross-session consistency includes correct updates, entity separation, and forgetting; it does not mean preserving every earlier answer.

Knowledge check

References

• Microsoft, What is Memory?
• Microsoft, Create and use memory in Foundry Agent Service