Durable Agent Memory on Azure: What Changes When Your Agent Remembers?

Agent memory is becoming one of those features that looks simple in a demo and gets complicated the moment you put it near real users.

This week Microsoft announced a preview integration that gives Microsoft Agent Framework agents durable, cross-session memory backed by Azure Cosmos DB. The Python package, agent-framework-azure-cosmos-memory, adds a CosmosMemoryContextProvider that can store conversation turns, extract durable facts, and recall relevant context in later runs.

That sounds like a small library announcement. It is not. It is a sign that agent platforms are moving past “chat history in a database” and toward a more explicit memory layer: one that has identity, retrieval, extraction, isolation, retention, observability, and cost as first-class design concerns.

In this post I want to look at what changed, why it matters, and how I would approach this if I had to ship it in a production Azure application.

The Short Version

The new Cosmos DB integration gives Python agents a managed pattern for long-term memory:

• Before an agent run, it searches for relevant memories and adds them to the model context.
• After the run, it stores the turn and lets the memory pipeline extract durable facts, summaries, profiles, and other useful records.
• Cosmos DB stores both the raw operational data and derived memory, using vector, full-text, and hybrid retrieval.
• A stable user_id scopes recall across sessions, so a new conversation can remember facts from an earlier one.

For Python teams already building on Microsoft Agent Framework and Foundry, this is a useful step forward. For .NET teams, the picture is more mixed: Hosted Agents and the C# hosting story have improved a lot, but first-party durable memory is still not symmetrical with Python. You can build it, but you need to own more of the plumbing.

The key point: durable memory is no longer a novelty feature. It is becoming infrastructure.

Why Agent Memory Is Hard

Most teams start with the obvious version of memory: save the transcript and replay some of it into the next request.

That works for a prototype. It does not scale well.

Conversation history is not the same thing as memory. A transcript says what happened. Memory should say what still matters. If a user spends ten minutes debugging a deployment issue, the agent does not need to recall every failed command forever. It might need to remember that the service runs in West Europe, the team uses managed identity, and the production environment cannot use public network access.

That distinction matters because agent context is expensive and limited, even when context windows keep getting bigger. Dumping everything back into the prompt creates noise, increases latency, and makes the agent more likely to act on stale information.

A useful memory system needs to answer a few practical questions:

• What should be remembered?
• Who is the memory about?
• When should it be recalled?
• How do we know it is still true?
• How can a user inspect, correct, or delete it?
• How do we prevent one user's memory from leaking into another user's context?

Those questions are product, architecture, and security questions at the same time.

What Microsoft Announced

The new preview package is focused on Python:

pip install --pre agent-framework-azure-cosmos-memory agent-framework-foundry

You configure Cosmos DB, Foundry, an embedding model, and a chat model, then attach CosmosMemoryContextProvider to the agent's context_providers collection. The provider participates in the Agent Framework lifecycle:

• before_run: retrieve relevant memories and add them to context.
• after_run: store the new interaction and trigger extraction/consolidation work.

Cosmos DB is doing more than acting as a document store. The design leans on its ability to support vector search, full-text search, and hybrid retrieval in the same operational database. That is important because memory lookup is rarely just semantic. Sometimes a precise term, project name, environment name, or ticket number matters more than embedding similarity.

Microsoft also published a sample that makes the difference between session history and long-term memory visible. In that model, a Foundry Agent Service session represents one conversation, while a stable user ID scopes durable memory across conversations. Start a new conversation and the session changes; keep the user ID and the memory remains available.

That separation is the right mental model. Sessions are temporary. Identity-scoped memory is durable.

Where This Fits In An Azure Agent Architecture

If you are building on Azure today, the surrounding agent platform has become more concrete over the last few months:

• Microsoft Foundry Hosted Agents provide a managed runtime for agent applications.
• The Foundry Responses protocol gives hosted agents an OpenAI-compatible endpoint.
• Foundry handles deployment concerns such as endpoint hosting, lifecycle, identity, scale, sessions, traces, and evaluations.
• Agent Framework gives you the application-level programming model.
• Cosmos DB can now act as the durable memory layer for Python agents.

A production shape might look like this:

1. The user talks to your app, Teams bot, Copilot extension, or internal portal.
2. Your backend invokes an agent hosted in Foundry or your own runtime.
3. The agent receives the stable user identity and current session context.
4. The memory provider retrieves only the most relevant durable memories.
5. The agent runs with tools, grounding, policy, and memory.
6. After the run, the provider stores the interaction and extracts durable memory asynchronously.
7. Observability captures what was recalled, what tools ran, and what changed.

The important detail is step 4. Memory should be selective. The goal is not to make the prompt bigger. The goal is to make it better.

Python Gets The Cleaner Path For Now

For Python developers, the announcement gives a relatively direct path:

memory = CosmosMemoryContextProvider(
    cosmos_endpoint=config.COSMOS_ENDPOINT,
    cosmos_database=config.COSMOS_DATABASE,
    foundry_endpoint=config.FOUNDRY_PROJECT_ENDPOINT,
    embedding_model=config.EMBEDDING_MODEL,
    chat_model=config.CHAT_MODEL,
    credential=credential,
)

agent = FoundryAgent(
    ...,
    context_providers=[memory],
)

The exact API may change before general availability, but the design is clear: memory becomes a composable provider in the agent lifecycle. That is the right abstraction. Application code should not manually stuff old facts into prompts or decide when to run embedding searches. It should configure the memory provider, pass identity correctly, and set policy around what can be remembered.

The practical concerns are still yours:

• Use stable, tenant-aware user IDs.
• Separate development, test, and production memory stores.
• Decide which memory types are enabled.
• Log what was recalled for debugging and audit.
• Give users a way to correct or delete memory.
• Set retention rules before the first production conversation.

The provider helps with mechanics. It does not replace governance.

The .NET Gap

The .NET story is interesting because it is strong in some places and behind in others.

On the hosting side, .NET developers have a much better route than they did a year ago. A C# console agent can be adapted for Foundry Hosted Agents with the Foundry hosting package, the Responses protocol, and Azure Developer CLI commands. Foundry then gives you managed infrastructure, scaling, identity, sessions, traces, evaluations, and versioning.

That is good progress.

But durable memory is not yet as clean as the Python path. If you want Cosmos-backed memory in .NET today, you should expect to build more yourself: chat history storage, semantic retrieval, memory extraction, consolidation, and the policy around when to recall it. You can do it with Cosmos DB, Microsoft.Extensions.VectorData, embeddings, and your own providers, but it is not the same as installing one first-party memory package and attaching it to the agent.

That does not mean .NET teams should wait. It means they should be honest about ownership.

If I were building this in .NET right now, I would keep the interface small:

public interface IAgentMemoryStore
{
    Task<IReadOnlyList<AgentMemory>> RecallAsync(
        string tenantId,
        string userId,
        string input,
        CancellationToken cancellationToken);

    Task RecordTurnAsync(
        string tenantId,
        string userId,
        AgentTurn turn,
        CancellationToken cancellationToken);
}

Then I would hide the Cosmos DB implementation behind that contract and treat the extraction pipeline as replaceable. The mistake would be scattering vector search, transcript storage, and prompt formatting across the application.

Memory Needs Boundaries

The biggest risk with durable memory is not technical failure. It is overreach.

A helpful agent remembering “Fanie prefers examples in C# and practical Azure guidance” is useful. An agent remembering sensitive incidental details from every conversation is not. The line can get blurry unless you design the system deliberately.

I would start with these rules:

• Remember preferences, stable project facts, and durable technical context.
• Do not remember secrets, credentials, raw access tokens, personal identifiers, or one-off emotional context.
• Prefer summaries over raw transcripts for long-term recall.
• Make tenant ID and user ID part of every memory key.
• Keep memory inspectable by support and, ideally, by the user.
• Record which memories were injected into each run.

That last point is underrated. If an agent gives a strange answer, you need to know whether the model invented it, a tool returned it, or memory injected stale context. Without recall tracing, memory bugs are painful to diagnose.

Cost And Latency

Memory adds work to every agent run. At minimum you now have:

• a retrieval query before the run;
• extra tokens in the model context;
• storage writes after the run;
• background extraction or summarization;
• embedding generation for new memory records.

None of that is free.

The right approach is to make memory conditional. A simple classification step can decide whether a run needs long-term recall at all. A password reset agent may not need durable memory. A project assistant, code assistant, sales copilot, or operations agent probably does.

Also be strict about the number of memories returned. Five relevant memories are often better than thirty loosely related ones. You want context with high signal, not a scrapbook.

How I Would Ship This

If a team asked me to add durable memory to an Azure agent, I would not start by enabling every memory type and hoping for the best. I would ship it in stages.

Stage 1: Explicit Memory Only

Start with facts the user or application explicitly marks as durable. For example:

• preferred language or framework;
• project name and environment names;
• repository URLs;
• deployment region;
• team conventions.

This gives you the storage, retrieval, identity, and observability path without immediately trusting automatic extraction.

Stage 2: Assisted Extraction

Let the memory pipeline suggest facts, but require confidence thresholds and filtering. Store the proposed memory with provenance: source conversation, timestamp, extraction model, and reason.

Do not skip provenance. Six months later, someone will ask why the agent thought a fact was true.

Stage 3: Personalization And Workflow Memory

Once the basics are stable, use memory to improve workflows:

• preselect default Azure regions;
• remember preferred stack choices;
• reuse architectural constraints;
• adapt examples to the user's common language;
• avoid repeating explanations the user already knows.

This is where memory starts feeling valuable instead of decorative.

What To Watch Next

I would keep an eye on three areas.

First, parity across languages. Python often gets the first version of agent capabilities. For enterprise Azure teams, .NET parity matters. A first-party Cosmos memory provider for .NET would make the architecture much cleaner.

Second, evaluation. Memory should be evaluated like any other retrieval system. You need tests for recall precision, stale memory, isolation, and harmful recall. “The agent remembered something” is not enough. It has to remember the right thing at the right time.

Third, user controls. The more useful memory becomes, the more important it is to expose it. Users should be able to say “forget that”, “remember this”, and “what do you know about this project?” without filing a support ticket.

Final Thought

Durable memory is one of the pieces that turns an agent from a stateless prompt runner into a useful long-term assistant. But it also turns the architecture into something that needs real engineering discipline.

The Cosmos DB integration for Microsoft Agent Framework is a good sign because it puts memory where it belongs: in the platform architecture, not in ad-hoc prompt glue. For Python teams, it is now practical enough to prototype seriously. For .NET teams, the hosting story is strong, but durable memory still needs careful custom implementation.

My advice is simple: do not add memory because it demos well. Add it where repeated context genuinely improves the workflow, keep the recall small and explainable, and treat every remembered fact as data your system is responsible for.

That is the difference between an agent that remembers and an agent that is safe to trust.