AI Agents’ Memory and State
Developing an AI agent can be challenging. It is quite difficult to keep it on course for a six-month deployment.
By design, LLMs are stateless. Every call begins anew, with no recollection of previous calls. In order to get around this, early agent developers just threw all of the conversation history into the context window and hoped for the best.
We already know that strategy fails quickly. Latency increases and the model’s capacity to use context deteriorates; essential facts are obscured, and there is no assurance that it will select the current version of a fact when two versions are available. Although rapid caching has lessened the blow for stable prefixes, token costs are still skyrocketing. Treating memory and state as intentional design choices rather than afterthoughts is the solution, not expanding the context window.
It’s important to clarify the meaning of those two terms before delving into the patterns, as they can be easily confused.
A state is a moment in time. It contains all of the information the agent presently has about a task, including the variables it is tracking, the step it is on, and the results of the most recent tool call. Consider it a whiteboard. Unless you consciously persist it, which is the focus of Pattern 2, it is updated continuously as the job moves forward and disappears at the conclusion of the session.
The system that transports information across boundaries—the next round, the next session, or an entirely different agent operating later—is called memory. Semantic and episodic memory span sessions, while working memory is the shortest-horizon example (turn to turn).
The two engage in a particular cycle of interaction. The agent loads pertinent information, relevant behavioral norms, and records of previous failures on related tasks from memory to construct its initial state at the beginning of a task. The agent continuously changes its state while working on the assignment. Select portions of that state are written back to memory as the work proceeds and ends, allowing the subsequent turn or session to take advantage of what just transpired. State feeds back into memory, and memory feeds into state.
Because the failure modes differ, this distinction is important. When an agent is in a broken condition, it loses focus while working on a task. When an agent has a broken memory, they are unable to learn, cannot customize, and approach each interaction as if it were their first. Production systems frequently experience both types of failures, which call for different solutions.
The following five patterns deal with both: Patterns 1 and 2 control state; Patterns 3 and 4 construct the persistent memory layer; and Pattern 5 limits both.
1. The Short-Term Execution In-Context Working Buffer
The Idea
The current session’s transient state, including the active prompt, recent conversational turns, and live tool outputs, are stored in working memory. Consider it the agent’s temporary scratch area, which is flushed at the conclusion of the session.
How It Operates
The working buffer functions as a sliding window instead of allowing the message list to expand endlessly. Immediate reasoning steps are written by the agent on a scratchpad. A summarization procedure removes the raw tool outputs and retains the logical conclusions while compressing older turns into a dense background summary when the buffer gets closer to a token limit. The buffer is flushed when the task is finished; everything valuable is taken out and stored for a long time, while the remainder is thrown away.
It’s important to note that mid-conversation summarization has the potential to rewrite the prompt prefix, invalidating the KV cache and causing a latency spike on the subsequent call. Designing around it is a true trade-off.
When to Apply It
This is essential for every agent. It serves as the foundation for managing multi-step reasoning during a session.
2. Execution Checkpointing (Pausing & Fault Tolerance)
After you have a plan for controlling the agent’s memory throughout a session, the next concern is what happens if the session is cut short.
The Idea
Long-term projects don’t work out. An agent may halt while waiting for a human to approve an activity, clock out, or reach a rate restriction. By storing the agent’s workflow information in a database, checkpointing allows execution to pick up where it left off without having to redo previously finished tasks.
How It Operates
Workflows are modeled as nodes and edges in graph-based frameworks. The framework saves the workflow information, including variables, history, and current position, to a durable store like PostgreSQL or SQLite after every step. In the event that the agent crashes, it restarts from the previous checkpoint.
One thing that frequently burns practitioners is that resumption does not provide exactly-once semantics. A node may resume execution if it sent an email or wrote a database row before crashing. Nodes that have side effects must be idempotent. Additionally, remember that client objects and open file handles cannot be checkpointed, which restricts what you may safely put in state.
When to Apply It
vital for any long-term activity vulnerable to network disruptions, regulated processes where activities require approval, and human-in-the-loop systems.
3. Cross-Session Knowledge (Semantic Memory)
Checkpointing manages task continuance. However, what about information that must endure over completely different sessions?
The Idea
The agent’s understanding of facts, user preferences, and domain knowledge that endures between separate sessions is known as semantic memory.
How It Operates
Asynchronously extracted facts are kept in an external database, which is often a vector store with metadata filtering and occasionally combined with a knowledge graph if relationship traversal is actually important. Before the model sees a question, the system retrieves the most pertinent information and inserts it into the prompt. Keep in mind that, depending on the architecture, extraction may require an extra LLM call or more, usually one every turn.
One issue that has to be resolved is that if a user says, “I use Postgres” in March and “we migrated to Snowflake” in July, both statements will be in the store. Retrieval may reveal either. The stale fact issue at the top is actually resolved by fact invalidation using TTLs, supersession logic, or recency weighting.
It’s also important to state clearly that secrets and credentials are not semantic memory. API keys should not be kept in a retrievable store. They may be released in a model response by a prompt injection or an overly eager retrieval. Secrets should be kept in a secrets manager, where the agent is given a credential handle that is never used.
Additionally, the inverse risk is important because untrusted input (such as a scraped page, a user message, or a tool output) that is extracted into semantic memory as a “fact” can continuously lead the agent in the incorrect path. The process is done via provenance tagging, which tracks the source of a fact and adjusts its scope of impact because there is no prompt equivalent of parameterization and no clear division between instructions and content.
When to Apply It
Enterprise agents, coding copilots, or personal assistants that must remember a user’s chosen code style, architectural rules, or database schema norms between sessions.
4. Historical Reflection: Episodic Event Logs
The agent’s knowledge is stored in semantic memory, whereas its actions are stored in episodic memory.
The Idea
The agent’s execution trajectory—Goal, Plan, Tool Calls, and Outcome—is chronologically recorded in episodic memory.
How It Operates
A background process tracks this entire trajectory after a workflow is complete. The agent queries this log prior to taking on a comparable assignment. The episodic memory brings up the context if it previously failed a database query because of a syntax issue, preventing the agent from making the same mistake twice.
Please note that recovered failure traces are not limitations but rather recommendations. They can be disregarded by the model. Additionally, there is a poisoning risk: you are continuously teaching the agent the wrong lesson if a single environmental failure is recorded as a strategy failure. Keep it in mind when you log in.
When to Apply It
Planning systems, data engineering pipelines, and autonomous coding agents that must learn from prior errors without human assistance.
5. Enterprise Privacy: Multi-Scope Segregation
The question is who can see the memory once it has persisted. Memory must be segregated as soon as your system is used by several users.
The Idea
There isn’t just one shared bucket of memory. User B must never discover a fact that was discovered while assisting User A.
How It Operates
Identity scopes such as user_id, session_id, and org_id are attached to each memory write. Retrieval is rigorously filtered using the auth token of the active user. Instead of depending only on application-layer query filters, try to enforce this at the store layer using row-level security or per-tenant namespaces. Storage-layer separation fails closed, and a forgotten WHERE clause fails open.
This is not the end goal, but rather a requirement for data privacy compliance. The more difficult issue is deletion: when a person exercises their right to erasure, you have to remove not only their raw data but also the summaries, embeddings, and facts that were taken from it.
When to Apply It
Any business deployment, multi-tenant system, or SaaS product where data boundaries need to be maintained.
Synopsis
Growth bounds are something that none of these patterns address by themselves. Semantic and episodic stores will accrue near-duplicates, out-of-date information, and noise over the course of a six-month deployment (the framing this post began with). As stores fill up, retrieval quality deteriorates, and costs increase accordingly. Operating memory at scale includes TTLs, consolidation jobs, and pruning strategies; these are not optional extras.
A database is not what the context window is. Decoupling memory into discrete components—semantic stores for facts, episodic logs for experience, and short-term buffers for execution—leads to systems that are able to learn, adhere to data boundaries, and function well in production.