Memory systems for agents

The five layers practitioners use

In-context memory

In-context memory is whatever you pass in the messages array this turn. It is the fastest layer—zero network—but bounded by the model context window and subject to lost-in-the-middle effects. Keep raw turns for the last 4–8 exchanges; compress older turns via summarization (see write strategies).

• Include: current user goal, recent tool results, active plan steps.
• Exclude: full PDF text, duplicate RAG chunks, PII you would not log.
• Pattern: system + summary block + last K turns + retrieved memory snippets.

External RAG (read-mostly knowledge)

External RAG is not chat memory—it is your knowledge base retrieved on demand. Agents extend RAG with tools: search_docs(query) instead of stuffing all chunks every turn. See RAG explained and vector databases for ingest and ANN details. Do not conflate “user said they prefer email” (entity memory) with “refund policy section 4.2” (RAG).

Entity memory

Entity memory stores typed slots extracted from dialogue: preferred_language=es, timezone=America/Chicago, subscription_tier=pro. Unlike episodic blobs, entities are queryable without embedding search—ideal for personalization and tool argument defaults.

from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass
class UserEntity:
    user_id: str
    key: str
    value: str
    confidence: float
    source_turn: int
    updated_at: datetime

def upsert_entity(conn, entity: UserEntity) -> None:
    """Merge only if confidence improves or value changed."""
    conn.execute(
        """
        INSERT INTO user_entities (user_id, key, value, confidence, source_turn, updated_at)
        VALUES (%(user_id)s, %(key)s, %(value)s, %(confidence)s, %(source_turn)s, %(updated_at)s)
        ON CONFLICT (user_id, key) DO UPDATE SET
          value = EXCLUDED.value,
          confidence = EXCLUDED.confidence,
          source_turn = EXCLUDED.source_turn,
          updated_at = EXCLUDED.updated_at
        WHERE EXCLUDED.confidence >= user_entities.confidence
           OR user_entities.value IS DISTINCT FROM EXCLUDED.value
        """,
        {
            "user_id": entity.user_id,
            "key": entity.key,
            "value": entity.value,
            "confidence": entity.confidence,
            "source_turn": entity.source_turn,
            "updated_at": datetime.now(timezone.utc),
        },
    )

public record UserEntity(
    String userId, String key, String value,
    double confidence, int sourceTurn, Instant updatedAt) {}

public void upsertEntity(JdbcTemplate jdbc, UserEntity e) {
  jdbc.update("""
    INSERT INTO user_entities (user_id, key, value, confidence, source_turn, updated_at)
    VALUES (?, ?, ?, ?, ?, ?)
    ON CONFLICT (user_id, key) DO UPDATE SET
      value = EXCLUDED.value,
      confidence = EXCLUDED.confidence,
      source_turn = EXCLUDED.source_turn,
      updated_at = EXCLUDED.updated_at
    WHERE EXCLUDED.confidence >= user_entities.confidence
       OR user_entities.value IS DISTINCT FROM EXCLUDED.value
    """,
    e.userId(), e.key(), e.value(), e.confidence(), e.sourceTurn(), e.updatedAt());
}

Episodic memory

Episodic memory captures events: “On March 3 the user escalated billing issue #8821 and we issued a credit.” Store a short summary + embedding + metadata (session_id, outcome, topics). Retrieve when the user references past sessions (“what did we decide last time?”) or when semantic similarity to the current query is high.

Procedural memory

Procedural memory is how the agent operates—not what it remembers about the user. System prompts, tool schemas, retry policies, and LangGraph node definitions are procedural. Version them in Git; load by environment; never let the model rewrite its own guardrails into procedural storage without human review.

How the types compose in one turn

1. Load procedural config (system prompt, tools) from registry.
2. Fetch entity slots for user_id from Postgres/Redis.
3. Retrieve episodic snippets (semantic + recency) and optional RAG chunks via tools.
4. Append in-context last K turns from session store.
5. Run agent loop; write new entities/episodes per policy.

Three policies compared

Always write

Append each turn to an immutable event log (Kafka, Postgres chat_events). Optionally embed asynchronously—decouple write path from the user-facing latency budget. Use compaction jobs to roll raw events into summaries nightly; raw events remain for compliance, not for retrieval.

Selective write

After each assistant response, run a lightweight classifier (small model or rules) with a structured output schema:

• should_persist: bool
• memory_type: entity | episodic
• facts: [{key, value, confidence}]
• summary: str | null

Gate on confidence and blocklist sensitive patterns (credit card fragments, health data) before any upsert.

import json
from openai import OpenAI

client = OpenAI()
EXTRACT_SCHEMA = {
    "type": "object",
    "properties": {
        "should_persist": {"type": "boolean"},
        "memory_type": {"type": "string", "enum": ["entity", "episodic", "none"]},
        "facts": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "key": {"type": "string"},
                    "value": {"type": "string"},
                    "confidence": {"type": "number"},
                },
                "required": ["key", "value", "confidence"],
            },
        },
        "summary": {"type": ["string", "null"]},
    },
    "required": ["should_persist", "memory_type", "facts", "summary"],
}

def extract_memories(user_msg: str, assistant_msg: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract durable user facts or session summaries. Ignore small talk."},
            {"role": "user", "content": f"User: {user_msg}\nAssistant: {assistant_msg}"},
        ],
        response_format={"type": "json_schema", "json_schema": {"name": "memory", "schema": EXTRACT_SCHEMA}},
    )
    return json.loads(resp.choices[0].message.content)

def apply_selective_write(store, user_id: str, payload: dict) -> None:
    if not payload.get("should_persist"):
        return
    for fact in payload.get("facts", []):
        if fact["confidence"] < 0.75:
            continue
        store.upsert_entity(user_id, fact["key"], fact["value"], fact["confidence"])
    if payload.get("memory_type") == "episodic" and payload.get("summary"):
        store.append_episode(user_id, payload["summary"])

// Spring AI + structured output — conceptual flow
public record MemoryExtract(
    boolean shouldPersist,
    String memoryType,
    List<Fact> facts,
    String summary) {
  public record Fact(String key, String value, double confidence) {}
}

public void applySelectiveWrite(MemoryStore store, String userId, MemoryExtract m) {
  if (!m.shouldPersist()) return;
  for (Fact f : m.facts()) {
    if (f.confidence() < 0.75) continue;
    store.upsertEntity(userId, f.key(), f.value(), f.confidence());
  }
  if ("episodic".equals(m.memoryType()) && m.summary() != null) {
    store.appendEpisode(userId, m.summary());
  }
}

Periodic summarization

Trigger summarization when turn_count % N == 0, when estimated tokens exceed a threshold, or on session.end webhook. The summary replaces (not duplicates) raw turns in the working-memory buffer; store the summary as a new episodic row with covers_turns=[12,24] metadata.

1. Collect turns 1–N from Redis session.
2. Call summarizer with “preserve decisions, IDs, user preferences; drop greetings.”
3. Write summary to episodic store; trim Redis to summary + turns N+1…current.
4. Log summarizer model version for regression tests.

Write strategy by memory type

Working memory: last K turns

Working memory is the sliding window of recent messages loaded from Redis (or in-process for single-node dev). Typical K is 4–10 turn pairs (user + assistant), not 4–10 messages if tool calls expand the trace.

import json
import redis

r = redis.Redis(decode_responses=True)
K_TURNS = 6

def session_key(session_id: str) -> str:
    return f"agent:session:{session_id}:turns"

def push_turn(session_id: str, role: str, content: str) -> None:
    key = session_key(session_id)
    r.rpush(key, json.dumps({"role": role, "content": content}))
    r.expire(key, 86_400)  # 24h TTL — see production section

def load_working_memory(session_id: str) -> list[dict]:
    key = session_key(session_id)
    raw = r.lrange(key, -(K_TURNS * 2), -1)  # approx K user+assistant pairs
    return [json.loads(x) for x in raw]

public List<ChatMessage> loadWorkingMemory(StringRedisTemplate redis, String sessionId, int kTurns) {
  String key = "agent:session:" + sessionId + ":turns";
  Long len = redis.opsForList().size(key);
  if (len == null || len == 0) return List.of();
  long start = Math.max(0, len - (kTurns * 2L));
  List<String> raw = redis.opsForList().range(key, start, -1);
  return raw.stream().map(json -> parseMessage(json)).toList();
}

Semantic retrieval (long-term)

Embed the current user message (or a rewritten “memory query”) and search episodic + entity-adjacent narrative stores. Apply metadata filters: user_id, memory_type=episodic, optional topic tags. Return top 3–5 snippets; inject under a <retrieved_memory> block—not mixed into system prompt without boundaries.

Time decay fusion

Pure cosine similarity favors old repeated facts over recent context. Fuse semantic score with recency using exponential decay:

final_score = α · cosine_sim + (1 − α) · exp(−λ · age_hours)

Tune α (0.6–0.85) toward semantic when users reference distant past; raise λ for fast-moving product domains. Re-rank with MMR if snippets are near-duplicates.

import math
from datetime import datetime, timezone

ALPHA = 0.75
LAMBDA = 0.08  # half-life ~8.7 hours at (1-alpha) term

def fusion_score(cosine_sim: float, created_at: datetime, now: datetime | None = None) -> float:
    now = now or datetime.now(timezone.utc)
    age_hours = max(0.0, (now - created_at).total_seconds() / 3600.0)
    recency = math.exp(-LAMBDA * age_hours)
    return ALPHA * cosine_sim + (1 - ALPHA) * recency

def rerank_memories(candidates: list[dict], top_k: int = 5) -> list[dict]:
    for c in candidates:
        c["score"] = fusion_score(c["cosine_sim"], c["created_at"])
    return sorted(candidates, key=lambda x: x["score"], reverse=True)[:top_k]

static double fusionScore(double cosineSim, Instant createdAt, Instant now) {
  double alpha = 0.75, lambda = 0.08;
  double ageHours = Duration.between(createdAt, now).toMinutes() / 60.0;
  ageHours = Math.max(0, ageHours);
  double recency = Math.exp(-lambda * ageHours);
  return alpha * cosineSim + (1 - alpha) * recency;
}

List<MemoryCandidate> rerank(List<MemoryCandidate> in, int topK) {
  Instant now = Instant.now();
  return in.stream()
      .peek(c -> c.setScore(fusionScore(c.getCosineSim(), c.getCreatedAt(), now)))
      .sorted(Comparator.comparingDouble(MemoryCandidate::getScore).reversed())
      .limit(topK)
      .toList();
}

Retrieval assembly order

1. Entity slots (deterministic, small).
2. Episodic snippets (fusion-ranked).
3. Working memory last K turns (recency).
4. RAG chunks only via explicit tool—not mixed silently with episodic.

Conceptual map

ConversationBufferMemory

Simplest: append every turn. Use return_messages=True for chat models. Pair with ConversationBufferWindowMemory (k=…) in LangChain to cap K explicitly.

from langchain.memory import ConversationBufferWindowMemory
from langchain_openai import ChatOpenAI
from langchain.chains import ConversationChain

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
memory = ConversationBufferWindowMemory(k=6, return_messages=True, memory_key="history")

chain = ConversationChain(llm=llm, memory=memory, verbose=False)
chain.predict(input="My name is Ada.")
chain.predict(input="What's my name?")  # uses last k turns only

// Spring AI chat client with in-memory MessageWindowChatMemory
// Equivalent to buffer window k=6 — not distributed; use Redis for prod

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.memory.MessageWindowChatMemory;

MessageWindowChatMemory memory = MessageWindowChatMemory.builder()
    .maxMessages(12)  // ~6 turn pairs
    .build();

ChatClient client = chatClientBuilder
    .defaultAdvisors(new MessageChatMemoryAdvisor(memory))
    .build();

client.prompt().user("My name is Ada.").call().content();
client.prompt().user("What's my name?").call().content();

ConversationSummaryMemory

Maintains a running summary string updated by the LLM after each turn. Injects summary plus recent messages into the prompt. Equivalent to our periodic summarization strategy with N=1 effective roll-up.

from langchain.memory import ConversationSummaryMemory
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
memory = ConversationSummaryMemory(
    llm=llm,
    memory_key="history",
    return_messages=True,
)

memory.save_context({"input": "I prefer dark mode and vim keybindings."}, {"output": "Noted."})
memory.save_context({"input": "What's my editor preference?"}, {"output": "Vim keybindings."})
# memory.load_memory_variables({}) contains compressed summary + recent context

// Spring AI: SummarizingChatMemoryAdvisor (conceptual — summarize when token threshold hit)
// Custom advisor: on after-call, if tokenCount > threshold, replace older messages with summary text

public class SummarizingMemoryAdvisor implements CallAdvisor {
  private final ChatModel summarizer;
  private final int tokenThreshold;

  // after each exchange: if history tokens > threshold, summarize prefix, keep tail
}

VectorStoreRetrieverMemory

Persists each input/output pair into a vector store; on new input, retrieves top similar past exchanges. Wire to Chroma/pgvector for persistence beyond process memory.

from langchain.memory import VectorStoreRetrieverMemory
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import PGVector

CONNECTION = "postgresql+psycopg://user:pass@localhost:5432/agents"
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

store = PGVector(
    connection_string=CONNECTION,
    embedding_function=embeddings,
    collection_name="agent_episodic",
)

retriever = store.as_retriever(search_kwargs={"k": 4, "filter": {"user_id": "u_123"}})
memory = VectorStoreRetrieverMemory(retriever=retriever, memory_key="history")

memory.save_context(
    {"input": "We shipped feature flags on Tuesday."},
    {"output": "I'll remember that milestone."},
)
# Next turn retrieves semantically related past inputs for the same user filter

// Spring AI VectorStoreChatMemoryAdvisor + PgVectorStore
// Store episodic summaries with metadata userId; retrieve topK before prompt build

PgVectorStore vectorStore = PgVectorStore.builder(jdbc, embeddingModel)
    .dimensions(1536)
    .tableName("agent_episodic")
    .build();

List<Document> related = vectorStore.similaritySearch(
    SearchRequest.query(currentUserMessage)
        .withTopK(4)
        .withFilterExpression("user_id == 'u_123'"));

LangGraph note

LangGraph prefers explicit state channels (messages, summary, entities) over opaque memory classes. Mirror Buffer/Summary/VectorStore as reducers on state—easier to test and serialize to Redis.

Reference architecture

1. API gateway attaches user_id, tenant_id, session_id.
2. Redis — working turns, optional summary blob, tool loop checkpoint; TTL 24–72h.
3. Postgres — user_entities table (JSONB or EAV).
4. pgvector — agent_episodes with embedding + created_at for decay.
5. Async worker — selective write + embed on session end; dead-letter queue for failures.

Redis session schema

pgvector long-term schema

EPISODE_DDL = """
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE agent_episodes (
  id          BIGSERIAL PRIMARY KEY,
  user_id     TEXT NOT NULL,
  tenant_id   TEXT NOT NULL,
  session_id  TEXT,
  summary     TEXT NOT NULL,
  topics      TEXT[] DEFAULT '{}',
  embedding   vector(1536) NOT NULL,
  created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX idx_episodes_user ON agent_episodes (tenant_id, user_id, created_at DESC);
CREATE INDEX idx_episodes_hnsw ON agent_episodes
  USING hnsw (embedding vector_cosine_ops);
"""

FUSION_QUERY = """
SELECT id, summary, created_at,
       1 - (embedding <=> %(query_vec)s) AS cosine_sim
FROM agent_episodes
WHERE tenant_id = %(tenant_id)s AND user_id = %(user_id)s
  AND created_at > now() - interval '90 days'
ORDER BY (
  0.75 * (1 - (embedding <=> %(query_vec)s))
  + 0.25 * exp(-0.08 * EXTRACT(EPOCH FROM (now() - created_at)) / 3600.0)
) DESC
LIMIT 5;
"""

/*
CREATE TABLE agent_episodes (
  id BIGSERIAL PRIMARY KEY,
  user_id TEXT NOT NULL,
  tenant_id TEXT NOT NULL,
  session_id TEXT,
  summary TEXT NOT NULL,
  topics TEXT[],
  embedding vector(1536) NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX idx_episodes_hnsw ON agent_episodes USING hnsw (embedding vector_cosine_ops);
*/

// Query with tenant + user filter — never omit tenant_id from WHERE
public List<Episode> retrieveEpisodes(String tenantId, String userId, float[] queryVec) {
  return jdbc.query(FUSION_QUERY, Map.of(
      "tenant_id", tenantId,
      "user_id", userId,
      "query_vec", queryVec), episodeMapper);
}

Session lifecycle

1. Start — generate session_id; set Redis TTL; load entities + top episodic hits.
2. Each turn — RPUSH turn; refresh TTL; optional selective write queue.
3. Idle timeout — worker consumes session.expired; summarize; embed episode; trim Redis.
4. User delete — DELETE entities + episodes WHERE user_id; purge Redis keys by pattern (scan, not KEYS in prod).

Monitoring

Production readiness checklist

• Redis session keys namespaced by tenant_id; TTL documented and tested
• pgvector queries always filter tenant_id + user_id
• GDPR/CCPA delete path removes entities, episodes, and Redis keys
• Summarizer + embed model versions pinned; migration runbook exists
• Selective write blocklist for secrets and injection patterns
• Golden sessions in eval CI—“remember my name” after 20-turn gap
• No in-process-only memory on horizontally scaled deployments
• Incident runbook: “agent forgot”—distinguish retrieval vs write vs TTL expiry