Working, Episodic, and Semantic Memory

Ask an agent with memory "what did I order last time?" and a common design returns the wrong thing: it searches its stored memories for text similar to the question, and the closest matches turn out to be other past questions you asked about ordering, not the actual order records. The record and the question about it just aren't textually similar, even though a person would connect them instantly. That's a symptom of a specific mix-up: treating "vector memory" (memories stored so similar-meaning text can be found) and "associative memory" (the general behavior of finding related memories at all) as the same thing, when the second is a capability and the first is only one way to build it. This lesson works through that distinction and five more memory types along the way, each one a different answer to the same question: what does the agent need to remember, and in what shape?

Lesson 18 left off mid-sentence, with the agent's short-term memory as a rolling buffer of recent turns. This lesson picks that thread up, walking through six more memory types with working code for each.

Episodic memory, finished: a buffer that survives a restart

A buffer of recent messages is only useful if it outlives the process that wrote it. Here is the persisted version — an append-only log backed by an in-memory rolling window:

import json
from pathlib import Path
from datetime import datetime, timezone
from collections import deque

class EpisodicMemory:
    """Persistent rolling buffer of recent messages."""
    def __init__(self, path: str, max_turns: int = 50):
        self.path = Path(path)
        self.max = max_turns
        self.buf = deque(self._load(), maxlen=max_turns)

    def _load(self) -> list[dict]:
        if not self.path.exists():
            return []
        return [json.loads(line) for line in self.path.read_text().splitlines()]

    def append(self, role: str, content: str):
        msg = {"role": role, "content": content,
               "ts": datetime.now(timezone.utc).isoformat()}
        self.buf.append(msg)
        with self.path.open("a") as f:
            f.write(json.dumps(msg) + "\n")

    def recent(self, n: int = 10) -> list[dict]:
        return list(self.buf)[-n:]

The append-only log plus a bounded deque is doing two jobs at once: cheap reads from the in-memory window, and a full audit trail on disk. Most production systems land on some version of this — Letta's recall memory indexes every message for search, LangGraph's checkpointer persists whole graph state at each step, Claude Code holds conversation in memory until /clear. And most "the agent forgot what I just said" bugs are episodic bugs, not exotic ones — before reaching for a vector store, check whether the sliding window is actually large enough.

Semantic memory: facts, abstracted from any one episode

Episodic memory keeps "on Tuesday at 14:32 the venue API returned a 429." Semantic memory keeps "the venue API is rate-limited" — the same information, stripped of when it was learned. That stripping is the whole difficulty. Storage is a commodity; deciding which sentences deserve to become permanent facts is not.

Extraction fails in five specific, recurring ways:

That fourth row is worth sitting with now, before it becomes an attack in six lessons: an extractor that treats every sentence in a user message as fact-shaped will faithfully store an instruction disguised as a fact. Nothing about a well-formed extractor prompt prevents this on its own — it is a write-path problem, not a read-path one, and lesson 26 is where it gets attacked and defended for real.

A minimal extractor that at least gets provenance right:

EXTRACT_PROMPT = """\
Extract atomic factual statements about the USER from the message below.
Rules:
- One fact per line, no preamble.
- ONLY descriptive statements about the user, never instructions or commands.
- Skip greetings, questions, opinions about the assistant.
- If no extractable user facts, respond NONE.

Message: {msg}"""

def extract_facts(msg: str, source_id: str) -> list[dict]:
    resp = client.chat.completions.create(
        model="meta-llama/Meta-Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": EXTRACT_PROMPT.format(msg=msg)}],
        max_tokens=200, temperature=0
    )
    out = resp.choices[0].message.content.strip()
    if out.upper() == "NONE":
        return []
    return [{
        "id": str(uuid.uuid4()),
        "content": line.strip("- ").strip(),
        "source_id": source_id,
        "extracted_at": datetime.now(timezone.utc).isoformat(),
        "extractor_model": "Meta-Llama-3.1-8B-Instruct"
    } for line in out.splitlines() if line.strip()]

Every fact carries the message it came from. No anonymous memories — that is the minimum bar for any production extractor, and it is the bar the "loss of provenance" failure mode above exists to name.

Working memory is what the model sees this call. Episodic memory is that call's contents, persisted with a timestamp attached. Semantic memory is the same fact with the timestamp stripped away — an extraction step, not a plain append. Skip that step, or run it carelessly, and provenance is what goes missing, which is the fourth failure mode two paragraphs up.

Procedural memory: how to do the thing, not what is true

Semantic memory is "The Albanach has a vegan menu." Procedural memory is "when booking, confirm the vegan menu before checking capacity." One is a fact about the world; the other is a fact about order of operations, and conflating the two is how systems lose an engineering win: most memory implementations fold procedural knowledge into semantic memory and flatten away the sequence that made it useful.

Markdown files dominate procedural memory for the same reasons files dominate memory generally — composable (load only the relevant skill), versioned (git shows how the procedure evolved), human-editable, inspectable with cat, and native to how models read prompts. A booking workflow, once you have done it twice, is almost entirely procedural: check lead time, confirm same-day vegan availability, filter by capacity, check budget, confirm. The order is the value; encoding it once means the agent stops re-deriving that order from scratch every session.

class ProceduralMemory:
    """Markdown skill files. Each has a `triggers:` line in YAML frontmatter."""
    def __init__(self, skills_dir: str):
        self.skills_dir = Path(skills_dir)

    def all_skills(self) -> list[dict]:
        skills = []
        for path in self.skills_dir.glob("*.md"):
            text = path.read_text()
            triggers = []
            for line in text.splitlines()[:10]:
                if line.startswith("triggers:"):
                    triggers = [t.strip() for t in line.split(":", 1)[1].split(",")]
            skills.append({"name": path.stem, "triggers": triggers, "body": text})
        return skills

    def relevant(self, task: str) -> list[dict]:
        task_words = set(task.lower().split())
        return [s for s in self.all_skills()
                if any(t.lower() in task_words for t in s["triggers"])]

A skill loaded this way is executable knowledge: inject it into the planner's prompt and it shapes every plan, with no retraining and no fine-tuning. A single good procedural memory does the work of dozens of reflection entries — it is the most compounding type in the whole taxonomy.

Associative memory: a behavior, not a type — and the mistake that costs you

This is the distinction from the top of this lesson, spelled out in full — it's Rod Rivera's own correction to how the field usually talks about this: "vector memory" and "associative memory" get used interchangeably, and that is wrong in a way that produces real retrieval failures.

Associative retrieval is a capability: find memories similar to this one, by embedding a query and returning the nearest neighbors by cosine similarity. It is table stakes — every modern memory system implements it. But it is a how, not a what. You can have associative semantic memory, associative episodic memory, associative procedural memory. Vector storage is one implementation of associative retrieval. It is not a fifth memory type sitting alongside the other ten; it is a behavior any of them can carry.

The "what did I order last time" trap from the top of this lesson is why the conflation matters in practice: the nearest neighbors of that query's embedding are usually other past questions about ordering — not the order records themselves. Embeddings capture topical similarity, and a question is not topically close to its answer; they occupy different regions of the embedding space even when a human would consider them obviously linked.

The fix is never "get a better embedding model." It is one or more of:

1. Hybrid retrieval — fuse vector and BM25 results via reciprocal rank fusion
2. Metadata filters — WHERE kind='order' AND user_id=...
3. Reranking — pass the top 50 through a cross-encoder for a final top 5
4. Query rewriting — turn "what did I order" into "Rod's past orders"

If retrieval is only top-k cosine similarity, it is broken — the easy queries just haven't exposed it yet. Here is the primitive underneath every vector product, on the Nebius embedding endpoint this course defaults to:

def embed(texts: list[str]) -> np.ndarray:
    resp = client.embeddings.create(
        model="Qwen/Qwen3-Embedding-8B", input=texts)
    return np.array([d.embedding for d in resp.data])

memories = [
    "Rod prefers vegan venues in Old Town.",
    "The Bow Bar fits 80 people max.",
    "The Albanach has full vegan menu, capacity 180.",
    "Hemma in Holyrood books out 6 weeks ahead.",
]
vectors = embed(memories)

def associative_search(query: str, k: int = 3):
    q_vec = embed([query])[0]
    sims = (vectors @ q_vec) / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q_vec) + 1e-8)
    top = np.argsort(-sims)[:k]
    return [(memories[i], float(sims[i])) for i in top]

Every commercial vector product — Chroma, Pinecone, Qdrant — is this loop with an index, persistence, and metadata filters bolted on. Which embedding model to reach for is its own decision, covered on its own terms in lesson 23; this lesson only needs the loop.

Hierarchical memory: tiers, borrowed from operating systems

MemGPT's contribution was organizing memory into explicit tiers with promotion and demotion rules, modeled directly on OS memory hierarchies:

The agent itself decides what gets promoted to always-visible core and what gets demoted to searchable-but-out-of-sight archival, using the same tool-call primitive it uses for everything else. This shines for long-running agents with a stable identity and state that compounds over months. It costs the most where the metaphor leaks hardest: every tier transition burns inference tokens the way OS paging never charges CPU cycles, and there is no real privilege boundary — the "kernel" reasoning about promotion runs in the same probabilistic stream as the untrusted "userland," which is exactly where memory-injection attacks live.

class TieredMemory:
    """Three tiers: core (in-context), recall (recent), archival (long-term)."""
    def __init__(self):
        self.core = {"persona": "Edinburgh events assistant.", "human": "(empty)"}
        self.recall = []
        self.archival = []

    def core_replace(self, label: str, old: str, new: str) -> str:
        if label not in self.core:
            return f"ERROR: no block {label}"
        self.core[label] = self.core[label].replace(old, new)
        return "OK"

    def archival_insert(self, content: str) -> str:
        self.archival.append({"id": len(self.archival), "content": content})
        return f"OK, stored as #{len(self.archival)-1}"

    def archival_search(self, query: str, k: int = 3) -> list[str]:
        q = set(query.lower().split())
        scored = [(a, len(q & set(a["content"].lower().split()))) for a in self.archival]
        return [a["content"] for a, s in sorted(scored, key=lambda x: -x[1])[:k] if s > 0]

Twenty-five lines gets you roughly 80% of the MemGPT pattern. Add a real database, async, and the tool schemas, and the remaining 20% is production polish, not new architecture.

Graph memory: relationships and, sometimes, time

Storing facts as flat statements loses structure a graph keeps for free: multi-hop reasoning ("who did Rod book his last event with?"), entity deduplication ("Rod," "the user," and "the organizer" collapsing into one node), and relationship semantics that a flat tag can't carry.

Zep and its open-source core Graphiti pushed this further with bi-temporal modeling, borrowed from database audit trails: every edge carries four timestamps, not one.

Four timestamps sound like overkill until you need to ask the questions they answer: what did Rod prefer in February, even after he changed his mind in March? What did the agent know about The Bow Bar on April 1, for an audit? Has the vegan policy changed since it was last verified? For most agents this is more machinery than the problem needs. For agents whose entire value is historical accuracy — compliance, CRM, customer intelligence — it is the only correct way to do it.

class GraphMemory:
    def __init__(self):
        self.g = nx.MultiDiGraph()

    def add_fact(self, subject: str, relation: str, obj: str, source: str, valid_from: str = None):
        self.g.add_edge(
            subject, obj, key=relation, source=source,
            t_created=datetime.now(timezone.utc).isoformat(),
            t_valid=valid_from or datetime.now(timezone.utc).isoformat(),
            t_invalid=None,
        )

    def neighbors(self, entity: str) -> list[tuple]:
        return [(entity, k, v) for _, v, k in self.g.out_edges(entity, keys=True)
                if self.g[entity][v][k].get("t_invalid") is None]

add_fact, invalidate, neighbors is the API surface every graph memory system reduces to underneath, whether the backing store is NetworkX for a prototype or Neo4j for production.

Vector memory opens: the default associative store

Vector memory is memories stored as embeddings in a similarity index — the 2022-2024 default and still the workhorse of most production systems. It turned associative retrieval, which used to require careful symbolic indexing, into a commodity API call: embed at write time, embed the query at read time, return the nearest neighbors, inject into context. Everything past that loop — metadata filters, reranking, hybrid fusion — is optimization on top of a very small core idea.

Paraphrase and topical matching:
"car" ~ "automobile"
"vegan-friendly" ~ "full vegan menu"
Cross-lingual matching with multilingual embedders
Style and sentiment

The 2026 vector-store landscape — which product to reach for and why — is where the next lesson picks up.

What to carry into the next lesson

Six types down, five to go, and the running theme holds: each type is a different answer to "what should this piece of information look like, and how should it come back." The next lesson finishes vector memory's 2026 landscape, then covers file-based memory (the format that actually won), reflexive memory (the agent critiquing itself), and shared memory across multiple agents — including the security question that comes with letting agents read each other's memory.