Skip to content

Markdown memory backend (#44 Phase A)#48

Merged
rdwj merged 1 commit into
mainfrom
feat/markdown-memory-backend
Apr 16, 2026
Merged

Markdown memory backend (#44 Phase A)#48
rdwj merged 1 commit into
mainfrom
feat/markdown-memory-backend

Conversation

@rdwj

@rdwj rdwj commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Addresses #44 Phase A.

Fills the Level-1 / Level-2 gap in the memory backend lineup — a human-readable, git-committed memory store for agents whose memory a developer wants to eyeball and edit by hand.

Design summary

Two modes selected by config:

  • Level 1 (file: ./memory.md) — one compound markdown doc, each memory is a ## <heading> section. The Karpathy LLM Wiki pattern.
  • Level 2 (dir: ./memories) — one .md file per topic, filename (without extension) is the memory id. Use when you have multiple functional areas to curate separately (e.g. standing instructions vs project memories).

Same backend, same code, the config picks which mode.

Memory IDs = section headings (Level 1) or filename stems (Level 2). The LLM can both append (write) and edit (update) using human-readable addresses rather than opaque UUIDs.

search(query="") returns every section/file in stable file order, intentionally designed for cache-friendly injection: agents load the whole memory set once at session start and inject it as a stable prefix so token positions don't shift between turns. Non-empty queries do a case-insensitive substring filter without ranking — no index, no magic. If you need ranked retrieval, jump to the sqlite backend.

report_contradiction is a log-only no-op; the human edits the file to resolve.

Positioning vs other backends

Added a decision tree to docs/architecture.md Memory Integration section:

Does this agent need memory across sessions?
├─ No  → backend: null
└─ Yes → How will you curate it?
   ├─ I'll read and edit the file by hand, commit it to git
   │  ├─ One topic       → markdown, file: ./memory.md    (Level 1)
   │  └─ Multiple areas  → markdown, dir:  ./memories      (Level 2)
   ├─ Agent needs searchable memory on one host → sqlite   (Level 3)
   ├─ Multiple agents share memory / vector     → pgvector (Level 4)
   └─ Audit trails, RBAC, retention             → memoryhub (Level 5)

Each jump is a real jump, not a sliding scale of features. Users should recognise themselves in exactly one row.

What this PR deliberately does NOT build

  • No search ranking beyond file order (use sqlite)
  • No regex (use sqlite)
  • No per-section frontmatter/metadata (it's sqlite in a costume)
  • No multi-level directory trees (flat dir only)
  • No auto-injection into BaseAgent.build_system_prompt — that belongs in the separate memory-prefix-slot follow-up (#47) because it applies to all backends, not just markdown

Scope-discipline check

Single-file module, no new runtime deps (pyyaml already present), pattern mirrors memory_sqlite.py / memory_pgvector.py exactly. The custom backend remains the escape hatch.

Test plan

  • 36 new tests in test_memory_markdown.py — factory (both modes, missing/malformed/both-keys/neither-key configs, relative paths), Level 1 write/search/update, Level 2 write/search/update/unsafe-filename rejection, stable sort order, dispatcher integration
  • Full fipsagents suite 429/429 green (393 previously + 36 new)
  • gitleaks clean

Follow-ups

Fills the Level-1/Level-2 gap in the memory backend lineup: a
human-readable, git-committed memory store for single-agent setups
where the developer curates the memory file by hand.

Two modes selected by config:

- Level 1 (`file: ./memory.md`) — one compound markdown document.
  Each memory is a `## <heading>` section. The Karpathy LLM Wiki
  pattern.
- Level 2 (`dir: ./memories`) — one `.md` file per topic. Filename
  (without extension) is the memory id. Use when you have separate
  functional areas (standing instructions vs project memories) to
  curate.

Memory IDs are section headings or filename stems so an LLM can both
append new memories (`write`) and edit existing ones (`update`) using
human-readable addresses rather than opaque UUIDs.

`search(query="")` returns every section/file in stable file order,
designed for cache-friendly injection: agents load the whole memory
set once at session start and inject it as a stable prefix so token
positions don't shift between turns. Non-empty queries do a
case-insensitive substring filter without ranking — no index, no
magic. If you need ranked retrieval, use the sqlite backend.

`report_contradiction` is a log-only no-op; the human edits the file
to resolve. Matches the backend's curated-by-hand ethos.

Also documents the memory-backend decision tree and the prefix-cache
pattern in `docs/architecture.md`. Makes the positioning of each
backend explicit so new users can recognise themselves in exactly one
row.

Scope-discipline check: single-file module, no new runtime deps
(pyyaml already present), pattern mirrors `memory_sqlite.py` /
`memory_pgvector.py` siblings exactly. The `custom` backend remains
the escape hatch for everything else.

Follow-up filed as #47: first-class memory-prefix slot in BaseAgent
so the cache-friendly injection pattern works automatically across
all backends, not just markdown.

Assisted-by: Claude Code (Opus 4.6)
@rdwj
rdwj merged commit 37168f1 into main Apr 16, 2026
@rdwj
rdwj deleted the feat/markdown-memory-backend branch April 16, 2026 23:14
rdwj added a commit that referenced this pull request Apr 16, 2026
Missed in #48. Keeps the in-repo decision-reference list in sync with
the backends actually supported by the dispatcher.

Assisted-by: Claude Code (Opus 4.6)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant