graphify extract silently skips semantically-stale files after graphify update has touched them (shared manifest.json)

[cnchg]

Summary

graphify extract --backend openai uses graphify-out/manifest.json to decide which files need semantic re-extraction. But the same manifest.json is also written by graphify update (the cheap AST-only path that the auto-installed post-commit git hook runs). When update stamps a file's new hash into the manifest, a later extract reads the hash, sees "already processed," and silently skips semantic extraction — even though the file has never been semantically re-processed at the new hash.

Net effect: semantic edges (e.g. --references-->) never get inferred for files edited between semantic-extract passes, and the graph silently goes stale. The user has no way to know via the CLI output: extract cheerfully reports "0 changed; N unchanged" and exits 0.

Reproduction

# Fresh state: assume `graphify extract . --backend openai` has run once.
echo "<!-- minor edit -->" >> CLAUDE.md
git add CLAUDE.md && git commit -m "edit CLAUDE.md"
# post-commit hook runs `graphify update` -> stamps CLAUDE.md's new hash into manifest.json

graphify extract . --backend openai
# Observed:
#   [graphify extract] 0 code, 0 docs, 0 papers, 0 images changed; 97 unchanged; 0 deleted
#   [graphify extract] incremental summary: 97 files cached/unchanged, 0 re-extracted, 0 deleted
# Expected: CLAUDE.md re-extracted semantically (its content changed since last semantic pass).

Confirmation that the manifest is the culprit (workaround):

# Manually invalidate the entry and re-run -> file is re-extracted correctly
python3 -c "
import json
m = json.load(open('graphify-out/manifest.json'))
del m['/abs/path/to/CLAUDE.md']
json.dump(m, open('graphify-out/manifest.json', 'w'))
"
graphify extract . --backend openai
# Observed:
#   [graphify extract] 0 code, 1 docs, 0 papers, 0 images changed; 96 unchanged; 0 deleted
#   [graphify extract] semantic extraction on 1 files via openai...
# -> new --references--> edges show up in graph.json

Root cause (suspected)

update and extract share graphify-out/manifest.json as their file-hash change-detection store, but they perform different work (update = AST-only, extract = AST + semantic LLM). When update runs first, it writes the new hash. When extract runs next, it sees the hash matches and skips the file — but the file's content has only been AST-processed, never semantically.

This is especially confusing because the hook installed by graphify hook install always runs update on every commit. So in any project using the hook, extract is permanently in this broken state for any file edited since the last full semantic pass.

Versions / environment

• graphify (PyPI graphifyy) — recent (Python 3.11 via pipx, project on Rocky Linux 9.6)
• Hook installed via graphify hook install
• Backend tested: --backend openai

Suggested fix directions

A few options, ordered by invasiveness:

1. Per-mode manifest tracking. Track AST-extract and semantic-extract hashes separately (manifest.json -> {path: {ast_hash, semantic_hash}}). update only updates ast_hash; extract checks semantic_hash. This is the cleanest fix.
2. Manifest version flag. Mark each manifest entry with the extraction mode that produced it; extract re-extracts any entry not marked "semantic."
3. Mode-specific cache files. Split manifest into manifest-ast.json and manifest-semantic.json so the two commands no longer share state.

Happy to PR option 1 or 2 if that helps — option 1 feels like the right long-term shape but option 2 is a one-line workaround.

Why this matters

For users following the documented workflow ("session logs get committed -> post-commit hook rebuilds graph -> semantic references edges connect logs to anchor docs"), the workflow silently degrades over time: new session logs become graph islands reachable only by content search, not by graphify path traversal from their anchor docs. The user has to manually invalidate manifest.json entries or rm -rf graphify-out/cache/ and pay the full-repo re-extract cost to recover.

Discovered while wiring up session-log linking on a real project; full session log of the investigation (with concrete repro and observed cost: ~7k tokens / $0.0041 to re-extract 3 docs after manual manifest fix) is in the repo at https://github.com/cnchg/digitalleafco/blob/main/docs/sessions/2026-05-13-graphify-extract-cache-bug.md.

[safishamsi]

Fixed in 972fdc1 — manifest entries split into ast_hash/semantic_hash; graphify update stamps kind='ast' only so graphify extract always re-runs semantic on files touched by update.