save_semantic_cache overwrites AST cache entries (same hash key), breaking `graphify update` on mixed code+docs corpora

[nskidan]

Summary

On mixed code+docs corpora, graphify update <path> triggers the v0.5.0 shrink guard ("refusing to overwrite graph.json") because cached AST results for code files have been silently overwritten by semantic-extraction results during the original full /graphify run.

Both save_cached and save_semantic_cache write to the same cache file path (graphify-out/cache/{file_hash}.json) with no AST-vs-semantic namespacing. When the semantic subagent emits a node whose source_file is a code file, save_semantic_cache overwrites that file's AST cache entry. On the next graphify update, extract() reads back the semantic-shaped data and treats it as AST output — losing real code nodes from the rebuild.

The shrink guard (#479, v0.5.0) and desync guard (#562, v0.5.1) prevent this from corrupting graph.json on disk, so users see "refusing to overwrite" warnings rather than data loss. Pure-code corpora are immune because the skill skips Part B (semantic) on code-only detection (skill.md:235).

Mechanism

Verified on main and on installed graphifyy 0.5.2.

1. Same hash key, no namespacing. save_cached (cache.py:73) writes to graphify-out/cache/{hash}.json where hash = file_hash(path, root) (cache.py:20) — SHA256 of file contents + relative path. save_semantic_cache (cache.py:143) groups nodes by source_file and calls save_cached(p, result, root) for each (cache.py:176) — same key, same directory.

2. AST and semantic run in parallel on overlapping file sets. skill.md:209 runs Part A (AST) and Part B (semantic) in parallel. skill.md:256 feeds all detected file types — including code — into the semantic chunker. The subagent prompt (skill.md:301-302) explicitly instructs: "Code files: focus on semantic edges AST cannot find."

3. Race in save_cached. AST extraction is per-file and fast; subagent calls return ~45s later and save_semantic_cache overwrites code-file hash entries with rationale-shape data. cache.py:88-98 unconditionally calls os.replace(tmp, entry) — no shape check, no merge, last-writer-wins.

4. extract() doesn't validate cache shape. extract.py:3339-3346:

   cached = load_cached(path, cache_root or root)
   if cached is not None:
       per_file.append(cached)
       continue

   Whatever shape load_cached returns gets unconditionally appended to per-file results. No filter on file_type, no rejection of hyperedges, no AST-shape signature.

5. graphify update reads back the wrong shape. __main__.py:1412-1419 dispatches update → _rebuild_code (watch.py:35-134), which calls extract(code_files, cache_root=watch_root) at watch.py:59. The cached semantic data is treated as fresh AST output. Real code nodes for cache-collided files are missing from the new graph.

6. Shrink guard fires. Since v0.5.0 (Prevent partial-chunk overwrite in --update: pre-build node-count assertion + build_merge() helper #479), to_json returns False when the new graph is smaller than the existing one, and _rebuild_code aborts the write (watch.py:102-104). User-visible result in 0.5.0+: warning, no graph.json mutation. In 0.4.23 and earlier: silent corruption.

Reproducer

mkdir -p /tmp/graphify-cache-collision-repro && cd $_
cat > main.py <<'EOF'
def reverse_string(s: str) -> str:
    """Return the input string with characters in reverse order."""
    return s[::-1]

def is_palindrome(s: str) -> bool:
    """True if the input string reads the same forwards and backwards."""
    return s == s[::-1]
EOF
cat > README.md <<'EOF'
# String Utils

Two helpers for working with strings: reversal and palindrome detection.
Used internally by the text-processing pipeline (see DESIGN.md for context).
EOF
cat > DESIGN.md <<'EOF'
# Design notes

The string utilities in main.py exist because we needed deterministic
character-level operations for our normalization layer.
EOF

# Step 1: full graphify run (writes both AST and semantic to cache,
#         with semantic overwriting AST for main.py because it appears
#         in the semantic file list per skill.md:256)
graphify  # via your AI assistant: /graphify .

# Step 2: confirm the corruption is in cache, not in graph.json
python3 -c "
import json, hashlib, pathlib
p = pathlib.Path('main.py')
content = p.read_bytes()
h = hashlib.sha256()
h.update(content)
h.update(b'\\x00')
h.update(b'main.py')
key = h.hexdigest()
entry = pathlib.Path('graphify-out/cache') / f'{key}.json'
data = json.loads(entry.read_text())
fts = {n.get('file_type') for n in data.get('nodes', [])}
has_hyperedges = bool(data.get('hyperedges'))
print(f'cache file_types for main.py: {fts}')
print(f'cache contains hyperedges:    {has_hyperedges}')
print(f'  expected (AST-only):        {{\"code\"}} / False')
print(f'  if collision occurred:      {{\"rationale\"}} / True')
"

# Step 3: run update — observe shrink guard fire
graphify update .

Expected (post-bug-fix): step 2 reports {"code"}, no hyperedges. Step 3 succeeds and writes a graph of ≥ same size.

Observed (current main): step 2 reports {"rationale"} with hyperedges present. Step 3 prints the shrink-guard warning and refuses to write graph.json.

Why this isn't caught by existing fixes

• watch: _rebuild_code overwrites semantic nodes instead of merging with existing graph #253 / watch: 0.4.3 merge fix still drops cross-type edges and loses code nodes #261 addressed _rebuild_code's in-memory merge logic, not the underlying cache-shape contamination.
• Prevent partial-chunk overwrite in --update: pre-build node-count assertion + build_merge() helper #479 (shrink guard) and graphify update desyncs GRAPH_REPORT.md and graph.html when refusing to overwrite graph.json #562 (desync guard) prevent the corrupted graph from being persisted, but they convert silent corruption into a graceful refusal — they don't fix the cache.
• commit c529e221 ("cache key collision") added the relative-path component to the hash to disambiguate identical-content files in different directories. It did not namespace AST vs. semantic.
• No test covers this scenario. tests/test_cache.py does not test mixed AST + semantic writes for the same source_file. tests/test_watch.py covers _notify_only and watchdog import handling but no end-to-end rebuild with semantic context.

Why active users probably haven't reported this specifically

1. Pure-code corpora are immune (skill.md:235 — Part B skipped on code-only detection). Most users are running graphify on code-only repos.
2. Shrink guard masks the symptom in 0.5.0+. Users see a warning, not data loss; many will assume they did something wrong rather than file a bug.
3. Race-dependent on parallel AST/semantic execution. AST sometimes wins the cache write, sometimes loses — non-deterministic, hard to reproduce reliably without instrumenting the cache.
4. Adjacent issues already filed (watch: _rebuild_code overwrites semantic nodes instead of merging with existing graph #253, watch: 0.4.3 merge fix still drops cross-type edges and loses code nodes #261, graphify update desyncs GRAPH_REPORT.md and graph.html when refusing to overwrite graph.json #562) made this region look "addressed."

Suggested fix

Namespace the cache directory by extractor type. Smallest-diff version:

# cache.py
def cache_dir(root: Path = Path("."), kind: str = "ast") -> Path:
    """Returns graphify-out/cache/{kind}/ - creates it if needed.

    kind in {"ast", "semantic"} prevents AST and semantic cache entries
    for the same source_file from sharing a hash key.
    """
    d = Path(root).resolve() / "graphify-out" / "cache" / kind
    d.mkdir(parents=True, exist_ok=True)
    return d


def load_cached(path: Path, root: Path = Path("."), kind: str = "ast") -> dict | None:
    ...
    entry = cache_dir(root, kind) / f"{h}.json"
    ...


def save_cached(path: Path, result: dict, root: Path = Path("."), kind: str = "ast") -> None:
    ...
    entry = cache_dir(root, kind) / f"{h}.json"
    ...

Then update save_semantic_cache (cache.py:175) to pass kind="semantic", leave AST callers (extract.py:3339, 3345) on the default. clear_cache should clear both subdirectories.

Migration: existing flat cache/{hash}.json entries can be left in place (treated as ast/) since semantic results from prior runs are not recoverable from the cache anyway — the next full /graphify will repopulate semantic/ correctly.

Tests to add:

• test_cache.py: write AST result for path, then write semantic result for same path via save_semantic_cache, confirm both are independently recoverable.
• test_watch.py: end-to-end — full extract → corrupted-cache scenario → _rebuild_code → assert resulting graph contains the original code nodes.

Environment

• graphifyy 0.5.2 (PyPI)
• Python 3.12.13
• macOS 14 (Darwin 25.4.0)
• Mixed corpus: ~62 code nodes (Python), 116 document nodes (markdown), 23 rationale nodes
• Reproduces deterministically on the synthetic 3-file repo above when AST cache is written before semantic cache

Severity / impact

• Functional: graphify update <path> is non-functional on mixed code+docs corpora since the v0.5.0 shrink guard. Users on 0.5.0+ must run a full /graphify . rebuild to refresh.
• Silent in 0.4.23 and earlier: corrupted graphs were persisted without warning, until the next full rebuild.
• Worked around in CLAUDE.md / skill mode, where /graphify . --update takes a different code path. CLI-only consumers (e.g. cron, post-commit hook rebuilds) are most exposed.

[safishamsi]

Fixed in v0.5.3. AST entries now write to cache/ast/, semantic entries to cache/semantic/. load_cached falls back to legacy flat cache/ for AST reads so existing entries survive the upgrade.