build_merge() silently deletes freshly re-extracted nodes when an edited file's content overlaps with its stale version

[erichkusuki]

Summary

graphify.build.build_merge() (used by /graphify --update) can silently delete nodes belonging to a file that was just edited, rather than updating them, when the new extraction for that file produces a node whose label/id exact-matches (or fuzzy-matches) a node already in the existing graph.json from the pre-edit version of the same file.

This is a data-loss bug, not a cosmetic one: the merged graph ends up missing the concept entirely (not renamed, not merged — gone), even though the concept is still present and correctly documented in the source file.

Root cause

Looking at the current implementation (v0.7.5, graphify/build.py):

def build_merge(
    new_chunks: list[dict],
    graph_path: str | Path = "graphify-out/graph.json",
    prune_sources: list[str] | None = None,
    *,
    directed: bool = False,
    dedup: bool = True,
) -> nx.Graph:
    ...
    all_chunks = base + list(new_chunks)   # base = existing graph.json reconstructed as extraction dict
    G = build(all_chunks, directed=directed, dedup=dedup)   # <-- dedup runs BEFORE prune

    # Prune nodes from deleted source files
    if prune_sources:
        to_remove = [
            n for n, d in G.nodes(data=True)
            if d.get("source_file") in prune_sources
        ]
        G.remove_nodes_from(to_remove)
    ...

The order is: merge old+new → dedup → prune by source_file.

When a file is edited (not deleted), the intended workflow is:

1. Re-extract the edited file → new nodes, still carrying the same source_file path.
2. Call build_merge(new_chunks=[new_extraction], prune_sources=[edited_file_path]) to drop the stale pre-edit nodes for that file and merge in the fresh ones.

But because dedup runs first, if the new extraction produces a node with the same (or fuzzy-similar) label as a node already in the old graph for that same file — which is extremely common, since editing a file usually keeps most of its concepts unchanged — build()'s dedup step merges the old and new node into a single node. That merged node's source_file is still the edited file's path (correctly). Then the prune_sources step removes every node whose source_file matches, including this just-merged node — deleting the concept from the graph entirely, even though it's present in both the old and new extraction.

Repro

1. Build a graph over a small corpus with a doc file docs/foo.md containing a section titled "Widget Cache Design" that becomes node foo_widget_cache (label "Widget Cache Design", source_file: "docs/foo.md"), with some edges to other nodes.
2. Edit docs/foo.md — add a new paragraph/cross-reference under the same "Widget Cache Design" heading, without renaming it.
3. Re-extract just that file into a new chunk JSON. Since the heading/title is unchanged, re-extraction naturally reproduces a node with the same label "Widget Cache Design" plus the new content/edges.
4. Call:

   from graphify import build
   G = build.build_merge(
       new_chunks=[new_chunk],
       graph_path="graphify-out/graph.json",
       prune_sources=["docs/foo.md"],
       directed=False,
       dedup=True,
   )

5. Inspect G.nodes: foo_widget_cache is missing entirely — not present under its old id, not present under a new id, not merged into a stub. The node and all its edges are gone.

In my own use (a ~185-file doc corpus), this affected a node I'd just edited to add a cross-reference to another file — the node vanished from the rebuilt graph along with all its edges, rather than being updated.

Suggested fix

Reorder so pruning happens before merging, not after:

def build_merge(new_chunks, graph_path=..., prune_sources=None, *, directed=False, dedup=True):
    ...
    if graph_path.exists():
        ...
        existing_nodes = [...]
        existing_edges = [...]
        if prune_sources:
            existing_nodes = [n for n in existing_nodes if n.get("source_file") not in prune_sources]
            kept_ids = {n["id"] for n in existing_nodes}
            existing_edges = [e for e in existing_edges if e["source"] in kept_ids and e["target"] in kept_ids]
        base = [{"nodes": existing_nodes, "edges": existing_edges}]
    else:
        base = []

    all_chunks = base + list(new_chunks)
    G = build(all_chunks, directed=directed, dedup=dedup)
    return G

i.e. drop stale nodes/edges from prune_sources before calling build(), so the dedup step only ever sees the new extraction's node for that file, never a stale duplicate that could get merged-then-pruned.

Workaround I used

Instead of calling build_merge() with graph_path pointing at the exported graph.json, I:

1. Loaded the raw pre-build semantic extraction (nodes/edges/hyperedges, before build() / dedup), not the built graph.
2. Filtered out nodes/edges whose source_file was in my edited-files set.
3. Concatenated in the fresh re-extraction chunk for those files.
4. Called build.build() once on the combined, already-pruned data.

This avoids the merge-then-prune ordering entirely, but it means --update can't safely be used as documented for edited files today — only for genuinely deleted files, or callers need to bypass build_merge and do the prune-then-build sequence manually as above.

Environment

• graphifyy (PyPI) 0.7.5
• Python 3.14, Windows

[safishamsi]

Fixed on v8 (0.9.13), credited to you. Two things:

Current build_merge already applies your suggested fix in spirit — it drops a re-extracted file's stale base nodes before the merge (replace-per-source), so an EDITED file passed only in new_chunks is handled correctly (verified: the same-label node survives, re-keyed but present). Your v0.7.5's merge→dedup→prune order was reworked since.

But your exact repro passes the edited file in both new_chunks and prune_sources, and the prune step still deleted its freshly-built nodes afterward — so the data loss reproduced. Fixed by excluding new_sources (anything in new_chunks) from the prune set: a re-extracted file is being replaced, never deleted, so replace wins over a contradictory delete. Genuine deletions (a file in prune_sources but not new_chunks) still prune. Verified on your Widget-Cache-Design scenario + a genuine-deletion control; regression tests added. Thanks for the precise root-cause and repro.