Prevent partial-chunk overwrite in --update: pre-build node-count assertion + build_merge() helper

[aturner71]

Summary

--update (and any multi-session incremental workflow) has a silent data-loss failure mode: calling build() with a partial chunk list silently replaces the existing graph.json with a smaller graph. There is no warning, no abort, and the resulting graph looks plausible enough that the problem isn't obvious until you notice god-node degrees have collapsed.

Two library-level changes would eliminate the entire failure class.

---

Root Cause (two-part)

Part A — build() has replace semantics, not merge semantics

build() constructs a graph from scratch from the provided chunk list. It does not load and merge with existing graph.json. In a multi-session incremental workflow:

1. Session 1: base extraction (123 nodes) → passed directly to build() → graph.json written. The extraction is not persisted as a chunk file.
2. Session 2: build() called with glob ".graphify_chunk_0*.json" → finds only new chunks (e.g. chunks 04, 05, 06) → builds a 55-node graph → silently overwrites the 123-node graph.json.

The output looks correct — nodes cluster, god nodes appear, GRAPH_REPORT.md generates — but 2/3 of the graph is gone.

Part B — Compounding silent edge drop from source_file field-name drift

Manual chunk files using field name "source" instead of the canonical "source_file" cause build() to silently drop all edges from those nodes (a warning is emitted, but the build continues and succeeds). In our case this dropped 107 edges from two chunks, leaving 62 edges from one conforming chunk. The graph had 55 nodes / 62 edges vs the expected 178 / 287.

Detection signal (had we known to look): post-build god-node list showed execution_ledger at #1 with 8 edges. Expected brain_orchestrator at 21+. A node-count check would have caught it before the file was written.

---

Proposed Fix 1: Pre-build node-count safety check

Before overwriting graph.json, assert that the new graph is not smaller than the existing one. If it is, abort with a clear error instead of silently overwriting.

# In build_from_json() or to_json(), before writing:
existing_path = Path("graphify-out/graph.json")
if existing_path.exists():
    existing_data = json.loads(existing_path.read_text())
    existing_n = len(existing_data.get("nodes", []))
    new_n = G.number_of_nodes()
    if new_n < existing_n:
        raise ValueError(
            f"graphify: refusing to overwrite graph.json — new graph has {new_n} nodes "
            f"but existing graph has {existing_n}. "
            f"You may be missing chunk files from a previous session. "
            f"Pass force=True to override, or use build_merge() to add nodes incrementally."
        )

This makes the failure loud instead of silent and gives the user an actionable error message.

---

Proposed Fix 2: build_merge() helper

A build_merge() function that loads the existing graph, adds new chunks, re-clusters, and saves — never replacing, only growing. This prevents the entire class of replace-semantics failures for incremental workflows.

Proposed signature:

def build_merge(
    new_chunks: list[dict],           # new extraction chunks to add
    graph_path: str | Path = "graphify-out/graph.json",
    prune_sources: list[str] | None = None,  # source_files to remove (deleted files)
) -> nx.Graph:
    """
    Load existing graph, merge new_chunks into it, and save back to graph_path.
    
    - Nodes from new_chunks are added; existing nodes are preserved.
    - Edges from new_chunks are added; existing edges are preserved.
    - If prune_sources is provided, nodes whose source_file is in that list are removed first.
    - Never replaces the graph; only grows it (or prunes deleted-file nodes).
    - Returns the merged graph for downstream use (clustering, report generation, etc.)
    """

The --update path in the SKILL.md already attempts a manual version of this using json_graph.node_link_graph() + G.update(), but it's complex, error-prone, and not tested. A first-class library function would be safer and simpler for skill authors.

---

Proposed Fix 3 (minor): Warn loudly on source vs source_file field name mismatch

Currently, when a node uses "source" instead of "source_file", graphify emits a warning and drops all edges from that node silently. The edge drop is the dangerous part — the node appears in the graph but all its connections are lost.

Suggested improvement: make the warning message explicitly state the edge count being dropped, e.g.:

WARNING: node "execution_ledger" uses field "source" instead of "source_file" — dropping 23 edges. Rename the field to "source_file" to retain them.

This makes field-name drift immediately visible rather than requiring post-hoc god-node analysis to detect.

---

Impact

	Before	After (with fixes)
Partial chunk overwrite	Silent data loss	Hard error before write
Incremental add	Error-prone manual merge in SKILL.md	Single build_merge() call
Field drift edge drop	Silent (warning only)	Loud warning with edge count

---

Our local workarounds (for context)

1. Always persist base extraction as a chunk file (chunk_01_base.json) before calling build() — so the next session's glob finds it. One line: Path(".graphify_chunk_01_base.json").write_text(json.dumps(extraction)).
2. Use source_file (not source) in all manual chunks — documented in our internal corpus notes.
3. Manual merge via json_graph.node_link_graph() + G.add_edges_from() + save — works but is fragile.

Fixes 1 and 3 from this list are local workarounds; fixes 1 and 2 in this issue are the library-level solutions that would protect all graphify users from the same failure.

---

Environment

• graphifyy v0.4.23
• networkx node_link format with edges='edges' key
• Multi-session incremental corpus expansion (153-node knowledge graph, 6 chunk files across 3 sessions)

[sergey-king]

+1

[safishamsi]

Fixed in v0.5.0. See #528 for release notes.