Skip to content

fix(dedup): warn on cross-chunk node ID collision to surface silent data loss#1508

Closed
nuthalapativarun wants to merge 1 commit into
Graphify-Labs:v8from
nuthalapativarun:fix/1504-warn-cross-chunk-node-id-collision
Closed

fix(dedup): warn on cross-chunk node ID collision to surface silent data loss#1508
nuthalapativarun wants to merge 1 commit into
Graphify-Labs:v8from
nuthalapativarun:fix/1504-warn-cross-chunk-node-id-collision

Conversation

@nuthalapativarun

Copy link
Copy Markdown
Contributor

Summary

  • deduplicate_entities() silently drops one of two nodes that share an ID when they come from different source files (first-writer-wins), with no indication in any log, counter, or output
  • This happens when two LLM extraction chunks each process a file with the same name in different directories (e.g. module-a/README.md and module-b/README.md) and independently generate the same stem-based IDs
  • Adds a stderr WARNING when the pre-dedup loop encounters a duplicate ID from a different source_file, telling the user exactly which files collided and pointing to the merge-graphs workflow

Changes

  • graphify/dedup.py — add collision detection to the pre-dedup loop; emit a WARNING to stderr when a duplicate ID comes from a different source_file
  • tests/test_dedup.py — two regression tests: one asserting the warning fires and only the first node survives; one asserting no warning for same-file duplicates

Verification

uv run pytest tests/test_dedup.py -q
34 passed

Closes #1504

…ata loss (Graphify-Labs#1504)

When two LLM extraction chunks each process a file with the same name in
different directories, they independently generate the same node IDs and
deduplicate_entities() silently drops one node (first-writer-wins). The
data loss had no indication in any log, counter, or output.

Adds a stderr WARNING when a duplicate ID comes from a different
source_file, telling the user which files collided and recommending the
per-subfolder extract + merge-graphs workflow to avoid it.
@sub4biz

sub4biz commented Jun 28, 2026

Copy link
Copy Markdown

Thanks @nuthalapativarun for the quick turnaround! The warning is a meaningful improvement — at least users will know data was silently dropped instead of getting a quietly incomplete graph.

One note for completeness: the warning fires after the collision has already happened — the first-writer-wins behavior in deduplicate_entities() is unchanged, so the losing node's data is still gone at that point. The warning tells you something went wrong but can't recover the dropped nodes.

The fully preventive fix would be computing stems library-side before any LLM call — passing stem= as an explicit attribute in the <untrusted_source> wrapper and instructing the model to use it verbatim. That way two files with identical names in different directories get deterministically different ID namespaces regardless of model behavior, and there is no collision to detect in the first place.

That said, this PR is a solid and safe incremental step. The warning + the existing merge-graphs workaround gives users a viable path forward while the deeper fix is considered.

@safishamsi

Copy link
Copy Markdown
Collaborator

Merged into v8 as 5320aa8 (your authorship). The warning fires only when IDs collide across different source files (real #1504 data loss), staying quiet for same-file dedup — your two tests pin exactly that. Full suite green. Ships next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent data loss: doc files with the same name in different directories produce colliding node IDs across extraction chunks

3 participants