fix(dedup): warn on cross-chunk node ID collision to surface silent data loss#1508
Conversation
…ata loss (Graphify-Labs#1504) When two LLM extraction chunks each process a file with the same name in different directories, they independently generate the same node IDs and deduplicate_entities() silently drops one node (first-writer-wins). The data loss had no indication in any log, counter, or output. Adds a stderr WARNING when a duplicate ID comes from a different source_file, telling the user which files collided and recommending the per-subfolder extract + merge-graphs workflow to avoid it.
|
Thanks @nuthalapativarun for the quick turnaround! The warning is a meaningful improvement — at least users will know data was silently dropped instead of getting a quietly incomplete graph. One note for completeness: the warning fires after the collision has already happened — the first-writer-wins behavior in The fully preventive fix would be computing stems library-side before any LLM call — passing That said, this PR is a solid and safe incremental step. The warning + the existing |
|
Merged into |
Summary
deduplicate_entities()silently drops one of two nodes that share an ID when they come from different source files (first-writer-wins), with no indication in any log, counter, or outputmodule-a/README.mdandmodule-b/README.md) and independently generate the same stem-based IDsstderrWARNING when the pre-dedup loop encounters a duplicate ID from a differentsource_file, telling the user exactly which files collided and pointing to themerge-graphsworkflowChanges
graphify/dedup.py— add collision detection to the pre-dedup loop; emit a WARNING to stderr when a duplicate ID comes from a differentsource_filetests/test_dedup.py— two regression tests: one asserting the warning fires and only the first node survives; one asserting no warning for same-file duplicatesVerification
Closes #1504