perf: fix O(n^2) -> O(n) LSH neighbor lookup in dedup by sirphilliptubell · Pull Request #1289 · Graphify-Labs/graphify

sirphilliptubell · 2026-06-12T14:36:24Z

Replaced O(n) linear scan next((n for n in candidates if n["id"] == neighbor_id)) with O(1) dict lookup via pre-built candidates_by_id. Also pre-caches _norm() results in norm_cache to avoid recomputing per inner iteration.

For a 36k-file codebase (~100k high-entropy candidates) this reduces the pass-2 loop from O(n^2B) (~30–100B iterations) to O(nB), eliminating the multi-minute CPU hang after AST extraction completes.

On my corporate repo the runtime (w/ instrumentation) went from 2.5 hours to 1.8 hours.

Before

[graphify extract] scanning C:\repo
[graphify extract] found 36456 code, 0 docs, 0 papers, 0 images
[graphify extract] AST extraction on 36456 code files...
  AST extraction: 100/36456 uncached files (0%) [20 workers]
  AST extraction: 200/36456 uncached files (0%) [20 workers]
<trimmed>
  AST extraction: 36300/36456 uncached files (99%) [20 workers]
  AST extraction: 36400/36456 uncached files (99%) [20 workers]
  AST extraction: 36456/36456 files (100%) [20 workers]
[graphify] Deduplicated 153300 node(s) (124857 exact, 23710 fuzzy).
[graphify extract] wrote C:\repo\graphify-out\graph.json: 483203 nodes, 940709 edges, 23340 communities
[graphify extract] wrote C:\repo\graphify-out\.graphify_analysis.json
[graphify extract] next: run `graphify cluster-only C:\repo` to generate GRAPH_REPORT.md and name communities

  _     ._   __/__   _ _  _  _ _/_   Recorded: 15:38:28  Samples:  1614555
 /_//_/// /_\ / //_// / //_'/ //     Duration: 8244.048  CPU time: 7561.906
/   _/                      v5.1.2

Program: graphify .

8244.133 <module>  ..\graphify\__main__.py:1
└─ 8244.051 main  ..\graphify\__main__.py:2070
   └─ 8243.813 main  ..\graphify\__main__.py:2070
      ├─ 6312.412 build  ..\graphify\build.py:276
      │  └─ 6263.680 deduplicate_entities  ..\graphify\dedup.py:133
      │     ├─ 4307.959 [self]  ..\graphify\dedup.py
      │     └─ 1904.307 <genexpr>  ..\graphify\dedup.py:239
      ├─ 772.906 extract  ..\graphify\extract.py:11293
      │  ├─ 434.855 _augment_symbol_resolution_edges  ..\graphify\extract.py:7827
      │  │  └─ 375.742 _collect_js_symbol_resolution_facts  ..\graphify\extract.py:7487
      │  │     └─ 308.992 _walk_js_tree  ..\graphify\extract.py:7194
      │  │        └─ 295.857 _walk_js_tree  ..\graphify\extract.py:7194
      │  │           └─ 283.294 _walk_js_tree  ..\graphify\extract.py:7194
      │  │              └─ 270.829 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                 └─ 258.413 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                    └─ 245.755 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                       └─ 233.292 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                          └─ 220.528 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                             └─ 207.755 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                └─ 194.176 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                   └─ 180.805 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                      └─ 166.745 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                         └─ 153.184 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                            └─ 136.511 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                               └─ 124.575 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                                  └─ 112.806 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                                     └─ 99.625 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                                        └─ 87.919 _walk_js_tree  ..\graphify\extract.py:7194
      │  ├─ 151.511 _extract_parallel  ..\graphify\extract.py:11173
      │  │  └─ 142.180 as_completed  concurrent\futures\_base.py:193
      │  │        [3 frames hidden]  threading, <built-in>
      │  └─ 107.138 load_cached  ..\graphify\cache.py:226
      ├─ 418.009 save_manifest  ..\graphify\detect.py:1229
      │  └─ 408.213 _md5_file  ..\graphify\detect.py:1149
      │     └─ 407.369 WindowsPath.open  pathlib\__init__.py:768
      │        └─ 407.280 open  <built-in>
      ├─ 357.090 detect  ..\graphify\detect.py:997
      │  └─ 311.410 _is_ignored  ..\graphify\detect.py:760
      │     └─ 289.128 _eval  ..\graphify\detect.py:783
      │        └─ 247.802 _matches  ..\graphify\detect.py:787
      │           └─ 215.876 fnmatch  fnmatch.py:22
      │              └─ 134.595 normcase  <frozen ntpath>:50
      ├─ 275.646 cluster  ..\graphify\cluster.py:86
      │  └─ 186.111 _partition  ..\graphify\cluster.py:22
      │     └─ 149.300 func  networkx\utils\decorators.py:783
      │        └─ 149.299 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │           └─ 149.299 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │              └─ 149.299 louvain_communities  networkx\algorithms\community\louvain.py:14
      │                 └─ 149.239 louvain_partitions  networkx\algorithms\community\louvain.py:133
      │                    └─ 112.915 _one_level  networkx\algorithms\community\louvain.py:227
      └─ 85.921 to_json  ..\graphify\export.py:484

To view this report with different options, run:
    pyinstrument --load-prev 2026-06-11T15-38-28 [options]

Elapsed: 8635.5s

After

[graphify extract] scanning C:\repo
[graphify extract] found 36456 code, 0 docs, 0 papers, 0 images
[graphify extract] AST extraction on 36456 code files...
  AST extraction: 100/36456 uncached files (0%) [20 workers]
  AST extraction: 200/36456 uncached files (0%) [20 workers]
<trimmed>
  AST extraction: 36300/36456 uncached files (99%) [20 workers]
  AST extraction: 36400/36456 uncached files (99%) [20 workers]
  AST extraction: 36456/36456 files (100%) [20 workers]
[graphify] Deduplicated 153308 node(s) (124857 exact, 23710 fuzzy).
[graphify extract] wrote C:\repo\graphify-out\graph.json: 483195 nodes, 940706 edges, 23603 communities
[graphify extract] wrote C:\repo\graphify-out\.graphify_analysis.json
[graphify extract] next: run `graphify cluster-only C:\repo` to generate GRAPH_REPORT.md and name communities

  _     ._   __/__   _ _  _  _ _/_   Recorded: 08:16:56  Samples:  1082577
 /_//_/// /_\ / //_// / //_'/ //     Duration: 6222.870  CPU time: 5191.547
/   _/                      v5.1.2

Program: graphify .

6223.129 <module>  ..\graphify\__main__.py:1
└─ 6223.041 main  ..\graphify\__main__.py:2070
   └─ 6222.467 main  ..\graphify\__main__.py:2070
      ├─ 4294.127 build  ..\graphify\build.py:276
      │  └─ 4247.563 deduplicate_entities  ..\graphify\dedup.py:133
      ├─ 580.815 detect  ..\graphify\detect.py:997
      │  ├─ 429.577 count_words  ..\graphify\detect.py:624
      │  │  └─ 428.059 WindowsPath.read_text  pathlib\__init__.py:785
      │  │     └─ 427.236 WindowsPath.open  pathlib\__init__.py:768
      │  └─ 141.300 _is_ignored  ..\graphify\detect.py:760
      │     └─ 134.949 _eval  ..\graphify\detect.py:783
      │        └─ 121.492 _matches  ..\graphify\detect.py:787
      │           └─ 107.489 fnmatch  fnmatch.py:22
      │              └─ 68.557 normcase  <frozen ntpath>:50
      ├─ 473.912 save_manifest  ..\graphify\detect.py:1229
      │  └─ 459.808 _md5_file  ..\graphify\detect.py:1149
      │     └─ 453.686 WindowsPath.open  pathlib\__init__.py:768
      │        └─ 453.513 open  <built-in>
      ├─ 437.577 cluster  ..\graphify\cluster.py:86
      │  ├─ 305.431 _partition  ..\graphify\cluster.py:22
      │  │  └─ 270.806 func  networkx\utils\decorators.py:783
      │  │     └─ 270.805 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │  │        └─ 270.805 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │  │           └─ 270.805 louvain_communities  networkx\algorithms\community\louvain.py:14
      │  │              └─ 270.694 louvain_partitions  networkx\algorithms\community\louvain.py:133
      │  │                 └─ 229.701 _one_level  networkx\algorithms\community\louvain.py:227
      │  │                    ├─ 107.482 [self]  networkx\algorithms\community\louvain.py
      │  │                    └─ 105.178 _neighbor_weights  networkx\algorithms\community\louvain.py:335
      │  └─ 121.545 _split_community  ..\graphify\cluster.py:191
      │     └─ 114.391 _partition  ..\graphify\cluster.py:22
      │        └─ 70.913 argmap_louvain_communities_5  <class 'networkx.utils.decorators.argmap'> compilation 9:1
      │           └─ 70.900 _dispatchable._call_if_no_backends_installed  networkx\utils\backends.py:541
      │              └─ 70.896 louvain_communities  networkx\algorithms\community\louvain.py:14
      │                 └─ 70.842 louvain_partitions  networkx\algorithms\community\louvain.py:133
      ├─ 335.515 extract  ..\graphify\extract.py:11293
      │  ├─ 155.039 _augment_symbol_resolution_edges  ..\graphify\extract.py:7827
      │  │  └─ 133.144 _collect_js_symbol_resolution_facts  ..\graphify\extract.py:7487
      │  │     └─ 109.682 _walk_js_tree  ..\graphify\extract.py:7194
      │  │        └─ 105.322 _walk_js_tree  ..\graphify\extract.py:7194
      │  │           └─ 100.924 _walk_js_tree  ..\graphify\extract.py:7194
      │  │              └─ 96.580 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                 └─ 92.190 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                    └─ 87.665 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                       └─ 83.035 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                          └─ 78.448 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                             └─ 73.571 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                └─ 68.792 _walk_js_tree  ..\graphify\extract.py:7194
      │  │                                   └─ 64.084 _walk_js_tree  ..\graphify\extract.py:7194
      │  └─ 100.371 _extract_parallel  ..\graphify\extract.py:11173
      │     └─ 96.388 as_completed  concurrent\futures\_base.py:193
      │           [3 frames hidden]  threading, <built-in>
      └─ 81.598 to_json  ..\graphify\export.py:484
         └─ 71.129 dump  json\__init__.py:120

To view this report with different options, run:
    pyinstrument --load-prev 2026-06-12T08-16-56 [options]

Elapsed: 6425.2s

Replaced O(n) linear scan `next((n for n in candidates if n["id"] == neighbor_id))` with O(1) dict lookup via pre-built `candidates_by_id`. Also pre-caches `_norm()` results in `norm_cache` to avoid recomputing per inner iteration. For a 36k-file codebase (~100k high-entropy candidates) this reduces the pass-2 loop from O(n^2*B) (~30–100B iterations) to O(n*B), eliminating the multi-minute CPU hang after AST extraction completes.

…bs#1207, Graphify-Labs#1208, Graphify-Labs#1176

safishamsi merged commit 418fab8 into Graphify-Labs:v8 Jun 12, 2026

safishamsi added a commit that referenced this pull request Jun 12, 2026

changelog: add unreleased entries for #1289, #1207, #1208, #1176

3929eca

rbelem pushed a commit to rbelem/graphify that referenced this pull request Jun 12, 2026

changelog: add unreleased entries for Graphify-Labs#1289, Graphify-La…

ad587a3

…bs#1207, Graphify-Labs#1208, Graphify-Labs#1176

sirphilliptubell deleted the perf/fix1 branch June 15, 2026 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

perf: fix O(n^2) -> O(n) LSH neighbor lookup in dedup#1289

perf: fix O(n^2) -> O(n) LSH neighbor lookup in dedup#1289
safishamsi merged 1 commit into
Graphify-Labs:v8from
sirphilliptubell:perf/fix1

sirphilliptubell commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

sirphilliptubell commented Jun 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants