Hi! I've been using graphify on a large Python monorepo (~10K files) and ran into a few performance bottlenecks at scale. I pulled 0.7.14 and reviewed the latest code before writing this — here's what I think could help:
1. Pipeline wall-clock benchmark
benchmark.py measures token reduction, which is great for quality. But there's no per-stage timing (detect → extract → build → cluster → analyze → export). On large codebases, knowing which stage dominates is really useful for profiling. I'd like to add a companion benchmark that instruments wall-clock per stage on synthetic graphs of increasing size.
Small, standalone PR. No new dependencies.
2. Sampled edge betweenness in _cross_community_surprises()
suggest_questions() already uses k=min(100, n) for sampling — nice fix. But _cross_community_surprises() (analyze.py ~line 302) still calls nx.edge_betweenness_centrality(G) unsampled, and bails out entirely for >5000 nodes — returning an empty list rather than approximate results.
Replacing the hard cutoff with approximate/sampled edge betweenness would let large single-source corpora get useful results instead of nothing.
3. Pre-computed degree dict in _surprise_score()
Cross-community pre-filter — after reading the composite _surprise_score() system, I think the scoring design is better than hard filtering (it correctly preserves cross-file same-community surprises).
One small optimization instead: _surprise_score() calls G.degree() per edge (~lines 211-212). Pre-computing a degree dict in _cross_file_surprises() and passing it through would avoid repeated lookups on large graphs.
Each would be an independent, backward-compatible PR with tests. Happy to start with whichever is most useful to you.
Hi! I've been using graphify on a large Python monorepo (~10K files) and ran into a few performance bottlenecks at scale. I pulled 0.7.14 and reviewed the latest code before writing this — here's what I think could help:
1. Pipeline wall-clock benchmark
benchmark.pymeasures token reduction, which is great for quality. But there's no per-stage timing (detect → extract → build → cluster → analyze → export). On large codebases, knowing which stage dominates is really useful for profiling. I'd like to add a companion benchmark that instruments wall-clock per stage on synthetic graphs of increasing size.Small, standalone PR. No new dependencies.
2. Sampled edge betweenness in
_cross_community_surprises()suggest_questions()already usesk=min(100, n)for sampling — nice fix. But_cross_community_surprises()(analyze.py ~line 302) still callsnx.edge_betweenness_centrality(G)unsampled, and bails out entirely for >5000 nodes — returning an empty list rather than approximate results.Replacing the hard cutoff with approximate/sampled edge betweenness would let large single-source corpora get useful results instead of nothing.
3. Pre-computed degree dict in
_surprise_score()Cross-community pre-filter— after reading the composite_surprise_score()system, I think the scoring design is better than hard filtering (it correctly preserves cross-file same-community surprises).One small optimization instead:
_surprise_score()callsG.degree()per edge (~lines 211-212). Pre-computing a degree dict in_cross_file_surprises()and passing it through would avoid repeated lookups on large graphs.Each would be an independent, backward-compatible PR with tests. Happy to start with whichever is most useful to you.