fix(import): resolve graph-index.json race condition in batch import - #67
Conversation
2f8ba1f to
6cb25d8
Compare
Code Review概览PR 解决了一个真实且严重的数据丢失问题:批量 import 并发写全局 🔴 P1 — 聚合时「先 merge 后 detect」会把单仓内部依赖误判为跨仓边
globalGraph = mergeGraphs(globalGraph, overlay);
const crossEdges = detectCrossRepoEdges(overlay, globalGraph, dir.name); // ⚠️ globalGraph 已含 overlay 节点被删除的旧代码顺序相反——先 detect 再 merge, const crossRepoEdges = detectCrossRepoEdges(overlay, existing ?? {...}, slug); // existing = merge 前
const merged2 = existing ? mergeGraphs(existing, overlay) : overlay;
后果:全局图被单仓内部依赖污染,边数虚高——宣称的 780 edges 很可能混入大量这种自匹配噪声。 修复(保留 merge 前的引用再检测): if (globalGraph) {
const crossEdges = detectCrossRepoEdges(overlay, globalGraph, dir.name); // 先用 merge 前的
globalGraph = mergeGraphs(globalGraph, overlay);
if (crossEdges.length > 0) {
globalGraph.edges.push(...crossEdges as GraphIndex['edges']);
}
} else {
globalGraph = overlay;
}🟠 P2 — 需确认1. batch 模式:聚合 markdown 在 push 之后才重建,落后一次未推送
本轮重建的 2.
if (!dryRun && !skipAutoPush && teamwikiRoot) { // 后台深度生成也被 skipAutoPush 关掉了
3. 测试只覆盖纯函数,未覆盖聚合本身
🟡 P3 / 观察
正向亮点
结论REQUEST_CHANGES — 主因是 P1 的 detect/merge 顺序回归,直接影响本 PR 核心产物(全局图边集)的正确性并使指标失真。修掉 P1 + 补一个覆盖聚合路径的测试,并确认 P2 的两处行为后即可合入。 |
f452772 to
abc6390
Compare
When import-repo-list runs with concurrency=3, multiple importFromRepo calls read-merge-write the global graph-index.json concurrently. The last writer wins, silently discarding other repos' nodes/edges. Fix: batch-then-aggregate strategy - Each repo now only writes its own graph to evidence/<slug>/.indices/ - After all repos complete, import-repo-list serially aggregates all per-repo graphs into the global graph-index.json in one pass - Cross-repo edge detection runs during aggregation with full context - A single autoPushTeamRepo call at the end replaces per-repo pushes Additional fixes: - autoPushTeamRepo: log.warn on failure instead of silent catch - explicitDomain from whitelist now overrides AI-inferred _domains.json - Add skipAutoPush option to suppress per-repo push in batch mode Closes Tencent/teamai-cli#(graph-race)
abc6390 to
671218e
Compare
Summary
Fixes critical data loss in
teamai import --from-org/--from-repo-list/--from-repowhen graph-index.json is written concurrently or without full aggregation.Problem
When
import-repo-listdispatches multipleimportFromRepocalls concurrently (default concurrency=3), the globalgraph-index.jsonread-merge-write cycle has no mutual exclusion. The last writer wins, silently discarding other repos' nodes/edges. Result: only the first repo's data survives in the global graph.Fix: batch-then-aggregate strategy + shared aggregation
importFromReponow only writes its own graph toevidence/<slug>/.indices/aggregateGlobalGraph()(src/graph-aggregate.ts): serially merges all per-repo graphs into one globalgraph-index.jsonwith cross-repo edge detectionReview feedback addressed
autoDetectInit()获取_newProject参数未使用as GraphIndex['edges']类型断言'DEPENDS_ON',断言已移除Additional fixes (from E2E testing)
DEFAULT_TIMEOUT_MS从 1200s → 120s + 超时诊断日志--skip-enrich选项: CLI + import-repo + codebase-extract,跳过 AI 只做 extract+graphautoPushTeamRepo:log.warnon failure instead of silentcatch {}explicitDomainfrom whitelist now overrides AI-inferred_domains.jsondetectCrossRepoEdges: compatible with bothid/label/fileandslug/title/typenode formats--skip-enrich选项说明E2E verification (TGit group 378710, 11 repos)
Test plan
npx tsc --noEmit— zero errorsnpx vitest run— 1587 passed (117 files)aggregateGlobalGraph: 4 integration tests (merge, cross-repo edges, P1 regression, empty dir)detectCrossRepoEdges: 4 unit tests (slug/title, config match, mixed format, no-match)import --from-org git.woa.com/378710— 11 repos, 1179 nodes, 780 edges--skip-enrich: import 跳过 AI 调用,仅做 extract+graph