Semantic layer is mostly disconnected from the AST graph

[cs-anonymous]

I think there is a structural issue in how graphify merges AST extraction and semantic extraction.

In practice, the semantic layer behaves more like a heuristic exploratory subgraph than a true semantic enrichment layer for the code graph. The two layers are often largely independent, and meaningful fusion seems to rely mostly on exact node-id collisions.

That is a very brittle condition.

Why this happens

AST and semantic extraction run in parallel, so the semantic pass does not know which AST nodes were produced in the current run.

Because of that, the semantic layer cannot intentionally attach edges to AST-only nodes. It can only emit its own nodes and edges, and later merge if some node ids happen to match exactly.

But exact id matching is hard in real cases:

• AST: SentenceTransformer, get_embedding()
• semantic: "sentence transformer", "get embedding"

These may describe the same thing, but they are unlikely to end up with the same id.

What I observed

I checked a generated graph locally:

OpenHarness: https://github.com/HKUDS/OpenHarness

• Node types include code, rationale, document, and concept
• I found effectively no code <-> document or code <-> concept bridges
• The code graph forms one large connected component
• The semantic layer forms a separate subgraph

The similar situation happens in many other cases, the final graph looks more like:

• one AST/code graph
• one semantic/document graph
• very limited real integration between them

Suggestion

It may need a stronger alignment step, such as:

• canonical labels / normalized concept ids
• post-merge entity resolution between AST nodes and semantic nodes
• explicit code-to-concept linking after both passes complete

At the moment, the semantic layer is useful as a heuristic exploration layer, but it does not seem to reliably enrich the AST graph itself.

[Lum1104]

Hi, you can try my skill here, the Understand-Anything that built for the codebase understanding.

[karthick1005]

Hi, I’ve been exploring this issue and I can reproduce the disconnect between AST and semantic layers.

I’d like to work on an alignment step that:

• normalizes identifiers (camelCase, snake_case → tokens)
• performs fuzzy matching between AST and semantic nodes
• introduces explicit bridging edges (code ↔ concept)

Goal is to ensure the semantic layer actually enriches the AST graph instead of forming a separate subgraph.

Happy to align on approach before starting implementation.

[cs-anonymous]

I’m glad this issue is receiving attention, and I’m really looking forward to your solution. I’m also experimenting with other tools, such as Understand-Anything, deepwiki and similar GraphRAG-style tools.

[safishamsi]

v0.4.3 fixed the most impactful part of this: relative import edges (e.g. ./foo) were silently dropped because AST node IDs used full paths but import target IDs used short names (#256). On large TypeScript codebases this meant 0 cross-file edges. That's now resolved, so the AST graph should be much more connected. The deeper point about AST↔semantic label matching is a harder problem — entity resolution / canonicalization is tracked in #207.

[Adrian-LSY]

Adding 0.7.7 evidence — this is still happening, and it has a clean drop-in fix.

Reproduction on a 39-file corpus

Corpus: Codex monorepo root (31 Python code files + 8 markdown/yaml docs, ~13k words). Standard /graphify invocation, two parallel Claude subagents handled semantic extraction.

After merge, every Python test function existed twice in the graph because the two passes label and locate identically-named entities differently:

Pass	label	source_file root	source_location
AST	test_settings_users_crud	e2e/tests/... (project-rooted)	None
Semantic	test_settings_users_crud()	tests/... (relative)	L69

The skill prompt says "IDs must be deterministic from the label alone" — but neither the AST nor the semantic side normalizes the label before hashing, so:

• AST id = test_org_admin_settings_users_crud
• Semantic id = tests_test_org_admin_test_settings_users_crud

Both pointing at line 69 of the same file. 84 such paren-twin pairs in this single 39-file corpus.

Knock-on effect: every --calls--> edge from navigate_to_tab() exists twice (once EXTRACTED to the AST node, once INFERRED to its phantom twin). navigate_to_tab() reported degree 31 — 16 real edges plus 15 duplicate phantoms.

Post-processing fix that works (no LLM tokens)

Two passes on graph.json:

def norm(label): return re.sub(r'\(\s*\)$', '', (label or '')).strip().lower()
def basename(p):  return (p or '').rsplit('/', 1)[-1]

# Group by (normalized_label, basename(source_file)) only — ignore source_location
# because AST often has it as None while semantic provides it.
groups = defaultdict(list)
for nid, n in nodes.items():
    key = (norm(n['label']), basename(n.get('source_file','')))
    if key[0] and key[1]:
        groups[key].append(nid)

# Canonical = label without trailing "()", then shortest id (AST nodes win)
remap = {}
for key, ids in groups.items():
    if len(ids) < 2: continue
    canonical = min(ids, key=lambda i: (1 if nodes[i]['label'].endswith('()') else 0, len(i)))
    for nid in ids:
        if nid != canonical:
            # Graft non-null source_location from absorbed twin onto canonical
            if not nodes[canonical].get('source_location') and nodes[nid].get('source_location'):
                nodes[canonical]['source_location'] = nodes[nid]['source_location']
            remap[nid] = canonical

# Apply remap to edges + hyperedges, drop self-loops, dedupe by (s, t, relation)
# Then `graphify cluster-only` to refresh communities.

Result

Metric	Before	After
Nodes	373	154
Edges	392	212
Communities	51	13
Weakly-connected (deg ≤ 1)	195 (52%)	52 (34%)
Singleton communities	23	2
navigate_to_tab() degree	31 (15 phantom)	16 (real)
Procurement E2E cohesion	(split across stubs)	0.39
Project Detail cohesion	(split across stubs)	0.36

Communities went from "mostly named after a single function" to clean topical clusters: Page Object Helpers & Auth Fixtures, Codex Submodule Architecture, Persona Fixtures & Playwright Doc, Login Flow E2E, Docker Compose Stack, etc. The Playwright-doc → fixtures → helpers cross-cluster bridge survived the cleanup intact, which suggests the remap is preserving real signal.

Suggested upstream placement

In merge_ast_semantic (or wherever the two extractions are unioned), apply the (norm_label, basename) canonicalization before inserting semantic nodes into the existing AST graph. The label pairing is robust because the semantic agent reliably emits function_name() labels and the AST extractor emits function_name. If a project diverges from that convention, falling back to (norm_label, basename, source_location) when source_location is non-null on both sides catches the rest.

Happy to PR this if you'd like — the change touches one merge step and doesn't alter any extraction code.