AST extractor misses cross-file inheritance and produces ghost-duplicate nodes for cross-file type references

[cfbrush]

AST extractor misses cross-file inheritance and produces ghost-duplicate nodes for cross-file type references

graphify version: 0.8.35
Python: 3.x (uv tool install)
Corpus: 891 Python files, 14,050 nodes, 17,822 edges, AST-only extraction

Summary

The AST extractor has a single root-cause gap that surfaces as two distinct bugs:

1. class X(Base): does not produce an inherits/EXTRACTED edge when Base is imported from another project file.
2. Type annotations and class-base references that cross a file boundary create duplicate "ghost" nodes with inconsistent ID formats instead of resolving to the canonical definition node.

Both stem from the same shortcoming: when AST sees a Name reference whose binding comes from an import of another project module, it does not resolve the reference back to the canonical node. Instead it emits a fresh node (often with a malformed ID) and/or downgrades the relation to uses/INFERRED at the import line.

Repro

Any Python codebase with cross-file inheritance is enough. Concrete observed examples below come from iwfm-py (procedural codebase with one Strategy/Backend hierarchy):

# iwfm/iwfm/hdf5/hdf_metadata_base.py
class HdfBackend(ABC):  # L64
    @abstractmethod
    def get_n_nodes(...): ...

# iwfm/iwfm/hdf5/hdf_metadata.py
from iwfm.iwfm.hdf5.hdf_metadata_base import HdfBackend, HdfMetadata  # L42
class HdfReader(HdfBackend):  # L47
    def get_n_nodes(self, ...) -> int: ...
    def get_metadata(self, ...) -> HdfMetadata: ...  # L284

# iwfm/iwfm/dll/dll_backend.py
from iwfm.iwfm.hdf5.hdf_metadata_base import HdfBackend, HdfMetadata  # L25
class DllBackend(HdfBackend):  # L29
    def get_metadata(self, ...) -> HdfMetadata: ...  # L261

Bug 1 — Cross-file inheritance silently dropped

Greping the corpus for all class X(Y): declarations and comparing to graph inherits edges:

Source declaration	Captured?	Why
class SimulationFiles(_DictAccessMixin):	✓	Same file
class PreprocessorFiles(_DictAccessMixin):	✓	Same file
class GroundwaterFiles(_DictAccessMixin):	✓	Same file
class RootzoneFiles(_DictAccessMixin):	✓	Same file
class BackendNotAvailableError(HdfError):	✓	Same file
class DataSourceError(HdfError):	✓	Same file
class HdfReader(HdfBackend):	✗	Cross-file
class DllBackend(HdfBackend):	✗	Cross-file

Capture rate:

• Same-file internal bases: 6/6 (100%)
• Cross-file internal bases: 0/2 (0%)
• Stdlib bases (ABC, Enum, Exception, str, object): all captured

What the graph contains for the missed cases:

HdfReader  --uses/INFERRED--> HdfBackend  @ L42  (the import line, not L47 where class is declared)
DllBackend --uses/INFERRED--> HdfBackend  @ L25  (the import line, not L29)

Expected:

HdfReader  --inherits/EXTRACTED--> HdfBackend  @ L47
DllBackend --inherits/EXTRACTED--> HdfBackend  @ L29

Impact on graph quality

Community detection has no way to bind a polymorphic family together without inheritance edges. In this corpus:

Community	What it contains	Should be
C4	HdfBackend ABC + tests, alone	Same family
C17	HdfReader + tests, alone	Same family
C147	DllBackend alone (no tests)	Same family

Three communities for one Strategy/Backend abstraction. The HDF backend hierarchy is invisible to any clustering or god_nodes computation that relies on structural edges.

Bug 2 — Ghost-duplicate nodes from cross-file type references

Cross-file references in type annotations and class bases produce duplicate nodes with malformed IDs instead of resolving to the canonical definition.

Four substantive ghosts observed in 14,050 nodes:

Ghost ID	Canonical ID	Source of ghost	In-edges absorbed
context (bare)	n/a (refers to external typer.Context)	def main(ctx: typer.Context, ...) @ main.py:L90	1
userlevel (bare)	cli_context_userlevel (20/5 edges)	def configure_logging(user_level: UserLevel) @ main.py:L67	2
iwfm_iwfm_dll_dll_backend_py_hdfmetadata	hdf5_hdf_metadata_base_hdfmetadata (19/0 edges)	def get_metadata(...) -> HdfMetadata: @ dll_backend.py:L261	1
iwfm_iwfm_hdf5_hdf_metadata_py_hdfmetadata	hdf5_hdf_metadata_base_hdfmetadata	def get_metadata(...) -> HdfMetadata: @ hdf_metadata.py:L284	1

The bare-name ghosts (context, userlevel) and stdlib bare names (object, str, enum, exception, abc) are present in the raw AST cache — not introduced during merge — so the bug is in the extractor.

Two distinct ID-format conventions coexist

Skill spec mandates {parent_dir}_{filename_stem}_{entity} (e.g. cli_main_userlevel). Observed formats:

1. Bare name — e.g. userlevel, context. No prefix at all. Used when the resolver gives up on a name.
2. Full-path-with-_py_ — e.g. iwfm_iwfm_dll_dll_backend_py_hdfmetadata. Includes full path from repo root, keeps .py extension as a literal _py_ segment. Used for type annotations referencing cross-file imports.
3. Clean spec — e.g. hdf5_hdf_metadata_base_hdfmetadata. Used for entity definitions inside the file.

Both (1) and (2) produce nodes that don't match (3), so canonical and ghost coexist for the same logical symbol.

Adjacent cosmetic issue

214 nodes have an iwfm_iwfm_* double-prefix because the codebase layout is iwfm/iwfm/*.py (package nested in repo root with same name). The ID generator joined two levels of parent path when it should have used only the immediate parent. These are mostly unique nodes — no ghost duplicates — but the format inconsistency obscures the spec.

Root cause hypothesis

When AST encounters a Name node (in a class base, type annotation, or other typed context) whose binding comes from from X import Y where X is a project-internal module:

• Resolver doesn't follow the import to the canonical definition node.
• Resolver emits a fresh node with whatever ID the local context produces (bare name, or full path with _py_ separator).
• The relation gets degraded — inherits becomes uses/INFERRED at the import line, type annotation becomes a references/EXTRACTED to the ghost.

Same-file references work because the resolver finds the definition in the current file's symbol table without needing import resolution. Stdlib references work because they bottom out at known external types that don't need canonical-node resolution.

Suggested fix direction

A single pass over from X import Y statements that builds a name → canonical-node-ID map for project-internal imports, applied before emitting any edge involving an imported name. With that map:

• class HdfReader(HdfBackend): resolves HdfBackend to hdf5_hdf_metadata_base_hdfbackend and emits inherits/EXTRACTED.
• def get_metadata(...) -> HdfMetadata: resolves HdfMetadata to hdf5_hdf_metadata_base_hdfmetadata and emits references/EXTRACTED without a fresh ghost.
• Bare-name fallback (userlevel, context) reserved for truly unresolvable references (stdlib, third-party, builtins).

ID-format inconsistency (1) vs (2) vs (3) is a separate normalization fix — pick one and apply everywhere.

Reproduction evidence

Audit Q&A from the iwfm-py corpus (saved to graphify-out/memory/):

• query_20260607_193207_audit_of_75_inferred_edges_on_read_next_line_value.md
• query_20260607_193602_why_does_context___uses____userlevel_appear_as_sur.md
• query_20260607_193903_sweep_for_ghost_duplicate_nodes_across_14050_node.md
• query_20260607_194331_why_is_hdf_reader_vs_metadata_backend_split_across.md
• query_20260607_194503_sweep_for_missing_inheritance_edges_across_corpus.md

Happy to provide the corpus or extracted graph.json for repro.

[safishamsi]

Progress: the C# cross-file type-reference resolver landed on v8 (PR #1466), the C# analogue of the existing Java resolver — it collapses ghost-duplicate shadow nodes for cross-file type refs onto the real definition, refusing ambiguous matches. Go (#1500) and same-label import stubs (#1462) were also fixed this cycle. Leaving open for the remaining languages/cases. Ships in the next release.

[safishamsi]

Progress: 0.9.0 (full-path node IDs) plus the earlier C#/Go cross-file resolvers address much of the ghost-duplicate / cross-file id-collision problem. Leaving open for the remaining cross-file inheritance/type-reference cases not yet covered by a per-language resolver.