fix(extract): skip Python primitive/library type-annotation nodes

[MasterKinjalk]

Problem

Type annotations (return_type, parameter_type, field, generic_arg, base class) materialize Python language primitives and common library types — str, int, Tensor, ndarray, Path, Module, np, torch — as graph nodes. They carry no domain meaning and become spurious high-degree hubs that pollute community clustering and clutter the report's node list.

Cause

_LANGUAGE_BUILTIN_GLOBALS already filtered these names at call sites, but not at type-reference sites, which route through ensure_named_node.

Fix (scoped to Python — see review thread)

• Add _PYTHON_TYPE_REF_PRIMITIVES = _LANGUAGE_BUILTIN_GLOBALS ∪ {stdlib / typing / NumPy / PyTorch annotation types}, gated per language via _TYPE_REF_PRIMITIVES_BY_LANG / _type_ref_primitives(ts_module).
• _extract_generic skips add_node for these names only when the language is Python. The node id is still returned, so reference edges still resolve (dangling-to-external, tolerated by build.py).
• The hand-written Julia / Fortran / Rust / PowerShell / Obj-C walkers no longer carry the guard — it only over-filtered foreign-named user types there.

Why Python-only

The same bare names are ordinary user-defined types in other languages (a Rust Response, a Julia Module, a Go Set). A single flat membership test applied in every extractor silently dropped them. Languages that need to suppress their own primitives already do so at the collector, scoped to the language — Go via _GO_PREDECLARED_TYPES, Rust by skipping primitive_type nodes, Python scalars via _PYTHON_ANNOTATION_NOISE (#1147). This PR layers the library/annotation names on top, for Python only.

Tests

tests/test_type_ref_primitives.py — per affected extractor (Python / Rust / Julia / Go): primitive dropped, a real same-named user type kept, edge still resolves, plus the cross-language over-filter regression.

Verification

Extracting a sample with Tensor / ndarray / int annotations:

• Before: Tensor, ndarray, int, np, torch appear as nodes.
• After: only real nodes remain (Net, .forward(), .to_np(), count()); no primitive leakage.

Rebased onto v8 (9e0b876). Full suite: no new failures vs the v8 base.

🤖 Generated with Claude Code

[safishamsi]

Thanks @MasterKinjalk — the idea (skip primitive/library type names so they don't become noise nodes) is sound, but a few things need addressing before this can merge:

1. Inconsistent coverage. The guard is added to 6 of the 7 ensure_named_node definitions but is missing in the Go extractor (extract_go), so Go type references will still materialize int/string etc. as nodes. Add it there or document why Go is excluded — the PR description says 'all language extractors', which isn't currently true.
2. False-positive over-filtering. The membership test is on the bare type name with no namespace/origin check, so a user-defined type named Path/Set/Type/Module/Dataset/Parameter/device/File/object is silently dropped as a node while its inheritance/reference edge still points at it — recreating the orphan/shadow-node problem AST extraction generates inconsistent node IDs for same symbol at definition vs. reference sites, causing edges to connect to orphan "shadow nodes" instead of actual definitions #1318 just fixed. Please gate by language and/or trim names that plausibly collide with user-defined types.
3. Scope. The filter only touches the hand-written walkers, not _extract_generic, so the high-traffic typed languages (Java/Swift/TS/JS) aren't covered. Either extend it to _extract_generic or narrow the description to match.
4. Tests. This is exactly the kind of change that silently drops nodes — please add a regression test per affected extractor (primitive dropped, a real same-named user type kept, edge still resolves).

Also please rebase onto current v8 (the branch predates #1318/#1327). Happy to re-review once these are addressed.

[MasterKinjalk]

Thanks for the detailed review. Rebased onto current v8 and reworked the approach after tracing how each extractor actually handles type references — addressed point by point below.

Scope reworked to Python-only. The root issue you flagged — a name-only membership test with no language/origin check — drove the redesign. The filter is no longer one flat set applied in every walker; it's keyed by language (_TYPE_REF_PRIMITIVES_BY_LANG) and populated for Python only.

1. Go / consistency. Go is intentionally excluded, with a reason rather than a guard: _go_collect_type_refs already filters Go's predeclared types via _GO_PREDECLARED_TYPES, so Go never materializes int/string/error as nodes. Adding the flat guard to Go would only over-filter user types (e.g. a Go type named Set). Same pattern already exists for Rust (primitive_type nodes are skipped) and Python scalars (_PYTHON_ANNOTATION_NOISE, #1147) — predeclared filtering lives at the per-language collector, which is exactly where Go does it.

2. Over-filtering / shadow nodes. Two findings:

• A user type defined in the same file is never dropped — ensure_named_node returns the stem-qualified id for in-file symbols and only gates the bare cross-file/external id. So the orphan/shadow-node problem AST extraction generates inconsistent node IDs for same symbol at definition vs. reference sites, causing edges to connect to orphan "shadow nodes" instead of actual definitions #1318 fixed does not recur for same-file defs (regression test included).
• The genuine over-filtering was cross-language: the flat set carried JS globals (Response, Request, File) and PyTorch names (Module, Tensor), which mis-dropped identically-named Rust/Julia/etc. user types. Scoping to Python removes that. test_rust_type_named_like_js_global_not_overfiltered covers it (red before, green after).

3. Scope / typed languages. _extract_generic is now the only place the guard lives, so Python (routed through it) is covered. Java/Swift/TS/JS and the other typed languages keep their primitive type nodes on purpose — their extractor tests assert those refs resolve as nodes (PowerShell string/void, Julia Float64, C++ string/vector, Obj-C NSString). Narrowed the PR description to "Python" to match.

4. Tests. Added tests/test_type_ref_primitives.py: per extractor (Python / Rust / Julia / Go) — primitive dropped, real same-named user type kept, edge still resolves, plus the cross-language over-filter regression.

5. Rebase. Done — rebased onto v8 (9e0b876), clean. #1318/#1327 are already in the base.

Full suite shows no new failures vs the v8 base. Happy to adjust scope if you'd prefer the filter extended to other languages' own primitives (that would mean updating those extractors' existing assertions, so I left them as-is).

[Synvoya]

Thanks for tackling this — the problem is real and correctly diagnosed. Library/annotation type names (Tensor, ndarray, np, torch, Module, Path, datetime) leaking into the graph as high-degree noise hubs is worth fixing, and the per-language scoping rationale in the description is exactly the right instinct (a flat global list would mis-drop a Rust Response or a Go Set). Two things before it can land, though — the patch as written can't apply to v8, and its mechanism collides with a fix that went in after this branch was cut.

1. The patch no longer applies to v8

git apply --check fails at extract.py:43. This was written against a pre-refactor extract.py: the hunk anchors on an ensure_named_node whose body is

nid = _make_id(name)
if nid not in seen_ids:
    add_node(nid, name, line)

but on current v8 (extract.py:3504) that function is now nid = _make_id(stem, ".".join(namespace_stack), name) and its body is a sourceless-stub append, not add_node. Needs a rebase before it can be assessed for merge.

2. Suppressing the node here would regress the #1402 phantom-duplicate fix

This is the blocking concern. On v8, ensure_named_node deliberately materializes a sourceless stub for cross-file type references (extract.py:3510-3525):

# The name isn't defined in this file, so this is a cross-file reference ...
# Emit a SOURCELESS stub ... so the corpus-level rewire can collapse it onto
# the real definition. A sourced stub here ... blocks the rewire, which is
# the phantom-duplicate-node bug (#1402).

The PR's change is "don't create the node at all, just return the id." For an imported real type (from .models import Dataset, used only in an annotation), that removes the very stub the corpus rewire keys on — so instead of collapsing onto the real Dataset definition, the reference edge now points at an id with no node and no rewire target. That reintroduces #1402 for real imported types. The same sourceless-stub pattern is load-bearing at seven sites (3516, 7690, 8007, 8270, 8625, 8989, 12274), so a suppression at ensure_named_node fights all of them.

3. The scalar names in _PYTHON_TYPE_REF_PRIMITIVES are already filtered upstream

Python parameter/return/generic annotations don't reach ensure_named_node raw — they go through _python_collect_type_refs (extract.py:651), which already drops _PYTHON_ANNOTATION_NOISE (str, int, float, bool, bytes, bytearray, complex, object, True, False, at 641) and _PYTHON_TYPE_CONTAINERS before the emit sites at 4384-4401. So int, str, bytearray, complex, object in the new set can never fire — they're filtered a layer earlier. The genuinely-new coverage is only the library names (Tensor, ndarray, np, torch, Module, Dataset, NDArray, Path, datetime, …).

Suggested direction: drop the ensure_named_node change and instead extend the existing collector filter — add the library/annotation names to _PYTHON_ANNOTATION_NOISE (or a sibling _PYTHON_LIBRARY_ANNOTATION_NOISE frozenset unioned into the same guards at 651-698). That filters the noise at the same point Python scalars are already filtered, keeps the fix Python-scoped without the _TYPE_REF_PRIMITIVES_BY_LANG indirection, and never suppresses a stub the #1402 rewire depends on — a name dropped at the collector never creates an edge in the first place, so there's no dangling target.

One caveat on that path: a name like Path or Dataset can legitimately be a user's own class. Filtering it at the collector means a real project-internal class Dataset referenced in an annotation would also lose that annotation edge. _PYTHON_ANNOTATION_NOISE already accepts that trade for scalars; worth a sentence on why it's acceptable for these library names too (they're overwhelmingly third-party), or gating on whether the name resolves to a local definition.

4. Tests assert behavior this diff doesn't change

tests/test_type_ref_primitives.py asserts Rust/Julia/Go outcomes, but the diff only adds a Python key to _TYPE_REF_PRIMITIVES_BY_LANG — every other language resolves to frozenset() and is untouched, so those cases test pre-existing collector behavior, not this change. The one test that exercises the new path (test_python_primitive_type_refs_not_materialized) is the one to keep; consider folding it into tests/test_languages.py to match the per-language convention the rest of the suite (and the merge-onto-v8 flow) uses.

Net: correct problem, right scoping instinct, but (2) is a real regression risk and (1) blocks the merge. Moving the filter to the collector resolves both and shrinks the diff. Happy to re-review after a rebase.