Skip to content

AST chunking never runs — core.ts ALL_LANGUAGES is an empty stub set #28

Description

@amondnet

Summary

src/chunking/core.ts defines ALL_LANGUAGES as an empty stub Set, while the real, populated set already exists in src/indexing/files.ts. Because chunk-source.ts gates AST chunking on isSupportedLanguage() (which reads the empty set), tree‑sitter AST chunking never runs — every file falls back to line chunking.

Evidence

  • src/chunking/core.ts:11-14
    // Stub for ALL_LANGUAGES until Unit 4 (language detection) lands.
    //   import { ALL_LANGUAGES } from '../indexing/files.ts'
    const ALL_LANGUAGES: ReadonlySet<string> = new Set<string>()
  • src/chunking/core.ts:87isSupportedLanguage(lang) => ALL_LANGUAGES.has(lang) → always false.
  • src/chunking/chunk-source.ts:35-36if (language !== null && isSupportedLanguage(language)) { chunkBoundaries = await chunk(...) } → branch never taken.
  • src/indexing/files.ts:436 — real export const ALL_LANGUAGES already exists (Unit 4 has landed).
  • @kreuzberg/tree-sitter-language-pack (^1.8.1) is installed but effectively bypassed.

Impact

CLAUDE.md specifies "Chunking: tree‑sitter AST‑based with line‑fallback". Currently it is line‑fallback only, so chunk boundaries ignore code structure — directly affecting chunk quality and therefore search relevance.

Acceptance criteria

  • core.ts imports ALL_LANGUAGES from ../indexing/files.ts and removes the empty stub set.
  • Verify no import cycle is introduced (files.tschunking/); restructure if needed.
  • Add a test asserting a supported language (e.g. typescript) takes the AST path (isSupportedLanguage('typescript') === true).
  • Manual/spot check that AST chunking actually produces structure‑aligned boundaries.

Found during a stub audit.

Metadata

Metadata

Assignees

No one assigned

    Labels

    p1Priority 1 - Hightype:bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions