feat(indexing): port BM25 enrichment + index from semble

[amondnet]

Summary

Adds packages/layer/modules/markdown-rewrite.ts — a build-time Nuxt module that injects rewrite rules into Vercel's build-output config.json so AI agents get raw markdown instead of the SPA shell.

Ports docus upstream commits:

• 6fd8686b — feat(llms): redirect homepage to /llms.txt
• 9ceafe6f — feat(llms): add docs page redirection to raw markdown for agents

See docs/docus-upstream-changes.md item #9.

Behaviour

• No-op on every non-Vercel preset. The check is preset.startsWith('vercel'), so it covers vercel, vercel-edge, vercel-static, etc.
• On Vercel: read <output.publicDir>/../config.json (the Vercel build-output config), confirm llms.txt was emitted, then unshift route pairs onto routes so they fire before the SPA fallback.
• Rules emitted when the request carries Accept: text/markdown or User-Agent: curl/*:
  • ^/$ → /llms.txt
  • ^/<locale>/?$ → /llms.txt (one per runtimeConfig.public.i18n.locales entry)
  • ^<page>$ → /raw<page>.md (one per /raw/...md link discovered in llms.txt)
• Vercel's has array is AND-ed, so OR semantics between the Accept and User-Agent matchers require emitting two rule entries per src → dest pair.
• Locale codes are regex-escaped before being joined into the alternation, so an exotic code can't break the pattern.

Conventions

• Module style follows the existing packages/layer/modules/{config,shadcn}.ts (defineNuxtModule + named module).
• TypeScript only; an inline VercelBuildOutputConfig / VercelRoute / VercelHeaderHas interface describes the parts we touch — no any.
• Module is registered in packages/layer/nuxt.config.ts immediately after ./modules/shadcn.

Verification

bun install
cd packages/layer && bun typecheck      # no errors in markdown-rewrite.ts
cd ../.. && bun lint                     # clean

# Vercel build
NITRO_PRESET=vercel bun --filter @pleaseai/docs-site build
jq '.routes[] | select(.has // empty)' apps/docs/.vercel/output/config.json

Output (homepage rules only, because the current nuxt-llms config emits links to canonical /docs/... URLs rather than /raw/...md — the per-page rules will start being emitted as soon as llms.txt carries raw-md links):

{
  "src": "^/$",
  "dest": "/llms.txt",
  "headers": { "content-type": "text/markdown; charset=utf-8" },
  "has": [{ "type": "header", "key": "accept", "value": "(.*)text/markdown(.*)" }],
  "continue": true
}
{
  "src": "^/$",
  "dest": "/llms.txt",
  "headers": { "content-type": "text/markdown; charset=utf-8" },
  "has": [{ "type": "header", "key": "user-agent", "value": "curl/.*" }],
  "continue": true
}

Default (cloudflare) build is unaffected — the module bails silently and dist/ is produced as before.

Notes

• Per-page rules require llms.txt to enumerate /raw/...md URLs. The current site config doesn't, so only the homepage routes are emitted today. This matches the upstream behaviour and avoids accidentally rewriting asset URLs. Wiring nuxt-llms to also emit raw-md URLs is tracked separately.
• Runtime e2e is Vercel-only (build-output rewrites apply at the edge), so verification here is limited to inspecting the generated config.json.

Follow-up to #27.

---

Summary by cubic

Ports a minimal, dependency-free BM25 index and helpers from Semble for sparse retrieval with path-aware enrichment. Adds optional mask and disk persistence, normalizes path separators for cross-platform consistency, and optimizes mask handling in scoring.

• New Features
  • Bm25Index with build/getScores, query de-dup, and optional weight mask.
  • Persistence via save(dir) and load(dir); scores are preserved on round-trip.
  • Helpers: enrichForBm25 (adds stem twice + last 3 dir parts) and selectorToMask (Uint8 mask, null-safe).

^{Written for commit f47f867. Summary will update on new commits.}

[gemini-code-assist]

Code Review

This pull request introduces a TypeScript port of a sparse BM25 index (Bm25Index) along with helper functions enrichForBm25 and selectorToMask, and a comprehensive suite of unit tests. The feedback highlights two valuable improvement opportunities: first, using path.posix.parse and normalizing backslashes in enrichForBm25 to ensure cross-platform consistency; second, optimizing the getScores method by skipping BM25 calculations for masked-out documents directly within the postings list loop rather than zeroing them out afterward.

[cubic-dev-ai]

No issues found across 2 files

Architecture diagram

sequenceDiagram
    participant App as Application Code
    participant Indexer as BM25 Indexer
    participant Enricher as enrichForBm25()
    participant Mask as selectorToMask()
    participant FS as File System

    Note over App,FS: Document Indexing Flow

    App->>Indexer: build(documents[])
    Indexer->>Indexer: Tokenize documents
    Indexer->>Indexer: Compute doc lengths, avg length
    Indexer->>Indexer: Build postings list
    Indexer->>Indexer: Compute document frequencies
    Indexer-->>App: Return Bm25Index instance

    Note over App,FS: Document Enrichment (pre-indexing)

    App->>Enricher: enrichForBm25(chunk)
    Enricher->>Enricher: Parse file path
    Enricher->>Enricher: Extract stem (repeated twice)
    Enricher->>Enricher: Extract last 3 directory parts
    Enricher-->>App: Return enriched content string

    Note over App,FS: Query Scoring with Optional Mask

    App->>Indexer: getScores(queryTokens[], weightMask?)
    Indexer->>Indexer: De-duplicate query tokens
    loop For each unique term
        Indexer->>Indexer: Lookup postings list
        Indexer->>Indexer: Compute IDF score
        loop For each matching document
            Indexer->>Indexer: Compute BM25 contribution
            Indexer->>Indexer: Accumulate score
        end
    end
    alt weightMask provided
        Indexer->>Indexer: Zero scores for masked-out docs
    end
    Indexer-->>App: Return Float32Array of scores

    Note over App,FS: Mask Conversion

    App->>Mask: selectorToMask(selector[], size)
    alt selector is null/undefined
        Mask-->>App: Return null
    else
        Mask->>Mask: Create Uint8Array of length size
        loop For each index in selector
            alt index < size
                Mask->>Mask: Set mask[index] = 1
            end
        end
        Mask-->>App: Return Uint8Array
    end

    Note over App,FS: Index Persistence

    App->>Indexer: save(dir)
    Indexer->>Indexer: Serialize state to JSON
    Indexer->>FS: mkdir(dir, recursive)
    Indexer->>FS: writeFile(dir/bm25.json)
    FS-->>Indexer: Confirm write
    Indexer-->>App: Promise<void>

    App->>FS: load(dir)
    FS-->>App: Read bm25.json
    App->>Indexer: Deserialize state
    Indexer->>Indexer: Reconstruct postings Map
    Indexer-->>App: Return Bm25Index instance

Loading

_{Re-trigger cubic}

[amondnet]

Applied 2 of 2 gemini-code-assist suggestions in f47f867:

1. enrichForBm25 — normalize backslashes and use path.posix.parse for cross-platform consistency on repo-relative paths. Added a backslash-normalization test.
2. Bm25Index.getScores — skip masked-out docs inside the postings-list loop instead of zeroing scores in a separate O(N) pass. Result is identical (Float32Array entries default to 0); avoids extra work.

cubic-dev-ai reported no issues. Deferred: 0. All 17 sparse tests pass.