llms.txt aggregate walker only descends one level, undercounts deeply-nested indexes

[SahilAujla]

Context

The aggregate .txt walker in src/helpers/get-page-urls.ts (walkAggregateLinks) only descends one level into nested llms.txt indexes. Sub-link .txt references at depth 2+ are explicitly filtered out (see “skip further .txt nesting” comment in the code).

This works for two-level patterns (Cloudflare per-product files, Supabase aggregate content files) but undercounts sites that use deeper progressive disclosure — which the spec encourages for large sites that would otherwise exceed llms-txt-size.

The visible symptom is llms-txt-freshness reporting very low coverage on sites whose llms.txt is actually exhaustive — the walker just isn’t reaching the leaves.

Concrete example

Site: alchemy.com/docs

Three-level structure (correctly organized per spec — sections split because a unified file would exceed llms-txt-size):

/docs/llms.txt                           # 6 section links, all .txt
  └─ /docs/get-started/llms.txt          # .md page links     [walked]
  └─ /docs/node/llms.txt                 # .md page links     [walked]
  └─ /docs/data/llms.txt                 # .md page links     [walked]
  └─ /docs/wallets/llms.txt              # .md page links     [walked]
  └─ /docs/rollups/llms.txt              # .md page links     [walked]
  └─ /docs/chains/llms.txt               # ~80 chain .txt links [walked, children dropped]
      └─ /docs/chains/ethereum/llms.txt  # eth_call, eth_chainId, …  [NOT REACHED]
      └─ /docs/chains/solana/llms.txt    # getBalance, getSlot, …    [NOT REACHED]
      └─ … 80 more chains                                            [NOT REACHED]

Five sections have a flat layout, so their pages are counted (~311 page URLs total). The Chains section has another nesting level for per-chain files, and all ~5,100 method pages live at depth 2. Those never make it into the URL pool.

Verbose afdocs output:

✗ llms-txt-freshness: llms.txt covers 311/5452 sitemap doc pages (6%); 5141 missing
      Fix: Your llms.txt covers less than 80% of your site's pages. ...

The 5,141 “missing” pages are mostly chain-specific RPC method docs:

• /docs/chains/ethereum/ethereum-api-endpoints/eth-call
• /docs/chains/solana/solana-api-endpoints/get-balance
• …

These are in the llms.txt tree — just one level deeper than the walker currently explores.

This also biases sampling for any check that flows through getUrlsFromCachedLlmsTxt:

• markdown-content-parity
• llms-txt-directive
• etc.

All sample only from the 5 flat sections, never from the ~5,100 chain method pages.

Suggested behaviors (in priority order)

1. Walk recursively, with bounded depth and file count

   • Stop at e.g. depth = 5 and ≤200 total fetches
   • Deduplicate visited .txt URLs so cycles and shared sub-indexes don’t cause repeated fetches
   • Current behavior is effectively “depth = 1 hardcoded”

2. Treat the aggregate walk uniformly
   The seed URLs from the canonical llms.txt and discovered sub-links should go through the same classification logic (page URL vs aggregate-to-walk), instead of the current setup where the inner loop uses a different filter than the outer one.

3. Surface the walked tree in details
   When verbose:

   • List which aggregate files were fetched
   • Include depth information
   • Show whether safety caps were hit

Workarounds tried

• None viable from the user side.
  The site’s llms.txt is structured exactly as the spec recommends; the bug is on the consumer (afdocs) side.

• Flattening llms.txt is technically possible, but defeats the purpose of the size-based recommendation to split files.

[dacharyc]

Thanks for the detailed writeup. You're right about the observed symptom here, but the one-level walk limit is an intentional design decision, not an oversight. I think I'll take a different direction for the fix given the design rationale driving the one-level walk.

I've designed afdocs to keep its HTTP footprint predictable. The aggregate walker fetches .txt files it discovers at depth 1, but stops there. For your Alchemy example, that's 6 aggregate fetches on top of the candidate probes, which is manageable. A recursive walk would bump that to ~86 (the ~80 per-chain files at depth 2). That's workable for one site, but the general case gets expensive fast.

Consider Microsoft Learn: 2.5 million+ URLs across 100+ products, each with their own llms.txt subtrees. Even scoped to a single product like Azure (which has sub-products for Compute, Storage, Networking, etc., each potentially with nested indexes), a recursive walk could hit hundreds of aggregate fetches before touching a single page. And the safety cap you'd need to prevent runaway fetches would silently truncate results, producing incomplete coverage numbers with no indication they're partial.

Organizations I've spoken with so far with large multi-product sites have been using afdocs at the per-product level: afdocs check https://alchemy.com/docs/chains/ethereum, rather than walking the entire tree from the docs root. That model keeps runs fast and results meaningful.

That said, you're right that the coverage check (llms-txt-freshness) unfairly penalizes sites that follow the spec's progressive disclosure recommendation. Reporting 6% coverage on a site whose llms.txt tree is actually exhaustive is a bug.

I'll plan a fix for this in an upcoming breaking change release that renames the check to llms-txt-coverage. The approach: when the depth-1 walk encounters .txt links it doesn't descend into (like your /docs/chains/ethereum/llms.txt), it extracts the path prefix and passes those to the coverage check as "omitted subtrees." We can present sitemap URLs under those prefixes as omitted from the check rather than missing. Zero additional HTTP requests, and the output will distinguish directly-verified pages from omitted coverage so it's transparent, and will recommend running afdocs per-section for sites that want deeper coverage.

For sampling checks (llms-txt-directive, content-negotiation, etc.), the 311 pages from your 5 flat sections are probably a reasonable pool. A 50-page random draw from that gives statistically useful signal. If you want sampling specifically from chain pages, afdocs check https://alchemy.com/docs/chains/ethereum is the right approach for now.

We may add an opt-in --walk-depth flag in a future release for users who want exhaustive discovery and accept the resource cost, but the omitted subtrees approach should resolve the coverage scoring issue without it.

[dacharyc]

Ok, I've just released v0.17.0 with support for omitted subtrees - does this solve your use case? The docs are here: https://afdocs.dev/checks/observability.html#omitted-subtrees

[SahilAujla]

It does, thank you, appreciate it 🙏🏻