Autogenerate comprehensive docs llms.txt

[Alek99]

What changed

• Generate a sectioned llms.txt from every page discovered by the XY docs configuration.
• Keep llms-full.txt generated from the same source content, including generated component API references.
• Add the canonical agent-discovery directive to every direct and trailing-slash Markdown asset.
• Add the same directive to each prerendered HTML docs page as screen-reader-only content.
• Extend production validators and tests to catch missing, stale, oversized, duplicated, or incomplete agent-facing docs assets.

Why

The existing generator produced a flat index and did not advertise llms.txt from individual HTML or Markdown pages. That left XY short of the agent-discovery behavior used by the main Reflex documentation and made regressions in the production export harder to catch.

Impact

The docs build now publishes a comprehensive nine-section index covering all 65 public pages. The generated index is 12,163 characters, below the 50,000-character threshold, and every indexed URL points to an agent-readable Markdown asset.

Validation

• ruff format --check on all changed files
• ruff check on all changed files
• pytest docs/app/tests -q — 92 passed
• reflex compile --dry
• Production Reflex build
• Generated sitemap validator
• Markdown links against generated sitemap
• Generated Markdown and LLM asset validator
• Prerendered HTML route and preview validator

Summary by CodeRabbit

• New Features
  • Added AI-agent discovery directives to documentation pages and Markdown assets.
  • Added structured, section-based navigation to llms.txt.
  • Added validation for llms.txt and llms-full.txt generated assets.
• Bug Fixes
  • Documentation builds now detect missing or outdated discovery directives and generated asset mismatches.
• Tests
  • Expanded coverage for Markdown directives, index structure, page links, and hidden HTML accessibility markup.

[coderabbitai]

📝 Walkthrough

Walkthrough

Documentation pages now expose agent discovery links in HTML and Markdown, generate structured llms.txt indexes, remove repeated directives from llms-full.txt, and validate these outputs through build checks and tests.

Changes

Agent documentation discovery

Layer / File(s)	Summary
Markdown directives and structured indexes docs/app/xy_docs/plugins.py	Markdown pages receive discovery directives, llms.txt is grouped by navigation section, and llms-full.txt removes repeated directives.
Hidden HTML directive wiring docs/app/xy_docs/xy_docs.py, docs/app/scripts/check_html_routes.py	Documentation layouts render a hidden complete-index link, and prerendered HTML validation requires that directive.
Generated asset validation docs/app/scripts/check_markdown_assets.py, docs/app/tests/test_agent_files.py	Build checks and tests validate Markdown prefixes, generated index contents, page URLs, and HTML accessibility markup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DocumentationLayout
  participant MarkdownGenerator
  participant IndexGenerator
  participant AssetChecks
  DocumentationLayout->>DocumentationLayout: render hidden llms.txt link
  MarkdownGenerator->>MarkdownGenerator: prepend agent discovery directive
  MarkdownGenerator->>IndexGenerator: provide page Markdown links and content
  IndexGenerator->>IndexGenerator: build llms.txt and llms-full.txt
  AssetChecks->>IndexGenerator: generate expected assets
  AssetChecks->>AssetChecks: validate generated HTML and Markdown outputs

Loading

Suggested reviewers: carlosabadia

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: generating a comprehensive docs llms.txt index.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent/autogenerate-comprehensive-llms-txt

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/app/scripts/check_html_routes.py`:
- Line 17: Update the HTML validation around LLMS_DIRECTIVE to require the
directive text inside its sr-only container and verify that its link href points
to the canonical llms.txt URL, rather than accepting matching text anywhere in
the document. Ensure missing or stale targets fail the check.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7011cf5d-82e5-4280-b0a5-76574c08a87f

📥 Commits

Reviewing files that changed from the base of the PR and between 077c53f and 71d0ac8.

📒 Files selected for processing (5)

• docs/app/scripts/check_html_routes.py
• docs/app/scripts/check_markdown_assets.py
• docs/app/tests/test_agent_files.py
• docs/app/xy_docs/plugins.py
• docs/app/xy_docs/xy_docs.py

[coderabbitai]

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Validate the canonical directive link, not just its text.

A matching phrase anywhere in the document passes this check; a stale or missing href to the canonical llms.txt URL would go undetected. Validate the expected link target (and the directive’s sr-only container) in the prerendered HTML.

Also applies to: 113-116

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/app/scripts/check_html_routes.py` at line 17, Update the HTML validation
around LLMS_DIRECTIVE to require the directive text inside its sr-only container
and verify that its link href points to the canonical llms.txt URL, rather than
accepting matching text anywhere in the document. Ensure missing or stale
targets fail the check.