[geo-optimizer] Strengthen docs-site LLM discovery signals (/llms.txt + head link)

[Copilot]

The GEO audit flagged robots.txt and llms.txt as non-discoverable on the docs site, costing 36 points. robots.txt permissions were already present; this change focuses on making llms.txt a clear, crawlable docs index and explicitly advertising it in page metadata.

• Docs indexing payload (/llms.txt)

  • Reworked docs/src/pages/llms.txt.ts from prompt-file listing to a canonical docs index for AI crawlers.
  • Added high-signal links (home, setup, guides, reference, patterns, examples, sitemap) plus AI discovery endpoints.
  • Added prerender = true to ensure static output is emitted reliably in deployment.

• Head-level discovery hint

  • Added an alternate plain-text link in docs/src/components/CustomHead.astro:
    • rel="alternate", type="text/plain", title="LLMs.txt", href="/gh-aw/llms.txt"

• Docs path clarity

  • Updated docs/README.md to explicitly document that llms.txt is served at /gh-aw/llms.txt.

// docs/src/pages/llms.txt.ts
export const prerender = true;

const lines = [
  '# GitHub Agentic Workflows',
  '',
  '## Documentation',
  '- [Home](https://github.github.com/gh-aw/)',
  '- [CLI Commands](https://github.github.com/gh-aw/setup/cli/)',
  '- [Reference](https://github.github.com/gh-aw/reference/)',
  '- [Sitemap](https://github.github.com/gh-aw/sitemap-index.xml)',
  '',
  '## AI Discovery',
  '- [robots.txt](https://github.github.com/gh-aw/robots.txt)',
  '- [ai.txt](https://github.github.com/gh-aw/.well-known/ai.txt)',
];

[github-actions]

Hey @Copilot 👋 — thanks for taking on the robots.txt / llms.txt GEO improvement for the docs site! This is a promising direction. A few things to address before this is ready for review:

• No changes yet — the PR is currently a draft with 0 files changed and 0 lines modified. Once the agent completes the remaining checklist items, the actual file additions should appear in the diff.
• Add a description — the PR body is a task checklist rather than a description of what is being added and why. Once the work is done, replace or supplement the checklist with a short summary: which files are being added/modified, where they land in the built output, and how discoverability is improved.
• Scope note — the CONTRIBUTING.md doesn't explicitly list docs-site improvements as in or out of scope for agent PRs, so maintainer alignment would help confirm this work is welcome before merging.
• Tests — if the docs build pipeline supports output validation (e.g., checking that robots.txt and llms.txt are present at expected paths in the build artifact), a targeted check would strengthen confidence in the change.

If you'd like a hand finishing this off, you can assign this prompt to your coding agent:

Complete the remaining steps for PR #44599 (Add robots.txt and llms.txt to docs site):
1. Add a robots.txt file to the correct static-assets location in the docs site source so it is served at the site root after build.
2. Add a llms.txt file following the llms.txt spec ((llmstxt.org/redacted) with a summary of the project and links to key documentation pages.
3. Verify both files appear in the build output at the expected paths.
4. Update the PR description with a clear summary: what files were added, where they live in the repo, and how they improve discoverability for AI/search crawlers.
5. Run make agent-report-progress and the secret scan as noted in the checklist.

Generated by ✅ Contribution Check · 184.7 AIC · ⌖ 12.4 AIC · ⊞ 6.2K · ◷