[WIP] Add robots.txt and llms.txt to docs site

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[geo-optimizer] Add robots.txt and llms.txt to docs site (github.github.com/gh-aw)</issue_title>
<issue_description>## GEO Improvement: Add robots.txt and llms.txt to the docs site

Source audit: GEO Audit Report — 2026-07-20
Audit date: 2026-07-20
Run: §29761233522

Finding

robots.txt score: 0/18 — file not found at https://github.github.com/gh-aw/robots.txt. All AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) have no explicit permission signal.
llms.txt score: 0/18 — file not found at https://github.github.com/gh-aw/llms.txt. No LLM-friendly structured index exists for the docs site.

Why this matters

These two files are the largest scoring categories (18 pts each, 36 pts combined out of 100). Their absence is the single biggest reason the docs site scores only 44/100 (Foundation band). Fixing both would push the site into the Good band (80+) and ensure every one of the 20+ sitemap pages benefits simultaneously, since they share the same domain.

Suggested fix

1. Add robots.txt to the docs site static assets (e.g. docs/public/robots.txt):

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap.xml

2. Generate and publish llms.txt at https://github.github.com/gh-aw/llms.txt:

geo llms --base-url https://github.github.com/gh-aw --output docs/public/llms.txt

Or author it manually following the [llms.txt spec]((llmstxt.org/redacted) At minimum include: project name, description, and links to key documentation pages.

3. Verify by re-running the GEO audit after deployment.

Generated by 🌍 GEO Optimizer Daily Audit · 31.7 AIC · ⌖ 9.91 AIC · ⊞ 4.9K · ◷

• expires on Jul 27, 2026, 8:55 AM UTC-08:00

• Closes [geo-optimizer] Add robots.txt and llms.txt to docs site (github.github.com/gh-aw) #46891

Comments on the Issue (you are @copilot in this section)

• Fixes [geo-optimizer] Add robots.txt and llms.txt to docs site (github.github.com/gh-aw) #46891