[WIP] Add robots.txt to docs site allowing AI crawlers

[Copilot]

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

---

This section details on the original issue you should resolve

<issue_title>[geo-optimizer] Add robots.txt to docs site allowing AI crawlers</issue_title>
<issue_description>### GEO Improvement: Add robots.txt to docs site allowing AI crawlers

Source audit: GEO Audit Report — 2026-08-08 (created in this run)
Audit date: 2026-08-08
Run: §31267168943

Finding

"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)" — the docs site robots_txt check scored 0/18 (found: false, no bots_allowed) across all 20 sampled pages (and applies to all 215 discovered docs URLs), while the GitHub README already scores 15/18 with 27 AI bots explicitly allowed (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, etc.).

Why this matters

Without a robots.txt, AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) have no explicit permission signal to index github.github.com/gh-aw/, which can suppress citation coverage for the entire documentation site — the single largest fixable point gap identified (0/18, site-wide, vs. the README's 15/18 for the same check).

Suggested fix

• Add a static/robots.txt (or equivalent) to the Docusaurus docs site source, mirroring the allow-list already used in the repo's .well-known/robots setup for the README/GitHub Pages, including at minimum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot.
• Ensure the file is served at https://github.github.com/gh-aw/robots.txt (root of the docs site, not nested under a subpath truncation).
• Optionally pair with a sitemap reference (Sitemap: https://github.github.com/gh-aw/sitemap.xml) to aid discovery.
• Re-run the GEO audit after deployment to confirm the robots_txt check moves from 0/18 to a passing score.

Generated by 🌍 GEO Optimizer Daily Audit · auto · 20.4 AIC · ⌖ 3.35 AIC · ⊞ 7.2K · ◷

• expires on Aug 15, 2026, 8:37 AM UTC-08:00

• Closes [geo-optimizer] Add robots.txt to docs site allowing AI crawlers #51418

Comments on the Issue (you are @copilot in this section)

• Fixes [geo-optimizer] Add robots.txt to docs site allowing AI crawlers #51418