[geo-optimizer] Add robots.txt with explicit AI bot Allow rules to the docs site

[github-actions]

GEO Improvement: Add robots.txt with AI bot Allow rules

Source audit: GEO Audit Report — 2026-07-24
Audit date: 2026-07-24
Run: §30110998991

Finding

"Create robots.txt with Allow rules for AI bots (GPTBot, ClaudeBot, PerplexityBot)"

The docs site at https://github.github.com/gh-aw/ currently has no robots.txt. Without explicit Allow rules, AI crawlers may not index the site, preventing it from being cited by AI engines such as ChatGPT, Claude, and Perplexity.

Why this matters

This is the highest-structural-signal fix available: AI crawlers check robots.txt before crawling. Without explicit Allow entries for GPTBot, ClaudeBot, and PerplexityBot, the entire docs site may be skipped for AI-engine citation — the docs site currently scores 44/100 (Foundation band), and fixing crawler access is a prerequisite for all other GEO improvements to take effect.

Suggested fix

Add a robots.txt file to the root of the docs site (static/robots.txt or equivalent) with at minimum:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap.xml

Verify by running:

curl https://github.github.com/gh-aw/robots.txt

Generated by 🌍 GEO Optimizer Daily Audit · sonnet46 · 23.2 AIC · ⌖ 19.6 AIC · ⊞ 5K · ◷

• expires on Jul 31, 2026, 9:02 AM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-07-31T17:02:40.587Z.

Closed by Workflow