docs: add Applebot-Extended to robots.txt AI crawler allow rules - #49599
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
Thanks for this robots.txt update! 🤖 The addition of This looks ready for review once the draft status is removed. The static config change is appropriate for this type of infrastructure improvement — no tests are needed here. Great work on improving docs site discoverability for AI crawlers!
|
PR Triage
Draft; single-line robots.txt addition, trivial and safe.
|
There was a problem hiding this comment.
Pull request overview
Adds explicit permission for Applebot-Extended to crawl the documentation site.
Changes:
- Added an
Applebot-Extendedrule with site-wide access.
Show a summary per file
| File | Description |
|---|---|
docs/public/robots.txt |
Allows Applebot-Extended to crawl all documentation pages. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Balanced
|
🎉 This pull request is included in a new release. Release: |
The GEO audit identified that the docs site lacked explicit AI crawler permissions in
robots.txt, reducing discoverability and citation potential for AI engines. The file already existed with most required bots;Applebot-Extendedwas the only missing entry from the audit's recommended set.Changes
docs/public/robots.txt: AddedApplebot-ExtendedAllow: /rule alongside the existing entries forGPTBot,ClaudeBot,PerplexityBot,Google-Extended, and others.