Fix broken legit_bots entries, add 16 new bots, and auto-sync the IP-based ones - #285
Open
blotus wants to merge 4 commits into
Open
Fix broken legit_bots entries, add 16 new bots, and auto-sync the IP-based ones#285blotus wants to merge 4 commits into
blotus wants to merge 4 commits into
Conversation
perplexitybot.json and openai-chatgpt-user.json had their contents swapped in b1e0e1e: - perplexitybot.json held OpenAI's ChatGPT-User list (254 Azure /28s): 91% overlap with OpenAI's current list, 0% overlap with Perplexity's. Perplexity crawls from AWS, so the Azure space was never theirs. - openai-chatgpt-user.json held PerplexityBot's 8 AWS prefixes, an exact match for perplexity.ai/perplexitybot.json. Both are repointed at their real upstream source. ChatGPT-User goes 8 -> 286 ranges; PerplexityBot goes 254 -> 8, which is the full list Perplexity publishes today. duckduckbot.json was also stale: it held 12 EC2 /32s from before DuckDuckGo moved to Azure. Refreshed to the current 481 prefixes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sourced from sourcecidr.com/catalog.json, with every endpoint and rDNS domain verified against the vendor directly. rDNS-based (forward-confirmed, self-maintaining): commoncrawl crawl.commoncrawl.org youbot search.you.com naver (Yeti) web.naver.com seznam seznam.cz semrush bot.semrush.com (Semrush publishes no IP list) linkedin fwd.linkedin.com internet-archive archive.org IP-based (no usable FCrDNS: generic cloud PTRs, no PTR, or PTRs that do not forward-confirm): anthropic 20 ranges duckassistbot 481 ranges openai-adsbot 2 ranges perplexity-user 4 ranges mistralai-index 2 ranges mistralai-user 4 ranges ahrefs 51 ranges lumar 66 ranges flipboard 136 ranges anthropic.json matches only claude-searchbot|claude-user. Anthropic publishes one shared IP list for all three of its crawlers, so ClaudeBot (training) is excluded on the user agent alone. DuckAssistBot and MistralAI-Index/User are kept in files separate from duckduckbot and each other so search crawling can be whitelisted without also allowing AI assistant fetches, even where the vendor serves one combined IP list. Skipped: PetalBot (documented aspiegel.com rDNS does not match the real crawler IPs, which reverse to generic Huawei Cloud names), SE Ranking (rDNS domain sr-srv.net is WHOIS-privacy registered with no provable link to the company), Mojeek (single /28, no usable rDNS), MJ12bot (distributed across volunteer machines). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three bugs fixed in 5e8fc1b (two swapped files, one stale list) were all the same failure mode: hand-copied IP snapshots drift silently. This polls the vendors instead. .github/legit-bots-sources.json lists the 17 bots that publish a stable machine-readable list. Each entry carries the user_agent regex and file comment to preserve, plus one or more URLs; JSON sources are reduced with a jq_filter, others are read as plain lines. Bots verified by reverse DNS are absent by design, since they have nothing to poll and never go stale. Unlike sync-external-sources.yml, opening a PR closes any still-open PR from an earlier run of this workflow. Each run regenerates from master, so a pending PR is always a strict subset of the next one; leaving both open just invites merging stale state. The supersede list is collected before the new PR is created and acted on only after it exists, so a failure mid-step leaves the old PR open rather than dropping updates. Safety properties, all covered by a local dry run: - a source returning nothing never overwrites a populated whitelist - one dead endpoint does not block the other 16 from updating - the run only fails outright if every source fails - ranges are sorted, so a vendor reshuffling its list produces no diff - a sharp shrink is applied but flagged in the PR body for review, since a legitimate one does happen (perplexitybot went 254 -> 8) Lumar is carried as disabled: it publishes only under a date-stamped wp-content URL that rotates, so it cannot be polled on a fixed URL. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
First run of .github/scripts/sync-legit-bots.py, plus the same range sort applied by hand to the two IP-based files it does not manage (lumar, twitterbot). Ranges are now sorted numerically, IPv4 before IPv6, so future syncs diff only on real upstream changes rather than on vendors reordering their lists. Comments drop the "fetched <date>; refresh periodically" notes, which the schedule now makes untrue. One real change came out of it: uptimerobot picks up 6 IPv6 addresses (200 -> 206) that the hand-maintained snapshot had missed. Every other managed file is byte-identical apart from ordering and the comment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
buixor
approved these changes
Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expands
whitelists/benign_bots/legit_bots/from 24 to 40 entries, fixes three broken ones, and puts the IP-based entries on a daily sync so they stop drifting.Candidates came from sourcecidr.com/catalog.json. Every endpoint and every rDNS domain below was verified against the vendor directly.
1. Fixes to existing entries (
5e8fc1b)perplexitybot.jsonandopenai-chatgpt-user.jsonhad their contents swapped in b1e0e1e:perplexitybot.jsonopenai-chatgpt-user.jsonperplexity.ai/perplexitybot.jsonChatGPT-User goes 8 → 286 ranges; PerplexityBot goes 254 → 8, which is the whole list Perplexity publishes today. The shrink looks wrong but isn't — those 254 were never theirs.
duckduckbot.jsonwas separately stale: 12 EC2 /32s from before DuckDuckGo moved to Azure. Now 481 prefixes.2. Sixteen new bots (
34abf84)Verified by forward-confirmed rDNS — self-maintaining, nothing to poll:
commoncrawl·youbot·naver(Yeti) ·seznam·semrush·linkedin·internet-archiveVerified by published IP list — no usable FCrDNS (generic cloud PTRs, no PTR, or PTRs that don't forward-confirm):
anthropic(20) ·duckassistbot(481) ·openai-adsbot(2) ·perplexity-user(4) ·mistralai-index(2) ·mistralai-user(4) ·ahrefs(51) ·lumar(66) ·flipboard(136)Two deliberate choices:
anthropicmatches onlyclaude-searchbot|claude-user. Anthropic publishes one shared IP list for all three of its crawlers, so ClaudeBot (training) is excluded on user agent alone.Skipped, with reasons: PetalBot (documented
aspiegel.comrDNS doesn't match the real crawler IPs, which reverse to generic Huawei Cloud names) · SE Ranking (rDNS domainsr-srv.netis WHOIS-privacy registered with no provable link to the company) · Mojeek (single /28, no usable rDNS) · MJ12bot (distributed across volunteer machines).3. Daily auto-sync (
6bd2d97,e92f1c6)All three bugs above were the same failure mode: hand-copied snapshots drift silently.
.github/legit-bots-sources.jsonnow drives a daily refresh of the 17 bots that publish a stable machine-readable list. rDNS-verified bots are absent by design.Behavior difference vs
sync-external-sources.yml: opening a PR closes any still-open PR from an earlier run of this workflow. Each run regenerates frommaster, so a pending PR is always a strict subset of the next one — leaving both open only invites merging stale state. The supersede list is collected before the new PR is created and acted on only after it exists, so a failure mid-step leaves the old PR open rather than dropping updates.Safety properties, each exercised in a local dry run:
perplexitybotabove shows, so this can't hard-failLumar is carried as
disabled: it publishes only under a date-stampedwp-contentURL that rotates, so it can't be polled on a fixed URL. The config records that and what to do instead.Reviewing
e92f1c6It touches 19 files but is almost entirely range sorting and comment rewording. The one substantive change:
uptimerobotgains 6 IPv6 addresses (200 → 206) that the hand-maintained snapshot had dropped — found by the sync on its first run. Every other managed file is byte-identical apart from ordering.Validation
All 40 entries check out: format, JSON parse, regex compilation, CIDR validity, no duplicate or non-public ranges, sorted order. Every user agent maps to exactly one entry, with no cross-matching and no match against a real browser UA — including a specific check that
ClaudeBotdoes not matchanthropic.Cross-file range overlap flags exactly two pairs, both expected:
duckassistbot/duckduckbot(DuckDuckGo serves one combined list — the deliberate cost of keeping them separately whitelistable) andgptbot/openai-searchbot(pre-existing, shared OpenAI infrastructure).🤖 Generated with Claude Code