fix(crawler): skip fan-out when crawl source delegates via ads.txt MANAGERDOMAIN

[bokelley]

Summary

Empirical bug found in production after #4851 deployed. The fan-out is wired at all four call sites, but the source publisher being crawled isn't always the manifest's host. When a crawl hits a publisher that delegates via ads.txt MANAGERDOMAIN=, the file comes from the manager — but the crawl source is the delegating child. Firing fan-out from those calls used the child as the manager_domain for all 6,800 siblings.

Observed in production (the smoking gun)

After triggering a cafemedia.com crawl-request:

$ curl '.../api/registry/publisher?domain=07.gg' | jq '{discovery_method, manager_domain}'
{ "discovery_method": "adagents_authoritative", "manager_domain": "2foodtrippers.com" }

07.gg's manager should be cafemedia.com. It's 2foodtrippers.com because a 2foodtrippers.com crawl (via managerdomain=cafemedia.com in 2foodtrippers' ads.txt) fired fan-out after the cafemedia crawl and overwrote sibling attribution. The publishers UPSERT preserves direct rows but happily overwrites adagents_authoritative rows.

Prospective fix (crawler.ts)

Gate all four fan-out call sites on validation.discovery_method !== 'ads_txt_managerdomain'. When the crawl source is a delegating child, skip fan-out; the manager's own crawl handles it with correct attribution.

The four sites (per #4851):

• crawler.ts:482 — main loop, registered publishers
• crawler.ts:570 — main loop, agent-claimed publishers
• crawler.ts:1356 — crawlSingleDomain (admin crawl-request)
• crawler.ts:1642 — crawlSingleDomainForCatalog (manager revalidation worker)

Retrospective fix (migration 487)

Recursive CTE walks child.manager_domain → grandparent → root and fixes every adagents_authoritative row whose manager_domain points to another adagents_authoritative row, pointing it at the first non-adagents_authoritative ancestor (the actual root manager — cafemedia.com for the Raptive set, discovery_method='direct'). Bounded recursion depth 10 to defend against cycles.

Out of scope

• More invasive crawler refactor to pass the manifest host (vs source publisher) through the helper chain. The gate above is the minimum surgical fix.

Test plan

• Typecheck passes
• Reviewer: confirm migration 487 backfill query plan on production publishers table size
• After deploy: re-trigger cafemedia.com crawl; probe a sample child (e.g., 07.gg) and confirm manager_domain = 'cafemedia.com' again
• After deploy: probe the directory endpoint and confirm discovery_methods: ["adagents_authoritative"] for the 6,800 Raptive children, all with manager_domain: "cafemedia.com"

🤖 Generated with Claude Code

[aao-release-bot]

LGTM. Real bug, minimum surgical fix, plus a backfill that resolves the existing damage. The gate predicate matches the failure mode: when the manifest came from the manager (via ads.txt MANAGERDOMAIN), the source publisher being processed is the wrong attribution anchor for fan-out.

Things I checked

• Gate predicate correct at all four sites (server/src/crawler.ts:488, 572, 1361, 1642). validation is the AdAgentsValidationResult from validateDomain() in every scope; the literal 'ads_txt_managerdomain' matches the enum at server/src/adagents-manager.ts:18.
• Correct to NOT also gate on 'authoritative_location'. That's the publisher's own stub redirecting to a canonical URL — the source publisher is still the authoritative entity for fan-out attribution.
• Coverage complete: fanOutPublisherPropertiesAuthorizations has exactly four call sites in crawler.ts, and recordChildPublisherFromManager is only called from fanOutPublisherPropertiesAuthorizations. No other write path can stamp manager_domain on an adagents_authoritative row.
• Migration 487 seed (p1 and p2 both adagents_authoritative) is the right base case — precisely the bug shape.
• Recursive step walks one hop past the chain (the row with non-adagents_authoritative provenance), and resolved picks it via WHERE hop_discovery_method != 'adagents_authoritative' + DISTINCT ON (child_domain) ORDER BY child_domain, depth DESC. Correct.
• Cycle defense (depth < 10) is sufficient: a pure-adagents_authoritative cycle never satisfies the resolved predicate, so the child is skipped rather than mis-updated. Fail-closed beats fail-open.
• Idempotency: re-runs produce no new seeds (fixed rows' parent is now direct, failing the seed predicate) and the manager_domain != root_manager_domain guard short-circuits the rest.

Follow-ups (non-blocking — file as issues)

• No unit test for the gate. Table-driven test asserting fanOutPublisherPropertiesAuthorizations is not called when validation.discovery_method === 'ads_txt_managerdomain', exercised at each of the four call sites, would lock the regression. Test plan has it post-deploy, which is fine for the data fix but doesn't lock the code path.
• Manager-not-crawled edge case. If a manager chain dead-ends at a row with discovery_method='adagents_authoritative' (no non-adagents_authoritative ancestor reachable in ≤10 hops), the migration leaves descendants unfixed and the live code never fans them out — because fan-out only fires from the manager's own crawl. Cafemedia is direct so the Raptive set is covered. Worth either enqueueing manager revalidation for orphan chain heads or a metric that surfaces them post-migration.

Minor nits (non-blocking)

1. Nested BEGIN;/COMMIT; in the migration. migrate.ts already wraps the body in a transaction. The inner BEGIN is a no-op warning, but the inner COMMIT closes the outer transaction — the runner's INSERT INTO schema_migrations then runs outside any transaction. Net effect: UPDATE and the migrations row are not atomic. Pre-existing pattern (migration 486 same shape), and the manager_domain != root_manager_domain idempotency guard makes re-runs safe — so this is a Minor here, but the repo-wide pattern is worth a cleanup pass at some point.

Safe to merge.