feat(crawler): schedule periodic re-validation for fan-out child publishers (#4850)

[bokelley]

Summary

Resolves #4850. recordChildPublisherFromManager (PR #4840) created publishers rows for fan-out children but never scheduled them for refresh — a manager-asserted child stamped last_validated = NOW() at fan-out time stayed at that timestamp indefinitely. This wires children into the existing manager_revalidation_queue worker.

Fix

Two-line addition to recordChildPublisherFromManager:

INSERT INTO manager_revalidation_queue
  (publisher_domain, manager_domain, enqueued_at, next_attempt_after, attempts, ...)
VALUES ($1, $2, NOW(), NOW() + INTERVAL '24 hours', 0, ...)
ON CONFLICT (publisher_domain) DO NOTHING

• 24h initial delay — prevents 6,800 cafemedia children from storming the crawler on top of the cafemedia fan-out itself
• ON CONFLICT DO NOTHING — preserves existing backoff (idempotent re-enqueue from subsequent fan-outs doesn't reset)
• Reuses existing queue (manager_revalidation_queue from migration 471) — same shape, same worker, same crawlSingleDomain action

What the worker does per child

crawlSingleDomain(child) runs the full discovery cascade:

1. Try child's own /.well-known/adagents.json (direct). If present, the child gets promoted to discovery_method='direct' with the cached blob. The divergence detector (filed in SDK companion tickets adcp-client-python#749 part 3, etc.) now has real data to compare manager's inline claim against child's own declaration.
2. Fall back to ads.txt MANAGERDOMAIN. Child's row stays as-is, queue row deletes.
3. 404 on both. recordFailedAdagentsFetch updates last_http_status. Queue row deletes — re-enqueue happens on next cafemedia rotation, no infinite loop.

Out of scope

• Long-interval periodic re-check for children that 404'd (the spec text's "30-day re-check"). Today the queue deletes on success; re-enqueue depends on the manager rotating. Worth a follow-up if it bites.
• Backoff for non-404 failures — already in place via existing markRevalidationFailed schedule (1h / 6h / 24h / 72h).

Test plan

• Typecheck passes
• Integration tests added: 24h initial delay, idempotent re-enqueue, self-attribute skip
• Reviewer (post-deploy): probe manager_revalidation_queue after a cafemedia crawl, confirm ~6,800 rows with next_attempt_after clustered ~24h out

🤖 Generated with Claude Code

[aao-release-bot]

LGTM. Follow-ups noted below. Right shape — reuses migration 471's queue rather than spinning a new table, and the 24h initial delay handles the cafemedia thundering-herd at the right layer.

Things I checked

• Column list on the new INSERT (server/src/db/publisher-db.ts:281) matches enqueueManagerRevalidation at :542 exactly — (publisher_domain, manager_domain, enqueued_at, next_attempt_after, attempts, last_attempted_at, last_error).
• Migration 471 declares publisher_domain TEXT PRIMARY KEY, so ON CONFLICT (publisher_domain) DO NOTHING is well-defined against the canonicalized string being passed.
• Idempotency test at crawler-publisher-properties-fanout.test.ts:126 exercises the DO NOTHING path correctly — seeds NOW+3d/attempts=2, re-calls, asserts state preserved.
• Self-attribute early return at publisher-db.ts:248 skips both writes; the queue-side assertion lives in the separate test at :159 (count = 0 for MANAGER == MANAGER).
• Caller wired at server/src/crawler.ts:1078 inside fanOutPublisherPropertiesAuthorizations, with manager-self drop already applied one line earlier — defense-in-depth, both layers reject self-attribute.
• Empty-frontmatter changeset matches repo convention (other .changeset/*.md files do the same). Not a missing-changeset block.
• No tenant/auth surface — publishers and manager_revalidation_queue are global registry tables.

Follow-ups (non-blocking — file as issues)

• Wrap the two INSERTs in BEGIN/COMMIT. publisher-db.ts:251-285 shares a getClient() handle across both INSERTs but never opens a transaction, unlike upsertAdagentsCache at :325. If the queue INSERT fails after the publishers INSERT commits (connection drop mid-write), the child row lands with no refresh schedule. Self-healing on the next fan-out (the queue INSERT retries via DO NOTHING and the publishers ON CONFLICT branch stays correct), so this isn't load-bearing — but matching the file's own transaction convention removes the orphan window entirely.
• Extend clearFixtures to cover manager_revalidation_queue. crawler-publisher-properties-fanout.test.ts:43 cleans publishers and agent_publisher_authorizations but not the queue. The new tests at :123 and :156 rely on inline DELETE for cleanup — if either throws before that line, subsequent test runs against the same DB inherit a stale row that fails the attempts: 0 assertion.
• Post-deploy probe. The test plan's unchecked item — confirming ~6,800 cafemedia children land with next_attempt_after clustered 24h out — is the right validation to actually run once this ships. Worth a calendar reminder, not a merge blocker.

Approving.