Skip to content

fix(crawler): also fan publisher_properties[] from crawlSingleDomain + catalog variant#4851

Merged
bokelley merged 1 commit into
mainfrom
bokelley/fanout-singledomain-hotfix
May 20, 2026
Merged

fix(crawler): also fan publisher_properties[] from crawlSingleDomain + catalog variant#4851
bokelley merged 1 commit into
mainfrom
bokelley/fanout-singledomain-hotfix

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

Hotfix to #4840. The fan-out was wired into the two periodic-crawl loops but missed the on-demand crawlSingleDomain path (called by POST /api/registry/crawl-request) and crawlSingleDomainForCatalog (called by the manager revalidation queue worker).

Symptom

Admin-triggered crawl of cafemedia.com on agenticadvertising.org refreshed the manifest cache (last_verified_at updated to 14:53 UTC) but didn't synthesize the 6,800 child rows. Production endpoint kept returning:

{
  "publishers": [{
    "publisher_domain": "cafemedia.com",
    "discovery_method": "direct",
    "properties_authorized": 0,
    "properties_total": 0,
    "last_verified_at": "2026-05-20T14:53:31.264Z"
  }]
}

Fix

Two lines: await this.fanOutPublisherPropertiesAuthorizations(authorizedAgent, domain) inside the crawlSingleDomain per-agent loop (crawler.ts:~1346) and the crawlSingleDomainForCatalog loop (~1620).

Verification after deploy

curl -X POST 'https://agenticadvertising.org/api/registry/crawl-request' \
  -H "Authorization: Bearer \$ADMIN_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"domain":"cafemedia.com"}'

# wait ~30s for the 2.6MB fetch + 6,800 fan-out writes

curl 'https://agenticadvertising.org/api/v1/agents/https%3A%2F%2Finterchange.io/publishers?limit=1000' | jq '.publishers | length'
# expect 1000 (page 1 of 7)

Test plan

  • Typecheck passes
  • Existing fan-out integration tests still cover the helper behavior; the new call sites use the same helper, no new test logic required
  • Reviewer (post-merge): re-run the cafemedia probe and report row counts

🤖 Generated with Claude Code

…+ catalog variant

#4840 wired the fan-out helper into the two periodic-crawl loops but
missed the on-demand crawlSingleDomain path (called by
POST /api/registry/crawl-request) and crawlSingleDomainForCatalog
(called by the manager revalidation queue worker). Both iterate
authorized_agents[] independently and need the same fan-out call.

Symptom: admin-triggered crawl of cafemedia.com refreshed the manifest
cache (last_verified_at updated) but didn't synthesize the 6,800 child
rows — the directory inverse-lookup kept returning a single
cafemedia.com row with properties_total=0.

Two lines added: fanOutPublisherPropertiesAuthorizations call at the
end of each per-agent loop, mirroring the periodic paths.

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean hotfix. The helper's idempotency plus early-return guards make these two new call sites safe by construction — the work is in proving the wiring matches, not in defending the helper.

Things I checked

  • Placement matches the periodic-crawl pattern. L1347 inside the crawlSingleDomain per-agent loop and L1624 inside crawlSingleDomainForCatalog mirror L482 and L562 exactly — same loop scope, same position after recordPropertiesForAgent, same args.
  • Helper safety holds at the new call sites. fanOutPublisherPropertiesAuthorizations (server/src/crawler.ts:1012) returns early on authorization_type !== 'publisher_properties' (L1016), skips ambiguous singular+plural selectors (L1033) and empty ones (L1034), refuses by_id + publisher_domains[] (L1050), and per-child writes are wrapped in try/catch (L1076-L1084). Idempotent per the doc at L1008.
  • No race with periodic crawl. Both paths converge on the same upserts in recordChildPublisherFromManager + recordAgentFromAdagentsJson. Manager-self drop at L1062-L1064 still applies.
  • Changeset present. Empty ---/--- frontmatter is used elsewhere in this repo for no-publish entries (e.g. web-blue-ramp-dedup-and-1000-stops.md). Categorization quibble below.

Follow-ups (non-blocking — file as issues)

  • No regression test on the path that broke. The existing fan-out tests in server/tests/integration/crawler-publisher-properties-fanout.test.ts call the helper directly via prototype reflection (L168, L238). The shipped bug was that crawlSingleDomain / crawlSingleDomainForCatalog didn't call it. The hotfix ships without a test that drives crawlSingleDomain('manager.example') end-to-end with a stubbed validateDomain returning a publisher_properties manifest and asserts the child rows were synthesized. Same refactor-blind-spot can silently unwire this again. Add the test in a follow-up.
  • Changeset frontmatter should bump @adcp/server patch. This changes the observable output of POST /api/registry/crawl-request (the 6,800-row symptom in the body). Empty frontmatter fits no-publish cosmetic changes; a server behavior fix has a more accurate release-note shape with "@adcp/server": patch. Not a block — this isn't a wire-schema change covered by the MUST-FIX rule, just release-note hygiene.
  • Commit body names the wrong method as the worker entry point. Says crawlSingleDomainForCatalog is called by "the manager revalidation queue worker"; the worker actually calls crawlSingleDomain (crawler.ts:1549). Both methods needed the fix so placement is still correct, but notable for future archaeologists.

Minor nits (non-blocking)

  1. Call-site style drift. L482 wraps the helper call across multiple lines with a five-line explanatory comment above; L1347 and L1624 are single-line, no comment. Pulling the L477-L481 comment up to a doc on the helper itself would let future call sites inherit the rationale without copy-paste.

Safe to merge. Re-run the cafemedia probe post-deploy as planned.

@bokelley
bokelley merged commit 450436a into main May 20, 2026
14 checks passed
@bokelley
bokelley deleted the bokelley/fanout-singledomain-hotfix branch May 20, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant