fix(crawler): also fan publisher_properties[] from crawlSingleDomain + catalog variant#4851
Merged
Merged
Conversation
…+ catalog variant #4840 wired the fan-out helper into the two periodic-crawl loops but missed the on-demand crawlSingleDomain path (called by POST /api/registry/crawl-request) and crawlSingleDomainForCatalog (called by the manager revalidation queue worker). Both iterate authorized_agents[] independently and need the same fan-out call. Symptom: admin-triggered crawl of cafemedia.com refreshed the manifest cache (last_verified_at updated) but didn't synthesize the 6,800 child rows — the directory inverse-lookup kept returning a single cafemedia.com row with properties_total=0. Two lines added: fanOutPublisherPropertiesAuthorizations call at the end of each per-agent loop, mirroring the periodic paths.
Contributor
There was a problem hiding this comment.
Clean hotfix. The helper's idempotency plus early-return guards make these two new call sites safe by construction — the work is in proving the wiring matches, not in defending the helper.
Things I checked
- Placement matches the periodic-crawl pattern. L1347 inside the
crawlSingleDomainper-agent loop and L1624 insidecrawlSingleDomainForCatalogmirror L482 and L562 exactly — same loop scope, same position afterrecordPropertiesForAgent, same args. - Helper safety holds at the new call sites.
fanOutPublisherPropertiesAuthorizations(server/src/crawler.ts:1012) returns early onauthorization_type !== 'publisher_properties'(L1016), skips ambiguous singular+plural selectors (L1033) and empty ones (L1034), refusesby_id+publisher_domains[](L1050), and per-child writes are wrapped in try/catch (L1076-L1084). Idempotent per the doc at L1008. - No race with periodic crawl. Both paths converge on the same upserts in
recordChildPublisherFromManager+recordAgentFromAdagentsJson. Manager-self drop at L1062-L1064 still applies. - Changeset present. Empty
---/---frontmatter is used elsewhere in this repo for no-publish entries (e.g.web-blue-ramp-dedup-and-1000-stops.md). Categorization quibble below.
Follow-ups (non-blocking — file as issues)
- No regression test on the path that broke. The existing fan-out tests in
server/tests/integration/crawler-publisher-properties-fanout.test.tscall the helper directly via prototype reflection (L168, L238). The shipped bug was thatcrawlSingleDomain/crawlSingleDomainForCatalogdidn't call it. The hotfix ships without a test that drivescrawlSingleDomain('manager.example')end-to-end with a stubbedvalidateDomainreturning apublisher_propertiesmanifest and asserts the child rows were synthesized. Same refactor-blind-spot can silently unwire this again. Add the test in a follow-up. - Changeset frontmatter should bump
@adcp/serverpatch. This changes the observable output ofPOST /api/registry/crawl-request(the 6,800-row symptom in the body). Empty frontmatter fits no-publish cosmetic changes; a server behavior fix has a more accurate release-note shape with"@adcp/server": patch. Not a block — this isn't a wire-schema change covered by the MUST-FIX rule, just release-note hygiene. - Commit body names the wrong method as the worker entry point. Says
crawlSingleDomainForCatalogis called by "the manager revalidation queue worker"; the worker actually callscrawlSingleDomain(crawler.ts:1549). Both methods needed the fix so placement is still correct, but notable for future archaeologists.
Minor nits (non-blocking)
- Call-site style drift. L482 wraps the helper call across multiple lines with a five-line explanatory comment above; L1347 and L1624 are single-line, no comment. Pulling the L477-L481 comment up to a doc on the helper itself would let future call sites inherit the rationale without copy-paste.
Safe to merge. Re-run the cafemedia probe post-deploy as planned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Hotfix to #4840. The fan-out was wired into the two periodic-crawl loops but missed the on-demand
crawlSingleDomainpath (called byPOST /api/registry/crawl-request) andcrawlSingleDomainForCatalog(called by the manager revalidation queue worker).Symptom
Admin-triggered crawl of cafemedia.com on agenticadvertising.org refreshed the manifest cache (
last_verified_atupdated to 14:53 UTC) but didn't synthesize the 6,800 child rows. Production endpoint kept returning:{ "publishers": [{ "publisher_domain": "cafemedia.com", "discovery_method": "direct", "properties_authorized": 0, "properties_total": 0, "last_verified_at": "2026-05-20T14:53:31.264Z" }] }Fix
Two lines:
await this.fanOutPublisherPropertiesAuthorizations(authorizedAgent, domain)inside thecrawlSingleDomainper-agent loop (crawler.ts:~1346) and thecrawlSingleDomainForCatalogloop (~1620).Verification after deploy
Test plan
🤖 Generated with Claude Code