feat(crawler): fan publisher_properties[].publisher_domains[] into per-child rows

[bokelley]

Summary

Write-path counterpart to #4838 (directory inverse-lookup read endpoint) and #4827 (inline-resolution spec rule). The directory endpoint currently returns one cafemedia.com row with properties_authorized=0 / properties_total=0 for any agent listed in a publisher_properties selector — the 6,800 represented publishers are invisible. This PR makes the crawler fan the selector's publisher_domains[] into per-child rows.

Files

• server/src/db/migrations/486_publisher_discovery_method_adagents_authoritative.sql — adds 'adagents_authoritative' to the publishers.discovery_method CHECK constraint. Spec value from feat(aao): define directory inverse-lookup endpoint spec (#4823) #4828 for inline-resolution-discovered publishers; the crawler can now emit it.
• server/src/adagents-manager.ts + server/src/db/federated-index-db.ts — DiscoveryMethod TS union widened.
• server/src/db/publisher-db.ts — new recordChildPublisherFromManager method. Upserts the child publishers row with source_type='community', discovery_method='adagents_authoritative', manager_domain=<host>, NO blob. Preserves stronger rows (a child that was independently crawled with discovery_method='direct' is NOT downgraded).
• server/src/crawler.ts — new fanOutPublisherPropertiesAuthorizations helper, called from both crawler loops (per-discovered-publisher + per-agent-claimed-publisher). Walks each authorized_agents[] entry whose authorization_type is publisher_properties, resolves singular publisher_domain + compact publisher_domains[] forms, and writes per-child rows in both tables. Per-child failures are logged but don't abort — partial progress beats silent total failure at managed-network scale.
• server/tests/integration/crawler-publisher-properties-fanout.test.ts — 8 tests covering DB invariants (child row provenance, no self-attribution, direct-wins-over-attribution upsert, canonicalization) and the crawler helper (singular vs compact selector forms, non-publisher_properties skip, manager-in-publisher_domains drop, idempotency).

What it produces on cafemedia.com

Before this PR	After this PR
1 row in agent_publisher_authorizations (interchange.io → cafemedia.com)	1 row for cafemedia.com (host) + 6,800 child rows
0 child rows in publishers	6,800 child publisher rows with discovery_method='adagents_authoritative' + manager_domain='cafemedia.com'
Directory endpoint returns 1 row with 0/0 counts	Directory endpoint returns 6,800 rows with per-publisher counts

Properties were already correctly scoped per-child by the existing recordPropertiesForAgent flow (the property's own publisher_domain field wins at crawler.ts:1002) — so once the child publishers + authz edges exist, properties_total and properties_authorized resolve correctly without further changes.

Trust profile of adagents_authoritative

Medium — the manager file names each represented publisher on each property's publisher_domain field (the same anchor the managerdomain fallback safety rule requires), but no bilateral confirmation from the publisher's own origin. Distinct from direct (origin-fetched), authoritative_location (publisher actively delegated), and ads_txt_managerdomain (delegation discovered via ads.txt fallback).

Idempotency

All writes use ON CONFLICT semantics on the existing unique constraints. Three consecutive crawls of the same manager file produce exactly 6,800 child rows, not 18,400.

Out of scope

• Per-child crawl of the 6,800 represented domains. Inline resolution from the manager file is sufficient per spec; per-child crawl is an optional escalation, separate concern.
• Revocation propagation through fan-out. The host's revoked_publisher_domains[] already short-circuits the directory's status emission via the existing JSONB walk on the host row.

Resolves the deferred follow-up flagged in #4838's PR description.

Test plan

• Typecheck passes (npm run typecheck)
• Reviewer: CI integration tests exercise the new helper (8 cases)
• Reviewer: confirm migration 486 ALTER TABLE / ADD CONSTRAINT is online-safe on a populated publishers table at production scale
• Reviewer (post-merge): re-run crawl against cafemedia.com and hit GET /api/v1/agents/https%3A%2F%2Finterchange.io/publishers — expect 6,800 rows

🤖 Generated with Claude Code

[aao-release-bot]

LGTM. Follow-ups noted below. The shape is right — the directory endpoint's job is discovery, and the schema at static/schemas/source/aao/agent-publishers.json:53-61 already names adagents_authoritative as the trust value that distinguishes "manager file said so" from a real origin fetch.

Things I checked

• Spec conformance. ad-tech-protocol-expert: sound. The new discovery_method value was specced in adcp#4823 / PR #4828 and the published schema enum at static/schemas/source/aao/agent-publishers.json:53-61 already lists it; this PR makes it emittable. The inline-resolution rule (adcp#4825 / PR #4827) is endorsed at docs/governance/property/adagents.mdx:496-522, and the managerdomain fallback safety rule (lines 242-249) gives the property's publisher_domain field the same trust-anchor role this fan-out relies on.
• Cross-publisher attribution split. projectAuthorizationToCatalog at server/src/db/publisher-db.ts:953-967 still refuses cross-publisher claims into the strict catalog. The new fan-out writes only to legacy agent_publisher_authorizations and stamps discovery_method='adagents_authoritative'. Both arms surface via federated-index-db.ts:265-356, but with distinct provenance — catalog-as-authorization, legacy-as-discovery. Right shape.
• Self-attribution. Belt-and-braces: crawler.ts:1042-1045 drops the manager from the child set after canonicalization, and publisher-db.ts:248 checks again at the writer. Both sides canonicalize, so case/whitespace/trailing-dot variants can't escape.
• Idempotency. The dedicated test at server/tests/integration/crawler-publisher-properties-fanout.test.ts:264-284 runs the fan-out 3× and asserts 2 rows, not 6. ON CONFLICT semantics on the existing unique constraints hold.
• Changeset. .changeset/4839-crawler-publisher-properties-fanout.md empty-frontmatter is consistent with the rest of the AAO series (.changeset/4823-aao-directory-inverse-lookup-endpoint.md, .changeset/4836-aao-directory-inverse-lookup-server.md).
• Test-plan items. The three unchecked boxes are explicitly Reviewer tasks, not Author. Author shipped what they signed up for.

Follow-ups (non-blocking — file as issues)

1. Stale fan-out rows are never reconciled. reconcileLegacyAdagentsAgents at federated-index-db.ts:593-625 deletes rows keyed on publisher_domain = <manager>. The fan-out writes rows keyed on publisher_domain = <child>. If a manager drops a child from publisher_domains[] on a later crawl, the (agent_url, child, source='adagents_json') row persists. The endpoint surfaces it indefinitely. Needs a fan-out-aware reconciler: compute the new desired child set per (manager, agent), DELETE FROM agent_publisher_authorizations WHERE agent_url=? AND source='adagents_json' AND publisher_domain NOT IN (desired) AND publisher_domain IN (SELECT domain FROM publishers WHERE manager_domain=? AND discovery_method='adagents_authoritative'). Same logic on the publishers row retirement.

2. First-writer-wins through transient child-origin failures. recordChildPublisherFromManager at publisher-db.ts:255-267 keys "stronger row" off adagents_json IS NULL. recordFailedAdagentsFetch at publisher-db.ts:285-303 writes the row without setting adagents_json (correct — there's no body). If attacker writes first and the child's origin is flapping 5xx for the directory's crawl window, attacker's manager_domain claim is durable across the failures. Mitigation: track last_direct_success_at separately and refuse downgrade when it's set.

3. Migration 486 takes an ACCESS EXCLUSIVE lock for a full-table scan whose outcome is guaranteed. 486_publisher_discovery_method_adagents_authoritative.sql:22-29 is DROP CONSTRAINT + ADD CONSTRAINT. The new value set is a strict superset of the old one — every existing row trivially satisfies the new constraint, so the validating scan is pure dead weight. Precedent in this repo (migration 473) uses ADD CONSTRAINT ... NOT VALID; VALIDATE CONSTRAINT; — second statement takes only SHARE UPDATE EXCLUSIVE. Worth backporting before this hits the next populated-table case.

4. property_tags silently dropped during fan-out. crawler.ts:1051-1056 calls recordAgentFromAdagentsJson with property_ids: undefined and has no equivalent argument for selection_type='by_tag' selectors' property_tags. For cafemedia this is operationally a no-op (the tag is the whole network), but a manager who uses tags to scope (['video-only']) gets over-reported authorization in the federated index. Document the limitation, or extend the index schema.

5. canonicalizePublisherDomain is canonicalization, not validation. It accepts IP literals, ports, and path-bearing strings (nyt.com/admin). The schema pattern at static/schemas/source/adagents.json is the spec; enforce it at the fan-out writer boundary so a malformed manager file can't pollute publishers.domain with rows that won't ever match a real crawl.

6. inline_properties not handled. Same inverse-lookup concern, same shape — inline_properties[*].publisher_domain identifies per-child publishers the same way. Either deliberate scope or an oversight; the PR title is specific to publisher_properties, so this is probably a separate ticket.

Minor nits (non-blocking)

1. Three docblocks stacked above one method. publisher-db.ts:214-241 now has the original (already-orphaned) "Cache an adagents.json manifest…" + "Record a failed adagents.json fetch…" + the new recordChildPublisherFromManager block, all in front of recordChildPublisherFromManager. recordFailedAdagentsFetch is now bare. The orphan situation predated this PR but the PR makes it worse. Move the new method below recordFailedAdagentsFetch, or relocate the two orphans to their actual targets while you're in the file.

2. Same orphan pattern at crawler.ts:984-1012. The new method's docblock is inserted between recordPropertiesForAgent's docblock at L984-986 and its body at L1069. Notable that the comment-relocation game now has scoreline 2-0 in this PR's favor.

3. Test seam reaches into a private method via prototype binding + any cast (crawler-publisher-properties-fanout.test.ts:127-150). Signal that the helper has no actual dependency on CrawlerService state — only on publisherDb and federatedIndex. Extracting it to a free function would let the test import directly. Optional.

4. Per-child catch logs but has no counter (crawler.ts:1057-1065). "Partial progress beats silent total failure" is the right call, but a metric (crawler_fanout_per_child_failures_total{manager_domain}) lets operators alert on rows that get swallowed.

5. source='adagents_json' on synthesized rows is technically accurate (the manager file IS the adagents.json source per the inline-resolution rule) but a consumer reading the column as "origin-attested" will be wrong. Worth a single line in federated-index-db.ts documenting that the value covers inline-resolution-synthesized rows.

Safe to merge.

[bokelley]

Issue #4841 proposes projecting fan-out rows into catalog_agent_authorizations (Option A: extend fanOutPublisherPropertiesAuthorizations to parallel-write with evidence='adagents_json') — same call site as this PR. #3177 cutover closed 2026-05-17, so the catalog table should exist. Consider folding before merge or confirm follow-up.

---

Generated by Claude Code

[aao-release-bot]

Two real correctness regressions on top of the spec-faithful fan-out. Both are fixable in this PR or as a fast-follow — but I want answers before approve, because the public, unauthenticated directory endpoint at server/src/routes/registry-api.ts:6924 is the consumer of every row this PR writes.

Things I checked

• Migration 486 wraps DROP+ADD CHECK in BEGIN/COMMIT. Constraint is expansion-only so no existing row can violate; validation pass holds ACCESS EXCLUSIVE for one sequential scan. Acceptable at current publishers scale; would prefer NOT VALID + async VALIDATE next time this constraint shape changes.
• XOR + by_id guards at server/src/crawler.ts:1029-1054 are faithful to the schema. Walked all eight test cases: singular-only on by_id (crawler.ts:1037) fans one row, by_id + publisher_domains[] skips at L1050, by_tag + publisher_domains[] fans the plural set, both-populated + neither-populated skip at L1033-1034. Tests at crawler-publisher-properties-fanout.test.ts:551-721 exercise every branch. ad-tech-protocol-expert confirmed static/schemas/source/core/publisher-property-selector.json:38-44,47-76 actually enforces the XOR + by_id rule the helper echoes — not over-stated.
• 'adagents_authoritative' is already on the wire. static/schemas/source/aao/agent-publishers.json:58-60 carries the enum on the directory response; docs/aao/directory-api.mdx:92,106 documents Medium trust. This PR widens the TS union and the CHECK constraint to let the crawler actually emit a value the response schema already accepts. Not a unilateral protocol extension.
• Revocation OR EXISTS at server/src/db/federated-index-db.ts:322-340 correctly implements the normative rule at docs/governance/property/adagents.mdx:520 ("Inline resolution MUST honor revoked_publisher_domains[] on the parent file"). LEFT JOIN to mgr is safe when pub.manager_domain IS NULL — COALESCE returns [], EXISTS returns false, OR short-circuits. Query plan at 6,800 children is 6,800 PK lookups against one manager row. Confirmed safe.
• recordChildPublisherFromManager direct-wins upsert at server/src/db/publisher-db.ts:251-263 preserves the stronger row when adagents_json IS NOT NULL. Test at crawler-publisher-properties-fanout.test.ts:469-497 covers it. Correct for the discovery_method / source_type axis.

Open questions — answer before I flip to approve

1. manager_domain last-writer-wins on fan-out children (server/src/db/publisher-db.ts:260-263). For a child whose row already exists with adagents_json IS NULL (i.e., it was previously minted by manager A), the upsert's CASE WHEN publishers.adagents_json IS NULL THEN EXCLUDED.manager_domain will overwrite manager_domain to whoever wrote last. After this PR the revocation join at federated-index-db.ts:340 reads revoked_publisher_domains[] from whichever manager won the race. Net effect: any subsequent manager file claiming the same child seizes revocation authority over that child's edges from the original manager. What's the AAO gate that prevents arbitrary domains from registering manager files? If there isn't one — or if registration is open by design and the trust label is the only signal — this needs WHERE publishers.manager_domain IS NULL (or OR = EXCLUDED.manager_domain) on the manager_domain branch of the CASE before the public directory consumes these rows.

2. discovered_agents.source_domain thrashing (server/src/federated-index.ts:248-252 called from server/src/crawler.ts:1070). upsertAgent keys on agent_url alone (server/src/db/federated-index-db.ts:653) and EXCLUDED.source_domain overwrites on conflict. For cafemedia's 6,800-child fan-out, the same agent_url gets source_domain rewritten 6,800 times in a single crawl pass — final value is the alphabetically-last child, not the manager that actually authorized it. The host-level call at crawler.ts:461/546 already established the agent row with source_domain=cafemedia.com; the fan-out's recordAgentFromAdagentsJson need only call upsertAuthorization, not upsertAgent. Two-line fix: split recordAgentFromAdagentsJson so the agent-side upsert is opt-out, or inline-call upsertAuthorization directly from the fan-out loop.

Inverse of (1) — caught by ad-tech-protocol-expert, file as follow-up

The same CASE preserves manager_domain (as NULL) on a direct crawled child that is later legitimately inlined by a manager. The manager-side revocation join then can't reach the manager's revoked_publisher_domains[] for that child — spec rule at docs/governance/property/adagents.mdx:520 is unenforceable. manager_domain is information-additive, not provenance-competing; the "direct wins" rule should apply to discovery_method / source_type but not to manager_domain. Net: a single-column manager_domain is the wrong shape; a manager_domains TEXT[] (or a separate publisher_manager_claims table) is the right shape long-term.

Follow-ups (non-blocking — file as issues)

• No length cap on publishers.domain — server/src/db/migrations/432_publishers_overlay.sql:27 declares domain TEXT PRIMARY KEY with no CHECK, while migrations/207_registry_requests.sql:6 has CHECK (length(domain) <= 253 AND domain = lower(domain)). A malicious manager file shipping 100KB-per-domain payloads can amplify into a PRIMARY KEY index. Belt-and-suspenders.
• Empty-canonical defense — canonicalizePublisherDomain("....") returns "", which fanOutPublisherPropertiesAuthorizations will happily insert into publishers.domain. Validator already defends in validator.ts:373-377; the writer should too. Add if (childCanonical === '') continue; after crawler.ts:1063.
• Aggregate failure counter on fan-out — crawler.ts:1076-1084 logs one warn per failed child. At 6,800-domain scale a poisoned manager file produces a flood indistinguishable from one Postgres retry. Tally per call, log one structured {manager, agentUrl, attempted, succeeded, failed} with a metric so an alarm can fire on rate spikes.
• Reconcile manager_domain on manager-side drops — when a child drops out of a manager's publisher_properties[].publisher_domains[] across crawls, the synthetic publishers.manager_domain is never NULLed. Stale manager attribution lingers.
• Test seam — crawler-publisher-properties-fanout.test.ts:154-167 reaches into a private method via (CrawlerService as any).prototype with a hand-rolled ctx. Tests SQL behavior cleanly, but locks the test to the helper's private API; a follow-up that exercises the full crawler loop through validateDomain would be more durable.

Minor nits (non-blocking)

1. Changeset out-of-scope claim contradicts what shipped. .changeset/4839-crawler-publisher-properties-fanout.md lists "Revocation propagation through fan-out … separate concern" as out-of-scope while federated-index-db.ts:322-340 implements exactly that. Reconcile the changeset language before release notes auto-generate.
2. Stale doc comment above fanOutPublisherPropertiesAuthorizations at crawler.ts:983-986 — the recordPropertiesForAgent block doc landed above the new method's doc.
3. insertPublisher helper in registry-agent-publishers-detail.test.ts:71-83 still types discovery_method as the 3-value union; the new test at L198-237 bypasses it with direct pool.query. Widen the helper for consistency.
4. Filename .changeset/4839-*.md for PR #4840 — a notable choice of off-by-one.

Holding at comment. Flip to approve once (1) the manager_domain overwrite is gated against last-writer-wins (or you confirm the AAO registration model makes that irrelevant) and (2) the discovered_agents.source_domain thrash is bounded.