feat(registry): crawler caches adagents.json and projects catalog (PR 2 of #3177)#3218
Merged
Conversation
… 2 of #3177) Wires the adagents.json crawler to write the publishers overlay (migration 432 from #3195) in lockstep with the existing discovered_properties / agent_property_authorizations writes, and projects parsed properties into catalog_properties + catalog_identifiers in the same transaction. Closes the gap surfaced by Setupad escalation #218: gatavo.com lives in discovered_properties but never reached the catalog because migration 336 was a one-time seed. Every successful crawl now lands in both places. Implementation mirrors brand registry's writer pattern: - new server/src/db/publisher-db.ts with upsertAdagentsCache, modeled on brand-db.ts upsertDiscoveredBrand - crawler.ts cacheAdagentsManifest helper called once per validated domain in both crawl paths (registered publishers, buying-agent claims), modeled on the upsertBrandProperties projection at crawler.ts ~200-237 - catalog rows tagged evidence='adagents_json', confidence='authoritative' to match the migration 336 seed so newly crawled properties are indistinguishable from seeded ones - normalizeIdentifier applied before catalog_identifiers insert so www-prefixed and mixed-case publisher inputs collapse to the canonical property_rid Old tables continue to be written via federated-index-db. Reader paths are untouched; PR 4 will swap reads to the new tables, and PR 5 will drop the legacy ones. Tests cover: publishers cache shape and ON CONFLICT preservation of org metadata; catalog_properties + catalog_identifiers materialization with the right evidence/confidence; identifier normalization; property_rid stability across re-crawls; and the dual-write fallback to discovered_properties + agent_property_authorizations. Refs #3177. Builds on #3195. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Code- and security-review on PR #3218 surfaced four real issues: 1. cacheAdagentsManifest was only called from populateFederatedIndex. crawlSingleDomain (on-demand HTTP API) and crawlSingleDomainForCatalog (catalog crawl-queue path) successfully validated adagents.json but skipped the new cache, so a domain crawled via either path landed only in legacy tables — the exact failure mode the changeset claims to close. Add the cache call to both paths. 2. Catalog identity hijack via shared identifiers. A malicious manifest could list a victim's identifier (domain:cnn.com) alongside its own and the writer's existence-check would reuse the victim's property_rid, silently binding the attacker's own identifier into the victim's property. The override layer (PR #3195) is agent-keyed, not property-rid-keyed, so it doesn't remediate this. The land-grab variant was equally bad: the attacker claims first, victim's later crawl inherits the attacker's adagents_url via COALESCE. Fix: existence query now joins catalog_properties.created_by. If any matching rid was authored by another publisher's adagents.json, refuse the projection rather than rebind the identifier or extend the conflicting property. If the identifier set spans multiple distinct own-side rids, also refuse — silent merging requires moderation (catalog_disputes). 3. rss_url silent rollback. normalizeRssUrl preserves URL path case; chk_identifier_lowercase requires the entire identifier_value to be lowercase. A publisher with a mixed-case path triggered a 23514 check_violation mid-transaction and the whole crawl rolled back into a one-line warn log. Two-part fix: lowercase identifier values defensively after normalizeIdentifier (matches migration 336's lower(value) seed behavior); and wrap each property projection in a SAVEPOINT so a constraint violation skips the offending property without losing the rest of the manifest or the cache write. 4. Smaller items: ORDER BY on the existence query for determinism, drop the unused PublisherDatabase singleton export, strip the historical "PR 2 of #3177" framing from the writer's docstring (CLAUDE.md flags this as no-historical-naming). Tests added for: cross-publisher claim refusal (the cnn.com hijack), multi-rid conflation refusal, isolated property failure not aborting the rest of the manifest, and rss_url lowercase. Cleanup widened to FK-safe property_rid join so per-test fixtures can vary identifier values. All 11 cases pass; existing 16 schema tests still green. Refs #3177, #3218. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second-pass security review found two more vectors the first round of
fixes didn't fully close:
1. Land-grab via first-claim. The cross-publisher refusal only blocked
the rebind direction (victim crawled first, attacker rebinds later).
Reverse ordering — attacker crawls first, claims the victim's domain
alongside its own — was unblocked: attacker mints a rid pointing
domain:victim.example at attacker.example/.well-known/adagents.json,
victim's later crawl is then refused, and catalog readers permanently
misattribute the victim's domain to the attacker.
2. Seed-rid takeover via adagents_url COALESCE. When a matched rid's
created_by was anything other than adagents_json:* (system seeds,
community/member_resolve, brand_json, emails), the conflicting filter
skipped it and the reuse branch ran UPDATE catalog_properties SET
adagents_url = COALESCE(NULL, $attacker_url). Attacker captured the
authority pointer on a victim's seed rid via a single bundle ID.
The fix: add a publisher-anchor rule to projectPropertyToCatalog.
- Cross-publisher domain claims are refused at the front of the function.
A `domain` or `subdomain` identifier in the property must equal the
publisher's domain or be a subdomain of it. This kills the land-grab.
- Foreign-rid reuse now requires an anchor identifier in the manifest.
A non-adagents seed/community rid can only be adopted when the
publisher's manifest carries at least one domain/subdomain anchor
proving authority. Without an anchor, the projection is refused
rather than COALESCE-rebinding adagents_url.
Legitimate flows still work:
- App publishers with no domain identifier still mint new rids for
bundle-only properties. ON CONFLICT DO NOTHING drops bundle IDs
already claimed (first-claimer wins on bundles is a known limitation
that needs App Store corroboration; that's beyond this PR).
- Re-crawls (created_by === adagents_json:{publisher}) skip the anchor
requirement — re-crawl semantics unchanged.
- Anchored manifests can adopt seed rids (modeling the migration 336
backfill case where a seed row had both a domain and a bundle id).
Tests added:
- Attacker-first land-grab: refused at the anchor stage; victim's later
crawl still lands cleanly with own created_by.
- Seed-rid takeover via unanchored bundle ID: refused; seed adagents_url
stays NULL.
- Seed-rid adoption with anchor: legitimate publisher takes ownership,
seed adagents_url updates to publisher's URL.
All 14 cases pass; the four other registry test files (registry-overlay
schema, pipeline client, search, feed; 52 cases combined) still green.
Refs #3177, #3218.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
bokelley
added a commit
that referenced
this pull request
Apr 26, 2026
… (#3244) * feat(registry): swap property readers to catalog UNION (PR 4a of #3177) Property-side readers in federated-index-db.ts and property-db.ts now union the catalog-side publishers cache (PR 1 of #3177) with the legacy discovered_properties / discovered_publishers tables. Crawl-sourced properties that landed via the new writer (PR 2 / #3218) but missed the legacy table — Setupad escalation #218's gatavo.com is the canonical example — now surface on the registry endpoints alongside legacy data. Functions changed: - hasValidAdagents: returns true if either publishers (with source_type='adagents_json') or discovered_publishers has a positive signal. Three-state contract preserved (null when neither has the domain; false only when legacy says false and catalog has no row). - getPropertiesForDomain (federated-index-db): UNION over discovered_properties and properties extracted from publishers.adagents_json JSONB, deduped on (publisher_domain, name, property_type) with legacy preferred so callers holding a discovered_properties.id keep working. Catalog-only rows surface property_rid as id (also a UUID; same call sites). - getDiscoveredPropertiesByDomain (property-db): same UNION shape. - getAllPropertiesForRegistry: three-way UNION (hosted + legacy crawl + catalog-only). Hosted wins over crawl (existing precedence carve-out preserved); legacy wins over catalog within the crawl side so the agent_count subquery against agent_property_authorizations remains meaningful during the dual-write window. - getPropertyRegistryStats: mirrors the registry's three-way UNION so a domain with both legacy and catalog rows is counted exactly once. Authorization-side readers (getPropertiesForAgent, findAgentsForPropertyIdentifier, getPublisherDomainsForAgent) stay on legacy tables — those need the authorization model decision still deferred from PR 1 and ship in PR 4b. PR 3 baseline coverage (registry-reader-baseline-* test files) all pass unchanged: 27 properties + 12 public-endpoints + 24 authorizations + 3 mcp = 66 cases. Sibling registry tests (registry-overlay-schema, registry-search, registry-feed, registry-pipeline-client, registry-crawler-cache) also pass — 66 more cases, no regressions. Refs #3177. Builds on #3195 (schema), #3218 (writer cutover), #3221 (reader baseline). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(registry): harden property readers against malformed adagents.json Code- and security-review on PR #3244 converged on one Must Fix and a companion writer-side defense. publishers.adagents_json is publisher- controlled JSONB (the migration 432 comment is explicit about it), and the AdAgentsManager validator only type-checks `authorized_agents` — nothing forces `properties`, `tags`, or `identifiers` to be arrays. A publisher serving a JSON-valid manifest with `properties: "x"` (string) or any non-array value used to crash the readers via: jsonb_array_elements(p.adagents_json->'properties') -- 22023 error jsonb_array_length(p.adagents_json->'properties') -- 22023 error jsonb_array_elements_text(prop->'tags') -- 22023 error For per-domain readers (getPropertiesForDomain, getDiscoveredProperties ByDomain) that's a per-tenant 500. For the registry list/stats readers the catalog_only CTE selects across every publisher, so one poisoned manifest takes down the entire `properties://registry` MCP listing. Two layers of defense: 1. SQL-side jsonb_typeof guards. Every jsonb_array_elements / jsonb_array_length call in the four readers now wraps its argument with `CASE WHEN jsonb_typeof(...) = 'array' THEN ... ELSE '[]'::jsonb END` (or `0` for length). Identifiers and tags get the same guard so a property with `tags: "shopping"` or `identifiers: null` no longer throws. 2. Writer-side normalization. publisher-db.upsertAdagentsCache now normalizes `properties` and `authorized_agents` to arrays before stringifying into JSONB. Defends against future validator regressions and rows that landed before this PR. Both belt and suspenders so the readers are safe regardless of writer hygiene, and the writer doesn't rely on validator details that may loosen later. Also adds: - migration 433 — partial index on catalog_properties(created_by, property_id) WHERE created_by LIKE 'adagents_json:%'. The new readers do a per-property LATERAL join against catalog_properties; without this index the lookup is a sequential scan per row. - registry-reader-poisoned-manifest.test.ts — 6 cases that seed malformed bodies directly via SQL (bypassing the writer's normalization) and assert each reader returns []/0 instead of throwing. All 66 PR 3 baseline cases still pass. 14 crawler-cache + 16 overlay- schema cases also pass. 6 new poisoned-manifest cases pass — 96 cases total green. Refs #3177, #3244. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR 2 of the property registry unification (tracking issue #3177). Builds on #3195, which merged the empty publisher overlay schema (migration 432).
This PR wires the adagents.json crawler to start writing the new tables:
evidence='adagents_json'andconfidence='authoritative'so newly crawled rows are indistinguishable from migration 336's one-time seed.The legacy
discovered_properties/agent_property_authorizationswrites continue alongside the new ones for one release. PR 4 will swap reads; PR 5 will drop the legacy tables.Closes the gap surfaced by Setupad escalation #218: gatavo.com lives in
discovered_propertiesbut never reached the catalog because migration 336 was a one-time seed. Every successful crawl now lands in both places.Implementation notes
server/src/db/publisher-db.tswithPublisherDatabase.upsertAdagentsCache, modeled onbrand-db.ts:upsertDiscoveredBrand(the closest analogue).crawler.tscacheAdagentsManifesthelper called once per validated domain in both crawl paths (registered publishers and buying-agent claims), modeled on the brand projection atcrawler.ts:200-237.normalizeIdentifierapplied beforecatalog_identifiersinsert sowww-prefixed or mixed-case publisher inputs collapse to one canonicalproperty_rid.publishersupsert preserve org/ownership metadata (workos_organization_id,created_by_email, etc.) on re-crawl while always refreshing the manifest body.Test plan
npm run test:server-unit— all existing tests still passserver/tests/integration/registry-crawler-cache.test.ts(7 cases):source_type='adagents_json'and JSONB manifest bodyadagents_urlandsource='authoritative'evidence='adagents_json'/confidence='authoritative'property_ridrather than forking identitydiscovered_propertiesandagent_property_authorizationsstill receive rowsregistry-overlay-schema.test.ts(16 cases) still passes against the same DBnpm run typecheckcleanRefs
🤖 Generated with Claude Code