feat(registry): persist managerdomain discovery provenance on publisher rows#4204
Merged
Conversation
…er rows Threads discovery_method + manager_domain from AdAgentsValidationResult (already populated by PR #4173) through all crawler write paths, events, and API endpoints for issue #4200 items 1, 3, 4. - Migration 470: ADD COLUMN discovery_method TEXT (CHECK constraint) + manager_domain TEXT to publishers table - UpsertAdagentsCacheInput extended; SQL upsert writes both columns - All 3 cacheAdagentsManifest call sites pass provenance fields - publisher.adagents_changed + publisher.adagents_discovered events now carry discovery_method + manager_domain in their payloads - /api/validate-publisher (http.ts) and /api/public/validate-publisher (registry-api.ts) responses include both fields - /api/registry/publisher reads discovery_method + manager_domain from the publishers row and includes them at the publisher level - PublisherLookupResultSchema and /api/public/validate-publisher OpenAPI schema updated to declare the new fields Non-breaking: all columns nullable, all response fields optional additive. Item 2 (reverse index + queue-backed fan-out) is a follow-up. Refs #4200 https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt
- /api/registry/publisher: discovery_method and manager_domain now use ?? undefined so absent values omit the key from JSON rather than serialising null - Event payloads: remove ?? undefined no-op (manager_domain is already string | undefined on AdAgentsValidationResult) https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt
Three small improvements based on expert review of #4204 before merge: 1. **Backfill `discovery_method = 'direct'` for previously-validated rows.** Pre-migration, every successfully-validated publisher was necessarily discovered via the direct path — the other two methods didn't exist before #4173/#4204 landed. Stamping `direct` on those rows eliminates the otherwise-confusing NULL window between this migration and the next 60-minute crawl cycle, so the API returns a stable provenance value immediately. 2. **Add a partial index on `manager_domain`.** Item 2 of #4200 (queue- backed fan-out when a manager rotates their adagents.json) needs a manager → publishers reverse lookup. Building the partial index here is essentially free — the column is mostly NULL, so the index footprint is tiny — and means item 2 ships without another schema migration. 3. **Tighten `discovery_method` to required on the validate-publisher response schema.** `AdAgentsValidationResult.discovery_method` is already required and unconditionally set on every code path that reaches this response. Marking the OpenAPI schema `.optional()` was loose-on-the-server, leaking `T | undefined` into generated client types for a guarantee the server already meets. Now consumers get the tight contract.
adagents_valid is an API-derived field, not a column on publishers. The right canonical signal for 'we have a successfully cached adagents.json for this row' is source_type = 'adagents_json' AND adagents_json IS NOT NULL.
bokelley
added a commit
that referenced
this pull request
May 8, 2026
Closes #4200 item 2. When a manager rotates its adagents.json, every publisher delegating via ads.txt MANAGERDOMAIN needs re-validation. Inline fan-out at managed-network scale would saturate crawler concurrency, so this PR adds a persistent queue and a bounded worker tick. Migration 471: manager_revalidation_queue table mirroring catalog_crawl_queue (idempotent insert, next_attempt_after backoff, partial index for due-row scan). cacheAdagentsManifest reads the previously-cached body before the upsert and compares the contributory subset (authorized_agents, properties) via recursive stable-key canonicalization. Only actual content drift triggers fan-out. processManagerRevalidationQueue worker tick drains up to 50 rows per 5-minute interval at concurrency 10, with exponential backoff (1h / 6h / 1d / 3d) on failure. Wired into http.ts bootstrap alongside startPeriodicCatalogCrawl. Refs #4200, #4173, #4204.
bokelley
added a commit
that referenced
this pull request
May 8, 2026
Closes #4200 item 5. New POST /api/registry/manager-revalidation-request short-circuits the 60-minute organic crawl cycle: when a manager rotates its adagents.json, ops can hit this endpoint and have every delegating publisher enqueued for immediate re-validation. Thin wrapper around enqueueManagerRevalidation (#4210). Body: { manager_domain }. Returns 202 with publishers_enqueued. Rate-limited via the shared validateAndRateLimitCrawl machinery; key namespaced (manager: prefix) so a manager request doesn't bypass an in-window publisher recrawl on the same domain. Per-agent source enum extension to 'adagents_json_via_manager' is closed as won't-fix on #4200: publishers.discovery_method (#4204) already lets consumers join through and discriminate, and a separate per-agent value would silently exclude managerdomain rows from existing readers filtering on source='adagents_json'. Refs #4200, #4173, #4204, #4210.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4200
Summary
Threads
discovery_methodandmanager_domainfromAdAgentsValidationResult(populated by #4173) through the crawler's cache write, event payloads, and API endpoints so buyers and members can distinguish direct origin-attestation from one-hop ads.txtMANAGERDOMAINdelegation.Scope: items 1, 3, 4 from #4200. Item 2 (reverse index + queue-backed fan-out) is a tracked follow-up — the reverse index needs a bounded work queue before shipping to avoid crawl storms at Raptive-scale managed networks. Item 5 (manager recrawl endpoint) defers until the reverse index exists.
Changes
server/src/db/migrations/470_publisher_discovery_provenance.sql):ALTER TABLE publishers ADD COLUMN discovery_method TEXT CHECK (...), ADD COLUMN manager_domain TEXT. Backfill implicit via 60-min crawl cadence; nulls degrade gracefully.server/src/db/publisher-db.ts:UpsertAdagentsCacheInputextended withdiscoveryMethod?andmanagerDomain?;upsertAdagentsCacheSQL writes both columns.server/src/crawler.ts: All threecacheAdagentsManifestcall sites pass provenance fields.publisher.adagents_changedandpublisher.adagents_discoveredevent payloads now includediscovery_method+manager_domain.server/src/http.ts:/api/validate-publisherresponse includes both fields.server/src/routes/registry-api.ts:/api/public/validate-publisherresponse and its OpenAPI schema include both fields./api/registry/publisherreadsdiscovery_method+manager_domainfrom the DB row and surfaces them at the publisher level (not per-agent).server/src/schemas/registry.ts:PublisherLookupResultSchemadeclaresdiscovery_methodandmanager_domainas nullable optional fields with.openapi()descriptions.Non-breaking justification
All DB columns are nullable with no backfill. All new API response fields are optional and additive. No existing field removed, renamed, or made required. The
sourceenum onPublisherAuthorizedAgentSchemais unchanged — per-agentsourceextension toadagents_json_via_manageris a follow-up that requires threadingdiscoveryMethodthroughrecordAgentFromAdagentsJsonandupsertAuthorization.Pre-PR review
res.json()response); one nit resolved (removed?? undefinedno-op from event payloads). Migration CHECK constraint cross-reference comment present.PublisherLookupResultSchemanow declares both fields; (2)/api/public/validate-publishernow emits both fields and its OpenAPI schema is updated. Placement of discovery provenance at publisher level (not per-agent) is correct — it describes how the manifest was found, not per-agent trust.Nits (not fixed — noted for reviewers)
cacheAdagentsManifest→ DB params and/api/registry/publisherresponse shape would harden this path.manager_domainhas no length guard at the DB write path; ads.txt hostnames are capped at 253 chars by RFC 1035 but no defensive truncation was added (consistent withmanager_domainbeing a hostname, not a URL).Session: https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt
Generated by Claude Code