Skip to content

feat(registry): persist managerdomain discovery provenance on publisher rows#4204

Merged
bokelley merged 4 commits into
mainfrom
claude/issue-4200-discovery-provenance
May 7, 2026
Merged

feat(registry): persist managerdomain discovery provenance on publisher rows#4204
bokelley merged 4 commits into
mainfrom
claude/issue-4200-discovery-provenance

Conversation

@bokelley

@bokelley bokelley commented May 7, 2026

Copy link
Copy Markdown
Contributor

Refs #4200

Summary

Threads discovery_method and manager_domain from AdAgentsValidationResult (populated by #4173) through the crawler's cache write, event payloads, and API endpoints so buyers and members can distinguish direct origin-attestation from one-hop ads.txt MANAGERDOMAIN delegation.

Scope: items 1, 3, 4 from #4200. Item 2 (reverse index + queue-backed fan-out) is a tracked follow-up — the reverse index needs a bounded work queue before shipping to avoid crawl storms at Raptive-scale managed networks. Item 5 (manager recrawl endpoint) defers until the reverse index exists.

Changes

  • Migration 470 (server/src/db/migrations/470_publisher_discovery_provenance.sql): ALTER TABLE publishers ADD COLUMN discovery_method TEXT CHECK (...), ADD COLUMN manager_domain TEXT. Backfill implicit via 60-min crawl cadence; nulls degrade gracefully.
  • server/src/db/publisher-db.ts: UpsertAdagentsCacheInput extended with discoveryMethod? and managerDomain?; upsertAdagentsCache SQL writes both columns.
  • server/src/crawler.ts: All three cacheAdagentsManifest call sites pass provenance fields. publisher.adagents_changed and publisher.adagents_discovered event payloads now include discovery_method + manager_domain.
  • server/src/http.ts: /api/validate-publisher response includes both fields.
  • server/src/routes/registry-api.ts: /api/public/validate-publisher response and its OpenAPI schema include both fields. /api/registry/publisher reads discovery_method + manager_domain from the DB row and surfaces them at the publisher level (not per-agent).
  • server/src/schemas/registry.ts: PublisherLookupResultSchema declares discovery_method and manager_domain as nullable optional fields with .openapi() descriptions.

Non-breaking justification

All DB columns are nullable with no backfill. All new API response fields are optional and additive. No existing field removed, renamed, or made required. The source enum on PublisherAuthorizedAgentSchema is unchanged — per-agent source extension to adagents_json_via_manager is a follow-up that requires threading discoveryMethod through recordAgentFromAdagentsJson and upsertAuthorization.

Pre-PR review

  • code-reviewer: approved — one blocker fixed (null → undefined for absent fields in res.json() response); one nit resolved (removed ?? undefined no-op from event payloads). Migration CHECK constraint cross-reference comment present.
  • adtech-product-expert: approved after two blockers fixed: (1) PublisherLookupResultSchema now declares both fields; (2) /api/public/validate-publisher now emits both fields and its OpenAPI schema is updated. Placement of discovery provenance at publisher level (not per-agent) is correct — it describes how the manifest was found, not per-agent trust.

Nits (not fixed — noted for reviewers)

  • No new unit tests for the crawler write paths or API responses. The change is narrow (threading existing values), but a regression test for cacheAdagentsManifest → DB params and /api/registry/publisher response shape would harden this path.
  • manager_domain has no length guard at the DB write path; ads.txt hostnames are capped at 253 chars by RFC 1035 but no defensive truncation was added (consistent with manager_domain being a hostname, not a URL).

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout <num>
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See #3121
for context.

Session: https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt


Generated by Claude Code

claude added 2 commits May 7, 2026 19:10
…er rows

Threads discovery_method + manager_domain from AdAgentsValidationResult
(already populated by PR #4173) through all crawler write paths, events,
and API endpoints for issue #4200 items 1, 3, 4.

- Migration 470: ADD COLUMN discovery_method TEXT (CHECK constraint) +
  manager_domain TEXT to publishers table
- UpsertAdagentsCacheInput extended; SQL upsert writes both columns
- All 3 cacheAdagentsManifest call sites pass provenance fields
- publisher.adagents_changed + publisher.adagents_discovered events now
  carry discovery_method + manager_domain in their payloads
- /api/validate-publisher (http.ts) and /api/public/validate-publisher
  (registry-api.ts) responses include both fields
- /api/registry/publisher reads discovery_method + manager_domain from
  the publishers row and includes them at the publisher level
- PublisherLookupResultSchema and /api/public/validate-publisher OpenAPI
  schema updated to declare the new fields

Non-breaking: all columns nullable, all response fields optional additive.
Item 2 (reverse index + queue-backed fan-out) is a follow-up.

Refs #4200

https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt
- /api/registry/publisher: discovery_method and manager_domain now use
  ?? undefined so absent values omit the key from JSON rather than
  serialising null
- Event payloads: remove ?? undefined no-op (manager_domain is already
  string | undefined on AdAgentsValidationResult)

https://claude.ai/code/cse_01SX6coXKTUA3sLPB6dfcjZt
@bokelley bokelley added the claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage. label May 7, 2026
bokelley added 2 commits May 7, 2026 17:46
Three small improvements based on expert review of #4204 before merge:

1. **Backfill `discovery_method = 'direct'` for previously-validated
   rows.** Pre-migration, every successfully-validated publisher was
   necessarily discovered via the direct path — the other two methods
   didn't exist before #4173/#4204 landed. Stamping `direct` on those
   rows eliminates the otherwise-confusing NULL window between this
   migration and the next 60-minute crawl cycle, so the API returns a
   stable provenance value immediately.
2. **Add a partial index on `manager_domain`.** Item 2 of #4200 (queue-
   backed fan-out when a manager rotates their adagents.json) needs a
   manager → publishers reverse lookup. Building the partial index here
   is essentially free — the column is mostly NULL, so the index
   footprint is tiny — and means item 2 ships without another schema
   migration.
3. **Tighten `discovery_method` to required on the validate-publisher
   response schema.** `AdAgentsValidationResult.discovery_method` is
   already required and unconditionally set on every code path that
   reaches this response. Marking the OpenAPI schema `.optional()` was
   loose-on-the-server, leaking `T | undefined` into generated client
   types for a guarantee the server already meets. Now consumers get
   the tight contract.
adagents_valid is an API-derived field, not a column on publishers.
The right canonical signal for 'we have a successfully cached
adagents.json for this row' is source_type = 'adagents_json' AND
adagents_json IS NOT NULL.
@bokelley bokelley marked this pull request as ready for review May 7, 2026 21:55
@bokelley bokelley merged commit a42f238 into main May 7, 2026
13 checks passed
@bokelley bokelley deleted the claude/issue-4200-discovery-provenance branch May 7, 2026 21:56
bokelley added a commit that referenced this pull request May 8, 2026
Closes #4200 item 2. When a manager rotates its adagents.json, every publisher delegating via ads.txt MANAGERDOMAIN needs re-validation. Inline fan-out at managed-network scale would saturate crawler concurrency, so this PR adds a persistent queue and a bounded worker tick.

Migration 471: manager_revalidation_queue table mirroring catalog_crawl_queue (idempotent insert, next_attempt_after backoff, partial index for due-row scan).

cacheAdagentsManifest reads the previously-cached body before the upsert and compares the contributory subset (authorized_agents, properties) via recursive stable-key canonicalization. Only actual content drift triggers fan-out.

processManagerRevalidationQueue worker tick drains up to 50 rows per 5-minute interval at concurrency 10, with exponential backoff (1h / 6h / 1d / 3d) on failure. Wired into http.ts bootstrap alongside startPeriodicCatalogCrawl.

Refs #4200, #4173, #4204.
bokelley added a commit that referenced this pull request May 8, 2026
Closes #4200 item 5. New POST /api/registry/manager-revalidation-request short-circuits the 60-minute organic crawl cycle: when a manager rotates its adagents.json, ops can hit this endpoint and have every delegating publisher enqueued for immediate re-validation.

Thin wrapper around enqueueManagerRevalidation (#4210). Body: { manager_domain }. Returns 202 with publishers_enqueued. Rate-limited via the shared validateAndRateLimitCrawl machinery; key namespaced (manager: prefix) so a manager request doesn't bypass an in-window publisher recrawl on the same domain.

Per-agent source enum extension to 'adagents_json_via_manager' is closed as won't-fix on #4200: publishers.discovery_method (#4204) already lets consumers join through and discriminate, and a separate per-agent value would silently exclude managerdomain rows from existing readers filtering on source='adagents_json'.

Refs #4200, #4173, #4204, #4210.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-triaged Issue has been triaged by the Claude Code triage routine. Remove to re-triage.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants