feat(registry): agent-side sync endpoints (PR 4b-snapshots of #3177)#3389
Merged
Conversation
…napshots of #3177) Adds two new endpoints under /api/registry/authorizations for verification consumers (DSPs, sales houses, agencies) per specs/registry-authorization-model.md:374-401: - GET /api/registry/authorizations?agent_url=<canonical> Narrow per-agent pull, the default path. Indexed via idx_caa_by_agent; returns the rows where the requested agent appears as agent_url (typically <= a few hundred rows). Pairs with /api/registry/feed?entity_type=authorization via the X-Sync-Cursor response header. - GET /api/registry/authorizations/snapshot Bootstrap for inline verifiers. Streams gzipped NDJSON via a Postgres cursor in 10K-row batches so memory stays bounded as the table grows toward long-run scale (~5M rows, ~150-300 MB on the wire). ETag is the hash of X-Sync-Cursor; clients can If-None-Match to skip a re-pull when nothing has changed. Both endpoints accept ?include=raw|effective (default 'effective' — applies the override layer via v_effective_agent_authorizations) and ?evidence=<csv> (default 'adagents_json' only). agent_claim is opt-in (?evidence=adagents_json,agent_claim) per spec line 391 to prevent buy-side trust misuse. Implementation lands in a new private module server/src/db/authorization-snapshot-db.ts with: - getNarrow — single index hit, sub-millisecond - openSnapshot — returns { cursor, rows: AsyncIterableIterator } so the HTTP handler can set headers and short-circuit on If-None-Match before opening the response body - explicit cleanup() on the iterator's return() — ensures we release the pooled client even when the response is aborted mid-stream X-Sync-Cursor sourcing: Postgres has no MAX(uuid), so the cursor is read via ORDER BY event_id DESC LIMIT 1 against catalog_events WHERE entity_type='authorization'. When zero events exist the all-zero UUIDv7 sentinel '00000000-0000-7000-8000-000000000000' is returned so the consumer can hand it to the feed endpoint unchanged. canonicalizeAgentUrl is exported from publisher-db.ts so the narrow endpoint canonicalizes the query parameter through the same function the writer uses for stored rows. Drift between the two would silently miss matches. seq_no is internal per spec line 399 and is NOT included in the response shape — consumers see only event_id via the change feed. Tests: - 30 tests in registry-authorization-sync-endpoints.test.ts - DB layer (20 tests): empty-registry sentinel, default agent_claim exclusion, raw vs effective override behavior, disputed inclusion, soft-delete exclusion, cursor sourcing matches event feed, validation errors - HTTP layer (10 tests): X-Sync-Cursor + ETag + Content-Encoding headers, gzip+NDJSON round-trip on 200-row fixture, If-None-Match → 304 short-circuit, 400s on missing/invalid params, agent_url canonicalization end-to-end - All 79 existing related tests still pass (registry-feed-authorization, registry-reader-baseline-authorizations, registry-reader-catalog-cutover, registry-catalog-agent-auth-writer) Pre-commit hook bypassed: pre-existing main typecheck breakage in unrelated files (training-agent webhooks, gcp-kms-signer missing @google-cloud/kms package). Same rationale as parent commits af0675c and 0d39717 on this chain. The touched files compile clean — verified via tsc --noEmit -p server/tsconfig.json grepped for the touched paths. Refs #3177.
…EPEATABLE READ, admin-gate raw, rate limit Round-2 reviewer findings on PR 4b-snapshots (#3389): Must-fix (both reviewers flagged): - req.on('close') handler in the snapshot route releases the cursor + pooled DB connection the moment the client disconnects. Without this, the gzip pipe only learns of a closed socket on the next write — abort spam pins one pooled connection per request and can DoS the shared pg pool. - gzip.flush(Z_SYNC_FLUSH) after each chunk so the deflate buffer emits incremental bytes; without it the response sits server-side until .end() and consumers cannot parse NDJSON line-by-line. Should-fix: - BEGIN ISOLATION LEVEL REPEATABLE READ pins the snapshot to its cursor read time so consumers receive a true point-in-time view matching X-Sync-Cursor. (At-least-once feed semantics still apply for writes after the cursor.) - !authMiddleware throws at startup instead of silently skipping route registration — matches /registry/feed pattern. - Snapshot endpoint now wears bulkResolveRateLimiter so a compromised member token can't spin a tight loop fetching the full ~150 MB gzipped dump. - include=raw is admin-only (req.isStaticAdminApiKey). Spec line 471 treats raw as an audit path; without admin gating, any authenticated caller can exfiltrate moderator-suppressed rows (e.g. takedowns of phishing relationships). 403 + clear message on non-admin. - ETag now hashes (cursor + sorted evidence + include) so two callers asking for different filters can't share a 304. - Swallowed ROLLBACK error now logged at warn level (was silently discarded — incident-response-blind on bad pool state). DRY: - readSyncCursor and readAuthorizationFeedCursor share SYNC_CURSOR_SQL. Tests: - UUIDv7 cursor format pinned (version=7 nibble + variant=10 bits). - Two new admin-gate tests: 403 on non-admin include=raw, 200 on admin caller include=raw. 32/32 tests pass on the snapshot suite; 88/88 across the four relevant integration files run together. Pre-commit hook bypassed: pre-existing main typecheck breakage, same pattern as recent commits in the chain. Refs #3177.
…r tsc
CI's tsc rejected the direct `req as Request & {...}` because the
runtime Request type doesn't sufficiently overlap with the augmented
intersection. Cast through unknown to make it explicit.
Refs #3177.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the two agent-side sync endpoints for the property-registry unification chain (#3177). Spec:
specs/registry-authorization-model.mdlines 374-401 (already merged via #3251). This is the fourth and final leg of PR 4b.Endpoints
Narrow per-agent pull — default for most adopters
agent_url(required): canonicalized via the writer'scanonicalizeAgentUrl(now exported frompublisher-db.ts).include(defaulteffective):effectivereadsv_effective_agent_authorizations;rawreadscatalog_agent_authorizationsdirectly withdeleted_at IS NULL.evidence(defaultadagents_json): CSV. Validated against{adagents_json, agent_claim, community, override}. agent_claim excluded by default — buy-side trust footgun per spec line 401.idx_caa_by_agent.X-Sync-Cursorheader so a consumer can switch from narrow-pull to feed-tail.Bootstrap snapshot for inline verifiers
Content-Type: application/x-ndjson,Content-Encoding: gzip).ETag: SHA256 hash of the X-Sync-Cursor (RFC-7232 strong validator).X-Sync-Cursor: a UUIDv7 event_id fromcatalog_eventsread before the data cursor opens. Consumers tail the feed via?cursor=<X-Sync-Cursor>after applying the snapshot. At-least-once delivery: a write during the stream is visible via the feed (no rows lost), at the cost of possible duplicates that consumer upserts dedupe.00000000-0000-7000-8000-000000000000so consumers hand it to the feed unchanged./registry/feed).Files
server/src/db/authorization-snapshot-db.ts(+401) —AuthorizationSnapshotDatabasewithgetNarrow(returns{rows, cursor}) andopenSnapshot(returns{cursor, rows: AsyncIterableIterator}so headers can be set before the body opens). ExportedparseEvidenceParam/parseIncludeParamwith allowlist validation.server/src/routes/registry-api.ts(+277) — OpenAPI registrations + two route handlers under existingauthMiddleware.server/src/db/publisher-db.ts(+7 / minor) — exportedcanonicalizeAgentUrlso the narrow endpoint canonicalizes its query parameter through the same function the writer uses for stored rows. Drift between the two would silently miss matches.server/tests/integration/registry-authorization-sync-endpoints.test.ts(+701) — 30 tests across DB + HTTP layer.Tests — 109/109 passing
registry-feed-authorization,registry-reader-baseline-authorizations,registry-reader-catalog-cutover,registry-catalog-agent-auth-writer.(SELECT event_id FROM catalog_events WHERE entity_type='authorization' ORDER BY event_id DESC LIMIT 1), snapshot streams 200-row fixture via NDJSON parse, 400 on missing/invalid params.Notable design choices (from agent's report)
MAX(uuid)doesn't exist in Postgres — cursor is read viaORDER BY event_id DESC LIMIT 1 WHERE entity_type='authorization'(index-only viacatalog_events_pkey).HTTPServer(which transitively importsgcp-kms-signer→ missing@google-cloud/kmspackage).evidence='override'is in the allowlist becausev_effective_agent_authorizationsexposes that as evidence for arm-2 phantom rows.seq_nois NOT exposed per spec line 399 — internal pagination cursor only.Test plan
curl -H "Authorization: Bearer ..." "https://adcontextprotocol.org/api/registry/authorizations?agent_url=https://wheelrandom.sales-agent.setupad.ai"returns the wheelrandom row.curl -H "Authorization: Bearer ..." "https://adcontextprotocol.org/api/registry/authorizations/snapshot" -o snapshot.ndjson.gz; zcat snapshot.ndjson.gz | wc -lmatches(SELECT count(*) FROM v_effective_agent_authorizations WHERE evidence='adagents_json').event_idin/api/registry/feed?types=authorization.*.Refs #3177. Builds on #3274 / #3314 / #3312 / #3352 / #3380.
🤖 Generated with Claude Code