docs(conformance, catalog): refresh universal-storyboards tables

.changeset/lint-universal-storyboard-doc-parity.md

@@ -0,0 +1,23 @@
+---
+---
+
+chore(compliance): lint that universal-storyboard doc tables match the filesystem
+
+New build-time + unit-test lint that prevents the drift #3099 just fixed from re-accumulating. Every graded universal storyboard MUST appear in both `docs/building/conformance.mdx` and `docs/building/compliance-catalog.mdx`; every backtick-quoted slug in those tables MUST resolve to a real graded storyboard on disk.
+
+**Why patch / `--empty` changeset.** Build-script + tests + new lint logic — no protocol-spec surface change. The lint encodes existing convention (the two index pages should agree with `static/compliance/source/universal/`), it doesn't introduce a new normative rule.
+
+**What it catches:**
+
+- New universal storyboard ships without a doc-table row → forward parity fails (specific "missing rows for X" error).
+- Doc keeps a row for a renamed/deleted storyboard → reverse parity fails ("references X but no graded storyboard exists" error).
+- Either index page loses its "Universal" heading → "missing expected heading" error.
+
+**Wiring:**
+
+- `scripts/lint-universal-storyboard-doc-parity.cjs` — new module exporting `lint({ sourceDir, repoRoot })` plus helpers, with a CLI entrypoint.
+- `scripts/build-compliance.cjs` — calls the lint inside `generateIndex`, between `verifyEnumParity` and `lintStoryboardIdempotency`. Build fails loudly if drift is present.
+- `tests/lint-universal-storyboard-doc-parity.test.cjs` — 10 tests: source-tree guard, clean-fixture, non-graded-fixture filtering, forward parity (both docs), reverse parity (both docs), missing-heading, helper unit tests.
+- `package.json` — new `test:storyboard-doc-parity` script wired into the umbrella `test` target alongside the other storyboard lints, so CI's existing `npm run test` invocation picks it up automatically.
+
+Identifies "graded" by the presence of a `phases:` array in the YAML. Filters out the three non-graded fixtures (`storyboard-schema.yaml`, `runner-output-contract.yaml`, `fictional-entities.yaml`) which live alongside graded storyboards but aren't run by the suite.

.changeset/refresh-universal-storyboard-tables.md

@@ -0,0 +1,20 @@
+---
+---
+
+docs(conformance, catalog): refresh universal-storyboards tables to match `static/compliance/source/universal/`
+
+Both `docs/building/conformance.mdx` and `docs/building/compliance-catalog.mdx` had stale universal-storyboards tables. The actual `static/compliance/source/universal/` directory now contains 9 graded storyboards; the docs listed 5–7. Drift accumulated as new storyboards landed without back-filling the index pages.
+
+**Adds (both files):**
+
+- `webhook-emission` — outbound webhook conformance (idempotency_key + RFC 9421 webhook signing). Runs for any agent accepting `push_notification_config`. Has been universal since #2417 / 3.0; never indexed in the catalog tables.
+- `pagination-integrity` — `cursor` ↔ `has_more` invariant. Recently landed; missing from both pages.
+
+**Adds (compliance-catalog only):**
+
+- `idempotency` — was missing entirely from the catalog table though present in conformance.mdx and shipped as universal in 3.0.
+- `signed-requests` — added in #3077 to the conformance.mdx table; the parallel catalog entry was missed.
+
+**Framing fix (compliance-catalog):** the lead-in said "Every agent runs every storyboard… regardless of which protocols or specialisms it claims," which is true in scope but confusing on capability-gated rows (`deterministic-testing`, `signed-requests`). Reworded to "every agent runs every storyboard, with a few capability-gated by an explicit `supported: true` advertisement" plus a closing paragraph noting that gated storyboards can't be partially implemented (advertise `false` if you don't ship the full surface).
+
+No new normative content. The two index pages now reflect the same 9 universal storyboards in the same order, matching the published `/compliance/{version}/universal/` directory.

docs/building/compliance-catalog.mdx

@@ -15,15 +15,23 @@ This page is the human-readable index of that taxonomy. The machine-readable equ
 
 ## Universal storyboards
 
-Every agent runs every storyboard in `/compliance/{version}/universal/` regardless of which protocols or specialisms it claims. Failing a universal storyboard fails overall compliance.
+Every agent runs every storyboard in `/compliance/{version}/universal/` regardless of which protocols or specialisms it claims. A few are *capability-gated* — they only run when the agent advertises the relevant capability — but the storyboard is still universal in scope: any agent claiming the capability is graded by it. Failing a universal storyboard fails overall compliance.
+
+{/* Lint: scripts/lint-universal-storyboard-doc-parity.cjs keeps this table in sync with static/compliance/source/universal/. Add a row (kebab-case slug) when you add a graded storyboard; remove the row when one is deleted. The build fails on drift. */}
 
 | Storyboard | Purpose |
 |-----------|---------|
-| `capability-discovery` | `get_adcp_capabilities` shape and version declaration |
-| `schema-validation` | Response schema conformance and temporal constraints |
-| `error-compliance` | Structured errors with correct codes and transport binding |
-| `deterministic-testing` | `comply_test_controller` state-machine verification (skipped if unsupported) |
+| `capability-discovery` | `get_adcp_capabilities` shape, protocol/specialism declarations, version advertising |
+| `schema-validation` | Request and response schema conformance, ISO 8601 timestamps, temporal invariants |
+| `error-compliance` | Structured error shape, published error codes, transport binding, no existence leaks across tenants |
+| `idempotency` | `idempotency_key` scoping, replay semantics, `IDEMPOTENCY_CONFLICT`, `replayed: true`, declared TTL |
 | `security` | **Authentication baseline — unauth rejection, API key enforcement, OAuth discovery + RFC 9728 audience binding.** See [Authentication](/docs/building/integration/authentication). |
+| `webhook-emission` | Outbound webhook conformance — stable `idempotency_key` across retries; RFC 9421 webhook signing (or HMAC fallback if the buyer opted in). Runs for any agent that accepts `push_notification_config` on any operation. |
+| `pagination-integrity` | `cursor` ↔ `has_more` invariant verified by walking a paginated `list_creatives` response from a continuation page through to terminal. |
+| `deterministic-testing` | `comply_test_controller` state-machine verification — skipped if `capabilities.compliance_testing.supported: false`. |
+| `signed-requests` | RFC 9421 transport-layer request-signing verification — skipped if `request_signing.supported: false`. |
+
+Capability-gated rows (`deterministic-testing`, `signed-requests`) are skipped only when the agent advertises the capability as `false`; they cannot be claimed and partially implemented. Declaring `supported: true` and failing the storyboard is non-conformant — declare `false` rather than ship a partial implementation.
 
 ## Protocols
 

docs/building/conformance.mdx

@@ -48,13 +48,17 @@ Agents MUST NOT declare a capability whose storyboards they do not pass. See the
 
 Every agent MUST pass every storyboard below.
 
+{/* Lint: scripts/lint-universal-storyboard-doc-parity.cjs keeps this table in sync with static/compliance/source/universal/. Add a row (snake_case YAML id) when you add a graded storyboard; remove the row when one is deleted. The build fails on drift. */}
+
 | Storyboard | What it verifies |
 |------------|------------------|
 | [`capability_discovery`](https://adcontextprotocol.org/compliance/latest/universal/capability-discovery) | `get_adcp_capabilities` shape, protocol/specialism declarations, version advertising |
 | [`schema_validation`](https://adcontextprotocol.org/compliance/latest/universal/schema-validation) | Request and response schema conformance, ISO 8601 timestamps, temporal invariants |
 | [`error_compliance`](https://adcontextprotocol.org/compliance/latest/universal/error-compliance) | Structured error shape, published error codes, transport binding, no existence leaks across tenants |
 | [`idempotency`](https://adcontextprotocol.org/compliance/latest/universal/idempotency) | `idempotency_key` scoping, replay semantics, `IDEMPOTENCY_CONFLICT`, `replayed: true`, declared TTL |
 | [`security_baseline`](https://adcontextprotocol.org/compliance/latest/universal/security) | Unauth rejection, API key enforcement, OAuth discovery + RFC 9728 audience binding |
+| [`webhook_emission`](https://adcontextprotocol.org/compliance/latest/universal/webhook-emission) | Outbound webhook conformance — stable `idempotency_key` across retries, RFC 9421 webhook signing (or opt-in HMAC fallback) on every delivery. Runs for any agent accepting `push_notification_config`. |
+| [`pagination_integrity`](https://adcontextprotocol.org/compliance/latest/universal/pagination-integrity) | `cursor` ↔ `has_more` invariant on paginated `list_creatives` responses, walked from a continuation page through to a terminal page |
 | [`deterministic_testing`](https://adcontextprotocol.org/compliance/latest/universal/deterministic-testing) | `comply_test_controller` state machine — skipped if `capabilities.compliance_testing.supported: false` |
 | [`signed_requests`](https://adcontextprotocol.org/compliance/latest/universal/signed-requests) | RFC 9421 transport-layer request-signing verification — skipped if `request_signing.supported: false`. |
 

package.json

@@ -38,6 +38,7 @@
     "test:storyboard-auth-shape": "node --test tests/lint-storyboard-auth-shape.test.cjs",
     "test:storyboard-test-kits": "node --test tests/lint-storyboard-test-kits.test.cjs",
     "test:storyboard-sample-request-schema": "node --test tests/lint-storyboard-sample-request-schema.test.cjs",
+    "test:storyboard-doc-parity": "node --test tests/lint-universal-storyboard-doc-parity.test.cjs",
     "test:pagination-invariant": "node --test tests/lint-pagination-invariant.test.cjs",
     "test:error-codes": "node scripts/lint-error-codes.cjs",
     "test:substitution-vector-names": "node scripts/lint-substitution-vector-names.cjs",
@@ -50,7 +51,7 @@
     "test:docker": "docker compose -f docker-compose.test.yml up --build --abort-on-container-exit",
     "test:docs-nav": "node tests/docs-nav-validation.test.cjs",
     "test:platform-agnostic": "node tests/check-platform-agnostic.cjs",
-    "test": "npm run test:docs-nav && npm run test:schemas && npm run test:examples && npm run test:extensions && npm run test:extension-schemas && npm run test:error-handling && npm run test:json-schema && npm run test:composed && npm run test:migrations && npm run test:hmac-vectors && npm run test:hmac-signer-conformance && npm run test:transport-errors && npm run test:targeting-overlay-vectors && npm run test:storyboard-scoping && npm run test:storyboard-branch-sets && npm run test:storyboard-contradictions && npm run test:storyboard-context-entity && npm run test:storyboard-auth-shape && npm run test:storyboard-test-kits && npm run test:storyboard-sample-request-schema && npm run test:pagination-invariant && npm run test:error-codes && npm run test:substitution-vector-names && npm run test:platform-agnostic && npm run test:unit && npm run test:server-unit && npm run test:openapi && npm run typecheck",
+    "test": "npm run test:docs-nav && npm run test:schemas && npm run test:examples && npm run test:extensions && npm run test:extension-schemas && npm run test:error-handling && npm run test:json-schema && npm run test:composed && npm run test:migrations && npm run test:hmac-vectors && npm run test:hmac-signer-conformance && npm run test:transport-errors && npm run test:targeting-overlay-vectors && npm run test:storyboard-scoping && npm run test:storyboard-branch-sets && npm run test:storyboard-contradictions && npm run test:storyboard-context-entity && npm run test:storyboard-auth-shape && npm run test:storyboard-test-kits && npm run test:storyboard-sample-request-schema && npm run test:storyboard-doc-parity && npm run test:pagination-invariant && npm run test:error-codes && npm run test:substitution-vector-names && npm run test:platform-agnostic && npm run test:unit && npm run test:server-unit && npm run test:openapi && npm run typecheck",
     "test:all": "npm run test:schemas && npm run test:examples && npm run test:extensions && npm run test:error-handling && npm run test:snippets && npm run typecheck",
     "precommit": "bash scripts/with-timeout.sh 60 npm run test:unit && npm run typecheck",
     "prepare": "husky",

scripts/build-compliance.cjs

@@ -345,10 +345,16 @@ function verifyEnumParity(specialisms, protocols) {
   }
 }
 
+const { lint: lintUniversalDocParity } = require('./lint-universal-storyboard-doc-parity.cjs');
+
 function generateIndex(version, sourceDir) {
   const specialisms = discoverSpecialisms(sourceDir);
   const protocols = discoverProtocols(sourceDir, specialisms);
   verifyEnumParity(specialisms, protocols);
+  const docParityErrors = lintUniversalDocParity({ sourceDir });
+  if (docParityErrors.length) {
+    throw new Error('Universal-storyboard doc parity drift:\n  - ' + docParityErrors.join('\n  - '));
+  }
   lintStoryboardIdempotency(sourceDir, SCHEMAS_DIR);
   const universalDir = path.join(sourceDir, 'universal');
   const universal = fs.existsSync(universalDir)

scripts/lint-universal-storyboard-doc-parity.cjs

@@ -0,0 +1,165 @@
+#!/usr/bin/env node
+/**
+ * Lint: docs index pages list every graded universal storyboard, and only
+ * reference storyboards that exist on disk.
+ *
+ * Why
+ * ---
+ * The catalog and conformance index pages each carry a table of universal
+ * storyboards. Both have drifted in the past — storyboards landed without
+ * back-filling the docs (#3099 caught webhook-emission, pagination-integrity,
+ * idempotency, signed-requests all missing in various places). This lint
+ * keeps them honest.
+ *
+ * Forward parity: every graded universal storyboard MUST appear in both
+ * tables. A new universal storyboard that ships without a row breaks the
+ * build.
+ *
+ * Reverse parity: every backtick-quoted slug appearing in a table row MUST
+ * resolve to a real graded storyboard on disk. A copy-paste typo or a row
+ * left behind after a rename breaks the build.
+ *
+ * "Graded" means the YAML has a `phases:` array. Filters out non-graded
+ * fixtures (storyboard-schema.yaml, runner-output-contract.yaml,
+ * fictional-entities.yaml) which live alongside graded storyboards but
+ * aren't run by the suite.
+ *
+ * Slug forms differ across the two docs by design (existing convention,
+ * not changed here):
+ *   - docs/building/conformance.mdx           uses snake_case YAML `id` values in markdown links
+ *   - docs/building/compliance-catalog.mdx    uses kebab-case filename slugs in plain backticks
+ */
+
+'use strict';
+
+const fs = require('node:fs');
+const path = require('node:path');
+const yaml = require('js-yaml');
+
+const REPO_ROOT = path.join(__dirname, '..');
+const DEFAULT_SOURCE_DIR = path.join(REPO_ROOT, 'static/compliance/source');
+
+const DOC_CHECKS = [
+  {
+    relpath: 'docs/building/conformance.mdx',
+    heading: '## Universal conformance',
+    tokenForItem: item => item.id,
+    tokenLabel: 'YAML `id`',
+  },
+  {
+    relpath: 'docs/building/compliance-catalog.mdx',
+    heading: '## Universal storyboards',
+    tokenForItem: item => item.slug,
+    tokenLabel: 'filename slug',
+  },
+];
+
+function discoverGradedUniversal(sourceDir) {
+  const universalDir = path.join(sourceDir, 'universal');
+  if (!fs.existsSync(universalDir)) return [];
+  return fs.readdirSync(universalDir)
+    .filter(f => f.endsWith('.yaml'))
+    .map(f => {
+      const slug = f.replace(/\.yaml$/, '');
+      let doc = null;
+      try {
+        doc = yaml.load(fs.readFileSync(path.join(universalDir, f), 'utf8'));
+      } catch {
+        // Malformed YAML is caught by other lints; skip it here so doc parity
+        // doesn't double-error on the same file.
+      }
+      const graded = doc && typeof doc === 'object' && Array.isArray(doc.phases);
+      const id = doc && typeof doc === 'object' && typeof doc.id === 'string'
+        ? doc.id
+        : slug.replace(/-/g, '_');
+      return { slug, id, graded };
+    })
+    .filter(item => item.graded);
+}
+
+function extractSection(content, heading) {
+  const start = content.indexOf(heading);
+  if (start === -1) return null;
+  const tail = content.slice(start + heading.length);
+  const nextHeadingIdx = tail.search(/\n## /);
+  return nextHeadingIdx === -1
+    ? content.slice(start)
+    : content.slice(start, start + heading.length + nextHeadingIdx);
+}
+
+function extractTableTokens(section) {
+  const tokens = new Set();
+  for (const line of section.split('\n')) {
+    if (!line.startsWith('|')) continue;
+    // First cell only; supports `[`token`](url)` (markdown link) and `` `token` `` (plain).
+    const m = line.match(/^\|\s*\[?`([a-z][a-z0-9_-]+)`/);
+    if (m) tokens.add(m[1]);
+  }
+  return tokens;
+}
+
+/**
+ * Run the lint. Returns an array of error strings (empty = clean).
+ * Pass `{ sourceDir, repoRoot }` to override default paths (used by tests).
+ */
+function lint({ sourceDir = DEFAULT_SOURCE_DIR, repoRoot = REPO_ROOT } = {}) {
+  const items = discoverGradedUniversal(sourceDir);
+  const errors = [];
+
+  for (const check of DOC_CHECKS) {
+    const filePath = path.join(repoRoot, check.relpath);
+    if (!fs.existsSync(filePath)) continue;
+    const content = fs.readFileSync(filePath, 'utf8');
+
+    const section = extractSection(content, check.heading);
+    if (section === null) {
+      errors.push(`${check.relpath}: missing expected heading "${check.heading}"`);
+      continue;
+    }
+
+    // Forward parity
+    const missingFromDoc = items
+      .map(item => check.tokenForItem(item))
+      .filter(token => !section.includes('`' + token + '`'));
+    if (missingFromDoc.length) {
+      errors.push(
+        `${check.relpath}: universal-storyboards table is missing rows for ${missingFromDoc.map(t => '`' + t + '`').join(', ')}.\n` +
+        `  Add a row to the table under "${check.heading}" using the ${check.tokenLabel} form. The runner will fail any agent that doesn't pass these.`
+      );
+    }
+
+    // Reverse parity
+    const knownTokens = new Set(items.map(item => check.tokenForItem(item)));
+    const tableTokens = extractTableTokens(section);
+    const ghostTokens = [...tableTokens].filter(t => !knownTokens.has(t));
+    if (ghostTokens.length) {
+      errors.push(
+        `${check.relpath}: universal-storyboards table references ${ghostTokens.map(t => '`' + t + '`').join(', ')} but no graded storyboard exists on disk under static/compliance/source/universal/.\n` +
+        `  Either add the storyboard YAML or remove the row. (Expected ${check.tokenLabel} form.)`
+      );
+    }
+  }
+
+  return errors;
+}
+
+function main() {
+  const errors = lint();
+  if (errors.length) {
+    console.error('Universal-storyboard doc parity drift:\n  - ' + errors.join('\n  - '));
+    process.exit(1);
+  }
+  console.log('✓ universal-storyboard doc parity: docs match graded universal storyboards');
+}
+
+if (require.main === module) {
+  main();
+}
+
+module.exports = {
+  lint,
+  discoverGradedUniversal,
+  extractSection,
+  extractTableTokens,
+  DOC_CHECKS,
+};