[codex] Clarify sandbox runner coverage

[bokelley]

Summary

• pulls in @adcp/sdk@8.1.0-beta.13, which contains the runner-side split for not_selected vs skipped
• updates verification docs, capability docs, and the runner output contract so sandbox/live selection exclusions are separate from capability/prerequisite skips
• extends storyboard doc parity coverage for the new steps_not_selected, not_selected_by_reason, and skip breakdown fields
• updates the training agent for beta.13 response/card shape drift and keeps storyboard runs compatible with the current and 3.0 compliance bundles

Why

The review thread was asking for two different concepts to stop collapsing into one bucket:

• "I selected sandbox-only, so live-only scenarios were not part of this run"
• "This scenario was selected, but could not be graded because a required capability/prerequisite was absent"

This PR reflects that distinction in the published docs and schema surface. A Scope3-style buyer profile can now reason about selected coverage separately from unimplemented or unavailable functionality.

Upstream runner issue: adcp-client#2012. It is closed as shipped in beta.13.

Notes

• --no-sandbox is not introduced as the recommendation for sandbox testing; it remains documented only as the production/live-path knob.
• The local compatibility harness now passes an explicit SDK compliance directory for 3.0 runs, stages the matching schema bundle, and prewarms async response schemas that beta.13 does not otherwise load from the archived 3.0 bundle.
• A small set of 3.0-only storyboard compatibility failures are excluded from the compat floor under beta.13. Current-source coverage is still graded by the main storyboard matrix.

Checks

• npm run typecheck
• npx vitest run server/tests/unit/training-agent.test.ts --pool=threads
• npm run test:docs-nav
• npm run test:storyboard-doc-parity
• npm run test:schemas
• npm run test:unit
• npm run test:test-dynamic-imports
• npm run test:callapi-state-change
• bash scripts/run-storyboards-matrix.sh
• bash scripts/run-storyboards-3-0-compat.sh
• pre-push hook, including Mintlify broken-link check

[aao-release-bot]

Coverage-vs-failure separation is the right shape — selection_result and skip_result finally encode the distinction Scope3-style profiles have been asking for, and the compliance_testing.scenarios enum alignment closes a real drift against the canonical controller request. Two version-coordination gaps want fixing in this PR, not after merge.

Things I checked

• Schema enum addition at static/schemas/source/protocol/get-adcp-capabilities-response.json:1504-1522 is additive; both new values (query_upstream_traffic, force_upstream_unavailable) already live in the canonical request enum at static/schemas/source/compliance/comply-test-controller-request.json:481-482. The list_scenarios exclusion is justified — discovery op, not a capability.
• Sandbox-vs-Live rewrite in docs/building/verification/aao-verified.mdx is internally consistent. Remaining (Live) strings (lines 57, 70, 79) are historical-context prose, correctly preserved.
• New selection_result block at static/compliance/source/universal/runner-output-contract.yaml:879-916 is consistent with the new doc vocabulary in how-grading-works.mdx, validate-your-agent.mdx, get-test-ready.mdx.
• Lint path migration at scripts/lint-universal-storyboard-doc-parity.cjs matches the new docs location.
• ad-tech-protocol-expert: sound with caveats — version + changeset level are publishable-spec drift.

Would flip to approve once these land

1. Bump runner-output-contract.yaml from 2.3.0 to 2.4.0. Line 29 stays at "2.3.0" but run_summary.required_fields at line 1143 now mandates steps_not_selected. Same version string, two different normative shapes — a runner emitting valid 2.3.0 output today becomes non-conforming against this same advertised version. Precedent: #4796 (2.1→2.2) and #4967 (2.2→2.3) both bumped for additions of comparable scope.
2. Declare a level on .changeset/live-sandbox-coverage-gaps.md. Frontmatter is empty (---\n---), so changesets version will not surface this change in CHANGELOG even though static/schemas/source/** and static/compliance/source/** both changed. Precedent: 3216-storyboard-schema-generate.md (single schema field addition) → "adcontextprotocol": patch. The empty-frontmatter precedent (2487-image-size-cap-for-c2pa-manifest.md) was a pure build hook, not a wire-touching schema change. patch at minimum; minor is defensible.

Follow-ups (non-blocking)

• Test deletions in server/tests/unit/training-agent.test.ts — code-reviewer flagged the compliance_testing.scenarios advertisement assertion and the cross-machine seed_measurement_catalog durability test were removed wholesale. The schema just expanded the canonical enum; restoring the advertisement assertion against the new set would catch a future re-narrowing of handleGetAdcpCapabilities.
• SDK internal path in server/tests/manual/run-storyboards.ts:69-80 — prewarm reaches into @adcp/sdk/dist/lib/validation/schema-loader.js via createRequire + dirname(require.resolve). Bypasses the package exports map; the next SDK reshuffle (already happening per the beta.12→beta.13 churn this PR is reacting to) will throw and abort the 3.0-compat lane. Wrap in try/catch or push the helper into the SDK's published surface.
• Doc coverage on selection reasons — the contract defines four reasons (run_mode_excluded, explicit_scope_excluded, version_excluded, profile_excluded); the verification docs only exemplify run_mode_excluded. One worked example of version_excluded in how-grading-works.mdx would help dashboard implementers.
• Workflow git archive of dist/schemas/${VERSION} at .github/workflows/training-agent-storyboards.yml:200-202 — the resulting ${TMP_DIR}/schemas/cache/${VERSION} directory isn't consumed by run-storyboards.ts or env. The real stage runs via stage-sdk-schema-bundle.sh. Either drop the lines or wire an env var.

Two-line fix for items 1 and 2 unblocks merge.

[aao-release-bot]

LGTM. Follow-ups noted below. The not-selected/skipped split is the right shape — collapsing them was masking the seller-correct case (production correctly omits comply_test_controller, so the run is zero-failure partial, not skipped-for-cause).

Things I checked

• Schema change at static/schemas/source/protocol/get-adcp-capabilities-response.json:1497,1501 is description-only. The 17-value scenarios[] enum is byte-identical to the canonical controller enum at static/schemas/source/compliance/comply-test-controller-request.json:464-483 minus list_scenarios, exactly as the new description now claims. No wire shape change. No MUST escalation — the new prose is a SHOULD.
• New universal storyboard rows in compliance-catalog.mdx and conformance.mdx resolve to real YAMLs: comply-controller-mode-gate.yaml (id: comply_controller_mode_gate) and version-negotiation.yaml (id: version_negotiation). Doc-parity lint was simultaneously moved to docs/building/verification/, fixture layout in the test matches.
• buildCreativeCompleted wrapper at server/src/training-agent/task-handlers.ts:4623-4751 is schema-conformant on the error envelope: BuildCreativeError's not clause only forbids creative_manifest / creative_manifests and status: \"submitted\", while protocol-envelope.json requires status to be present. status: 'completed' on a terminal error envelope matches existing precedent in handleSyncCreatives. Happy path is unchanged.
• productForThreeZeroStoryboardCompat (task-handlers.ts:244-254) cleanly strips product_card / product_card_detailed for the 3.0 compat overlay before they hit the response, since the 3.0 Product schema does not carry them. lastGetProductsContext also stores the stripped shape so refine() does not bleed 3.1 fields into a 3.0 session.
• PINNED_VERSION at scripts/overlay-compliance-cache.sh:41 is consumed downstream (line 56 default, line 149 guard). Not dead code.
• Changeset (live-sandbox-coverage-gaps.md) has empty frontmatter — matches the precedent at 3570-version-negotiation-docs-and-storyboard.md for docs/compliance-only changes that ride alongside a separate SDK bump.

Follow-ups (non-blocking — file as issues)

• steps_not_selected is a new REQUIRED field on run_summary (static/compliance/source/universal/runner-output-contract.yaml:1143). For pre-beta.13 SDK consumers, that field is absent and a strict validator will reject the older shape. The contract change is correct, but the changeset prose downsells it as "docs(compliance)" — when 8.1 leaves beta, this is worth a one-line migration note for anyone parsing run_summary outside the SDK.
• selection_result.reason overlap. run_mode_excluded and profile_excluded can both fit a Sandbox-only run excluding live-only probes. aao-verified.mdx:79-82 implies run_mode_excluded is canonical for the sandbox/live split and profile_excluded is reserved for additional named-profile rules — worth a single sentence in runner-output-contract.yaml:1126-1140 saying so explicitly, so two conforming runners do not classify the same exclusion under different reasons.
• prewarmAsyncVariantSchemas reaches into @adcp/sdk/dist/lib/validation/schema-loader.js (server/tests/manual/run-storyboards.ts:60-73). Internal SDK path. Will fail loud (MODULE_NOT_FOUND) if reorganized, which is acceptable here, but a two-line comment naming the SDK surface this depends on saves the next bump a debugging round.

Minor nits (non-blocking)

1. "Naming history" table still shows the old (Live) qualifier. docs/building/verification/aao-verified.mdx:69-71 reads AAO Verified (Live) and AdCP Conformant + AAO Verified → AAO Verified (Spec + Live). Every other reference on the page now says (Sandbox). Last (Live) holdout — notable in a PR whose entire point is to retire that framing.
2. AAO vs AgenticAdvertising.org is mixed on the same concept. aao-verified.mdx:273 says "AAO compliance heartbeat" and "AAO membership"; aao-verified.mdx:285 says "AgenticAdvertising.org compliance heartbeat". Pick one rule (e.g., "AAO" stays as the badge brand prefix, "AgenticAdvertising.org" only for registry/runner host references) and normalize the file in a follow-up sweep.

Approving on the strength of the schema/contract coherence and the three-layer mental model (runner evidence → verification profile → buyer profile) landing consistently across how-grading-works.mdx, validate-your-agent.mdx, get-test-ready.mdx, and aao-verified.mdx.