Training-agent baseline regressions exposed by @adcp/sdk@6.7.0 bump (PR #3962)

[bokelley]

PR #3962 bumps `@adcp/sdk` from 6.0.0 to 6.7.0 to unblock storyboard adoption of `upstream_traffic` (the runner-side support landed in 6.7.0 / closed adcp-client#1253). The bump exposes a cluster of baseline regressions in the training agent that aren't related to the contract additions but need fixing to restore the previous floors. Floors lowered in #3962 to ship the bump; this issue tracks the catch-up work.

Per-tenant deltas (clean / passed)

Tenant	Before (6.0.0)	After (6.7.0)	Delta
signals	59 / 23	59 / 23	0
sales	55 / 159	56 / 181	+1 / +22
governance	57 / 62	57 / 62	0
creative	58 / 44	51 / 44	−7 / 0
creative-builder	55 / 37	49 / 37	−6 / 0
brand	59 / 14	58 / 13	−1 / −1

Failure classes (first-pass diagnosis)

Sampled across all 6 tenants:

1. `deterministic_testing` context echo missing — `comply_test_controller` responses no longer include `context` field, breaking `field_present path: 'context'` and `field_value path: 'context.correlation_id'` validations on every scenario (`list_scenarios`, `unknown_scenario`, `missing_params`, `not_found_entity`).

2. `UNKNOWN_SCENARIO` returned where `INVALID_PARAMS` / `NOT_FOUND` expected — error-code mapping for scenario validation got coarser; `comply_test_controller` returns the catch-all `UNKNOWN_SCENARIO` for missing-params and entity-not-found cases the storyboards expect to discriminate.

3. `signed_requests` discovery_failed against `/mcp-strict` — a strict-mode endpoint the training agent doesn't expose. May predate the bump but surfaces as a hard failure on every tenant.

4. `idempotency_key` capture path not resolvable in `create_media_buy_initial` — the runner's new synthesized `capture_path_not_resolvable` validation correctly fires when the response omits a captured field. Either training-agent stopped echoing `idempotency_key` on `create_media_buy` responses, or the storyboard's capture path drifted.

5. `force_create_media_buy_arm` directive acknowledgement missing — `forced.arm` returns `undefined` instead of the registered `submitted` arm. `comply_test_controller` directive registration may have changed shape.

6. `seed_creative_format` ack returns `success: false` — seed scenario for creative-format fixture not implemented or returning rejection.

7. `brand_baseline` error code mismatch — `get_brand_identity` for unknown brand returns `REFERENCE_NOT_FOUND` (the new SDK convention) where storyboard expects `brand_not_found` / `BRAND_NOT_FOUND` / `NOT_FOUND` (older expectations). Either the storyboard expectation or the SDK convention should win.

Suggested split

Each class above is a separate small fix:

• A. Restore `context` echo in `comply_test_controller` responses across all scenarios (one handler change).
• B. Tighten error-code mapping in scenario validation (controller-side).
• C. Either expose `/mcp-strict` on each tenant or remove the assertion from `signed_requests`.
• D. Restore `idempotency_key` echo on `create_media_buy` response (or drop the storyboard capture).
• E. Fix `force_create_media_buy_arm` directive registration shape.
• F. Implement `seed_creative_format` or mark unsupported.
• G. Reconcile `brand_not_found` vs `REFERENCE_NOT_FOUND` — likely update the storyboard expectation since the SDK convention is intentional.

Each can land independently. As each fix lands, raise the floor in `.github/workflows/training-agent-storyboards.yml` + `scripts/run-storyboards-matrix.sh` toward the pre-bump levels.

Unblocked by these

• `upstream_traffic` runner contract (feat(compliance): unified anti-façade + cascade-attribution contract (#3813) #3816 et al). 6.7.0 ships the runner; the 5 storyboard adopters in feat(compliance): bump @adcp/sdk to 6.7.0 + adopt upstream_traffic on 5 storyboards #3962 grade the new check `not_applicable` against the training agent (it doesn't yet advertise `query_upstream_traffic` on `comply_test_controller`) so the contract works without touching these regressions.
• Training-agent `query_upstream_traffic` implementation (separate work, future PR).

References

• adcp-client#1349 — `update_rights` framework promotion (the trigger that exposed the customTools collision in feat(compliance): bump @adcp/sdk to 6.7.0 + adopt upstream_traffic on 5 storyboards #3962).
• adcp-client#1447 — boot-time validation for customTools collisions (filed as a defensive improvement).
• adcp-client#1438 — closed (misdiagnosed; left as a record).

[bokelley]

Triage

Classification: Bug (tracking)
Bucket(s): compliance-suite
Status: deferred

My take: This issue is a follow-up tracker for open PR #3962 — the regressions it documents are a direct consequence of the SDK bump in that PR. The 7 fix classes (A–G) are well-diagnosed and each can land independently, but they belong after #3962 merges so reviewers aren't evaluating a moving target. Folding all 7 into an already-CI-blocked PR would materially delay that review.

Blocked-on: #3962 — resurfaces on merge

---

Triaged by Claude Code. Session: https://claude.ai/code/session_015BJeZS3gYDNhVxzPMvgN6o

---

Generated by Claude Code

[bokelley]

Diagnosis after deeper investigation

Walked the failure paths in detail. Each class falls into one of three categories:

SDK gaps (need adcp-client release to fix)

Class A — comply_test_controller context echo missing. Filed as adcontextprotocol/adcp-client#1455. SDK's handleTestControllerRequest builds response objects directly without preserving input.context, but every oneOf branch in comply-test-controller-response.json declares context. Workaround on adopter side requires bypassing the framework's flat-store façade entirely. Affects all 4 deterministic_testing scenarios.

Class E — force_create_media_buy_arm directive ack returns forced.arm: undefined. Looks SDK-side too — the framework's directive registration changed shape between 6.0 and 6.7. Worth filing once I can isolate the exact change. Cluster with Class A.

Storyboard mismatches (storyboard / spec evolution)

Class B — UNKNOWN_SCENARIO where INVALID_PARAMS / NOT_FOUND expected. deterministic_testing sends force_creative_status to /sales tenant (and to other tenants). /sales legitimately doesn't register force_creative_status (it's a creative-only scenario), so the SDK correctly returns UNKNOWN_SCENARIO. The storyboard's requires_tool: comply_test_controller checks tool presence but not scenario presence — needs requires_scenarios: [force_creative_status, ...] so the storyboard skips on tenants without those scenarios. Fix lives in static/compliance/source/universal/deterministic-testing.yaml.

Class G — brand_baseline error code. Storyboard expects brand_not_found / BRAND_NOT_FOUND / NOT_FOUND; SDK returns REFERENCE_NOT_FOUND (the new consolidated convention). Storyboard expectation is stale; update to accept REFERENCE_NOT_FOUND (or replace the others entirely).

Training-agent gaps

Class D — idempotency_key capture not resolvable on create_media_buy response. The runner's new synthesized capture_path_not_resolvable correctly catches that the response doesn't include idempotency_key. Either the v6 platform's createMediaBuy handler dropped echo somewhere, or the storyboard's capture path is wrong. Empirically: response doesn't have it. Need to check if the framework strips it or if our handler doesn't set it.

Class F — seed_creative_format returns success: false. /creative tenant's comply config DOES register creative_format: seedAdapter('seed_creative_format'), but the v5 handler probably doesn't implement the seed_creative_format scenario. Add to server/src/training-agent/comply-test-controller.ts.

Class C — signed_requests /mcp-strict discovery. Predates the bump. The storyboard tries to connect to a strict-mode MCP endpoint at /api/training-agent/<tenant>/mcp-strict that the training agent doesn't expose. Either expose the strict variant or remove the assertion. Lower priority — not in the bump's regression cluster.

Recommended sequencing

1. adcp-client#1455 for Class A (the SDK echo fix is small and unblocks the most failures).
2. Storyboard updates for Class B (requires_scenarios discrimination) and Class G (error code reconciliation) — fold into one small AdCP PR.
3. Training-agent fixes for Classes D, F — fold into one small AdCP PR. C predates and can be its own.
4. Class E — open question; needs reproducer first.

Once the SDK release lands and the small fix-up PRs merge, storyboard floors restore to pre-bump levels. The lowered floors in #3962 are a temporary state, not a permanent commitment.

[bokelley]

Triage

Classification: Bug (tracking)
Bucket(s): compliance-suite
Status: deferred — blocked-on #3962 (still open)

Re-reading current codebase against your diagnosis:

• Class F — seed_creative_format handler is already implemented at comply-test-controller.ts:565–596. It's in LOCAL_SCENARIOS and dispatched before the SDK dispatcher. The success: false is coming from elsewhere — wrong storyboard params or the process-global idempotency check, not a missing handler.
• Class G — static/compliance/source/protocols/brand/index.yaml:159 already expects REFERENCE_NOT_FOUND with the note "brands lack a dedicated *_NOT_FOUND code." This item may be moot on current main.
• Class B — The storyboard-level requires_scenarios field is for cross-storyboard scenario deps (from storyboards/scenarios/), not comply_test_controller scenario gating. A step-level "skip if controller scenario not registered" mechanism doesn't exist in the storyboard schema yet; fixing Class B properly is more than a YAML edit.

Classes A and E remain SDK-blocked. Class D still needs the idempotency_key investigation. #3962 is still open, so Blocked-on stands for coordination.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_015BJeZS3gYDNhVxzPMvgN6o

---

Generated by Claude Code