Skip to content

docs + lint(storyboards): test-kit flavors + bimodal-partition guard (closes #2721)#2747

Merged
bokelley merged 1 commit into
mainfrom
bokelley/kit-shapes-docs
Apr 21, 2026
Merged

docs + lint(storyboards): test-kit flavors + bimodal-partition guard (closes #2721)#2747
bokelley merged 1 commit into
mainfrom
bokelley/kit-shapes-docs

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

Closes #2721 — filed from protocol review of PR #2723 (the resolved-auth fingerprint shape).

The investigation

#2721 worried that three runner-style test kits (`signed-requests-runner`, `substitution-observer-runner`, `webhook-receiver-runner`) have no `auth:` block and could produce ambiguous fingerprints if a storyboard composed them with a brand kit. The issue offered three schema changes (A single-kit + separate `coordination_runner:` field, B array form of `prerequisites.test_kit`, C distinct top-level `coordination:` field).

None of the three are needed. The composition pattern already exists via step-level `requires_contract:` — used today by:

  • `specialisms/sales-catalog-driven/index.yaml` — `prerequisites.test_kit: acme-outdoor.yaml` + `requires_contract: substitution_observer_runner` on `expect_substitution_safe` steps
  • `specialisms/creative-generative/index.yaml` — same pattern

Pure-harness storyboards (`specialisms/signed-requests/index.yaml`, `universal/webhook-emission.yaml`) point `prerequisites.test_kit` directly at the runner contract and use `task_default:` when `$test_kit.operations.` refs resolve to null.

`test_kit=` in the contradiction-lint fingerprint separately discriminates brand kits from runner contracts, so `auth=kit_default` never collides across flavors.

What ships

Docs (primary):

  • Adds a "Test kit flavors" section to `static/compliance/source/universal/storyboard-schema.yaml` — formalizes the bimodal partition (brand kit vs runner contract), names today's 5 brand kits and 3 runner contracts, documents the `requires_contract:` composition seam with the sales-catalog-driven example, and notes two future cases: a branded runner is a legitimate shape (so "no credentials" is temporal, not structural), and two-runner-contract composition would need `requires_contract:` to accept a list.
  • Clarifies the `prerequisites.test_kit` field description to explicitly state the single-path invariant.

Lint (enforcement, ~70 lines):

  • `scripts/lint-storyboard-test-kits.cjs` — fails the build when a kit file declares neither `auth.api_key` nor `applies_to`. Tolerates both (future-branded-runner). `classify(doc)` is exported for testability; `lint(dir = TEST_KITS_DIR)` accepts an override so the synthetic-tree test can run without touching real source.
  • Wired into `build:compliance` after the auth-shape lint, and into the aggregate `npm test` script alongside its siblings.
  • `tests/lint-storyboard-test-kits.test.cjs` — 9 tests: source-tree guard, classify branch coverage (brand-only, runner-only, hybrid, unclassified, null/non-object defensive, probe_task-without-api_key), synthetic-tree end-to-end aggregation with `{file, rule}` record-shape check, and error-message docs-pointer anchor.

Also fixes #2721's "verify osei-natural auth" note: it does have auth (`demo-osei-natural-v1`).

Expert review

Two rounds pre-push:

  • Docs expert: ship with two tightenings (applied). Trimmed the 6-line fingerprint-justification paragraph to one sentence; dropped a forward-reference that pointed at Step-section docs that don't exist (the `requires_contract:` field is task-scoped, not Step-scoped) and pointed readers at the actual task definitions instead.
  • Ad-tech protocol expert: docs-only would be "advisory" without enforcement — pushed for the small lint that pins the invariant mechanically. Applied. Also applied the "runner contracts carry no credentials today" softening (temporal, not structural) and the one-line future note about list-form `requires_contract:`.
  • Code reviewer (second pass on the lint): ship with one JSDoc nit (applied). Documented the deliberate asymmetry between `hasApiKey` (requires string `auth.api_key` — distinguishes a complete brand kit from an incomplete one with only `probe_task`) and `hasAppliesTo` (presence check — `applies_to:` is already polymorphic across runner contracts).

One bug found + fixed during test authoring: the initial `classify()` used `doc && typeof doc === 'object' && ...` chains that returned `undefined` on short-circuit instead of `false`. Three tests failed on strict equality. Rewrote with an `isObject` gate and explicit boolean returns.

Test plan

  • `npm run test:storyboard-test-kits` — 9/9 pass
  • `npm run build:compliance` — all six storyboard lints green (scoping, branch-sets, contradictions, context-entity, auth-shape, test-kits), 56 storyboards build
  • `npm run test:unit` — 631/631 pass (precommit)
  • `npm run typecheck` — clean (precommit)

🤖 Generated with Claude Code

closes #2721)

#2721 worried that three runner-style test kits (signed-requests,
substitution-observer, webhook-receiver) have no auth: block and
could produce ambiguous fingerprints if a storyboard composed them
with a brand kit. The issue offered three schema changes (A, B, C).

Investigation: none of the three are needed. The composition
pattern already exists via step-level requires_contract: — used
today by sales-catalog-driven and creative-generative to declare
a brand kit in prerequisites.test_kit AND opt into the substitution-
observer runner contract on specific expect_substitution_safe steps.
Pure-harness storyboards (signed-requests/index, webhook-emission)
point prerequisites.test_kit directly at the runner contract and
use task_default: when $test_kit.operations.<name> resolves to
null. `test_kit=<path>` in the contradiction-lint fingerprint
separately discriminates brand kits from runner contracts, so
auth=kit_default never collides across flavors.

This PR:

- Adds "Test kit flavors" section to storyboard-schema.yaml
  formalizing the bimodal partition, naming today's 5 brand kits
  and 3 runner contracts, documenting the requires_contract:
  composition seam, softening "runner contracts carry no
  credentials" to "carry no credentials today" (temporal, not
  structural — future branded runner is a legitimate shape), and
  noting that two-runner-contract composition would need
  requires_contract: to become a list.

- Clarifies prerequisites.test_kit field description to explicitly
  state the single-path invariant.

- Adds scripts/lint-storyboard-test-kits.cjs — a small enforcement
  lint that fails the build when a kit file declares neither
  auth.api_key nor applies_to. Tolerates both (future-branded-runner
  shape). Parameterized lint(dir) for synthetic-tree testing. Nine
  tests cover classify branch asymmetry (string-shape vs presence),
  hybrid tolerance, defensive null/non-object input, and synthetic-
  tree end-to-end aggregation.

- Wires the lint into build:compliance after auth-shape, and into
  the aggregate npm test script.

Fixes #2721's note that osei-natural "may have auth — verify": it
does (demo-osei-natural-v1).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley merged commit fd884b2 into main Apr 21, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Storyboard schema: runner test kits without auth: block — define composition model

1 participant