docs(compliance): operator index + skill § Validate-locally across all 8 skills#709
Merged
Merged
Conversation
bokelley
added a commit
that referenced
this pull request
Apr 21, 2026
New operator-facing docs/guides/VALIDATE-YOUR-AGENT.md: five-command checklist plus command-by-command references for storyboard runner, adcp fuzz (T1/T2/T3), adcp grade request-signing, multi-instance, webhook conformance, schema-driven validation hooks, custom --invariants, and SubstitutionEncoder/Observer. BUILD-AN-AGENT.md and root CLAUDE.md link to it so builders find the full surface without reading source. New scripts/manual-testing/skill-matrix.json + run-skill-matrix.ts driver, exposed as `npm run compliance:skill-matrix`. Fans out the existing agent-skill-storyboard harness across 14 skill × storyboard pairs (seller, brand-rights, creative, generative-seller, governance, retail-media, si, signals) with --filter, --parallel, --stop-on-first-fail. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-agent Replaces skill-specific § Validation sections (which varied in depth and didn't mention fuzz, webhook-receiver, multi-instance, substitution, --invariants, or the skill-matrix harness) with a consistent § Validate Locally block per skill. Each block points at docs/guides/VALIDATE-YOUR-AGENT.md for the full picture, then covers specialism-specific commands: canonical storyboard IDs, cross-cutting bundles (security_baseline, idempotency, schema_validation, error_compliance), rejection-surface fuzz with the right --tools list, and a decoder for the most common per-specialism failure modes. Skill-specific extras: - retail-media: SubstitutionEncoder.encode_for_url_context wiring for catalog-driven macro URLs; mentions substitution_encoding_violation / substitution_binding_missing grader failures. - seller: specialism bundle enumeration (sales_guaranteed, broadcast_tv, streaming_tv, social, etc.); signed-requests grader pointer; multi- instance pointer. - governance: --auto-seed fuzz (Tier 3 update-tool coverage); 3.0 GA authority_level → human_review_required reminder. Every skill also points at `npm run compliance:skill-matrix -- --filter <name>` as the port-less fallback — invokes the dogfood harness in an isolated workspace when local binding isn't available. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ontroller skill refs Matrix perf: - New `--shared-node-modules <path>` flag on `agent-skill-storyboard.ts` symlinks the scratch workspace's `node_modules` to a cached template instead of running `npm install` per pair. The matrix driver prepares the template once in `.cache/harness-template/` (~30s) and every subsequent pair skips install (~20s saved per pair). - `run-skill-matrix.ts` parallel default auto-picks `min(4, cpus/2)` — was `1`. 14-pair full matrix drops from ~34 min serial to ~9 min on a typical laptop. - Fast-fail up-front if the `claude` CLI isn't on PATH (saves every pair from hitting the same wall). - Summary line now prints wall, cpu, and speedup so regressions are obvious. - `--no-shared-install` escape hatch for debugging template drift. Skills (createComplyController is the new scaffold from #708): - build-seller-agent/SKILL.md § Compliance Testing now leads with `createComplyController` (adapter-based, handles dispatch + idempotency + sandbox gating). `registerTestController` demoted to the "low-level alternative" footnote. - build-creative-agent/SKILL.md gains § Deterministic Testing covering seed.creative + force.creative_status adapters, required for `creative_generative/seller` and `deterministic_testing` storyboards. - docs/guides/VALIDATE-YOUR-AGENT.md gains § Deterministic testing section pointing at `createComplyController` with the canonical adapter example. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
force-pushed
the
bokelley/compliance-docs-and-harness
branch
from
ab3e35e to
3ffd7d1
Compare
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the compliance-testing discoverability gap: the SDK has a mature surface (
adcp storyboard run,adcp fuzzT1/T2/T3,adcp grade request-signing,--webhook-receiver,--invariants, validation hooks,SubstitutionEncoder, multi-instance, dogfood harness) but skills barely referenced it and operators had to read source to find the right command.Two commits
1.
docs(validation): VALIDATE-YOUR-AGENT index + skill-matrix harnessdocs/guides/VALIDATE-YOUR-AGENT.md— five-command operator checklist plus deep command reference for storyboard runner,adcp fuzz(Tier 1/2/3), request-signing grader, multi-instance, webhook conformance, schema-driven validation, custom--invariants, andSubstitutionEncoder/Observer.BUILD-AN-AGENT.md § Compliance Checkand rootCLAUDE.mdlink to the new doc.scripts/manual-testing/skill-matrix.json— 14 skill × storyboard pairs.scripts/manual-testing/run-skill-matrix.ts— driver exposed asnpm run compliance:skill-matrix. Supports--filter,--parallel,--stop-on-first-fail,--keep-workspaces.2.
docs(skills): unified § Validate Locally section across all 8 build-*-agentAll 8 skills (seller, brand-rights, creative, generative-seller, governance, retail-media, si, signals) get a consistent § Validate Locally block pointing at
VALIDATE-YOUR-AGENT.md, then listing:security_baseline,idempotency,schema_validation,error_compliance)adcp fuzzwith the right--toolslistnpm run compliance:skill-matrix -- --filter <name>as the port-less fallbackSkill-specific extras: retail-media gets
SubstitutionEncoderwiring; seller gets specialism-bundle enumeration + signed-requests + multi-instance pointers; governance gets Tier-3--auto-seedfuzz + 3.0 GAauthority_level → human_review_requiredreminder.Test plan
npm run compliance:skill-matrix -- --help— worksnpm run compliance:skill-matrix -- --filter seller— correctly selects 5 pairsnpm run compliance:skill-matrix -- --filter nonexistentfilter— exits 2 with helpful message## Validate Locallyheading (no leftover## Validation)🤖 Generated with Claude Code