Skip to content

feat(conformance): structured StoryboardStepHint taxonomy — strict + monotonic kinds (#935)#959

Merged
bokelley merged 4 commits into
mainfrom
bokelley/pr937-followup
Apr 25, 2026
Merged

feat(conformance): structured StoryboardStepHint taxonomy — strict + monotonic kinds (#935)#959
bokelley merged 4 commits into
mainfrom
bokelley/pr937-followup

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Supersedes #937. Closes #935.

#937 implemented the first hint kind (shape_drift) but only the first part of issue #935's vision. This PR closes the loop:

Why #937 didn't go far enough

#935 proposed making StoryboardStepHint the canonical surface for every runner-side diagnostic with structured fields. #937 added ShapeDriftHint in parallel to the existing ValidationResult.warning prose (so the same diagnosis fires twice), and no consumer in the SDK actually read the new structured fields. The taxonomy was added; the contract wasn't.

What this PR does

1. Base interface + four new kinds. StoryboardStepHintBase { kind, message } constrains every hint. Union now includes ShapeDriftHint, MissingRequiredFieldHint, FormatMismatchHint, and MonotonicViolationHint alongside ContextValueRejectedHint. auth_misconfiguration is deferred — #766 is detection-net-new, not a wrap-existing-prose task.

2. De-dup shape-drift. Detection moved to shape-drift-hints.ts as the canonical surface; the legacy detectShapeDriftHint (string) in validations.ts delegates to it so the two can't drift. Removed the shape-drift prose from ValidationResult.warning — it lives only on step.hints[] going forward.

3. Strict-AJV findings as structured hints. strict-validation-hints.ts partitions strict issues:

  • keyword: required → grouped by parent instance_path into one MissingRequiredFieldHint per path
  • everything else → one FormatMismatchHint per issue, capped at 5

The strict prose in ValidationResult.warning is kept for one minor for back-compat with consumers that scrape it (existing storyboard-strict-validation.test.js expectations preserved).

4. Assertion → hint plumbing. AssertionResult gained an optional hint?: StoryboardStepHint that the runner mirrors into the owning step's hints[] for scope: \"step\" results. First user: status.monotonic emits a MonotonicViolationHint with resource_type, resource_id, from_status, to_status, from_step_id, legal_next_states[], enum_url.

5. CLI renders structured fields. bin/adcp-step-hints.js branches on hint.kind and prints per-kind detail lines under each prose hint:

   💡 Hint: media_buy mb-1: active → pending_creatives is not in the lifecycle graph...
            media_buy mb-1: active → pending_creatives
            from step: create
            legal next states: canceled, completed, paused

Unknown kind literals fall back to prose only — forward-compat per StoryboardStepHintBase.

Wire compatibility

  • Adding union members is non-breaking; existing consumers that only render message keep working.
  • Shape-drift content removed from ValidationResult.warning. Consumers scraping that surface specifically for shape-drift recipes need to switch to step.hints[]. The strict-AJV warning content stays one minor for back-compat (called out in the changeset).
  • No spec-level change. StoryboardStepHint is a runner-internal contract; the structured fields mirror existing taxonomies (SchemaValidationError.instance_path / RFC 6901, enums/*-status.json URLs).

Tests

  • test/lib/storyboard-shape-drift-hints.test.js — new, covers the structured ShapeDriftHint shape per branch
  • test/lib/storyboard-strict-validation-hints.test.js — new, covers the required-grouping + format-cap + schema_url passthrough
  • test/lib/storyboard-default-invariants.test.js — extended with three new tests for MonotonicViolationHint (forward / terminal-state / silent-on-pass)
  • test/lib/cli-step-hints.test.js — updated to expect prose + per-kind structured detail lines, plus a new suite covering each of the four new hint kinds and the unknown-kind forward-compat fallback

All 5,898 unit tests pass; tsc --project tsconfig.lib.json is clean; prettier --check passes.

Test plan

  • npm run build (tsc + version sync)
  • npm test (full suite — 5898 pass, 6 skipped, 0 fail)
  • npx prettier --check (clean across touched files)
  • Reviewer: spot-check step.hints[] content on a real conformance run by pointing the CLI at an agent that exercises shape drift / strict-AJV findings (any sample seller agent + adcp storyboard run)

🤖 Generated with Claude Code

… + monotonic kinds (#935)

Issue #935 proposed making `StoryboardStepHint` the canonical surface for
every runner-side diagnostic with structured fields. PR #937 shipped the
first member (`shape_drift`) but in parallel to the existing prose
`ValidationResult.warning`, and no consumer rendered the structured fields.

This change supersedes #937 and closes the broader vision:

- `StoryboardStepHintBase { kind, message }` constrains every hint
- Hint union extended to include `MissingRequiredFieldHint`,
  `FormatMismatchHint`, `MonotonicViolationHint` alongside the existing
  `ContextValueRejectedHint` and the new `ShapeDriftHint`
- Shape-drift detection moved to `shape-drift-hints.ts` as the canonical
  module; the legacy string `detectShapeDriftHint` delegates to it so the
  two surfaces can't drift apart. Redundant shape-drift prose removed
  from `ValidationResult.warning`
- Strict-AJV `response_schema` issues partition into structured hints via
  `strict-validation-hints.ts` (required-fields grouped by parent path,
  format/pattern/etc. emitted per-issue with a 5-hint cap)
- `AssertionResult.hint` lets cross-step assertions attach a structured
  hint that the runner mirrors into the owning step's `hints[]`. First
  user: `status.monotonic` emits a `MonotonicViolationHint` carrying
  resource_type / resource_id / from_status / to_status / from_step_id /
  legal_next_states / enum_url
- `bin/adcp-step-hints.js` branches on `hint.kind` to render per-kind
  detail lines under each prose hint; unknown kinds fall back to prose
  only (forward-compat)

Closes #935; supersedes #937.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley changed the title feat(conformance): close #935 vision — strict-AJV + monotonic hints, dedup with warning, CLI structured rendering feat(conformance): structured StoryboardStepHint taxonomy — strict + monotonic kinds (#935) Apr 25, 2026
bokelley and others added 2 commits April 25, 2026 07:45
…eview feedback)

- strict-validation-hints: emit a sentinel `format_mismatch` hint with
  `keyword: 'truncated'` when the per-validation cap (5 hints) trips, so
  the per-step surface doesn't silently disagree with
  `strict_validation_summary`. Renderers can branch on the sentinel kind
  to flag the run-level summary as authoritative.
- runner: comment the intentional shape_drift + format_mismatch
  co-emission for the same root cause (e.g. bare-array list responses).

Addresses code-reviewer + ad-tech-protocol-expert non-blocking suggestions
on #959.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hint taxonomy: extend StoryboardStepHint beyond context_value_rejected

1 participant