feat(conformance): structured StoryboardStepHint taxonomy — strict + monotonic kinds (#935)#959
Merged
Merged
Conversation
… + monotonic kinds (#935) Issue #935 proposed making `StoryboardStepHint` the canonical surface for every runner-side diagnostic with structured fields. PR #937 shipped the first member (`shape_drift`) but in parallel to the existing prose `ValidationResult.warning`, and no consumer rendered the structured fields. This change supersedes #937 and closes the broader vision: - `StoryboardStepHintBase { kind, message }` constrains every hint - Hint union extended to include `MissingRequiredFieldHint`, `FormatMismatchHint`, `MonotonicViolationHint` alongside the existing `ContextValueRejectedHint` and the new `ShapeDriftHint` - Shape-drift detection moved to `shape-drift-hints.ts` as the canonical module; the legacy string `detectShapeDriftHint` delegates to it so the two surfaces can't drift apart. Redundant shape-drift prose removed from `ValidationResult.warning` - Strict-AJV `response_schema` issues partition into structured hints via `strict-validation-hints.ts` (required-fields grouped by parent path, format/pattern/etc. emitted per-issue with a 5-hint cap) - `AssertionResult.hint` lets cross-step assertions attach a structured hint that the runner mirrors into the owning step's `hints[]`. First user: `status.monotonic` emits a `MonotonicViolationHint` carrying resource_type / resource_id / from_status / to_status / from_step_id / legal_next_states / enum_url - `bin/adcp-step-hints.js` branches on `hint.kind` to render per-kind detail lines under each prose hint; unknown kinds fall back to prose only (forward-compat) Closes #935; supersedes #937. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts: # src/lib/version.ts
…eview feedback) - strict-validation-hints: emit a sentinel `format_mismatch` hint with `keyword: 'truncated'` when the per-validation cap (5 hints) trips, so the per-step surface doesn't silently disagree with `strict_validation_summary`. Renderers can branch on the sentinel kind to flag the run-level summary as authoritative. - runner: comment the intentional shape_drift + format_mismatch co-emission for the same root cause (e.g. bare-array list responses). Addresses code-reviewer + ad-tech-protocol-expert non-blocking suggestions on #959. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #937. Closes #935.
#937 implemented the first hint kind (
shape_drift) but only the first part of issue #935's vision. This PR closes the loop:Why #937 didn't go far enough
#935 proposed making
StoryboardStepHintthe canonical surface for every runner-side diagnostic with structured fields. #937 addedShapeDriftHintin parallel to the existingValidationResult.warningprose (so the same diagnosis fires twice), and no consumer in the SDK actually read the new structured fields. The taxonomy was added; the contract wasn't.What this PR does
1. Base interface + four new kinds.
StoryboardStepHintBase { kind, message }constrains every hint. Union now includesShapeDriftHint,MissingRequiredFieldHint,FormatMismatchHint, andMonotonicViolationHintalongsideContextValueRejectedHint.auth_misconfigurationis deferred — #766 is detection-net-new, not a wrap-existing-prose task.2. De-dup shape-drift. Detection moved to
shape-drift-hints.tsas the canonical surface; the legacydetectShapeDriftHint(string) invalidations.tsdelegates to it so the two can't drift. Removed the shape-drift prose fromValidationResult.warning— it lives only onstep.hints[]going forward.3. Strict-AJV findings as structured hints.
strict-validation-hints.tspartitions strict issues:keyword: required→ grouped by parentinstance_pathinto oneMissingRequiredFieldHintper pathFormatMismatchHintper issue, capped at 5The strict prose in
ValidationResult.warningis kept for one minor for back-compat with consumers that scrape it (existingstoryboard-strict-validation.test.jsexpectations preserved).4. Assertion → hint plumbing.
AssertionResultgained an optionalhint?: StoryboardStepHintthat the runner mirrors into the owning step'shints[]forscope: \"step\"results. First user:status.monotonicemits aMonotonicViolationHintwithresource_type,resource_id,from_status,to_status,from_step_id,legal_next_states[],enum_url.5. CLI renders structured fields.
bin/adcp-step-hints.jsbranches onhint.kindand prints per-kind detail lines under each prose hint:Unknown
kindliterals fall back to prose only — forward-compat perStoryboardStepHintBase.Wire compatibility
messagekeep working.ValidationResult.warning. Consumers scraping that surface specifically for shape-drift recipes need to switch tostep.hints[]. The strict-AJVwarningcontent stays one minor for back-compat (called out in the changeset).StoryboardStepHintis a runner-internal contract; the structured fields mirror existing taxonomies (SchemaValidationError.instance_path/ RFC 6901,enums/*-status.jsonURLs).Tests
test/lib/storyboard-shape-drift-hints.test.js— new, covers the structuredShapeDriftHintshape per branchtest/lib/storyboard-strict-validation-hints.test.js— new, covers the required-grouping + format-cap + schema_url passthroughtest/lib/storyboard-default-invariants.test.js— extended with three new tests forMonotonicViolationHint(forward / terminal-state / silent-on-pass)test/lib/cli-step-hints.test.js— updated to expect prose + per-kind structured detail lines, plus a new suite covering each of the four new hint kinds and the unknown-kind forward-compat fallbackAll 5,898 unit tests pass;
tsc --project tsconfig.lib.jsonis clean;prettier --checkpasses.Test plan
npm run build(tsc + version sync)npm test(full suite — 5898 pass, 6 skipped, 0 fail)npx prettier --check(clean across touched files)step.hints[]content on a real conformance run by pointing the CLI at an agent that exercises shape drift / strict-AJV findings (any sample seller agent +adcp storyboard run)🤖 Generated with Claude Code