Skip to content

feat(conformance): add FormatMismatchHint to StoryboardStepHint taxonomy#956

Closed
bokelley wants to merge 2 commits into
mainfrom
claude/issue-947-format-mismatch-hint
Closed

feat(conformance): add FormatMismatchHint to StoryboardStepHint taxonomy#956
bokelley wants to merge 2 commits into
mainfrom
claude/issue-947-format-mismatch-hint

Conversation

@bokelley

@bokelley bokelley commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

Closes #947

Summary

  • Extends StoryboardStepHint from ContextValueRejectedHint to ContextValueRejectedHint | FormatMismatchHint
  • Adds FormatMismatchHint interface with: kind, message, tool, instance_path (RFC 6901 pointer — renamed from issue's field_path to match SchemaValidationError.instance_path convention), expected_format, and optional observed_value
  • New pure detection module format-mismatch-hints.ts (parallel to rejection-hints.ts): fires only on strict_only_failure validations — lenient Zod passed, strict AJV rejected on keyword === 'format'
  • Runner wires detectFormatMismatchHints as a separate parallel pass outside the stepFailed guard, since format hints are most valuable when the step is green (silent strictness gap)
  • Exports FormatMismatchHint and detectFormatMismatchHints from the public storyboard and testing indexes
  • 13 unit tests covering: gate logic, pointer decoding (RFC 6901 ~0/~1 escapes, array indices), observed_value extraction and truncation, expected_format regex fallback, edge cases

What's not changing

  • Pass/fail is unaffected — hints[] stays non-fatal
  • ValidationResult.warning string still fires (unchanged) — the hint is additive
  • No protocol-side change; this is a runner diagnostic contract
  • Existing detectContextRejectionHints gate (stepFailed) is unchanged

Implementation notes

  • instance_path: renamed from field_path (issue spec) to instance_path to align with SchemaValidationError.instance_path — this type is a filtered projection of that type
  • expected_format: parsed from AJV's stable message template must match format "<name>" via regex; falls back to '(unknown)' on mismatch (avoids confusing must match format "format" when using the keyword as fallback)
  • observed_value: resolved by converting RFC 6901 pointer → dot-bracket notation, then resolvePath(). Strings only, truncated at 200 chars via code-point slicing. Non-string values (objects, arrays, numbers) omitted
  • No stepFailed guard needed: skipped/probe/webhook-assertion steps have empty validations[] so no hints emit

Nits noted (not fixed)

  • pointerToDotPath in format-mismatch-hints.ts duplicates normalizeFieldPath in rejection-hints.ts. Both could be extracted to path.ts. Deferring to keep this PR scoped.

What was tested

  • TypeScript compiles clean (pre-existing @types/node + moduleResolution env errors only — same as PR feat(conformance): add ShapeDriftHint to StoryboardStepHint taxonomy #937)
  • prettier --check passes
  • 13/13 new unit tests pass (node --test test/lib/storyboard-format-mismatch-hints.test.js)
  • Baseline test pass/fail count unchanged vs main (496 failures, all pre-existing dist/build-artifact issues)

Pre-PR review

  • ad-tech-protocol-expert: approved — 1 blocker fixed (field_pathinstance_path); nit on pointerToDotPath duplication noted above; no protocol change confirmed; minor changeset classification correct
  • code-reviewer: approved — no blockers; fixed misleading (unknown) fallback and added guard comment; added 13 unit tests covering all non-trivial cases flagged in review

Session: https://claude.ai/code/session_01DuuYnsXwSs3znAi9b3bbHH

- Fix misleading expected_format fallback: 'format' → '(unknown)' when
  AJV message template doesn't match (avoids "must match format 'format'")
- Add comment explaining why no stepFailed guard is needed for format hints
- Add RFC 6901 §3 decode-order comment in pointerToDotPath
- Add unit tests for detectFormatMismatchHints (13 cases: gate, extraction,
  pointer decoding, truncation, fallback, edge cases)

https://claude.ai/code/session_01DuuYnsXwSs3znAi9b3bbHH
@bokelley

Copy link
Copy Markdown
Contributor Author

Superseded by #959 — landed there via strict-AJV partitioning in .

@bokelley bokelley closed this Apr 25, 2026
@bokelley

Copy link
Copy Markdown
Contributor Author

Superseded by #959FormatMismatchHint landed there via strict-AJV partitioning in strict-validation-hints.ts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(conformance): add FormatMismatchHint to StoryboardStepHint taxonomy

2 participants