refactor: rename assert: to assertions: in EVAL.yaml schema (#603)#604
Merged
refactor: rename assert: to assertions: in EVAL.yaml schema (#603)#604
Conversation
Rename the YAML key `assert:` to `assertions:` across the codebase. The old `assert:` key is preserved as a deprecated backward-compatible alias with a console warning when used at the suite level. Changes: - eval-file.schema.ts: add `assertions:` as primary field on EvalFileSchema - yaml-parser.ts: read `assertions ?? assert` for suite/test-level fields; emit deprecation warning for old `assert` key - eval-validator.ts: validate `assertions ?? assert`; update error message locations to reference `assertions` - evaluator-parser.ts: update warning messages to reference `assertions` - All examples/*.yaml: rename `assert:` → `assertions:` - All test fixtures: update YAML strings and primary object keys - All docs/README/skills: update YAML code snippets - Regenerate eval-schema.json to reflect updated Zod schema
christso
added a commit
that referenced
this pull request
Mar 15, 2026
Align with the global rename from PR #604.
christso
added a commit
that referenced
this pull request
Mar 15, 2026
…ator (#597) * feat(providers,evaluators): add claude-cli provider and trigger-judge evaluator (#593) - Add ClaudeCliProvider that spawns `claude -p` as a subprocess, streams output via --output-format stream-json --include-partial-messages, and extracts tool calls, token usage, and cost from stream events - Rename existing SDK provider class to ClaudeSdkProvider (claude-sdk.ts) with kind 'claude-sdk' for explicit opt-in to the Agent SDK path - Register 'claude' and 'claude-cli' as aliases for ClaudeCliProvider; 'claude-sdk' maps to ClaudeSdkProvider - Add 'claude-cli' and 'claude-sdk' to ProviderKind, AGENT_PROVIDER_KINDS, KNOWN_PROVIDERS, and ResolvedTarget union - Add TriggerJudgeEvaluator that checks whether the agent invoked a named skill by scanning tool calls for Skill invocations (args.skill match) or skill file reads (.claude/commands/, .claude/skills/) - Register trigger-judge in evaluator parser, schema, builtin registry, and EvaluatorConfig union - Regenerate eval-schema.json to include trigger-judge schema - Add unit tests for trigger-judge evaluator and claude provider aliases * fix(providers): guard stdio access for null safety in claude-cli provider * fix(providers): add --verbose flag to claude-cli subprocess invocation --output-format stream-json requires --verbose when using -p (--print) mode. Without it the CLI exits with code 1 immediately. Also adds E2E tests validating output, tokenUsage, durationMs, and log file emission parity between claude-cli and claude-sdk providers. * refactor(evaluators): move trigger-judge from built-in to .agentv/judges/ example Removes TriggerJudgeEvaluator from core built-ins (violates Principles 1 & 2: Claude-Code-specific, expressible as a code-judge script) and adds: - packages/core/src/evaluation/registry/judge-discovery.ts: new discoverJudges() function, mirroring discoverAssertions() but scans .agentv/judges/ - Wired discoverJudges into orchestrator alongside discoverAssertions - Exported discoverJudges from core public API and registry/index.ts - examples/features/agent-skills-evals/.agentv/judges/trigger-judge.ts: reference implementation as a code-judge script using defineCodeJudge - Regenerated eval-schema.json (trigger-judge removed from EvaluatorSchema union) * style: format eval-schema.json with biome * docs: use assert instead of evaluators in trigger-judge example comment * refactor(judges): align trigger-judge detection with skill-creator run_eval.py * docs(judges): update trigger-judge example comment assert: → assertions: Align with the global rename from PR #604.
christso
added a commit
that referenced
this pull request
Mar 17, 2026
Aligns documentation with the assert: → assertions: YAML key rename completed in PR #604. Updates prose references, YAML examples, table entries, SDK code samples, skill docs, and agent prompts across 17 files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
christso
added a commit
that referenced
this pull request
Mar 17, 2026
Aligns documentation with the assert: → assertions: YAML key rename completed in PR #604. Updates prose references, YAML examples, table entries, SDK code samples, skill docs, and agent prompts across 17 files. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
assert:→assertions:in EVAL.yaml files, schema, parser, validator, tests, and all docsassert:as a deprecated backward-compatible alias: the parser still accepts it and emits a console warning when used at the suite leveleval-schema.jsonto reflect the updated Zod schemaFiles changed
eval-file.schema.ts): addsassertions:as primary field onEvalFileSchema, keepsassert:as@deprecatedaliasyaml-parser.ts): updatesRawTestSuiteandRawEvalCasetype definitions; readsassertions ?? assert; emits deprecation warning for old keyevaluator-parser.ts): updates warning messages to referenceassertionseval-validator.ts): validatesassertions ?? assert; updates error message locationsassert:→assertions:eval-validator.test.ts,evaluator-parser.test.ts,criteria-optional.test.ts): primary tests updated to useassertions:, backward-compat test added forassert:README.md,apps/cli/README.md,apps/web/, plugins SKILL.md files, examples READMEs): all YAML code snippets updatedRisk
Low — pure rename with backward-compatible fallback. No logic changed. All CI checks pass.
Closes #603