Skip to content

feat(yaml-parser): add input_files shorthand to EVAL.yaml test cases#605

Merged
christso merged 1 commit intomainfrom
feat/602-input-files-shorthand
Mar 15, 2026
Merged

feat(yaml-parser): add input_files shorthand to EVAL.yaml test cases#605
christso merged 1 commit intomainfrom
feat/602-input-files-shorthand

Conversation

@christso
Copy link
Copy Markdown
Collaborator

Summary

  • Adds input_files: as a shorthand at the test level in EVAL.yaml that expands to type: file content blocks prepended before a type: text block in the user message
  • Implementation lives entirely in the parser/loader layer (shorthand-expansion.ts) — no provider or evaluator changes
  • Only supported with a string input in v1; multi-turn array inputs return undefined (no expansion)
  • Paths are resolved using the same convention as existing explicit type: file blocks

Expansion rule:

# shorthand
input_files:
  - evals/files/sales.csv
input: "Summarize the monthly trends in this CSV."

expands to:

input:
  - role: user
    content:
      - type: file
        value: evals/files/sales.csv
      - type: text
        value: "Summarize the monthly trends in this CSV."

Changes

  • packages/core/src/evaluation/loaders/shorthand-expansion.ts — new expandInputFilesShorthand() function; updated resolveInputMessages() to check input_files first
  • packages/core/src/evaluation/yaml-parser.ts — added input_files?: JsonValue to RawEvalCase type
  • packages/core/test/evaluation/loaders/shorthand-expansion.test.ts — 8 new unit tests for expandInputFilesShorthand, 4 new tests for resolveInputMessages with input_files
  • packages/core/test/evaluation/input-files-shorthand.test.ts — 4 new yaml-parser integration tests (including parity test between shorthand and explicit form)
  • examples/features/input-files-shorthand/ — new example eval demonstrating single-file, multi-file, and explicit-form equivalence

Risk

Low — purely additive parser change. No existing behavior changes. All 1300 tests pass.

closes #602

Adds `input_files:` as a shorthand at the test level that expands to
type:file content blocks prepended before the type:text block in the
user message. Only supported with a string `input` in v1.

Expansion rule (file blocks first, text block last):

  input_files:
    - path/to/file.csv
  input: "Summarize this."

expands to:

  input:
    - role: user
      content:
        - type: file
          value: path/to/file.csv
        - type: text
          value: "Summarize this."

Paths are resolved using the same convention as explicit type:file blocks.
Multi-turn array inputs are not supported in v1 (returns undefined).
@christso christso merged commit e661eb2 into main Mar 15, 2026
1 check was pending
@christso christso deleted the feat/602-input-files-shorthand branch March 15, 2026 04:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: input_files shorthand in EVAL.yaml test cases

1 participant