-
Notifications
You must be signed in to change notification settings - Fork 0
feat: input_files shorthand in EVAL.yaml test cases #602
Copy link
Copy link
Closed
Description
Objective
Add input_files: as a shorthand at the test level that expands to type: file content blocks in the user message. This reduces boilerplate for evals that pass files as context.
Motivation
The current syntax for attaching a file to an eval input requires verbose content-block notation:
tests:
- id: analyze-sales
input:
- role: user
content:
- type: file
value: evals/files/sales.csv
- type: text
value: "Summarize the monthly trends in this CSV."The proposed shorthand collapses this to:
tests:
- id: analyze-sales
input_files:
- evals/files/sales.csv
input: "Summarize the monthly trends in this CSV."Proposed Expansion
At YAML parse time, when input_files: is present on a test case:
- Expand each path into a
{ type: file, value: <path> }content block. - If
inputis a plain string, wrap it as{ type: text, value: <input> }. - Combine into a single user message with the file blocks first, text block last.
- Replace
inputwith the expanded message array before passing to the provider.
This is a pure syntactic sugar transform — no provider or evaluator changes needed.
Architecture Boundary
YAML parser / loader layer (yaml-parser.ts or equivalent pre-processing step). No changes to the runner, providers, or evaluators.
Design Latitude
Implementer chooses:
- Whether
input_files:is allowed alongside multi-turninputarrays (or only with stringinput) - Error messaging when
input_files:paths don't resolve - Whether relative paths are resolved from the YAML file location or the working directory (follow the existing convention for
type: filepaths)
Acceptance Signals
input_files: [path]with a stringinputproduces identical runtime behaviour to the explicittype: file+type: textcontent-block formagentv validateaccepts the shorthand without errors- Existing explicit
type: filecontent-block tests continue to pass unchanged - At least one example eval in
examples/uses the shorthand
Non-Goals
- No changes to evals.json format (the transpiler in feat: create EVAL.yaml to evals.json transpiler #598 should expand
input_files:→files[]using the same logic) - No new provider or evaluator APIs
- No support for
input_files:on multi-turninputarrays in v1 (can be revisited)
Related
- feat: create EVAL.yaml to evals.json transpiler #598 — EVAL.yaml → evals.json transpiler (will need to handle
input_files:expansion consistently)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels