Skip to content

feat: input_files shorthand in EVAL.yaml test cases #602

@christso

Description

@christso

Objective

Add input_files: as a shorthand at the test level that expands to type: file content blocks in the user message. This reduces boilerplate for evals that pass files as context.

Motivation

The current syntax for attaching a file to an eval input requires verbose content-block notation:

tests:
  - id: analyze-sales
    input:
      - role: user
        content:
          - type: file
            value: evals/files/sales.csv
          - type: text
            value: "Summarize the monthly trends in this CSV."

The proposed shorthand collapses this to:

tests:
  - id: analyze-sales
    input_files:
      - evals/files/sales.csv
    input: "Summarize the monthly trends in this CSV."

Proposed Expansion

At YAML parse time, when input_files: is present on a test case:

  1. Expand each path into a { type: file, value: <path> } content block.
  2. If input is a plain string, wrap it as { type: text, value: <input> }.
  3. Combine into a single user message with the file blocks first, text block last.
  4. Replace input with the expanded message array before passing to the provider.

This is a pure syntactic sugar transform — no provider or evaluator changes needed.

Architecture Boundary

YAML parser / loader layer (yaml-parser.ts or equivalent pre-processing step). No changes to the runner, providers, or evaluators.

Design Latitude

Implementer chooses:

  • Whether input_files: is allowed alongside multi-turn input arrays (or only with string input)
  • Error messaging when input_files: paths don't resolve
  • Whether relative paths are resolved from the YAML file location or the working directory (follow the existing convention for type: file paths)

Acceptance Signals

  • input_files: [path] with a string input produces identical runtime behaviour to the explicit type: file + type: text content-block form
  • agentv validate accepts the shorthand without errors
  • Existing explicit type: file content-block tests continue to pass unchanged
  • At least one example eval in examples/ uses the shorthand

Non-Goals

  • No changes to evals.json format (the transpiler in feat: create EVAL.yaml to evals.json transpiler #598 should expand input_files:files[] using the same logic)
  • No new provider or evaluator APIs
  • No support for input_files: on multi-turn input arrays in v1 (can be revisited)

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions