Skip to content

feat(evaluation): expose evaluationReferenceInputs on EvaluatorInput - #540

Merged
jariy17 merged 3 commits into
mainfrom
feat/evaluator-reference-inputs
Jun 23, 2026
Merged

feat(evaluation): expose evaluationReferenceInputs on EvaluatorInput#540
jariy17 merged 3 commits into
mainfrom
feat/evaluator-reference-inputs

Conversation

@jariy17

@jariy17 jariy17 commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Issue

Closes #539

Description of changes

The Lambda event for code-based evaluators carries fields that EvaluatorInput dropped, so evaluator functions had no typed access to them. This adds the missing ones (see input schema).

Ground truth (evaluationReferenceInputs)

  • A typed ReferenceInput model for a single reference entry (context, expected_response, assertions, expected_trajectory), with camelCase aliases matching the wire format and an expected_response_text convenience property.
  • A reference_inputs: List[ReferenceInput] field on EvaluatorInput, populated by the decorator from event["evaluationReferenceInputs"].
  • ReferenceInput is exported from both custom_code_based_evaluators and the top-level evaluation package.

Evaluator identity (evaluatorId / evaluatorName)

  • Optional evaluator_id / evaluator_name fields on EvaluatorInput, mapped by the decorator. Lets an evaluator log or branch on which evaluator config invoked it.

Usage

from bedrock_agentcore.evaluation import (
    custom_code_based_evaluator, EvaluatorInput, EvaluatorOutput,
)

@custom_code_based_evaluator()
def exact_match(inp: EvaluatorInput, context) -> EvaluatorOutput:
    expected = next(
        (r.expected_response_text for r in inp.reference_inputs if r.expected_response_text),
        None,
    )
    if expected is None:
        return EvaluatorOutput(label="Skip", explanation="No ground truth provided")
    ...

Notes

  • Backward compatible — all new fields default to empty/None; existing evaluators are unaffected.
  • expected_response is modeled as an object ({"text": "..."}), not a bare string, matching the contract. The expected_response_text property encodes this so callers don't misread it.
  • ReferenceInput uses extra="allow", so unknown/future keys in a reference entry are preserved (reachable via ref.model_extra) instead of dropped.
  • Naming: the package already exports ReferenceInputs (plural, client-side ground-truth input). This adds ReferenceInput (singular, the parsed evaluator-side entry) — opposite ends of the same contract.

Testing

uv run pytest tests/bedrock_agentcore/evaluation/custom_code_based_evaluators/ — 31 passed. New tests cover decorator pass-through (reference inputs + evaluator id/name), default-empty/None, alias coercion, extra-key preservation, and the expected_response_text property. ruff check and ruff format --check clean.

Code-based evaluators now receive ground-truth reference inputs in the
Lambda event's evaluationReferenceInputs field, but EvaluatorInput dropped
it. Add a typed ReferenceInput model and a reference_inputs list on
EvaluatorInput, populated by the @custom_code_based_evaluator decorator.

Backward compatible: existing evaluators ignore the new field.

Closes #539
@jariy17
jariy17 requested a review from a team June 18, 2026 19:24
@github-actions

github-actions Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

✅ No Breaking Changes Detected

No public API breaking changes found in this PR.

@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: the review did not analyze this PR (model took 0 turns). See the run for details; a later push or re-run is needed.

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026
…uatorInput

The Lambda event carries evaluatorId and evaluatorName (top-level, per the
code-based-evaluator input schema) but EvaluatorInput dropped both. Map them
to optional evaluator_id/evaluator_name fields so evaluators can log or
branch on which evaluator config invoked them.
@agentcore-devx-automation agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: the review did not analyze this PR (model took 0 turns). See the run for details; a later push or re-run is needed.

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 18, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: the review did not analyze this PR (model took 0 turns). See the run for details; a later push or re-run is needed.

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Jun 23, 2026
@jariy17
jariy17 merged commit ff55a2a into main Jun 23, 2026
39 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose evaluationReferenceInputs (ground truth) on EvaluatorInput for code-based evaluators

2 participants