fix(agent-eval): render targets from the task instruction (one canonical input) - #700
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (7)
🚧 Files skipped from review as they are similar to previous changes (6)
📝 WalkthroughWalkthroughAgent evaluation now passes task inputs directly, uses ChangesAgent input templating
Sequence Diagram(s)sequenceDiagram
participant AgentEvalTask
participant AgentEvaluator
participant HTTPInvocation
participant ModelEndpoint
AgentEvalTask->>AgentEvaluator: provide direct task inputs
AgentEvaluator->>HTTPInvocation: render instruction or generic body template
HTTPInvocation->>ModelEndpoint: send prompt or rendered payload
ModelEndpoint-->>HTTPInvocation: return model response
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
SandyChapman
force-pushed
the
generic-agent-body-from-task-inputs/schapman
branch
from
July 15, 2026 15:56
29cc91b to
bfdc806
Compare
Contributor
|
SandyChapman
force-pushed
the
generic-agent-body-from-task-inputs/schapman
branch
from
July 15, 2026 16:34
bfdc806 to
bc72d94
Compare
SandyChapman
changed the title
SandyChapman
force-pushed
the
generic-agent-body-from-task-inputs/schapman
branch
from
July 15, 2026 16:40
bc72d94 to
1cfc6aa
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py`:
- Around line 646-650: Update _task_row to validate the canonical instruction
input by invoking the existing AgentEvalTask.agent_prompt() check before
returning the verbatim task inputs and task_id. Preserve the current returned
row shape, but ensure blank inputs["instruction"] values fail fast rather than
reaching prompt rendering.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 352367b8-b00b-4be7-a821-a4ecf44d5ad2
⛔ Files ignored due to path filters (1)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.pyis excluded by!sdk/**
📒 Files selected for processing (6)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_callable_runtime.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_harbor_runtime.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_tasks.pypackages/nemo_evaluator_sdk/tests/test_agent_inference.py
…cal input)
Three related fixes so an agent-eval target's request derives cleanly from the
task inputs, keyed on a single canonical input.
1. GenericAgent (HTTP agent) no longer inherits the model-oriented chat
template. Its request now passes the task row through ("{{ item }}"), so its
`body` renders directly against the task inputs (e.g. `{{ instruction }}`),
with no chat/completions shape assumed. Model and NeMo Agent Toolkit targets
are unchanged.
2. `instruction` is the single canonical task input. Previously the online/Model
path synthesized a `prompt` key (`inputs["prompt"] or inputs["instruction"]`)
and the Model default template referenced `{{item.prompt}}` — so a Model was
fed via `prompt` while every runner used `instruction` (see
AgentEvalTask.agent_prompt). Drop the synthesized `prompt`; the Model default
templates now render `{{item.instruction}}`. (The dataset-driven path keeps
`prompt` — that is a dataset column, a separate concern.)
3. Harbor task discovery no longer puts the instruction text in `intent`.
`intent` is human-facing metadata never shown to the agent; it now holds the
task name. The instruction from instruction.md goes to `inputs["instruction"]`
(the field the agent actually acts on).
Tests migrated to the canonical `instruction` input; adds coverage for a generic
agent's body rendering from task inputs and for the completions-endpoint default
template.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
generic-agent-body-from-task-inputs/schapman
branch
from
July 15, 2026 16:51
1cfc6aa to
940a979
Compare
ngoncharenko
approved these changes
Jul 15, 2026
This was referenced Jul 15, 2026
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Three related rough edges in how an agent-eval task's request is built:
1. HTTP agents assumed a chat shape. A
GenericAgentinherited the model-oriented default prompt template —{"messages": [{"role": "user", "content": ...}]}— so itsbodycould only reach the instruction via{{ messages[-1].content }}. That leaks a chat/completions assumption onto arbitrary JSON endpoints.2. Two input keys.
instructionis the canonical task input everywhere real —AgentEvalTask.agent_prompt()reads exactlyinputs["instruction"], and every runner goes through it. But the online/Model path synthesized a separatepromptkey (inputs["prompt"] or inputs["instruction"]) and the Model default template referenced{{item.prompt}}— so aModelwas fed viapromptwhile everything else usedinstruction.3. Harbor leaked the instruction into
intent.discover_harbor_tasksreadinstruction.mdinto a variable namedintentand put it in both the task'sintent(human-facing metadata, never shown to the agent) andinputs["instruction"]. Theintentfield was carrying the full instruction text.Fix
"{{ item }}"), sobodyrenders directly against the task inputs — e.g.body={"query": "{{ instruction }}"}. No chat/completions shape assumed.instructionis the single canonical input. Dropped the synthesizedpromptin_task_row; Model default templates (chat + completions) now render{{item.instruction}}.intentto the task name (human metadata) and puts theinstruction.mdcontent ininputs["instruction"]only.ModelandNemoAgentToolkitAgentrequest shapes are unchanged — only the source field becomesinstruction.The dataset-driven path is untouched and still keys on
prompt(a dataset column — separate from agent-eval task inputs).Tests
_resolve_http_agent_invocationrendersbodyfrom task inputs ({{ instruction }}, arbitrary fields).messageswrapper).instruction.intent== task name,inputs["instruction"]== instruction.md content.instructioninput.Verification
nemo_evaluator_sdksuite passes (1088 passed, 2 skipped).ruff check,ruff format --check, full-projectty, andmake vendor(vendored SDK sync) all clean.Lint allgreen in CI.Note
Behavior change for existing agent-eval configs that used
inputs={"prompt": ...}or{{ messages[-1].content }}in aGenericAgentbody — switch toinstruction/{{ instruction }}. Agent-eval is beta. Docs for the deployed-HTTP-agent workflow land alongside this and assume the new format.🤖 Generated with Claude Code
Summary by CodeRabbit
instructioninputs and added coverage for generic-agent invocation payload resolution.