feat(studio): move agent eval over to evaluator endpoints - #775
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughChangesAgent evaluation workflow
Upload picker timestamps
Phishing-agent token limits
Sequence Diagram(s)sequenceDiagram
participant SubmitEvaluationModal
participant EvaluationHelpers
participant EvaluatorJobsAPI
participant ResultFilesAPI
SubmitEvaluationModal->>EvaluationHelpers: parse eval-config.json and build request
EvaluationHelpers->>EvaluatorJobsAPI: submit agent-evaluate job
EvaluatorJobsAPI-->>SubmitEvaluationModal: return job
SubmitEvaluationModal->>EvaluatorJobsAPI: fetch terminal result
EvaluationHelpers->>ResultFilesAPI: download task bundle JSONL
ResultFilesAPI-->>EvaluationHelpers: return tasks, trials, and scores
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 16
🧹 Nitpick comments (1)
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md (1)
1-5: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSplit explanation from API reference and add the required page framing.
This page mixes conceptual explanation with endpoint, payload, result, and legacy reference specifications. Keep one Diataxis quadrant, move reference material to a linked page, list prerequisites first, and add Next Steps.
As per coding guidelines, each page must fit one Diataxis quadrant, list prerequisites at the top, and end with Next Steps.
Also applies to: 70-110, 229-307
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md` around lines 1 - 5, Restructure the Agent Evaluation documentation to keep this page focused on conceptual explanation: add a prerequisites section near the top, move endpoint, payload, result, and legacy specifications into a separate linked API reference page, and finish with a Next Steps section. Apply the same separation and framing to the referenced sections while preserving the existing evaluation guidance.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@web/packages/common/src/components/UploadModal/Context/useUploadModalReducer.ts`:
- Line 55: Update the RESET handling in useUploadModalReducer so the reset state
preserves the existing showUpdatedAt option alongside acceptableFileTypes and
allowNewDataset. Ensure consumers that enabled updated-at timestamps retain that
setting after RESET.
In `@web/packages/studio/public/sample-agents/calculator/eval-config.json`:
- Around line 58-59: Remove the embedded judge model and max_concurrent_tasks
bindings from the reusable evaluation configs. In
web/packages/studio/public/sample-agents/calculator/eval-config.json at lines
58-59 and 88, and
web/packages/studio/public/sample-agents/email-phishing-analyzer/eval-config.json
at lines 167-168 and 198, delete those fields while preserving the surrounding
configuration so submission can inject the selected judge model and effective
concurrency.
In
`@web/packages/studio/public/sample-agents/email-phishing-analyzer/eval-config-dataset.json`:
- Around line 2-10: Update
web/packages/studio/public/sample-agents/email-phishing-analyzer/eval-config-dataset.json
lines 2-10 to remove dataset_file and prompt_template, replacing them with
inline tasks[] entries compatible with agent-evaluate. Update
web/packages/studio/src/api/evaluation/eval-config-fileset.ts lines 27-68 to
replace the seeded react-eval.yml with an agent-evaluate-compatible
configuration using inline tasks[] and an inline metric; do not reference a
dataset, Fileset, or Taskset.
In
`@web/packages/studio/public/sample-agents/email-phishing-analyzer/eval-spec.wip.json`:
- Around line 12-50: Update the eval spec’s metrics configuration, including all
remaining task metric blocks, to use a single root inline metric with type
“llm-judge” instead of per-task “answer_accuracy” metrics. Remove embedded judge
model, target, and concurrency settings from the spec; rely on submission-time
injection for the judge model, agent target, and effective max_concurrent_tasks.
In `@web/packages/studio/src/api/evaluation/agent-evaluations.ts`:
- Around line 236-260: Update the trial-to-score join in the surrounding mapping
logic so scores are matched by trial_id rather than task_id alone. Either return
separate details for each trial or, if retaining one selected trial per task,
filter taskScores to that trial before building scores and diagnostics, ensuring
responseText and attached scores come from the same trial.
- Around line 42-46: Update agentNameForJob to remove only the current job
workspace prefix from the returned agent name before useAgentPanel compares it
with the bare name. Preserve the existing target validation and return null
behavior, and avoid stripping unrelated portions of the agent name.
In `@web/packages/studio/src/api/evaluation/eval-config-fileset.ts`:
- Around line 124-127: Update the catch block in the fileset creation flow to
re-throw every creation error except an explicit HTTP 409 conflict; retain
cancellation propagation and only ignore the confirmed concurrent-creation
conflict, rather than treating 403 or 5xx responses as successful.
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsx`:
- Around line 194-202: Replace empty-string fallbacks passed to RelativeTime
with conditional rendering: in
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsx
lines 194-202, render each KVPair fallback when created_at or updated_at is
absent; in
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsx
line 213 and
web/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/AgentDetailsContent.tsx
line 212, render — instead of RelativeTime when created_at is absent.
- Around line 89-100: Update the result and bundle queries in
AgentEvaluationDetailRoute to use a bounded refetch interval whenever their
respective data is null, allowing newly available artifacts to be discovered
after a terminal job. Stop polling once non-null data is returned, while
preserving the existing query gating and fetch behavior.
In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md`:
- Around line 38-53: Update all JSON examples in AGENTS.md, including the shared
metric and task examples, to be valid executable JSON by removing comments and
non-JSON placeholders; use concrete representative values where needed rather
than changing the fences to jsonc. Apply this consistently to the referenced
example sections.
- Line 3: Replace every hardcoded “Studio” product-name reference in AGENTS.md,
including the introductory text and the additional referenced sections, with the
repository’s configured Sphinx product-name substitution. Preserve the existing
wording and formatting aside from using the substitution consistently.
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsx`:
- Around line 23-40: Update the score rendering in AgentEvalScoresPanel’s
scores.map callback to branch on each score’s score_type. For rubric scores,
render the rubric_distribution and mode_category data in addition to the shared
numeric statistics, while preserving the existing numeric-score rendering. Add
or update a rubric-focused test covering these fields.
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx`:
- Around line 118-131: Update the MODE_DEFAULT branch in SubmitEvaluationModal
so name collisions are rejected rather than returning the newly selected sample
content after ensureEvalConfigFileset preserves an existing eval-config.json.
Validate that the fileset name is available before submission and surface the
existing validation/error path, or retrieve and return the persisted file
content when the name already exists; ensure the submitted content always
matches the referenced fileset.
- Around line 189-208: Update SubmitEvaluationModal’s submission and validation
flow to derive the metric type from the config being submitted, including
fileset mode, rather than relying only on the preview query. Keep the judge
selector visible and required for llm-judge metrics, block submission while the
metric type/config is unresolved, and inject the selected judge model into the
submitted config at submission time.
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts`:
- Around line 120-129: Update buildAgentEvalRequestBody so
spec.max_concurrent_tasks is injected at submission with an effective value of
1, rather than reading config.max_concurrent_tasks or
DEFAULT_MAX_CONCURRENT_TASKS from the reusable EvalConfig. Keep the remaining
request construction unchanged.
- Around line 132-145: Update parseEvalConfig to fully validate the parsed
EvalConfig before returning it: reject malformed task entries, invalid inline
payload structures, unsupported metric types, and numeric options outside their
allowed bounds. Ensure invalid metric payloads and task definitions fail at this
JSON boundary rather than reaching request construction, while preserving the
existing required non-empty tasks and metric checks.
---
Nitpick comments:
In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md`:
- Around line 1-5: Restructure the Agent Evaluation documentation to keep this
page focused on conceptual explanation: add a prerequisites section near the
top, move endpoint, payload, result, and legacy specifications into a separate
linked API reference page, and finish with a Next Steps section. Apply the same
separation and framing to the referenced sections while preserving the existing
evaluation guidance.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 2f7fd330-bbcc-4d83-9d1b-e2b8426f41a5
📒 Files selected for processing (44)
plugins/nemo-agents/examples/email-phishing-analyzer/src/nat_email_phishing_analyzer/email-phishing-agent.ymlweb/packages/common/src/components/DatasetFileSelect/ControlledDatasetFileSelect.tsxweb/packages/common/src/components/DatasetFileSelect/DatasetFileSelect.tsxweb/packages/common/src/components/UploadModal/Context/useUploadModalReducer.tsweb/packages/common/src/components/UploadModal/DatasetUploader/Select.tsxweb/packages/common/src/components/UploadModal/InlineUploadPicker.tsxweb/packages/common/src/components/UploadModal/types.tsweb/packages/studio/public/sample-agents/calculator/eval-config.jsonweb/packages/studio/public/sample-agents/email-phishing-analyzer/agent.ymlweb/packages/studio/public/sample-agents/email-phishing-analyzer/eval-config-dataset.jsonweb/packages/studio/public/sample-agents/email-phishing-analyzer/eval-config.jsonweb/packages/studio/public/sample-agents/email-phishing-analyzer/eval-spec.wip.jsonweb/packages/studio/src/api/evaluation/agent-evaluations.test.tsweb/packages/studio/src/api/evaluation/agent-evaluations.tsweb/packages/studio/src/api/evaluation/eval-config-fileset.tsweb/packages/studio/src/components/agents/AgentBlockingInput/EvalConfigBlockingInput.test.tsxweb/packages/studio/src/components/agents/AgentBlockingInput/EvalConfigBlockingInput.tsxweb/packages/studio/src/components/evaluation/JudgeModelSelect.tsxweb/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/AgentDetailsContent.tsxweb/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/index.test.tsxweb/packages/studio/src/components/sidePanels/AgentPanels/AgentPanel/useAgentPanel.tsweb/packages/studio/src/constants/sampleAgents.test.tsweb/packages/studio/src/constants/sampleAgents.tsweb/packages/studio/src/mocks/handlers.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.mdweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationDetailRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AgentEvaluationsListRoute.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/api.test.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/api.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.test.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalTaskResultsPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/EvalConfigFilesPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/EvaluatorOutputPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/EvaluatorReasoning.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/WorkflowOutputPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/api.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/constants.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/useOptimizerSuggestions.test.tsxweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/useOptimizerSuggestions.tsweb/packages/studio/src/routes/agents/AgentSuggestionsRoute/utils.ts
💤 Files with no reviewable changes (8)
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/EvaluatorReasoning.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/EvalConfigFilesPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/api.test.ts
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/EvaluatorOutputPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/WorkflowOutputPanel.tsx
- web/packages/studio/src/routes/agents/AgentSuggestionsRoute/constants.ts
- web/packages/studio/src/routes/agents/AgentSuggestionsRoute/api.ts
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/api.ts
Contributor
|
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts (1)
113-121: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDeclare the request-body return contract.
buildAgentEvalRequestBodyis an exported API with a cross-layer payload contract, but has no explicit return type. Co-locate a named request-body interface and annotate it.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts` around lines 113 - 121, Define a named interface for the request body returned by buildAgentEvalRequestBody, colocated with that function, covering the description field and nested spec payload. Annotate buildAgentEvalRequestBody with this interface as its explicit return type while preserving the existing payload construction.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts`:
- Around line 113-121: Define a named interface for the request body returned by
buildAgentEvalRequestBody, colocated with that function, covering the
description field and nested spec payload. Annotate buildAgentEvalRequestBody
with this interface as its explicit return type while preserving the existing
payload construction.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8b80d453-524a-47d4-b01b-0286c024b2b3
📒 Files selected for processing (8)
plugins/nemo-agents/examples/email-phishing-analyzer/src/nat_email_phishing_analyzer/email-phishing-agent.ymlweb/packages/common/src/components/UploadModal/Context/useUploadModalReducer.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.mdweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.test.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts
💤 Files with no reviewable changes (1)
- plugins/nemo-agents/examples/email-phishing-analyzer/src/nat_email_phishing_analyzer/email-phishing-agent.yml
🚧 Files skipped from review as they are similar to previous changes (3)
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/AgentEvalScoresPanel.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.ts (1)
19-51: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRename module fixtures to
SCREAMING_SNAKE_CASE.
metricandconfigare constants and violate the project naming rule.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.ts` around lines 19 - 51, Rename the module-level fixtures `metric` and `config` to `SCREAMING_SNAKE_CASE` constants, updating every reference including `configWithBakedJudge` and any test usages while preserving their values and behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.ts`:
- Around line 19-51: Rename the module-level fixtures `metric` and `config` to
`SCREAMING_SNAKE_CASE` constants, updating every reference including
`configWithBakedJudge` and any test usages while preserving their values and
behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 43942fb8-5343-4875-81e5-a0975dc247c1
📒 Files selected for processing (4)
web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.mdweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsxweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.test.tsweb/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts
🚧 Files skipped from review as they are similar to previous changes (3)
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/AGENTS.md
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/submitEvaluationSpec.ts
- web/packages/studio/src/routes/agents/AgentEvaluationsRoute/components/SubmitEvaluationModal.tsx
nv-odrulea
force-pushed
the
od/agent-evals-refactor-evaluator
branch
from
July 20, 2026 21:54
a6a00f9 to
5d5a07f
Compare
steramae-nvidia
approved these changes
Jul 21, 2026
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Octavian Drulea <odrulea@nvidia.com>
nv-odrulea
force-pushed
the
od/agent-evals-refactor-evaluator
branch
from
July 21, 2026 20:18
5e5eab8 to
6a6555c
Compare
nv-odrulea
enabled auto-merge
July 21, 2026 20:19
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Eval Modal: Use premade example - will save eval config to new Fileset
New Eval Modal: Use saved eval config from an existing Fileset
Completed Eval
Highlights
/apis/agents/.../jobs/evaluatepath + local types; use generated evaluator SDK, POST{spec:{tasks,target}}to/agent-evaluate/jobs.AgentEvalResult.scores[]; new scores panel; handle serializedNaN+ aggregate coverage.eval-config.json; drop legacy dataset-driven ref file./generate— workaround for NAT agent bug where streaming response was not supported within eval processSummary by CodeRabbit
Summary
New Features
Bug Fixes
Documentation