feat(evaluator): add agent-eval engine (AgentEvaluator) - #369
Merged
Conversation
Contributor
|
arpitsardhana
force-pushed
the
aalgo-272-agent-engine/arpsingh
branch
2 times, most recently
from
June 17, 2026 08:54
1fd0172 to
b3660c2
Compare
arpitsardhana
marked this pull request as ready for review
June 17, 2026 08:54
arpitsardhana
commented
Jun 17, 2026
arpitsardhana
force-pushed
the
aalgo-272-agent-engine/arpsingh
branch
from
June 17, 2026 16:21
b3660c2 to
9256690
Compare
Contributor
📝 WalkthroughWalkthroughAdds ChangesAgent Evaluation Pipeline
Sequence Diagram(s)sequenceDiagram
participant Caller
participant AgentEvaluator
participant _generate_trials
participant _score_trials
participant persist_run
participant write_dashboard
Caller->>AgentEvaluator: run(tasks, trials|target, config)
AgentEvaluator->>AgentEvaluator: validate inputs, assign run_id
alt target provided
AgentEvaluator->>_generate_trials: generate trials concurrently
_generate_trials-->>AgentEvaluator: trials[]
end
AgentEvaluator->>_score_trials: score(tasks, trials, config)
_score_trials-->>AgentEvaluator: scores[]
AgentEvaluator->>AgentEvaluator: build summary + benchmark metadata
opt output_dir configured
AgentEvaluator->>persist_run: persist result artifacts
persist_run-->>AgentEvaluator: updated AgentEvalResult
opt write_dashboard configured
AgentEvaluator->>write_dashboard: render and write HTML
end
end
AgentEvaluator-->>Caller: AgentEvalResult
Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py (1)
65-66: TODO noted: task capability requirements.This would help skip tasks incompatible with certain targets (e.g., file-based tasks on text-only models).
Want me to open an issue to track this capability-matching feature?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py` around lines 65 - 66, The TODO comment in the tasks.py file notes that Tasks may need to define required_capabilities to allow the runtime to skip incompatible tasks. While this is a valid enhancement idea for the capability-matching feature, it is currently just a TODO comment without any associated tracking or implementation. Consider whether to create a separate issue in your project's tracking system to formally document this feature request so it can be prioritized and implemented as part of the task evaluation framework, or alternatively, enhance the TODO comment with more specific requirements for what required_capabilities should include and how the runtime should use them to filter tasks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py`:
- Line 4: Remove the `from __future__ import annotations` import statement from
the top of the test_dashboard.py file. This import defers all type hints and
stores them as strings at runtime, which conflicts with the coding guideline
requiring concrete runtime type hints. Simply delete this line to ensure type
hints are evaluated as concrete types at runtime.
In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py`:
- Line 4: Remove the `from __future__ import annotations` import statement at
the top of the file. Since the codebase targets Python 3.11+ and already uses
modern type hint syntax like PEP 604 unions (|) and PEP 585 built-in generics
(list[], dict[]), this future import is unnecessary and converts all type hints
to strings, which contradicts the goal of using concrete type hints. Simply
delete this line and the type annotations will continue to work as concrete
types with the native Python 3.11+ support.
---
Nitpick comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py`:
- Around line 65-66: The TODO comment in the tasks.py file notes that Tasks may
need to define required_capabilities to allow the runtime to skip incompatible
tasks. While this is a valid enhancement idea for the capability-matching
feature, it is currently just a TODO comment without any associated tracking or
implementation. Consider whether to create a separate issue in your project's
tracking system to formally document this feature request so it can be
prioritized and implemented as part of the task evaluation framework, or
alternatively, enhance the TODO comment with more specific requirements for what
required_capabilities should include and how the runtime should use them to
filter tasks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 680e33f6-66d5-4a1e-aec4-318b438c149d
⛔ Files ignored due to path filters (5)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/execution/samples.pyis excluded by!sdk/**
📒 Files selected for processing (7)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
arpitsardhana
force-pushed
the
aalgo-272-agent-engine/arpsingh
branch
from
June 17, 2026 16:44
9256690 to
0848abc
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py`:
- Around line 129-132: The _jsonish function currently only catches TypeError
when calling json.dumps on the value, but json.dumps can also raise ValueError
for issues like circular references. Modify the except clause to catch both
TypeError and ValueError (or a broader Exception) instead of just TypeError, so
that any serialization failure will fall back to the str(value) conversion
instead of crashing the dashboard rendering.
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py`:
- Around line 56-58: The _write_jsonl function currently materializes the entire
file content in memory by joining all rows at once, which can cause memory
spikes with large datasets. Instead of using "\n".join() to create a single
payload string, refactor the function to open the file in write mode and iterate
through each row, writing each serialized row followed by a newline directly to
the file in a streaming fashion. This avoids loading the complete content into
memory before writing.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 48cbefec-99c3-4558-92b1-785878391b66
⛔ Files ignored due to path filters (5)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/execution/samples.pyis excluded by!sdk/**
📒 Files selected for processing (7)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
✅ Files skipped from review due to trivial changes (1)
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py
🚧 Files skipped from review as they are similar to previous changes (4)
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
SandyChapman
approved these changes
Jun 17, 2026
arpitsardhana
force-pushed
the
aalgo-272-agent-engine/arpsingh
branch
from
June 17, 2026 16:51
0848abc to
a6c2ec6
Compare
Add the standalone agent-eval orchestration engine on top of the domain model: AgentEvaluator (generation + scoring), run-bundle persistence, and an HTML dashboard. Adapted to the refactored domain model: trial/score naming, status enums, AggregatedMetricResult-based summary, and from_scores aggregation. Online inference seams (inference fns, clients, default headers) are injected on the AgentEvaluator instance rather than AgentEvalRunConfig. Also maps candidate evidence through build_metric_input so metrics can consume trial evidence. Signed-off-by: Arpit Singh (SW-CLOUD) <arpsingh@nvidia.com>
arpitsardhana
force-pushed
the
aalgo-272-agent-engine/arpsingh
branch
from
June 17, 2026 16:56
a6c2ec6 to
59f1c1a
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
♻️ Duplicate comments (1)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py (1)
129-132:⚠️ Potential issue | 🟠 Major | ⚡ Quick winCatch
ValueErrorin_jsonishfallback path.
json.dumpscan raiseValueError(e.g., circular refs), which currently escapes and can break rendering.Diff
def _jsonish(value: Any) -> str: if isinstance(value, BaseModel): value = value.model_dump(mode="json") try: return json.dumps(value, indent=2, sort_keys=True) - except TypeError: + except (TypeError, ValueError): return str(value)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py` around lines 129 - 132, The `_jsonish` function only catches `TypeError` when calling `json.dumps`, but `json.dumps` can also raise `ValueError` for cases like circular references, which currently escapes the exception handler and breaks rendering. Modify the except clause to catch both `TypeError` and `ValueError` exceptions so that both cases are handled by the fallback `str(value)` conversion.
🧹 Nitpick comments (1)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py (1)
6-6: ⚡ Quick winRemove postponed annotations.
This makes every annotation string-based in this module; the shown types can be imported normally.
As per coding guidelines,
**/*.py: “Always prefer concrete type hints over string-based ones in Python code; do not import types under TYPE_CHECKING, instead import types as regular imports when possible.”Proposed fix
-from __future__ import annotations -🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py` at line 6, Remove the `from __future__ import annotations` import statement at the top of the evaluator.py file. This import makes all type annotations string-based, which violates the coding guideline that requires concrete type hints over string-based ones. After removing this line, ensure all type hints in the module use concrete imported types rather than string-based forward references, and import any necessary types directly as regular imports instead of under TYPE_CHECKING blocks.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py`:
- Around line 208-214: The _score_id() function is generating non-unique IDs for
metrics because it omits the metric index or configuration details. When
multiple metrics of the same type exist on a single task/trial, the IDs collide.
Modify the _score_id() function to include the metric's index position or
configuration details (such as metric name or any unique metric identifier) in
the ID generation to ensure each metric instance produces a unique score ID.
Also check the code at lines 437-438 for the same issue and apply the same fix
there.
- Around line 235-243: Add validation in the client initialization logic to
reject mismatched injected clients before generation proceeds. When self.client
is not None, verify that it matches the target type - for a Model target,
self.client should be an httpx.AsyncClient; for an Agent target, it should be an
AsyncOpenAI client. If there is a mismatch, raise an appropriate error to
prevent None from being passed to inference. Apply this same validation logic in
the other relevant section mentioned around lines 301-318 to ensure consistency
across all client handling paths.
- Around line 541-542: The _new_run_id() function uses only a second-level
timestamp, which can create collisions in parallel or rapid runs, causing score
ID and bundle identifier reuse. Modify the function to generate a
collision-resistant ID by appending a unique random component (such as a UUID or
random string) to the timestamp, ensuring each run ID is guaranteed to be unique
even when multiple runs occur within the same second.
---
Duplicate comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.py`:
- Around line 129-132: The `_jsonish` function only catches `TypeError` when
calling `json.dumps`, but `json.dumps` can also raise `ValueError` for cases
like circular references, which currently escapes the exception handler and
breaks rendering. Modify the except clause to catch both `TypeError` and
`ValueError` exceptions so that both cases are handled by the fallback
`str(value)` conversion.
---
Nitpick comments:
In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py`:
- Line 6: Remove the `from __future__ import annotations` import statement at
the top of the evaluator.py file. This import makes all type annotations
string-based, which violates the coding guideline that requires concrete type
hints over string-based ones. After removing this line, ensure all type hints in
the module use concrete imported types rather than string-based forward
references, and import any necessary types directly as regular imports instead
of under TYPE_CHECKING blocks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d1a60727-4a5c-4e5b-ac4b-7430f9f82766
⛔ Files ignored due to path filters (5)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/execution/samples.pyis excluded by!sdk/**
📒 Files selected for processing (7)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
✅ Files skipped from review due to trivial changes (1)
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py
🚧 Files skipped from review as they are similar to previous changes (3)
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
arpitsardhana
enabled auto-merge
June 17, 2026 17:00
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py (1)
6-6: ⚡ Quick winRemove postponed-annotation mode in this module.
Drop
from __future__ import annotationsand keep concrete runtime-resolved annotations only, per repo rule.As per coding guidelines: “Always prefer concrete type hints over string-based ones in Python code; do not import types under TYPE_CHECKING, instead import types as regular imports when possible.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py` at line 6, Remove the `from __future__ import annotations` import statement from the top of the module and ensure all type annotations use concrete runtime-resolved types instead of string-based annotations. Review any type hints throughout the module and convert them to use regular imports of the required types rather than importing them under TYPE_CHECKING blocks, so that the annotations are fully resolvable at runtime.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.py`:
- Line 6: Remove the `from __future__ import annotations` import statement from
the top of the module and ensure all type annotations use concrete
runtime-resolved types instead of string-based annotations. Review any type
hints throughout the module and convert them to use regular imports of the
required types rather than importing them under TYPE_CHECKING blocks, so that
the annotations are fully resolvable at runtime.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ebc2892a-c4ab-4a8f-8808-ecbc8c813438
⛔ Files ignored due to path filters (5)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/dashboard.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/evaluator.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/persistence.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/tasks.pyis excluded by!sdk/**sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/execution/samples.pyis excluded by!sdk/**
📒 Files selected for processing (7)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/dashboard.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/persistence.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
✅ Files skipped from review due to trivial changes (1)
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/tasks.py
🚧 Files skipped from review as they are similar to previous changes (4)
- packages/nemo_evaluator_sdk/tests/agent_eval/test_dashboard.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/execution/samples.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_evaluator.py
- packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/evaluator.py
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jun 17, 2026
This was referenced Jun 19, 2026
This was referenced Jun 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the standalone agent-eval orchestration engine on top of the domain model: AgentEvaluator (generation + scoring), run-bundle persistence, and an HTML dashboard.
Adapted to the refactored domain model: trial/score naming, status enums, AggregatedMetricResult-based summary, and from_scores aggregation. Online inference seams (inference fns, clients, default headers) are injected on the AgentEvaluator instance rather than AgentEvalRunConfig. Also maps candidate evidence through build_metric_input so metrics can consume trial evidence.
Summary by CodeRabbit
Release Notes
evidencein candidate payloads.