feat: extract Eval Author into nemo-eval-author-plugin - #932
Merged
Conversation
Move Eval Author, the Harbor evaluator stack, staging, and trace analysis into a library plugin so Experimentalist depends on it one-way like Insights, without Eval Author depending on Experimentalist. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
force-pushed
the
ase-eval-author-plugin/akhoury
branch
from
July 27, 2026 21:55
2a1a837 to
ea2fc80
Compare
Contributor
|
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughChangesThe PR extracts Eval Author into Eval Author plugin extraction
Sequence Diagram(s)sequenceDiagram
participant EvalAuthorRun
participant EnvBridge
participant ModelConfig
participant Experimentalist
EvalAuthorRun->>EnvBridge: import-time AUTHOR_* bridging
EvalAuthorRun->>ModelConfig: resolve model and credentials
ModelConfig->>ModelConfig: create or reuse CompletionClient
EvalAuthorRun->>Experimentalist: load shared evaluator, trace, and tool components
Experimentalist-->>EvalAuthorRun: execute extracted Eval Author flow
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
🔧 Fix failing CI
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (4)
plugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.py (1)
98-106: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMove
_redact_urlimport to top-level and stop reaching into a private symbol.
_redact_urlis imported locally, inside anexceptblock, from a leading-underscore (private) name inrepository.py. It's already imported at the top of this file alongsideAgentSource,clone_agent_repo, etc. (lines 23-29) — add it there instead of a function-local import, and consider exposing it without the underscore if it's meant to be used cross-module.♻️ Proposed fix
from nemo_eval_author_plugin.repository import ( AgentSource, clone_agent_repo, looks_like_git, split_agent_spec, split_git_ref, + _redact_url, ) @@ async def _clone_git_agent(self, agent: str, dest: Path, *, clone_depth: int | None = None) -> AgentSource: try: return await asyncio.to_thread(clone_agent_repo, agent, dest, clone_depth=clone_depth) except subprocess.CalledProcessError: remote, _ = split_git_ref(split_agent_spec(agent)[0]) - from nemo_eval_author_plugin.repository import _redact_url - raise ValueError(f"failed to fetch --agent {_redact_url(remote)!r}") from None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.py` around lines 98 - 106, Move the _redact_url import from the _clone_git_agent exception block to the module-level imports alongside AgentSource and clone_agent_repo. Continue using the existing _redact_url symbol in the error message, without changing unrelated cloning or error-handling behavior.plugins/nemo-eval-author/README.md (1)
1-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep this page to one Diátaxis quadrant.
It currently mixes API reference, installation how-to, and internal TODO content. Add prerequisites before the first section, separate the API reference from setup instructions, and add a
Next Stepssection with cross-links.As per coding guidelines, each documentation page must fit one Diátaxis quadrant, list prerequisites at the top, and include a Next Steps section with cross-links.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/README.md` around lines 1 - 36, Restructure the README into a single Diátaxis quadrant by choosing either API reference or setup/how-to as its primary purpose, moving or removing content from the other category. Add a prerequisites section before the first main section, retain only relevant content for the chosen quadrant, and finish with a “Next Steps” section containing cross-links to related documentation. Update the README structure and headings while preserving the accurate public API or installation details.Source: Coding guidelines
plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md (1)
1-23: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMissing "Next Steps" cross-links.
Add a closing section linking to
eval_author/README.md/ usage docs.As per coding guidelines: "Include 'Next Steps' section at the end with cross-links to related documentation content."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md` around lines 1 - 23, Add a closing “Next Steps” section to the EvalAuthorResult reference document, including cross-links to eval_author/README.md and the relevant usage documentation. Keep it as the final section after the existing content and use repository-relative documentation links.Source: Coding guidelines
plugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.py (1)
36-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDefault model names duplicated between getters and
log_model_config.Same fallback strings hardcoded in 4 places; a future default change in one getter won't be reflected in the log output.
♻️ Proposed fix
+_SMART_MODEL_DEFAULT = "openai/openai/openai/gpt-5.5" +_MID_MODEL_DEFAULT = "openai/gcp/google/gemini-3.5-flash" +_FAST_MODEL_DEFAULT = "openai/openai/openai/gpt-5-mini" + `@functools.cache` def get_smart_model() -> CompletionClient: ... - name = _optional_env("EXPERIMENTALIST_SMART_MODEL_NAME", "openai/openai/openai/gpt-5.5") + name = _optional_env("EXPERIMENTALIST_SMART_MODEL_NAME", _SMART_MODEL_DEFAULT)Apply similarly to the other two getters and to
log_model_config.Also applies to: 53-53, 70-70, 95-97
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.py` at line 36, Centralize the default model-name values used by the getters and reuse those symbols in log_model_config. Update the getters around EXPERIMENTALIST_SMART_MODEL_NAME and the other two model configuration keys, plus log_model_config, so each fallback is defined once and logging reflects any future default change.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-eval-author/pyproject.toml`:
- Around line 5-12: Resolve the Python-version mismatch between the project
metadata and util.py: either raise requires-python to exclude Python 3.11, or
make nooa available for Python 3.11 by removing its conditional dependency
marker and ensuring util.py’s unconditional import remains valid.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/cache.py`:
- Around line 109-117: Move the path.parent.mkdir call in the cache write flow
into the existing try block that handles tmp.write_text and tmp.replace. Ensure
OSError from cache-directory creation is caught by the existing warning log and
the optional cache write is skipped without propagating the exception.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.py`:
- Line 79: Update the mode argument docstring in run_eval_author to reference
make_eval_author_backend instead of Experimentalist’s backend factory, keeping
the documentation aligned with the current backend creation contract.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/models.py`:
- Around line 138-155: Update __aenter__ to call _stop_started_runtime only
after the dependency start command has actually been attempted successfully;
when self._runtime.start is None, raise ValueError without invoking the stop
command. Preserve cleanup and exception propagation for failures occurring
during or after run_dependency_command(self._runtime.start, "start").
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/tools.py`:
- Around line 110-123: Update list_agents sorting to handle directory names
without a "-" separator before accessing the numeric suffix. Preserve natural
numeric ordering for agent-N names while safely assigning a fallback sort key to
other directories containing metadata.json.
- Line 171: Update the read_analysis_file method signature to use the concrete
str | int type hint directly instead of a quoted string, leaving the limit
annotation and method behavior unchanged.
- Around line 189-208: Update get_metadata to honor its documented fallback
contract: catch missing-file, JSON parsing, read, and Candidate validation
failures, then return a minimal Candidate with default values. Preserve the
existing successful parsing and entity_id assignment behavior for valid
metadata.
---
Nitpick comments:
In `@plugins/nemo-eval-author/README.md`:
- Around line 1-36: Restructure the README into a single Diátaxis quadrant by
choosing either API reference or setup/how-to as its primary purpose, moving or
removing content from the other category. Add a prerequisites section before the
first main section, retain only relevant content for the chosen quadrant, and
finish with a “Next Steps” section containing cross-links to related
documentation. Update the README structure and headings while preserving the
accurate public API or installation details.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.py`:
- Around line 98-106: Move the _redact_url import from the _clone_git_agent
exception block to the module-level imports alongside AgentSource and
clone_agent_repo. Continue using the existing _redact_url symbol in the error
message, without changing unrelated cloning or error-handling behavior.
In
`@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md`:
- Around line 1-23: Add a closing “Next Steps” section to the EvalAuthorResult
reference document, including cross-links to eval_author/README.md and the
relevant usage documentation. Keep it as the final section after the existing
content and use repository-relative documentation links.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.py`:
- Line 36: Centralize the default model-name values used by the getters and
reuse those symbols in log_model_config. Update the getters around
EXPERIMENTALIST_SMART_MODEL_NAME and the other two model configuration keys,
plus log_model_config, so each fallback is defined once and logging reflects any
future default change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f0f898a7-d0e3-4a28-8e28-2ed1863c554e
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (66)
plugins/README.mdplugins/nemo-eval-author/README.mdplugins/nemo-eval-author/pyproject.tomlplugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/cache.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/client.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/dataset_staging.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/config.yamlplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/__init__.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/base.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/factory.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/harbor.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/holdout_utils.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/py.typedplugins/nemo-eval-author/src/nemo_eval_author_plugin/rationale.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/repository.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/tools.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/trace_analyzer.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/trace_explorer.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/util.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/workspace_entities.pyplugins/nemo-eval-author/tests/test_dataset_staging.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_eval_author_materialization.pyplugins/nemo-eval-author/tests/test_eval_author_run.pyplugins/nemo-eval-author/tests/test_evaluator_base.pyplugins/nemo-eval-author/tests/test_evaluator_factory.pyplugins/nemo-eval-author/tests/test_evaluator_harbor.pyplugins/nemo-eval-author/tests/test_trace_explorer_from_ref.pyplugins/nemo-experimentalist/AGENTS.mdplugins/nemo-experimentalist/benchmarks/run.pyplugins/nemo-experimentalist/pyproject.tomlplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/coder.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/goal_tree.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/rationalizer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_scorer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/deps.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/otlp.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/run.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/resolve.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_eval_author_repair_e2e.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/test_deps.pyplugins/nemo-experimentalist/tests/test_eval_author_config.pyplugins/nemo-experimentalist/tests/test_experiment_cli.pyplugins/nemo-experimentalist/tests/test_experimentalist_analyzer.pyplugins/nemo-experimentalist/tests/test_experimentalist_backend.pyplugins/nemo-experimentalist/tests/test_experimentalist_benchmark.pyplugins/nemo-experimentalist/tests/test_experimentalist_run.pyplugins/nemo-experimentalist/tests/test_otlp.pypyproject.toml
Contributor
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 7
🧹 Nitpick comments (4)
plugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.py (1)
98-106: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMove
_redact_urlimport to top-level and stop reaching into a private symbol.
_redact_urlis imported locally, inside anexceptblock, from a leading-underscore (private) name inrepository.py. It's already imported at the top of this file alongsideAgentSource,clone_agent_repo, etc. (lines 23-29) — add it there instead of a function-local import, and consider exposing it without the underscore if it's meant to be used cross-module.♻️ Proposed fix
from nemo_eval_author_plugin.repository import ( AgentSource, clone_agent_repo, looks_like_git, split_agent_spec, split_git_ref, + _redact_url, ) @@ async def _clone_git_agent(self, agent: str, dest: Path, *, clone_depth: int | None = None) -> AgentSource: try: return await asyncio.to_thread(clone_agent_repo, agent, dest, clone_depth=clone_depth) except subprocess.CalledProcessError: remote, _ = split_git_ref(split_agent_spec(agent)[0]) - from nemo_eval_author_plugin.repository import _redact_url - raise ValueError(f"failed to fetch --agent {_redact_url(remote)!r}") from None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.py` around lines 98 - 106, Move the _redact_url import from the _clone_git_agent exception block to the module-level imports alongside AgentSource and clone_agent_repo. Continue using the existing _redact_url symbol in the error message, without changing unrelated cloning or error-handling behavior.plugins/nemo-eval-author/README.md (1)
1-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep this page to one Diátaxis quadrant.
It currently mixes API reference, installation how-to, and internal TODO content. Add prerequisites before the first section, separate the API reference from setup instructions, and add a
Next Stepssection with cross-links.As per coding guidelines, each documentation page must fit one Diátaxis quadrant, list prerequisites at the top, and include a Next Steps section with cross-links.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/README.md` around lines 1 - 36, Restructure the README into a single Diátaxis quadrant by choosing either API reference or setup/how-to as its primary purpose, moving or removing content from the other category. Add a prerequisites section before the first main section, retain only relevant content for the chosen quadrant, and finish with a “Next Steps” section containing cross-links to related documentation. Update the README structure and headings while preserving the accurate public API or installation details.Source: Coding guidelines
plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md (1)
1-23: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMissing "Next Steps" cross-links.
Add a closing section linking to
eval_author/README.md/ usage docs.As per coding guidelines: "Include 'Next Steps' section at the end with cross-links to related documentation content."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md` around lines 1 - 23, Add a closing “Next Steps” section to the EvalAuthorResult reference document, including cross-links to eval_author/README.md and the relevant usage documentation. Keep it as the final section after the existing content and use repository-relative documentation links.Source: Coding guidelines
plugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.py (1)
36-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDefault model names duplicated between getters and
log_model_config.Same fallback strings hardcoded in 4 places; a future default change in one getter won't be reflected in the log output.
♻️ Proposed fix
+_SMART_MODEL_DEFAULT = "openai/openai/openai/gpt-5.5" +_MID_MODEL_DEFAULT = "openai/gcp/google/gemini-3.5-flash" +_FAST_MODEL_DEFAULT = "openai/openai/openai/gpt-5-mini" + `@functools.cache` def get_smart_model() -> CompletionClient: ... - name = _optional_env("EXPERIMENTALIST_SMART_MODEL_NAME", "openai/openai/openai/gpt-5.5") + name = _optional_env("EXPERIMENTALIST_SMART_MODEL_NAME", _SMART_MODEL_DEFAULT)Apply similarly to the other two getters and to
log_model_config.Also applies to: 53-53, 70-70, 95-97
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.py` at line 36, Centralize the default model-name values used by the getters and reuse those symbols in log_model_config. Update the getters around EXPERIMENTALIST_SMART_MODEL_NAME and the other two model configuration keys, plus log_model_config, so each fallback is defined once and logging reflects any future default change.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-eval-author/pyproject.toml`:
- Around line 5-12: Resolve the Python-version mismatch between the project
metadata and util.py: either raise requires-python to exclude Python 3.11, or
make nooa available for Python 3.11 by removing its conditional dependency
marker and ensuring util.py’s unconditional import remains valid.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/cache.py`:
- Around line 109-117: Move the path.parent.mkdir call in the cache write flow
into the existing try block that handles tmp.write_text and tmp.replace. Ensure
OSError from cache-directory creation is caught by the existing warning log and
the optional cache write is skipped without propagating the exception.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.py`:
- Line 79: Update the mode argument docstring in run_eval_author to reference
make_eval_author_backend instead of Experimentalist’s backend factory, keeping
the documentation aligned with the current backend creation contract.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/models.py`:
- Around line 138-155: Update __aenter__ to call _stop_started_runtime only
after the dependency start command has actually been attempted successfully;
when self._runtime.start is None, raise ValueError without invoking the stop
command. Preserve cleanup and exception propagation for failures occurring
during or after run_dependency_command(self._runtime.start, "start").
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/tools.py`:
- Around line 110-123: Update list_agents sorting to handle directory names
without a "-" separator before accessing the numeric suffix. Preserve natural
numeric ordering for agent-N names while safely assigning a fallback sort key to
other directories containing metadata.json.
- Line 171: Update the read_analysis_file method signature to use the concrete
str | int type hint directly instead of a quoted string, leaving the limit
annotation and method behavior unchanged.
- Around line 189-208: Update get_metadata to honor its documented fallback
contract: catch missing-file, JSON parsing, read, and Candidate validation
failures, then return a minimal Candidate with default values. Preserve the
existing successful parsing and entity_id assignment behavior for valid
metadata.
---
Nitpick comments:
In `@plugins/nemo-eval-author/README.md`:
- Around line 1-36: Restructure the README into a single Diátaxis quadrant by
choosing either API reference or setup/how-to as its primary purpose, moving or
removing content from the other category. Add a prerequisites section before the
first main section, retain only relevant content for the chosen quadrant, and
finish with a “Next Steps” section containing cross-links to related
documentation. Update the README structure and headings while preserving the
accurate public API or installation details.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.py`:
- Around line 98-106: Move the _redact_url import from the _clone_git_agent
exception block to the module-level imports alongside AgentSource and
clone_agent_repo. Continue using the existing _redact_url symbol in the error
message, without changing unrelated cloning or error-handling behavior.
In
`@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.md`:
- Around line 1-23: Add a closing “Next Steps” section to the EvalAuthorResult
reference document, including cross-links to eval_author/README.md and the
relevant usage documentation. Keep it as the final section after the existing
content and use repository-relative documentation links.
In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.py`:
- Line 36: Centralize the default model-name values used by the getters and
reuse those symbols in log_model_config. Update the getters around
EXPERIMENTALIST_SMART_MODEL_NAME and the other two model configuration keys,
plus log_model_config, so each fallback is defined once and logging reflects any
future default change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f0f898a7-d0e3-4a28-8e28-2ed1863c554e
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (66)
plugins/README.mdplugins/nemo-eval-author/README.mdplugins/nemo-eval-author/pyproject.tomlplugins/nemo-eval-author/src/nemo_eval_author_plugin/backend.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/cache.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/client.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/dataset_staging.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/REFERENCE.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/config.yamlplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/__init__.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/base.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/factory.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/harbor.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/holdout_utils.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/py.typedplugins/nemo-eval-author/src/nemo_eval_author_plugin/rationale.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/repository.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/tools.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/trace_analyzer.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/trace_explorer.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/util.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/workspace_entities.pyplugins/nemo-eval-author/tests/test_dataset_staging.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_eval_author_materialization.pyplugins/nemo-eval-author/tests/test_eval_author_run.pyplugins/nemo-eval-author/tests/test_evaluator_base.pyplugins/nemo-eval-author/tests/test_evaluator_factory.pyplugins/nemo-eval-author/tests/test_evaluator_harbor.pyplugins/nemo-eval-author/tests/test_trace_explorer_from_ref.pyplugins/nemo-experimentalist/AGENTS.mdplugins/nemo-experimentalist/benchmarks/run.pyplugins/nemo-experimentalist/pyproject.tomlplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/entities.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/coder.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/goal_tree.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/rationalizer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_scorer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/deps.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/otlp.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/run.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/resolve.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_eval_author_repair_e2e.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/test_deps.pyplugins/nemo-experimentalist/tests/test_eval_author_config.pyplugins/nemo-experimentalist/tests/test_experiment_cli.pyplugins/nemo-experimentalist/tests/test_experimentalist_analyzer.pyplugins/nemo-experimentalist/tests/test_experimentalist_backend.pyplugins/nemo-experimentalist/tests/test_experimentalist_benchmark.pyplugins/nemo-experimentalist/tests/test_experimentalist_run.pyplugins/nemo-experimentalist/tests/test_otlp.pypyproject.toml
🛑 Comments failed to post (1)
plugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/models.py (1)
138-155: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Stop command runs even when start was never attempted.
If
self._runtime.start is None,ValueErroris raised before any command runs, but theexcept BaseExceptionbranch still calls_stop_started_runtime(), executingself._runtime.stop(if set) against dependencies that were never started.🐛 Proposed fix
async def __aenter__(self) -> DependencyRuntime | None: """Start dependencies and return the runtime that was entered.""" if self._runtime is None: return None + if self._runtime.start is None: + raise ValueError("DependencyRuntime requires start") try: - if self._runtime.start is None: - raise ValueError("DependencyRuntime requires start") await run_dependency_command(self._runtime.start, "start")📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.async def __aenter__(self) -> DependencyRuntime | None: """Start dependencies and return the runtime that was entered.""" if self._runtime is None: return None if self._runtime.start is None: raise ValueError("DependencyRuntime requires start") try: await run_dependency_command(self._runtime.start, "start") if self._runtime.readiness is not None: await run_dependency_command(self._runtime.readiness, "readiness") except BaseException: try: await self._stop_started_runtime() except Exception: pass raise return self._runtime🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/evaluator/models.py` around lines 138 - 155, Update __aenter__ to call _stop_started_runtime only after the dependency start command has actually been attempted successfully; when self._runtime.start is None, raise ValueError without invoking the stop command. Preserve cleanup and exception propagation for failures occurring during or after run_dependency_command(self._runtime.start, "start").
Own only the agent package and AUTHOR_* config in the plugin; restore evaluator/staging/trace helpers to Experimentalist and install both via the experimentalist uv group to avoid a circular package dependency. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py (1)
185-203: 🎯 Functional Correctness | 🔴 Critical | ⚡ Quick winMake
task_slugstatic or acceptself.
self.task_slug(index, trace_ref)bindsself, but the method declares only(index, trace_ref). Every staging call raisesTypeErrorbefore copying tasks.Proposed fix
class InsightSuite: + `@staticmethod` def task_slug(index: int, trace_ref: str) -> str:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py` around lines 185 - 203, Update the task_slug method declaration to accept self, or mark it as static while preserving its current call from stage. Ensure the signature matches the invocation in InsightSuite.stage so staging proceeds without a TypeError.
🧹 Nitpick comments (2)
plugins/nemo-eval-author/README.md (1)
1-45: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winSeparate the README’s Diátaxis concerns.
This page mixes architecture explanation, API reference, and a credentials how-to. Move the API material to
REFERENCE.md; keep this page as a how-to with prerequisites first, parallel Python/CLI guidance where supported, and a Next Steps section.As per coding guidelines, each documentation page must fit one Diátaxis quadrant and include prerequisites and Next Steps.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-eval-author/README.md` around lines 1 - 45, Restructure the README as a how-to guide: add prerequisites first, retain standalone credential setup, provide parallel Python and CLI guidance where the CLI is supported, and finish with a Next Steps section. Move the “Public API” imports and API details to a new REFERENCE.md, and remove architecture/explanatory material that belongs outside the how-to while preserving the documented standalone and Experimentalist credential behavior.Source: Coding guidelines
plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py (1)
156-156: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftRetain equivalent orchestration coverage.
The deleted test covered staging train/validation/template before Eval Author execution, awaiting
shell.close, and preventing generated-task side effects after failure. Port it to the current runner or verify equivalent coverage intest_dataset_staging_loop.py.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py` at line 156, Restore the orchestration coverage removed from the dataset staging tests by porting it to the current runner or confirming equivalent coverage in test_dataset_staging_loop.py: verify train, validation, and template staging occurs before Eval Author execution, await shell.close, and ensure generated-task side effects do not occur after failure.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-eval-author/tests/conftest.py`:
- Around line 16-19: Update the AUTHOR_API_BASE and EXPERIMENTALIST_API_BASE
defaults in the test environment setup to use HTTPS URLs with non-routable
.invalid hosts. Keep the existing API key placeholders unchanged and ensure both
endpoints prevent accidental transmission of real credentials to an external
service.
---
Outside diff comments:
In
`@plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.py`:
- Around line 185-203: Update the task_slug method declaration to accept self,
or mark it as static while preserving its current call from stage. Ensure the
signature matches the invocation in InsightSuite.stage so staging proceeds
without a TypeError.
---
Nitpick comments:
In `@plugins/nemo-eval-author/README.md`:
- Around line 1-45: Restructure the README as a how-to guide: add prerequisites
first, retain standalone credential setup, provide parallel Python and CLI
guidance where the CLI is supported, and finish with a Next Steps section. Move
the “Public API” imports and API details to a new REFERENCE.md, and remove
architecture/explanatory material that belongs outside the how-to while
preserving the documented standalone and Experimentalist credential behavior.
In `@plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.py`:
- Line 156: Restore the orchestration coverage removed from the dataset staging
tests by porting it to the current runner or confirming equivalent coverage in
test_dataset_staging_loop.py: verify train, validation, and template staging
occurs before Eval Author execution, await shell.close, and ensure
generated-task side effects do not occur after failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 703ea3d1-98f3-4e8b-af48-131d6967557c
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (38)
plugins/nemo-eval-author/.example.envplugins/nemo-eval-author/README.mdplugins/nemo-eval-author/pyproject.tomlplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.mdplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/models.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.pyplugins/nemo-eval-author/tests/conftest.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_eval_author_materialization.pyplugins/nemo-eval-author/tests/test_eval_author_run.pyplugins/nemo-eval-author/tests/test_model_config.pyplugins/nemo-experimentalist/AGENTS.mdplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/coder.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/insight_promotion.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/loop.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/rationale.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/rationalizer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_analyzer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_scorer.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/experimentalist_backend.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/run.pyplugins/nemo-experimentalist/src/nemo_experimentalist_plugin/resolve.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging.pyplugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.pyplugins/nemo-experimentalist/tests/experimentalist/test_eval_author_repair_e2e.pyplugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.pyplugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.pyplugins/nemo-experimentalist/tests/experimentalist/test_trace_explorer_from_ref.pyplugins/nemo-experimentalist/tests/test_experiment_cli.pyplugins/nemo-experimentalist/tests/test_experimentalist_analyzer.pyplugins/nemo-experimentalist/tests/test_experimentalist_backend.pyplugins/nemo-experimentalist/tests/test_experimentalist_run.pypyproject.toml
🚧 Files skipped from review as they are similar to previous changes (9)
- plugins/nemo-experimentalist/tests/test_experimentalist_analyzer.py
- plugins/nemo-eval-author/pyproject.toml
- plugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/README.md
- plugins/nemo-experimentalist/src/nemo_experimentalist_plugin/experimentalist/components/trace_scorer.py
- plugins/nemo-experimentalist/tests/experimentalist/test_dataset_staging_loop.py
- plugins/nemo-eval-author/tests/test_eval_author_agent.py
- plugins/nemo-experimentalist/tests/experimentalist/test_eval_author_repair_e2e.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_insight_suite.py
- plugins/nemo-experimentalist/tests/experimentalist/test_loop_reporting.py
…entalist Eval Author ships in the 3.12-only experimentalist group, so its tests must be skipped when root CI collects on 3.11 and its nooa/harbor imports and agent method stubs need the same ty overrides Experimentalist already has. Also apply ruff import sorting and formatting missed by the module move. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
A plaintext placeholder base paired with a real key from the environment could send that key unencrypted on an unmocked call. Also document why the plugin declares a 3.11 floor while nooa and harbor are gated to 3.12. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Restore the repo lint config for the plugin. A local [tool.ruff] table had made the plugin's pyproject the root ruff config for that subtree, which disabled import sorting and reflowed the moved Eval Author modules at 88 columns, hiding that the extraction was a pure move. The E402 waiver that table existed for is gone too, replaced by an _env_bridge module imported for its side effect, so isort maintains the ordering the credential bridge depends on rather than a comment asking readers not to reshuffle imports. Only forward INFERENCE_API_KEY to the NVIDIA gateway over HTTPS, matching nemo_experimentalist_plugin.cli. Matching on hostname alone would have sent the key to a plain-http base in cleartext. Declare Experimentalist's dependency on Eval Author, which insight mode already imports at module scope. That is the arrow that survives, so pin the modules Eval Author still borrows in tests/test_plugin_boundary.py to keep the reverse coupling shrinking, and tag the transitional credential fallback and env bridge TODO(eval-author-standalone). Cover the previously untested bridge with a subprocess test that imports the agent with EXPERIMENTALIST_* stripped, move the Eval Author e2e canary into the plugin and fix a credential gate that could never skip, drop dead code and the unused pyyaml dependency, and rename .example.env to .env.example to match the rest of the repo. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-eval-author/.env.example`:
- Around line 5-7: Update the setup instructions in .env.example to remove the
implication that copying the file to .env enables automatic loading, and instead
direct users to export or source the variables in their shell until CLI
environment-file loading is implemented.
In `@plugins/nemo-eval-author/tests/test_eval_author_repair_e2e.py`:
- Around line 35-37: Update the test credential gate to resolve the API key
through the same model-config contract used by model_config._api_key(),
including supported INFERENCE_API_KEY configurations for the HTTPS NVIDIA
gateway. Preserve the existing base-URL resolution and placeholder checks, while
ensuring valid live-canary credentials are not skipped.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 96ebb518-de69-483b-9b10-e20d194c881d
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (19)
.gitignoreplugins/nemo-eval-author/.env.exampleplugins/nemo-eval-author/README.mdplugins/nemo-eval-author/pyproject.tomlplugins/nemo-eval-author/src/nemo_eval_author_plugin/_env_bridge.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/agent.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/materialization.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/eval_author/run.pyplugins/nemo-eval-author/src/nemo_eval_author_plugin/model_config.pyplugins/nemo-eval-author/tests/conftest.pyplugins/nemo-eval-author/tests/test_eval_author_agent.pyplugins/nemo-eval-author/tests/test_eval_author_materialization.pyplugins/nemo-eval-author/tests/test_eval_author_repair_e2e.pyplugins/nemo-eval-author/tests/test_eval_author_run.pyplugins/nemo-eval-author/tests/test_model_config.pyplugins/nemo-eval-author/tests/test_plugin_boundary.pyplugins/nemo-experimentalist/AGENTS.mdplugins/nemo-experimentalist/pyproject.tomlpyproject.toml
💤 Files with no reviewable changes (1)
- plugins/nemo-eval-author/pyproject.toml
🚧 Files skipped from review as they are similar to previous changes (4)
- plugins/nemo-eval-author/tests/conftest.py
- plugins/nemo-eval-author/README.md
- pyproject.toml
- plugins/nemo-experimentalist/AGENTS.md
…triple The _completion_client docstring called its arguments a "model triple", which made a reader decode the term before learning what the cache is keyed on. Name the three values, and say why the base URL and key are part of the key rather than leaving that implicit: reusing a client across endpoints or credentials would be a correctness bug, not just a missed optimization. Also record which alternative was rejected. The previous note asserted that the duplicate client Experimentalist holds "is the price of Eval Author not importing from it", which only reads as a reason if you already know an earlier revision shared one factory and was reverted to keep this module standalone. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
main dropped Python 3.11 (requires-python is now >=3.12 and ty resolves on 3.12), which turns both conflicting hunks in the root pyproject.toml into obsolete 3.11 accommodations rather than changes worth keeping: - the experimentalist dependency group no longer needs "; python_full_version >= '3.12'", so it lists nemo-experimentalist-plugin and nemo-eval-author-plugin unmarked. - the unresolved-import = "ignore" ty override existed only because nooa and harbor went unresolved when ty resolved on 3.11. main deleted it and this merge does not resurrect it; verified that neither plugin reports an unresolved-import diagnostic without it. Carried the same cleanup into the two plugin manifests, matching what main already did for Experimentalist. nemo-eval-author declared >=3.11 while hard-depending on nemo-experimentalist-plugin, which main raised to >=3.12, so it claimed support for a Python version where its own dependency cannot be installed. Raised to >=3.12,<3.14, dropped the now-vacuous harbor and nooa markers, and dropped the vacuous marker on Experimentalist's dependency back on nemo-eval-author-plugin. Reworded the two root test-discovery exclusion reasons, which described themselves as 3.11 artifacts. The find_spec guard is still correct, since the exclusion fires whenever the optional experimentalist group is not synced into the root environment, so only the text changed. It now matches the version-agnostic phrasing already used for insights. uv.lock was re-resolved from main's copy rather than hand-merged; the only delta is the added nemo-eval-author-plugin entry. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
ryana
approved these changes
Jul 29, 2026
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
plugins/nemo-eval-author/(nemo-eval-author-plugin) that owns the agent package (eval_author/) plus anAUTHOR_*model_config.EvalAuthorat runtime when both packages are installed viauv sync --group experimentalist.Dependency model
experimentalistuv groupThis is intentionally not the Insights packaging pattern. Insights works as Experimentalist → Insights because Analyst does not need Experimentalist internals. Eval Author does, so the package arrow flips; the product relationship (Experimentalist consumes Eval Author in insight mode) stays the same via co-install.
What lives where
Eval Author plugin owns
eval_author/—EvalAuthorNOOA agent, materialization, models,run_eval_authormodel_config.py— prefersAUTHOR_*, falls back toEXPERIMENTALIST_*, bridges into unset Experimentalist slots for shared helpers; acceptsINFERENCE_API_KEYon the NVIDIA gateway.example.env— standalone credential template (CLI not wired yet)Experimentalist keeps
evaluator/(Harbor + models + factory)dataset_staging.pytrace_analyzer.py/trace_explorer.pytools.py,cache.py,client.py, and other shared NOOA runtime helpersTests / CI
plugins/nemo-eval-author/tests/.EvolutionaryOptimizerConfignesting / legacycuratorrejection stay in Experimentalist.tests/discovery_exclusions.py);tyoverrides cover Eval Author'snooa/harborimports and agent empty bodies.Test plan
uv sync --group experimentalistuv run --frozen pytest -q plugins/nemo-eval-author/tests plugins/nemo-experimentalist/tests(575 passed, 2 credential-gated skips)uv run ruff check/ruff format --checkclean for touched pluginstyCI ignores for Eval Author mirror ExperimentalistEvalAuthorSummary by CodeRabbit
New Features
Documentation
Tests