feat: remove run execution - #601
Closed
mckornfield wants to merge 1 commit into
Closed
Conversation
Signed-off-by: Matt Kornfield <mkornfield@nvidia.com>
Contributor
|
🌿 Preview your docs: https://nvidia-preview-run-submit-delete-mck.docs.buildwithfern.com/nemo-platform |
Contributor
|
Contributor
📝 WalkthroughWalkthroughChangesPR removes local ("run") execution across the NeMo platform plugin framework and all product plugins (Data Designer, Anonymizer, Auditor, Evaluator, Agents), standardizing on Platform Plugin Framework
Product Plugins (Data Designer, Anonymizer, Auditor, Evaluator, Agents) Each plugin's CLI/SDK/job code drops Sequence Diagram(s)sequenceDiagram
participant CLI
participant Scheduler as NemoJobScheduler
participant Service as PluginService
CLI->>Scheduler: submit_remote(job_cls, spec)
Scheduler->>Service: POST /jobs/{job}
Service-->>Scheduler: job response
Scheduler-->>CLI: submitted job
sequenceDiagram
participant Client
participant Executor
participant JobService
Client->>Executor: evaluate(metric, dataset)
Executor->>JobService: submit job
JobService-->>Executor: completed result
Executor-->>Client: EvaluationResult
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Tools execution failed with the following error: Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error) Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
plugins/nemo-data-designer/tests/unit/test_spec.py (1)
42-42: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winDuplicate test function name drops coverage.
test_local_file_seeds_are_rejectedis defined twice (Line 42 and Line 89). The second definition shadows the first, so pytest never collects/runs the original test.🐛 Proposed fix: rename one of the tests
-def test_local_file_seeds_are_rejected() -> None: +def test_local_file_seeds_are_rejected_with_clear_error() -> None: with tempfile.NamedTemporaryFile(suffix=".parquet") as tmpfile: config = DataDesignerJobConfig(Apply to the second occurrence (Line 89).
Also applies to: 89-89
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-data-designer/tests/unit/test_spec.py` at line 42, The unit test suite has a duplicate function name, so the second test shadows the first and prevents both from being collected. Rename the later test definition of test_local_file_seeds_are_rejected in test_spec to a unique, descriptive name while keeping the existing assertions and behavior unchanged.plugins/nemo-agents/src/nemo_agents_plugin/skills/skills-optimization/SKILL.md (1)
107-120: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAdd
outputto theevaluate-suitequick-start example.The submit schema requires
output, so this command will 422 as written.🛠️ Proposed fix
- nemo agents evaluate-suite submit --spec '{ - "evals": "tests/agentic-use", - "agent": ".", - "filter_glob": "auth-authorization-cli", - "concurrency": 1 - }' + nemo agents evaluate-suite submit --spec '{ + "evals": "tests/agentic-use", + "agent": ".", + "output": "./runs/batch-<timestamp>", + "filter_glob": "auth-authorization-cli", + "concurrency": 1 + }'🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/src/nemo_agents_plugin/skills/skills-optimization/SKILL.md` around lines 107 - 120, The `evaluate-suite submit` quick-start example is missing the required `output` field, so the command will fail validation. Update the example in `SKILL.md` under the `evaluate-suite submit` snippet to include an explicit `output` value in the JSON passed to `nemo agents evaluate-suite submit`, keeping the rest of the example unchanged.plugins/nemo-auditor/README.md (1)
82-95: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winStale paragraph contradicts the new submit flow.
Lines 82-88 now describe
run()as submitting a job (print(job["name"])), but lines 91-95 immediately below still sayrun()"shells out to a pre-installed garak interpreter" and registers results "under a temp directory managed by the local scheduler." That describes the old in-process behavior and directly contradicts the paragraph above it.Proposed fix
-`run()` shells out to a pre-installed garak interpreter (default -`~/.auditor/.venv/bin/python`, override via `$NEMO_AUDITOR_GARAK_PYTHON`) -and registers the resulting JSONL / HTML / hitlog reports as job results -under a temp directory managed by the local scheduler. +`run()` submits an `auditor.audit` job to the Jobs backend, which shells out +to a pre-installed garak interpreter (default `~/.auditor/.venv/bin/python`, +override via `$NEMO_AUDITOR_GARAK_PYTHON`) and registers the resulting +JSONL / HTML / hitlog reports as job artifacts.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-auditor/README.md` around lines 82 - 95, The README text for the auditor submit flow is outdated and conflicts with the new `client.auditor.run()` usage shown above. Update the paragraph after the example to describe the current submission behavior instead of saying it shells out to a pre-installed garak interpreter and registers reports under the local scheduler’s temp directory; keep the description aligned with `run()`, `client.auditor.run`, and the persisted-entity/job submission flow.
🧹 Nitpick comments (9)
plugins/nemo-auditor/tests/test_sdk_resources.py (1)
294-340: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAssertions correctly validate POST-based submission (URL, JSON spec shape, workspace default, non-object rejection). Repeated
_ok_response({"name": "audit-job", "status": "created"}, status_code=201)across tests could be a fixture, but low payoff for test code.Also applies to: 364-375
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-auditor/tests/test_sdk_resources.py` around lines 294 - 340, The repeated _ok_response setup in TestSyncRun is duplicated across multiple test methods and should be factored out for maintainability. Add a shared fixture or helper in the test module for the common successful submission response, then update AuditorPluginResource.run tests to use it consistently while keeping the existing assertions unchanged.plugins/nemo-anonymizer/tests/unit/test_run_job.py (1)
126-188: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDuplicate registry/request setup across two tests.
Same
ModelProviderRegistry/NDDModelProvider/AnonymizerRequestconstruction repeated in both tests. Extract to a fixture or helper.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-anonymizer/tests/unit/test_run_job.py` around lines 126 - 188, The two tests repeat the same ModelProviderRegistry, NDDModelProvider, and AnonymizerRequest setup, so factor that shared setup into a reusable fixture or helper. Update test_run_submit_model_configs_uses_injected_async_sdk and test_run_submit_serialized_step_config_can_be_revalidated to call the shared setup instead of constructing the registry and request inline, keeping only the test-specific monkeypatching and assertions in each test.plugins/nemo-agents/examples/agent-improver.example.yml (1)
4-10: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winStale "run"/"locally" wording after submit-only migration.
Line 4 says "and run:" and line 8 says "instead of running locally" but both example commands (line 6, line 10) now use
submit. Since local execution is removed, this framing no longer matches — clarify the distinction is submit-without-cluster vs submit-with-cluster.📝 Suggested wording fix
-# retarget the paths, and run: +# retarget the paths, and submit: # # nemo agents optimize-skills submit --spec-file .agent-improver.yml # -# To submit to a cluster instead of running locally: +# To submit to a remote cluster instead of the default target:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/examples/agent-improver.example.yml` around lines 4 - 10, The example text in agent-improver.example.yml still refers to “run” and “running locally” even though the commands now use nemo agents optimize-skills submit. Update the surrounding wording near the optimize-skills submit examples to describe a submit-only flow, using the existing example command blocks to distinguish submitting with a spec file versus submitting to a cluster, and remove any local-execution framing.plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py (1)
9-23: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract shared submit-and-wait helper.
evaluate_benchmark(sync at Lines 306-316, async at Lines 490-500) duplicates the create/wait_until_done/get_result sequence already used inevaluate_remote(Lines 215-224, 429-438). Consider a shared private helper (e.g._create_and_wait(spec) -> job) reused by both, since only the result-extraction step differs.♻️ Example extraction (sync side)
+ def _create_and_wait(self, spec: EvaluateInputSpec) -> EvaluatorJobResource: + job = self.create( + spec=spec, workspace=http_utils.resolve_workspace(self._platform, self._workspace, strict=True) + ) + job.wait_until_done( + poll_interval_seconds=self._poll_interval_seconds, + job_timeout_seconds=self._job_timeout_seconds, + pending_timeout_seconds=self._pending_timeout_seconds, + ) + return job + def evaluate_remote(...) -> EvaluationResult: ... - job = self.create( - spec=spec, workspace=http_utils.resolve_workspace(self._platform, self._workspace, strict=True) - ) - job.wait_until_done( - poll_interval_seconds=self._poll_interval_seconds, - job_timeout_seconds=self._job_timeout_seconds, - pending_timeout_seconds=self._pending_timeout_seconds, - ) + job = self._create_and_wait(spec) return job.get_result(aggregate_fields=aggregate_fields) def evaluate_benchmark(...) -> BenchmarkEvaluationResult: ... - job = self.create( - spec=spec, workspace=http_utils.resolve_workspace(self._platform, self._workspace, strict=True) - ) - job.wait_until_done( - poll_interval_seconds=self._poll_interval_seconds, - job_timeout_seconds=self._job_timeout_seconds, - pending_timeout_seconds=self._pending_timeout_seconds, - ) + job = self._create_and_wait(spec) result = job.get_result()Same pattern applies to the async executor. Otherwise the remote submission logic (create → wait → result → normalize) looks correct.
Also applies to: 118-118, 237-250, 282-317, 440-464, 466-501
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py` around lines 9 - 23, The submit-and-wait flow is duplicated across EvaluatorJobResource and AsyncEvaluatorJobResource methods, especially in evaluate_benchmark and evaluate_remote. Extract the repeated create_job/request_job -> wait_until_done -> get_result sequence into a shared private helper (for example, a _create_and_wait-style method on the executor classes) and reuse it from both sync and async paths, keeping only the final result-normalization step in the caller.packages/nemo_platform_plugin/tests/test_dispatcher.py (1)
338-391: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRepeated local import of
RunDependencyError.Imported inline in three separate test methods (lines 343, 366, 381). Hoisting to module-level would remove duplication.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_platform_plugin/tests/test_dispatcher.py` around lines 338 - 391, The tests in the dispatcher module repeat the same inline import of RunDependencyError in multiple methods, creating avoidable duplication. Move that import to the top of the test file and update the affected tests (including test_run_dependency_error_propagates, test_run_dependency_error_from_job_run_propagates, and test_unsupported_required_run_param_raises_run_dependency_error) to use the shared module-level symbol instead of importing it locally.packages/nemo_platform_plugin/tests/test_commands.py (1)
72-84: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider using
_patch_job_submithelper consistently.Several tests (lines 247, 291) hand-roll a
submit_remotemonkeypatch instead of reusing_patch_job_submit. Minor duplication; not urgent.Also applies to: 141-141, 170-170, 192-192, 199-199, 247-247, 291-291, 698-698
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_platform_plugin/tests/test_commands.py` around lines 72 - 84, Several tests are duplicating the same submit_remote monkeypatch logic instead of reusing the existing _patch_job_submit helper. Update the affected tests to call _patch_job_submit consistently and use the returned captured data where needed, rather than hand-rolling a NemoJobScheduler.submit_remote patch. This should be applied in the test_commands.py cases that currently define their own submit_remote monkeypatch so the test setup stays centralized and easier to maintain.docs/anonymizer/tutorials/index.mdx (1)
80-80: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winHardcoded product name; use
{{platform_name}}.Rest of this table/file uses the substitution (e.g. adjacent doc row "{{platform_name}} Fileset"). This row hardcodes "NeMo Platform" instead.
✏️ Fix
-| **Artifacts** | Platform-managed | `run submit` stores artifacts in NeMo Platform job storage. | +| **Artifacts** | Platform-managed | `run submit` stores artifacts in {{platform_name}} job storage. |🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/anonymizer/tutorials/index.mdx` at line 80, The Artifacts row in the documentation table hardcodes the product name instead of using the existing substitution. Update the row in the tutorial table to use {{platform_name}} consistently, matching the nearby rows and the rest of the file, so the text reads with the same placeholder style as the adjacent {{platform_name}} references.packages/nemo_platform_plugin/src/nemo_platform_plugin/commands.py (1)
227-255: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winEntry-point key discarded here, forcing a redundant re-discovery scan per submit/explain.
add_job_commandsiteratesjobs.values(), dropping the entry-point key it already has (jobs: dict[str, type[NemoJob]]). Downstream,scheduler._api_segment_forhas to re-rundiscover_jobs()— a full re-scan/import of everynemo.jobsentry point — on everysubmit/explaininvocation just to look the key back up. Threading the known key through_register_job_subgroup→_add_submit_command/_add_explain_command→NemoJobSchedulerwould avoid re-discovering plugins that are already enumerated.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/commands.py` around lines 227 - 255, add_job_commands is dropping the entry-point key by iterating jobs.values(), which forces NemoJobScheduler._api_segment_for to rediscover all jobs later. Thread the existing job key from add_job_commands through _register_job_subgroup and the submit/explain command builders into NemoJobScheduler so the scheduler can use the known key directly instead of calling discover_jobs() again.packages/nemo_platform_plugin/src/nemo_platform_plugin/scheduler.py (1)
230-264: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winSilent
except Exceptionin_api_segment_forhides discovery failures.If
discover_jobs()raises for any reason, the fallback silently kicks in with no log line — on a path that runs for every submit/explain call and can produce a subtly-wrong URL segment (per the documented_plugin-suffix caveat) with no trace to debug it. Add a debug/warning log in the except block.Based on learnings: "Broken plugin imports should only trigger a warning and cause the plugin to be skipped at startup; the platform should continue running" — the same warn-don't-swallow-silently principle should apply here.
🪵 Proposed fix
try: registered = discover_jobs() - except Exception: + except Exception: + logger.debug("discover_jobs() failed while resolving API segment for %s", job_cls, exc_info=True) registered = {}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/scheduler.py` around lines 230 - 264, The broad except in _api_segment_for silently hides failures from discover_jobs(), making submit/explain URL derivation hard to debug when the fallback is used. Update the exception handler to emit a warning or debug log with the exception details before falling back to the module-name heuristic. Keep the existing fallback behavior in _api_segment_for, but make the failure observable so incorrect API segments can be traced.Source: Learnings
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/anonymizer/sdk-resources.mdx`:
- Line 78: The wording for download_artifacts in the sdk-resources docs is
inaccurate because it says the AnonymizerJobResults object loads parquet/JSON
artifacts into memory eagerly. Update the text near download_artifacts to
describe it as a results wrapper that supports on-demand/lazy loading of
artifacts instead of implying immediate in-memory loading, keeping the phrasing
aligned with AnonymizerJobResults behavior.
In `@plugins/nemo-agents/src/nemo_agents_plugin/improvement/README.md`:
- Around line 30-36: The README example under the NemoJob submit/explain verbs
uses `.agent-improver.yml` for the `analyze` command, but `analyze` expects a
batch spec with `batch` and `format`. Update the example tied to the `nemo
agents analyze submit` usage to reference the correct analyze spec instead of
the improvement spec, while leaving the other job examples unchanged.
In `@plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.md`:
- Around line 315-318: The evaluate-suite example is missing the required output
field, so update the inline payload used with nemo agents evaluate-suite submit
to include output along with evals and agent. Make this change in the SKILL.md
example that shows the EvaluateSuiteSubmitConfig usage so it matches the current
validation requirements and does not fail schema checks.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md`:
- Around line 15-22: The autopilot flow in the anonymizer workflow should stop
immediately if the service mount check fails instead of continuing to later
steps. Update the mount-check logic in the anonymizer autopilot instructions so
that after the `curl`/`jq`/`grep` probe, a missing `/apis/anonymizer/` path ends
the flow and tells the user to run `nemo services run`; keep the existing
preview/run guidance separate so it only applies after the mount is confirmed.
In `@plugins/nemo-evaluator/src/nemo_evaluator/docs/index.md`:
- Line 12: The page has conflicting descriptions of how the job runs: the table
entry for `nemo.jobs:evaluator.evaluate` says it backs durable platform job
submission, while the “Current Job” section still describes direct `Evaluator`
execution. Update the docs in `index.md` so the `Current Job` section and the
job table agree on one execution model, using the existing
`nemo.jobs:evaluator.evaluate` and `Evaluator` references to align the wording
consistently.
---
Outside diff comments:
In
`@plugins/nemo-agents/src/nemo_agents_plugin/skills/skills-optimization/SKILL.md`:
- Around line 107-120: The `evaluate-suite submit` quick-start example is
missing the required `output` field, so the command will fail validation. Update
the example in `SKILL.md` under the `evaluate-suite submit` snippet to include
an explicit `output` value in the JSON passed to `nemo agents evaluate-suite
submit`, keeping the rest of the example unchanged.
In `@plugins/nemo-auditor/README.md`:
- Around line 82-95: The README text for the auditor submit flow is outdated and
conflicts with the new `client.auditor.run()` usage shown above. Update the
paragraph after the example to describe the current submission behavior instead
of saying it shells out to a pre-installed garak interpreter and registers
reports under the local scheduler’s temp directory; keep the description aligned
with `run()`, `client.auditor.run`, and the persisted-entity/job submission
flow.
In `@plugins/nemo-data-designer/tests/unit/test_spec.py`:
- Line 42: The unit test suite has a duplicate function name, so the second test
shadows the first and prevents both from being collected. Rename the later test
definition of test_local_file_seeds_are_rejected in test_spec to a unique,
descriptive name while keeping the existing assertions and behavior unchanged.
---
Nitpick comments:
In `@docs/anonymizer/tutorials/index.mdx`:
- Line 80: The Artifacts row in the documentation table hardcodes the product
name instead of using the existing substitution. Update the row in the tutorial
table to use {{platform_name}} consistently, matching the nearby rows and the
rest of the file, so the text reads with the same placeholder style as the
adjacent {{platform_name}} references.
In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/commands.py`:
- Around line 227-255: add_job_commands is dropping the entry-point key by
iterating jobs.values(), which forces NemoJobScheduler._api_segment_for to
rediscover all jobs later. Thread the existing job key from add_job_commands
through _register_job_subgroup and the submit/explain command builders into
NemoJobScheduler so the scheduler can use the known key directly instead of
calling discover_jobs() again.
In `@packages/nemo_platform_plugin/src/nemo_platform_plugin/scheduler.py`:
- Around line 230-264: The broad except in _api_segment_for silently hides
failures from discover_jobs(), making submit/explain URL derivation hard to
debug when the fallback is used. Update the exception handler to emit a warning
or debug log with the exception details before falling back to the module-name
heuristic. Keep the existing fallback behavior in _api_segment_for, but make the
failure observable so incorrect API segments can be traced.
In `@packages/nemo_platform_plugin/tests/test_commands.py`:
- Around line 72-84: Several tests are duplicating the same submit_remote
monkeypatch logic instead of reusing the existing _patch_job_submit helper.
Update the affected tests to call _patch_job_submit consistently and use the
returned captured data where needed, rather than hand-rolling a
NemoJobScheduler.submit_remote patch. This should be applied in the
test_commands.py cases that currently define their own submit_remote monkeypatch
so the test setup stays centralized and easier to maintain.
In `@packages/nemo_platform_plugin/tests/test_dispatcher.py`:
- Around line 338-391: The tests in the dispatcher module repeat the same inline
import of RunDependencyError in multiple methods, creating avoidable
duplication. Move that import to the top of the test file and update the
affected tests (including test_run_dependency_error_propagates,
test_run_dependency_error_from_job_run_propagates, and
test_unsupported_required_run_param_raises_run_dependency_error) to use the
shared module-level symbol instead of importing it locally.
In `@plugins/nemo-agents/examples/agent-improver.example.yml`:
- Around line 4-10: The example text in agent-improver.example.yml still refers
to “run” and “running locally” even though the commands now use nemo agents
optimize-skills submit. Update the surrounding wording near the optimize-skills
submit examples to describe a submit-only flow, using the existing example
command blocks to distinguish submitting with a spec file versus submitting to a
cluster, and remove any local-execution framing.
In `@plugins/nemo-anonymizer/tests/unit/test_run_job.py`:
- Around line 126-188: The two tests repeat the same ModelProviderRegistry,
NDDModelProvider, and AnonymizerRequest setup, so factor that shared setup into
a reusable fixture or helper. Update
test_run_submit_model_configs_uses_injected_async_sdk and
test_run_submit_serialized_step_config_can_be_revalidated to call the shared
setup instead of constructing the registry and request inline, keeping only the
test-specific monkeypatching and assertions in each test.
In `@plugins/nemo-auditor/tests/test_sdk_resources.py`:
- Around line 294-340: The repeated _ok_response setup in TestSyncRun is
duplicated across multiple test methods and should be factored out for
maintainability. Add a shared fixture or helper in the test module for the
common successful submission response, then update AuditorPluginResource.run
tests to use it consistently while keeping the existing assertions unchanged.
In `@plugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.py`:
- Around line 9-23: The submit-and-wait flow is duplicated across
EvaluatorJobResource and AsyncEvaluatorJobResource methods, especially in
evaluate_benchmark and evaluate_remote. Extract the repeated
create_job/request_job -> wait_until_done -> get_result sequence into a shared
private helper (for example, a _create_and_wait-style method on the executor
classes) and reuse it from both sync and async paths, keeping only the final
result-normalization step in the caller.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f56b864c-8bb2-403b-b3e2-b17126e2e5c9
📒 Files selected for processing (141)
docs/agents/index.mdxdocs/agents/optimization.mdxdocs/agents/plugins.mdxdocs/anonymizer/cli.mdxdocs/anonymizer/index.mdxdocs/anonymizer/sdk-resources.mdxdocs/anonymizer/tutorials/index.mdxdocs/anonymizer/tutorials/preview.mdxdocs/anonymizer/tutorials/run.mdxdocs/auditor/configs/schema.mdxdocs/auditor/index.mdxdocs/auditor/sdk-resources.mdxdocs/auditor/tutorials/index.mdxdocs/auditor/tutorials/run-audit-locally.mdxdocs/data-designer/cli.mdxdocs/data-designer/execution-modes.mdxdocs/data-designer/index.mdxdocs/data-designer/migration.mdxdocs/data-designer/sdk-resources.mdxdocs/data-designer/tutorials/basics.mdxdocs/data-designer/tutorials/index.mdxdocs/data-designer/tutorials/seeding.mdxdocs/evaluator/index.mdxdocs/evaluator/metrics/agent-configuration.mdxdocs/evaluator/metrics/agentic.mdxdocs/evaluator/metrics/llm-as-a-judge.mdxdocs/evaluator/metrics/manage-metrics.mdxdocs/evaluator/metrics/model-configuration.mdxdocs/evaluator/metrics/rag.mdxdocs/evaluator/metrics/remote.mdxdocs/evaluator/metrics/results.mdxdocs/evaluator/metrics/similarity.mdxdocs/evaluator/sdk-resources.mdxdocs/evaluator/test_doc_examples.pydocs/evaluator/tutorials/define-run-custom-python-metrics.mdxdocs/evaluator/tutorials/run-llm-judge-evaluation.mdxpackages/nemo_platform_plugin/AGENTS.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/creating-a-plugin/SKILL.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-function/SKILL.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/.agents/skills/plugin-job/SKILL.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/README.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/cli.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/cli_renderer.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/cli_state.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/commands.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/discovery.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/docs/ARCHITECTURE.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/docs/JOB.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/docs/QUICKSTART.mdpackages/nemo_platform_plugin/src/nemo_platform_plugin/function.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/function_context.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/job.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/job_context.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/job_results.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/run_dependencies.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/scheduler.pypackages/nemo_platform_plugin/src/nemo_platform_plugin/tasks/dispatcher.pypackages/nemo_platform_plugin/tests/test_cli_hooks.pypackages/nemo_platform_plugin/tests/test_cli_renderer.pypackages/nemo_platform_plugin/tests/test_cli_state.pypackages/nemo_platform_plugin/tests/test_commands.pypackages/nemo_platform_plugin/tests/test_dispatcher.pypackages/nemo_platform_plugin/tests/test_run_dependencies.pypackages/nemo_platform_plugin/tests/test_scheduler.pyplugins/example-plugin/src/nemo_example_plugin/jobs/say_hello.pyplugins/example-plugin/tests/test_say_hello_job.pyplugins/nemo-agents/README.mdplugins/nemo-agents/examples/agent-improver.example.ymlplugins/nemo-agents/examples/calculator-agent/src/calculator_agent/calculator-eval.ymlplugins/nemo-agents/examples/calculator-agent/src/calculator_agent/calculator-optimize.ymlplugins/nemo-agents/examples/react-agent/react-eval.ymlplugins/nemo-agents/examples/react-agent/react-optimize.ymlplugins/nemo-agents/openapi/openapi.yamlplugins/nemo-agents/src/nemo_agents_plugin/improvement/GETTING_STARTED.mdplugins/nemo-agents/src/nemo_agents_plugin/improvement/README.mdplugins/nemo-agents/src/nemo_agents_plugin/jobs/analyze_batch.pyplugins/nemo-agents/src/nemo_agents_plugin/jobs/evaluate_agent.pyplugins/nemo-agents/src/nemo_agents_plugin/jobs/evaluate_suite.pyplugins/nemo-agents/src/nemo_agents_plugin/jobs/optimize_agent.pyplugins/nemo-agents/src/nemo_agents_plugin/jobs/optimize_skills.pyplugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.mdplugins/nemo-agents/src/nemo_agents_plugin/skills/skills-optimization/SKILL.mdplugins/nemo-agents/tests/unit/conftest.pyplugins/nemo-anonymizer/README.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/app/context.pyplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/functions/preview.pyplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/jobs/run.pyplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/SKILL.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/inputs.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/model-configs.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/preview-review.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/replace-strategies.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/references/rewrite-mode.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/interactive.mdplugins/nemo-anonymizer/src/nemo_anonymizer_plugin/tasks/anonymizer/run.pyplugins/nemo-anonymizer/tests/unit/test_preview_function.pyplugins/nemo-anonymizer/tests/unit/test_run_job.pyplugins/nemo-auditor/README.mdplugins/nemo-auditor/src/nemo_auditor/jobs/audit.pyplugins/nemo-auditor/src/nemo_auditor/sdk.pyplugins/nemo-auditor/src/nemo_auditor/skills/auditor/SKILL.mdplugins/nemo-auditor/tests/test_sdk_resources.pyplugins/nemo-data-designer/README.mdplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/inputs.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/main.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/renderers.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/cli/validate.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/functions/_preview_logs.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/functions/_types.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/functions/preview.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/jobs/bridge.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/jobs/create.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/jobs/run.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/jobs/spec.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/sdk/resources.pyplugins/nemo-data-designer/src/nemo_data_designer_plugin/sdk/validation.pyplugins/nemo-data-designer/tests/integration/test_preview_local_cli.pyplugins/nemo-data-designer/tests/integration/test_validate_cli.pyplugins/nemo-data-designer/tests/integration/test_validate_sdk.pyplugins/nemo-data-designer/tests/unit/test_create_job.pyplugins/nemo-data-designer/tests/unit/test_preview_function.pyplugins/nemo-data-designer/tests/unit/test_spec.pyplugins/nemo-evaluator/README.mdplugins/nemo-evaluator/examples/plugin_examples.pyplugins/nemo-evaluator/src/nemo_evaluator/docs/index.mdplugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/metric_resolution.pyplugins/nemo-evaluator/src/nemo_evaluator/jobs/result_persistence.pyplugins/nemo-evaluator/src/nemo_evaluator/metric_refs.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/_executor.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/fs_utils.pyplugins/nemo-evaluator/src/nemo_evaluator/sdk/resources.pyplugins/nemo-evaluator/src/nemo_evaluator/shared/metric_bundles/defaults.pyplugins/nemo-evaluator/tests/integration/test_agent_evaluate_job.pyplugins/nemo-evaluator/tests/test_agent_evaluate.pyplugins/nemo-evaluator/tests/test_evaluate_job.pyplugins/nemo-evaluator/tests/test_inline_bundle_execution.pyplugins/nemo-evaluator/tests/test_result_persistence.pyplugins/nemo-evaluator/tests/test_sdk.py
💤 Files with no reviewable changes (3)
- plugins/nemo-data-designer/src/nemo_data_designer_plugin/jobs/bridge.py
- plugins/nemo-evaluator/src/nemo_evaluator/sdk/fs_utils.py
- plugins/nemo-data-designer/tests/integration/test_preview_local_cli.py
| results = AnonymizerJobResults(Path("/path/to/persistent/results/artifacts")) | ||
| dataset = results.load_dataset() | ||
| ``` | ||
| `download_artifacts` returns an `AnonymizerJobResults` object that loads parquet / JSON artifacts into memory. |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use on-demand loading wording here.
download_artifacts() returns a results wrapper; it doesn't eagerly load the parquet/JSON payloads into memory. Rephrase this so it matches the lazy-loading behavior.
Proposed wording
-`download_artifacts` returns an `AnonymizerJobResults` object that loads parquet / JSON artifacts into memory.
+`download_artifacts` returns an `AnonymizerJobResults` object that lets you load the parquet / JSON artifacts on demand.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| `download_artifacts` returns an `AnonymizerJobResults` object that loads parquet / JSON artifacts into memory. | |
| `download_artifacts` returns an `AnonymizerJobResults` object that lets you load the parquet / JSON artifacts on demand. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/anonymizer/sdk-resources.mdx` at line 78, The wording for
download_artifacts in the sdk-resources docs is inaccurate because it says the
AnonymizerJobResults object loads parquet/JSON artifacts into memory eagerly.
Update the text near download_artifacts to describe it as a results wrapper that
supports on-demand/lazy loading of artifacts instead of implying immediate
in-memory loading, keeping the phrasing aligned with AnonymizerJobResults
behavior.
Comment on lines
+30
to
+36
| NemoJob with `submit` / `explain` verbs: | ||
|
|
||
| ```bash | ||
| # Run locally, in-process — daily use | ||
| nemo agents evaluate-suite run --spec-file ./.agent-improver.yml | ||
| nemo agents analyze run --spec-file ./.agent-improver.yml | ||
| nemo agents optimize-skills run --spec-file ./.agent-improver.yml | ||
|
|
||
| # Submit to a cluster | ||
| nemo agents optimize-skills submit --spec-file ./.agent-improver.yml --cluster <url> | ||
| # Submit improvement workflow jobs | ||
| nemo agents evaluate-suite submit --spec-file ./.agent-improver.yml | ||
| nemo agents analyze submit --spec-file ./.agent-improver.yml | ||
| nemo agents optimize-skills submit --spec-file ./.agent-improver.yml |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use an analyze spec here, not .agent-improver.yml.
analyze expects a batch spec (batch + format), so this example will mislead users as written.
🛠️ Proposed fix
- nemo agents analyze submit --spec-file ./.agent-improver.yml
+ nemo agents analyze submit --spec '{"batch": "./runs/batch-<timestamp>", "format": "md"}'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| NemoJob with `submit` / `explain` verbs: | |
| ```bash | |
| # Run locally, in-process — daily use | |
| nemo agents evaluate-suite run --spec-file ./.agent-improver.yml | |
| nemo agents analyze run --spec-file ./.agent-improver.yml | |
| nemo agents optimize-skills run --spec-file ./.agent-improver.yml | |
| # Submit to a cluster | |
| nemo agents optimize-skills submit --spec-file ./.agent-improver.yml --cluster <url> | |
| # Submit improvement workflow jobs | |
| nemo agents evaluate-suite submit --spec-file ./.agent-improver.yml | |
| nemo agents analyze submit --spec-file ./.agent-improver.yml | |
| nemo agents optimize-skills submit --spec-file ./.agent-improver.yml | |
| NemoJob with `submit` / `explain` verbs: | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-agents/src/nemo_agents_plugin/improvement/README.md` around
lines 30 - 36, The README example under the NemoJob submit/explain verbs uses
`.agent-improver.yml` for the `analyze` command, but `analyze` expects a batch
spec with `batch` and `format`. Update the example tied to the `nemo agents
analyze submit` usage to reference the correct analyze spec instead of the
improvement spec, while leaving the other job examples unchanged.
Comment on lines
+315
to
+318
| nemo agents evaluate submit --agent <name> --eval-config <yaml> | ||
| nemo agents optimize submit --agent <name> --optimize-config <yaml> | ||
| nemo agents optimize-skills submit --spec-file .agent-improver.yml | ||
| nemo agents evaluate-suite submit --spec '{"evals": "<dir>", "agent": "<name>"}' |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Add the required output field to the evaluate-suite example.
EvaluateSuiteSubmitConfig now requires output, so this inline payload will fail validation as written.
🛠️ Proposed fix
- nemo agents evaluate-suite submit --spec '{"evals": "<dir>", "agent": "<name>"}'
+ nemo agents evaluate-suite submit --spec '{"evals": "<dir>", "agent": "<name>", "output": "./runs/batch-<timestamp>"}'📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| nemo agents evaluate submit --agent <name> --eval-config <yaml> | |
| nemo agents optimize submit --agent <name> --optimize-config <yaml> | |
| nemo agents optimize-skills submit --spec-file .agent-improver.yml | |
| nemo agents evaluate-suite submit --spec '{"evals": "<dir>", "agent": "<name>"}' | |
| nemo agents evaluate submit --agent <name> --eval-config <yaml> | |
| nemo agents optimize submit --agent <name> --optimize-config <yaml> | |
| nemo agents optimize-skills submit --spec-file .agent-improver.yml | |
| nemo agents evaluate-suite submit --spec '{"evals": "<dir>", "agent": "<name>", "output": "./runs/batch-<timestamp>"}' |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-agents/src/nemo_agents_plugin/skills/agents-optimize/SKILL.md`
around lines 315 - 318, The evaluate-suite example is missing the required
output field, so update the inline payload used with nemo agents evaluate-suite
submit to include output along with evals and agent. Make this change in the
SKILL.md example that shows the EvaluateSuiteSubmitConfig usage so it matches
the current validation requirements and does not fail schema checks.
Comment on lines
+15
to
+22
| - **Preview surface**: `nemo anonymizer preview submit`. | ||
| - **Run surface**: `nemo anonymizer run submit`. | ||
| - **Input source**: use an HTTP(S) URL or fileset reference. If the user provided a local path and no upload target exists, ask one short question before proceeding. | ||
| - **Model configs**: required. Default to `nvidia-build` as the provider (or the provider the user named) with these aliases: | ||
| - `gliner-pii-detector` → `nvidia/gliner-pii` | ||
| - `gpt-oss-120b` → `openai/gpt-oss-120b` | ||
| - `nemotron-30b-thinking` → `nvidia/nemotron-3-nano-30b-a3b` | ||
| 3. **(If using a plugin-service surface) Confirm the service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then continue. Skip this step entirely for `preview run` / `run run`. | ||
| 3. **Confirm the service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then continue. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Stop after the mount check fails.
nemo services run is a prerequisite here; continuing past this check just guarantees the later preview/run commands fail.
Suggested fix
- If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then continue.
+ If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then stop and rerun this check before proceeding.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - **Preview surface**: `nemo anonymizer preview submit`. | |
| - **Run surface**: `nemo anonymizer run submit`. | |
| - **Input source**: use an HTTP(S) URL or fileset reference. If the user provided a local path and no upload target exists, ask one short question before proceeding. | |
| - **Model configs**: required. Default to `nvidia-build` as the provider (or the provider the user named) with these aliases: | |
| - `gliner-pii-detector` → `nvidia/gliner-pii` | |
| - `gpt-oss-120b` → `openai/gpt-oss-120b` | |
| - `nemotron-30b-thinking` → `nvidia/nemotron-3-nano-30b-a3b` | |
| 3. **(If using a plugin-service surface) Confirm the service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then continue. Skip this step entirely for `preview run` / `run run`. | |
| 3. **Confirm the service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then continue. | |
| - **Preview surface**: `nemo anonymizer preview submit`. | |
| - **Run surface**: `nemo anonymizer run submit`. | |
| - **Input source**: use an HTTP(S) URL or fileset reference. If the user provided a local path and no upload target exists, ask one short question before proceeding. | |
| - **Model configs**: required. Default to `nvidia-build` as the provider (or the provider the user named) with these aliases: | |
| - `gliner-pii-detector` → `nvidia/gliner-pii` | |
| - `gpt-oss-120b` → `openai/gpt-oss-120b` | |
| - `nemotron-30b-thinking` → `nvidia/nemotron-3-nano-30b-a3b` | |
| 3. **Confirm the service is mounted.** Run `curl -s http://localhost:8080/openapi.json | jq -r '.paths | keys[]' | grep '^/apis/anonymizer/'`. If nothing prints, tell the user to run `nemo services run` (no `--services` flag) — `nemo setup` does not mount this plugin — then stop and rerun this check before proceeding. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@plugins/nemo-anonymizer/src/nemo_anonymizer_plugin/skills/anonymizer/workflows/autopilot.md`
around lines 15 - 22, The autopilot flow in the anonymizer workflow should stop
immediately if the service mount check fails instead of continuing to later
steps. Update the mount-check logic in the anonymizer autopilot instructions so
that after the `curl`/`jq`/`grep` probe, a missing `/apis/anonymizer/` path ends
the flow and tells the user to run `nemo services run`; keep the existing
preview/run guidance separate so it only applies after the mount is confirmed.
| | Service | `nemo.services:evaluator` | `jobs`, `healthz` paths. | | ||
| | SDK | `nemo.sdk:evaluator` | Adds `client.evaluator.plugin_status() and run(), submit() interfaces`. | | ||
| | Job | `nemo.jobs:evaluator.evaluate` | Backs local `run` through in-process execution and `submit` through durable platform job submission. | | ||
| | Job | `nemo.jobs:evaluator.evaluate` | Backs durable platform job submission. | |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Align the job description with the rest of the page.
This row now says the job backs durable platform submission, but the “Current Job” section still says it calls Evaluator directly. Update one side so the page describes a single execution model.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/nemo-evaluator/src/nemo_evaluator/docs/index.md` at line 12, The page
has conflicting descriptions of how the job runs: the table entry for
`nemo.jobs:evaluator.evaluate` says it backs durable platform job submission,
while the “Current Job” section still describes direct `Evaluator` execution.
Update the docs in `index.md` so the `Current Job` section and the job table
agree on one execution model, using the existing `nemo.jobs:evaluator.evaluate`
and `Evaluator` references to align the wording consistently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
submitcommands across Anonymizer, Evaluator, Data Designer, Auditor, and Agents workflows.Bug Fixes
Tests