feat(evaluator-sdk)!: install NeMo Fabric 0.1.0rc6 from published wheels and align the runtime

[SandyChapman]

Summary

Installs NeMo Fabric 0.1.0rc6 from published wheels and migrates the Fabric agent-eval runtimes onto the API that release actually ships.

Supersedes and closes #778. The two were originally stacked (packaging, then API alignment), but rc-series Fabric removes API the packaging half's own code depends on, so neither is correct without the other.

Packaging

The Fabric SDK and the harness adapters are split by weight, measured against the base SDK:

	delta	where
nemo-fabric — typed config surface	+2 MB, 3 pkgs	base dependency
claude adapter	+231 MB	fabric extra
codex adapter	+299 MB	fabric extra

Both of those ship an agent CLI binary, so someone installing nemo-evaluator-sdk for LLM-as-judge or RAG metrics should not pay ~530 MB for coding agents they never invoke. Without the extra the SDK still imports, composes a full Fabric config and type-checks against Fabric's real models; only Fabric.plan/run fail, with available adapters: [].

fabric = [
  "nemo-fabric[claude,codex]>=0.1.0rc6,<0.2.0",
  "nemo-fabric-adapters-hermes>=0.1.0rc6,<0.2.0; python_version < '3.14'",
]

claude and codex now ride the metapackage extras. Earlier revisions of this PR pinned every adapter directly, because rc4's extras required nemo-fabric-adapters-*==0.1.0 — a version that was never published, making them unresolvable. rc6 fixes that; the extras now pin ==0.1.0rc6. Going back through them is the better shape: the adapters and the SDK we type against are held on one release by construction, which is exactly the class of skew this PR exists to clean up.

• hermes stays pinned directly, because it must omit [harness] and the metapackage's hermes-agent extra applies it. [harness] pulls hermes-agent, whose exact requests==2.33.0 still conflicts with the workspace's requests>=2.33.1 floor (confirmed against hermes-agent 0.19.0). The adapter alone is what the runtime needs. It cannot drift from the rest regardless — every adapter pins nemo-fabric-adapters-common==<its own version>, so one shared common forces one shared rc.
• deepagents is omitted to match nemo-agents (AIRCORE-952); nothing here exercises it.
• The base dep is plain nemo-fabric — rc4 folded the runtime in and dropped the runtime extra, which until then was silently warning "does not have an extra named runtime".
• No platform marker — rc2+ publishes a macosx_11_0_arm64 wheel alongside manylinux, so it resolves on every environment in [tool.uv] environments.
• nemo-relay>=0.6.0,<0.7, which is load-bearing: the runtime builds AtofConfig(sinks=[...]), a 0.6+ shape that 0.5.x accepts at install time and fails on at runtime.

Runtime alignment

Five independent API drifts, all in the same family — Fabric's models are extra="allow", so every one was accepted silently and simply did nothing:

1. enable_relay — dropped config=; now enable_relay(observability=RelayObservabilityConfig(...)) built from Fabric's typed models.
2. ATOF sinks — the destination moved onto a typed sink list. The old flat form lands in the extras bag, leaves sinks=None, and exports nothing.
3. Profiles deleted outright — FabricProfileConfig unexported, Fabric.run/plan take no profiles, and FabricConfig.from_mapping raises on a profiles key. Everything composes into one config; _eval_lock_profiles is gone (nothing left to defend against); skills use add_skill_path, which appends and de-duplicates, making "treated arm silently drops preconfigured skills" structurally impossible; the container runtime merges into the single /in/agent.yaml since fabric run no longer takes --profile.
4. ModelConfig — FabricConfig.models is dict[str, ModelConfig]; a raw dict skipped validation and warned on serialization.
5. RuntimeConfig — mode/transport are not declared fields; they reach the native core via extras, so examples build them with from_mapping.

Separately, nemo-relay 0.6 made the same ATOF change, so both relay call sites migrated to AtofConfig(sinks=[AtofFileSinkConfig(...)]).

⚠️ Breaking change

FabricRunnerTarget.profiles is removed from the job spec and the OpenAPI schema, as is the profiles= argument to FabricAgentRuntime and FabricContainerRuntime. The upstream concept no longer exists — fold any overlay into config.

Why this kept slipping through, and the guard

Every hermetic fabric test fakes nemo_fabric, so the fakes encoded whatever API their author wrote and kept passing as the real one moved. The profiles removal reached us as a CI failure, not a test failure.

test_fabric_surface.py is a real-wheel contract test driving the runtime's own call sites — enable_relay's shape, the composed ATIF/ATOF values (not merely is not None, which would pass with a stale field name), adapter resolution, and that the composed config is complete by Fabric's own rules.

It also pins skill routing for both harnesses, which caught a sixth drift: the shipped codex adapter now declares skills in its accepts, so it plans harness_native like hermes. That makes SKILL_MODE_CODEX_SKILLS_DIR (staging into <workspace>/.agents/skills/) unreachable in production — bundles land in the native stage dir and the post-run cleanup never runs. Nothing broke, because resolve_skill_mode asks the planner rather than hardcoding, but the docstring claimed codex "only accepts models" and the fakes still encode the old world. The branch is kept as a fallback for adapters that route skills unsupported; the docstring is corrected.

This is what the rc6 bump was checked against. Six drifts across rc2→rc4 were each found the hard way; rc6 passed the contract tests unchanged, which is the first time a Fabric bump on this branch has cost nothing.

fabric-wheel-smoke runs those tests and is the only real-Fabric coverage in CI — --extra fabric appears nowhere else — which the workflow now records so it does not read as redundant.

Tooling

script/dev-install-fabric.sh drops from 165 to 100 lines, scoped to the one thing wheels cannot provide: the nemo-relay gateway binary (the pip package is bindings-only — no console script, no executable). Its version is derived from the installed bindings, so daemon and bindings cannot drift.

Test plan

• 5/5 test_fabric_surface.py against real rc6 wheels installed from the lock — no API drift in this release
• 274 passed, 5 skipped — packages/nemo_evaluator_sdk/tests/agent_eval
• 127 passed — plugins/nemo-agents/tests/unit/test_fabric_*.py
• All six Fabric packages resolve to rc6 in the lock (nemo-fabric, -runtime, -adapters-{claude,codex,hermes,common}); nemo-relay 0.6.0
• uv run ruff check / ruff format --check / uv lock --check / make vendor / make update-licenses clean
• Live end-to-end — run on rc3, not re-run for rc6: real codex 0.142.5 + relay gateway 0.6.0 → COMPLETED trial, evidence {relay_atif, relay_atof, relay_config, result, stdout, trace, workspace}, a 1.0 MB ATIF containing steps, has-trajectory scoring True. relay_atof being present is live proof the ATOF sink migration actually exports. rc6's codex adapter moves the harness pin to openai-codex 0.144.4, so that binary version is covered by the contract tests rather than a live trial.

Note on CI

The Python integration tests job has two failures that are pre-existing on main, not from this branch:

• test_daemonize_services_spawns_child_that_becomes_ready — fixed separately in fix(platform-ext): wait for the instance flock before reporting a stop #987 (also red on main at d47846975, bef8f45a2).
• test_never_deployment_outlives_observe_wait_then_succeeds — a too-thin timing budget in the docker backend tests; also red on main at 434c0db15. Not addressed here.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	28180/36009	78.3%	62.7%
Integration Tests	16758/34727	48.3%	20.9%

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Fabric support now uses published wheels and typed configuration. Profile overlays are replaced with composed configs and skill paths, adapter IDs and Relay telemetry wiring are updated, and CI adds wheel smoke coverage.

Changes

Fabric runtime migration

Layer / File(s)	Summary
Packaging and CI support .github/..., packages/.../pyproject.toml, plugins/nemo-agents/pyproject.toml, script/dev-install-fabric.sh, third_party/licenses.jsonl	Adds Fabric change detection, dependency extras, updated Relay constraints, verified gateway installation, license records, and wheel smoke validation.
Single-config runtime execution packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/*	Removes profile overlays, composes evaluator-controlled configuration directly, uses typed Relay settings, and runs Fabric without profile arguments.
Skill path delivery packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py	Replaces native skill overlays with ordered, deduplicated skill_paths; Codex skills remain workspace-delivered.
Evaluator contracts and validation plugins/nemo-evaluator/..., packages/nemo_evaluator_sdk/tests/agent_eval/*, packages/nemo_evaluator_sdk/examples/*	Removes FabricRunnerTarget.profiles, updates adapter IDs and examples, and adds hermetic and real-API contract coverage.

Sequence Diagram(s)

sequenceDiagram
  participant AgentEvalJob
  participant FabricAgentRuntime
  participant FabricConfig
  participant Fabric
  AgentEvalJob->>FabricAgentRuntime: create runtime from config
  FabricAgentRuntime->>FabricConfig: compose settings and skill paths
  FabricAgentRuntime->>Fabric: plan and run composed config
  Fabric-->>FabricAgentRuntime: return result and adapter metadata

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#778: Related Fabric dependency and wheel-smoke CI work.
• NVIDIA-NeMo/nemo-platform#604: Introduced the Fabric runtime stack being refactored here.
• NVIDIA-NeMo/nemo-platform#816: Related Fabric skill-injection changes.

Suggested labels: ci, build

Suggested reviewers: ngoncharenko, crookedstorm, arpitsardhana, tylersbray

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (2 warnings, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 27.37% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	⚠️ Warning	The title matches the wheel-install/runtime alignment work, but the Fabric version is wrong: the PR targets rc2/rc3, not rc6.	Change the version to the release actually used in this PR, or remove the version number from the title.
Linked Issues check	❓ Inconclusive	Most objectives are covered, but the mirrored SDK and lockfile updates cannot be verified because sdk/python/nemo-platform/pyproject.toml and uv.lock were filtered out.	Expose the excluded sdk/python/nemo-platform/pyproject.toml and uv.lock diffs, or adjust filters so the dependency and lock updates can be checked.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Out of Scope Changes check	✅ Passed	The changes stay on Fabric dependency, runtime, CI, docs, and test alignment; no unrelated scope stands out.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fabric-runtime-api-alignment/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🧹 Nitpick comments (4)

packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_skills.py (1)

100-119: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Stale test name. Overlay assertions are gone; rename to test_install_native_stages_named_dir_and_skill_path.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_skills.py` around
lines 100 - 119, Rename the test function
test_install_native_stages_named_dir_and_overlay to
test_install_native_stages_named_dir_and_skill_path, preserving its existing
assertions and implementation.

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py (1)

312-330: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Re-deriving skill_paths from provenance["location"] duplicates install_skill's contract.

install_skill already returns the staged roots; discarding them and reconstructing from location couples this to the incidental fact that native location == staged root. Accumulate what install_skill returned instead.

♻️ Proposed refactor

     require_unique_skill_names(skills)
     provenances: list[SkillProvenance] = []
+    staged_paths: list[str] = []
     staged_roots: list[Path] = []
     try:
         for skill in skills:
@@
-            provenance = install_skill(
+            installation = install_skill(
                 skill=skill,
                 adapter_id=adapter_id,
                 mode=mode,
                 workspace_dir=workspace_dir,
                 skill_stage_dir=skill_stage_dir,
-            ).provenance
-            provenances.append(provenance)
+            )
+            provenances.append(installation.provenance)
+            staged_paths.extend(installation.skill_paths)
     except Exception:
         for root in staged_roots:
             shutil.rmtree(root, ignore_errors=True)
         raise
 
-    skill_paths: list[str] = []
-    if mode == SKILL_MODE_NATIVE:
-        # Each staged bundle root, order-preserved and de-duplicated (a native provenance's
-        # ``location`` is its absolute staged skill root).
-        skill_paths = list(dict.fromkeys(prov["location"] for prov in provenances))
-    return SkillsInstallation(skill_paths=skill_paths, provenances=provenances)
+    # Order-preserved and de-duplicated; the codex branch contributes none.
+    return SkillsInstallation(skill_paths=list(dict.fromkeys(staged_paths)), provenances=provenances)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py`
around lines 312 - 330, Update the skill installation flow to accumulate the
staged roots returned directly by install_skill alongside each provenance,
rather than rebuilding skill_paths from provenance["location"]. Use the
accumulated install_skill results for native mode while preserving ordering,
de-duplication, cleanup behavior, and the existing SkillsInstallation return
contract.

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/_common.py (1)

115-121: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Docstring still describes profile overlays.

Profiles are gone in Fabric 0.1.0rc2; both callers now inline this into config["telemetry"]. Reword "trajectory profile" / "Callers wrap this in a profile".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/_common.py`
around lines 115 - 121, Update the docstring of trajectory_telemetry to describe
the returned telemetry block directly, removing references to Fabric trajectory
profiles and callers wrapping the result in a profile. Retain the existing
explanation of Relay’s typed configuration ownership and relay_dir’s
exporter-file behavior.

packages/nemo_evaluator_sdk/examples/skill_eval/README.md (1)

46-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Restructure this as a HOW-TO page.

Move prerequisites directly below the title, keep task execution here, move conceptual material to a linked explanation, add Python SDK/CLI tab sets, and end with Next Steps links.

As per coding guidelines, “Always list prerequisites at the top,” “Each documentation page should fit ONE Diataxis quadrant,” “Provide both Python SDK and CLI examples in tab-sets,” and “Include ‘Next Steps’ section at the end.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/examples/skill_eval/README.md` around lines 46 -
48, Restructure the skill evaluation README as a HOW-TO: move prerequisites
immediately below the title, retain only task-execution guidance, and move
conceptual/background content into a linked explanation page. Add tab sets with
equivalent Python SDK and CLI examples for applicable tasks, and finish with a
“Next Steps” section containing relevant links.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb`:
- Line 554: Update the preceding notebook guidance for the Fabric target to
remove the stale claim that it passes base_dir to a Fabric checkout. Describe
that the adapter is resolved from the installed wheel instead, while leaving the
codex_via_fabric and FabricAgentRuntime construction unchanged.

In `@packages/nemo_evaluator_sdk/pyproject.toml`:
- Around line 85-87: Update the fabric dependency declaration in the pyproject
configuration to add an upper bound of <0.2 while retaining the existing extras
and lower bound, ensuring the fabric extra remains on the tested 0.1 API line.
- Line 33: Update the nemo-relay dependency constraint in pyproject.toml from a
0.5.x-compatible lower bound to >=0.6.0 while preserving the existing <0.7 upper
bound, ensuring only Relay 0.6+ versions are admitted.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py`:
- Around line 230-241: Update the _LIVE_READY capability check used by
requires_live_fabric to also require
importlib.util.find_spec("nemo_relay.observability") is not None, so
environments lacking Relay observability bindings are skipped before trajectory
capture runs.

In `@packages/nemo_platform/pyproject.toml`:
- Line 297: Update the nemo-relay dependency constraint in pyproject.toml from
>=0.5.0,<0.7 to >=0.6.0,<0.7, ensuring the runtime requires the AtofConfig and
AtofFileSinkConfig API-compatible Relay release.

---

Nitpick comments:
In `@packages/nemo_evaluator_sdk/examples/skill_eval/README.md`:
- Around line 46-48: Restructure the skill evaluation README as a HOW-TO: move
prerequisites immediately below the title, retain only task-execution guidance,
and move conceptual/background content into a linked explanation page. Add tab
sets with equivalent Python SDK and CLI examples for applicable tasks, and
finish with a “Next Steps” section containing relevant links.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/_common.py`:
- Around line 115-121: Update the docstring of trajectory_telemetry to describe
the returned telemetry block directly, removing references to Fabric trajectory
profiles and callers wrapping the result in a profile. Retain the existing
explanation of Relay’s typed configuration ownership and relay_dir’s
exporter-file behavior.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py`:
- Around line 312-330: Update the skill installation flow to accumulate the
staged roots returned directly by install_skill alongside each provenance,
rather than rebuilding skill_paths from provenance["location"]. Use the
accumulated install_skill results for native mode while preserving ordering,
de-duplication, cleanup behavior, and the existing SkillsInstallation return
contract.

In `@packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_skills.py`:
- Around line 100-119: Rename the test function
test_install_native_stages_named_dir_and_overlay to
test_install_native_stages_named_dir_and_skill_path, preserving its existing
assertions and implementation.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 10e50f5c-fd90-4335-9e84-1bb8dd912ee2

📥 Commits

Reviewing files that changed from the base of the PR and between 3df96dd and e5d4cb2.

⛔ Files ignored due to path filters (7)

• sdk/python/nemo-platform/pyproject.toml is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/_common.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/image.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/skills.py is excluded by !sdk/**
• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (26)

• .github/actions/changes/action.yaml
• .github/workflows/ci.yaml
• packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb
• packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py
• packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py
• packages/nemo_evaluator_sdk/examples/skill_eval/README.md
• packages/nemo_evaluator_sdk/examples/skill_eval/run_skill_eval.py
• packages/nemo_evaluator_sdk/pyproject.toml
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/_common.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/image.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_skills.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_surface.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_skill_used_metric.py
• packages/nemo_platform/pyproject.toml
• plugins/nemo-agents/pyproject.toml
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
• plugins/nemo-evaluator/tests/test_agent_evaluate.py
• script/dev-install-fabric.sh

💤 Files with no reviewable changes (1)

• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py`:
- Around line 319-355: Update the _composed_config telemetry construction to
pass agent_version="fabric" while preserving agent_name=_RUNTIME_NAME and the
existing relay configuration.

In `@script/dev-install-fabric.sh`:
- Around line 48-57: Update the installed_relay_ver assignment in the nemo-relay
PATH-check block to tolerate a failing nemo-relay --version pipeline under set
-euo pipefail, matching the existing guarded version-detection patterns
elsewhere in the script. Ensure a failed lookup leaves the version empty or
unknown so execution reaches the mismatch message and reinstall path instead of
aborting.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 91b8c29c-46aa-45df-a58c-fa56ed3a7fce

📥 Commits

Reviewing files that changed from the base of the PR and between 8e4ca69 and 73c4135.

⛔ Files ignored due to path filters (7)

• sdk/python/nemo-platform/pyproject.toml is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/_common.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/image.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/skills.py is excluded by !sdk/**
• uv.lock is excluded by !**/*.lock

📒 Files selected for processing (27)

• .github/actions/changes/action.yaml
• .github/workflows/ci.yaml
• packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb
• packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py
• packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py
• packages/nemo_evaluator_sdk/examples/skill_eval/README.md
• packages/nemo_evaluator_sdk/examples/skill_eval/run_skill_eval.py
• packages/nemo_evaluator_sdk/pyproject.toml
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/_common.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/image.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_skills.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_surface.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_skill_used_metric.py
• packages/nemo_platform/pyproject.toml
• plugins/nemo-agents/pyproject.toml
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
• plugins/nemo-evaluator/tests/test_agent_evaluate.py
• script/dev-install-fabric.sh
• third_party/licenses.jsonl

💤 Files with no reviewable changes (1)

• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_evaluate.py

🚧 Files skipped from review as they are similar to previous changes (19)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/image.py
• packages/nemo_evaluator_sdk/examples/fabric_container/run_e2e.py
• packages/nemo_evaluator_sdk/examples/fabric_harness_runtimes.py
• third_party/licenses.jsonl
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/_common.py
• .github/workflows/ci.yaml
• plugins/nemo-evaluator/openapi/openapi.yaml
• plugins/nemo-agents/pyproject.toml
• packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb
• plugins/nemo-evaluator/tests/test_agent_evaluate.py
• packages/nemo_evaluator_sdk/pyproject.toml
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_surface.py
• plugins/nemo-evaluator/src/nemo_evaluator/jobs/agent_spec.py
• .github/actions/changes/action.yaml
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_integration.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_skills.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/skills.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py

[crookedstorm]

Looks ok from devops side.