build(evaluator): install NeMo Fabric 0.1.0rc2 from published wheels via a fabric extra - #778
Closed
SandyChapman wants to merge 5 commits into
Closed
build(evaluator): install NeMo Fabric 0.1.0rc2 from published wheels via a fabric extra#778SandyChapman wants to merge 5 commits into
fabric extra#778SandyChapman wants to merge 5 commits into
Conversation
Contributor
|
SandyChapman
marked this pull request as ready for review
July 21, 2026 11:57
SandyChapman
force-pushed
the
fabric-published-wheels/schapman
branch
from
July 21, 2026 12:02
f7dddc5 to
1e85de4
Compare
Contributor
📝 WalkthroughWalkthroughFabric dependency constraints, development installation, relay provisioning, and Linux CI smoke validation were updated. Aggregate CI status now includes the Fabric wheel smoke job. ChangesFabric installation flow
Sequence Diagram(s)sequenceDiagram
participant CI
participant uv
participant nemo_evaluator_sdk
participant nemo_fabric
CI->>uv: Sync nemo-evaluator-sdk[fabric] with frozen lockfile
uv->>nemo_evaluator_sdk: Resolve Fabric extra
nemo_evaluator_sdk->>nemo_fabric: Provide Fabric package
CI->>nemo_fabric: Import module and Fabric symbols
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yaml:
- Around line 909-910: Update the actions/checkout step in the CI workflow to
set persist-credentials to false, preventing GITHUB_TOKEN from being stored in
.git/config while preserving the existing checkout configuration.
In `@script/dev-install-fabric.sh`:
- Line 78: Update the installation flow around RELAY_BIN_DIR so the installed
nemo-relay binary is discoverable by the live integration test. Either install
it into a directory already on PATH or print a shell PATH export that prepends
RELAY_BIN_DIR after installation, while preserving support for custom CARGO_HOME
values.
- Around line 79-80: Update the existing nemo-relay detection in the
provisioning flow to honor NEMO_RELAY_VERSION: obtain the installed binary’s
version, compare it with the requested version, and only skip installation when
they match. When they differ, replace the binary or fail explicitly instead of
treating any PATH match as already provisioned.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b76ccb50-3940-4bfc-8f7b-258b56a9b7d3
⛔ Files ignored due to path filters (2)
sdk/python/nemo-platform/pyproject.tomlis excluded by!sdk/**uv.lockis excluded by!**/*.lock
📒 Files selected for processing (5)
.github/workflows/ci.yamlpackages/nemo_evaluator_sdk/pyproject.tomlpackages/nemo_platform/pyproject.tomlpyproject.tomlscript/dev-install-fabric.sh
SandyChapman
marked this pull request as draft
July 21, 2026 13:11
Contributor
Author
|
Holding this as a draft until NeMo Fabric ships a public, non-prerelease release. The change itself is complete and CI is green, but it currently pulls Fabric's daily alpha wheels, which drag two Fabric-owned prerelease pins into the workspace lock ( Those clear automatically once Fabric cuts a stable release and its Parking rather than merging also avoids interim CI flakiness from Fabric alpha wheels being garbage-collected from the index. |
Merged
6 tasks
…ux `fabric` extra NeMo Fabric now publishes wheels to PyPI, so the Fabric agent-eval runtime no longer needs a source build from a local checkout on Linux. - Add a Linux-gated `fabric` extra on nemo-evaluator-sdk pulling nemo-fabric[claude,codex,deepagents,runtime]. The native nemo-fabric-runtime wheel is manylinux-only (no macOS wheel yet), so the sys_platform == 'linux' marker keeps macOS lock resolution intact; macOS-native dev still uses script/dev-install-fabric.sh (also updated to point Linux devs at the extra). - Bump nemo-relay 0.4.x -> 0.5.x (stable, all-platform wheels) to align with Fabric's relay line; mirror the bump in the bundle-package and vendored-SDK dependency blocks. - Allow the fabric subtree's prereleases via explicit constraint pins (openai-codex, openai-codex-cli-bin, sqlite-vec), matching the existing safetensors pattern; no global --prerelease=allow needed. - Add a Linux fabric-wheel-smoke CI job (gated on the deps path filter) that installs the extra from the lock and imports the nemo_fabric surface the runtime uses, validating the manylinux_2_39 wheel installs on the runner glibc. hermes is intentionally omitted: hermes-agent pins requests==2.33.0 exactly, which conflicts with the workspace's requests>=2.33.1 floor. The nemo-relay gateway binary is still source-only (not on PyPI), so dev-install-fabric.sh remains required for live ATIF trajectory capture. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
…rom GitHub releases - Drop the sqlite-vec explicit-prerelease constraint. It's a LangGraph dep (langgraph-checkpoint-sqlite>=0.1.6), so it resolves to stable 0.1.9 instead of the 0.1.10 alpha (re-locked via uv lock --upgrade-package sqlite-vec). Only the two Fabric-owned codex prereleases (openai-codex, openai-codex-cli-bin) remain, and they clear when Fabric bumps off the beta. - dev-install-fabric.sh: download the prebuilt nemo-relay gateway from the NeMo-Relay GitHub releases (checksum-verified, pinned via NEMO_RELAY_VERSION) instead of a cargo build from a local NeMo-Relay checkout. Source-build fallback for platforms with no prebuilt asset (Intel macOS) or NEMO_RELAY_REPO. - dev-install-fabric.sh: install the `runtime` extra as well. Fabric's metapackage split means [codex,relay] no longer provides the importable nemo_fabric module; and make --uninstall remove nemo-fabric-runtime + adapters so it actually restores the CI-equivalent state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>
- ci.yaml (fabric-wheel-smoke): set persist-credentials: false on checkout so
GITHUB_TOKEN isn't left in .git/config while the job installs + imports
third-party wheels (matches the repo convention used by other jobs).
- dev-install-fabric.sh: honor NEMO_RELAY_VERSION — compare an existing
nemo-relay's version and skip provisioning only on a match, otherwise
(re)install, instead of short-circuiting on any PATH match.
- dev-install-fabric.sh: warn when the gateway install dir isn't on PATH so the
live test's shutil.which("nemo-relay") can resolve it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
Fabric cut 0.1.0rc2, which publishes a macosx_11_0_arm64 wheel for nemo-fabric-runtime alongside the manylinux ones. The `sys_platform == 'linux'` marker on the nemo-evaluator-sdk `fabric` extra only existed because no macOS wheel was published, so it is gone — the extra now resolves on every environment in [tool.uv] environments. rc2's codex adapter also moved from `openai-codex==0.1.0b3` to the stable `openai-codex==0.144.4`, so the two explicit-prerelease constraint-dependencies that existed to let uv accept that beta are no longer needed and were dropped from the root pyproject. Refreshes the stale "no macOS wheel" / "Linux-gated" notes in script/dev-install-fabric.sh and the fabric-wheel-smoke CI job. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
… 0.6 The nemo-relay floor moved to >=0.5.0,<0.7 in this branch, and relocking onto current main resolves 0.6.0 (published after this branch was first written). Relay 0.6 moved ATOF's destination out of `AtofConfig` and into a typed sink list, so `AtofConfig(output_directory=..., filename=..., mode=...)` now raises TypeError. Both call sites — the host runtime's `_relay_config` and the shared container-profile helper in `_common.py` — now build `AtofConfig(sinks=[AtofFileSinkConfig(...)])`. ATIF is unchanged. test_fabric_runtime.py's fake nemo_relay module encoded the old flat API and so masked the break; it now exports AtofFileSinkConfig and renders nested sink lists. The two tests that exercise the *real* relay types — test_trajectory_profile_built_from_relay_types and test_fabric_runner_eval_exposes_trajectory_to_metric — are what caught this, and their assertions were updated to the sink shape. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
fabric-published-wheels/schapman
branch
from
July 28, 2026 17:20
ac52a6c to
4b22954
Compare
SandyChapman
added a commit
that referenced
this pull request
Jul 28, 2026
… API (enable_relay, adapter ids) Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis) and were masked because every fabric unit test fakes nemo_fabric. L7 — enable_relay: the dropped `config=` keyword now raises, and Fabric wants its own typed relay models. `_relay_config` now builds `RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))` from nemo_fabric's own types and passes it via `enable_relay(observability=...)`, mirroring Fabric's Harbor integration (Fabric owns the schema). ATOF's destination lives on a typed sink list (`RelayAtofFileSinkConfig`), not flat on the config — and because Fabric's models are `extra="allow"`, the stale flat form is accepted *silently* and simply never exports, so only asserting the composed values catches it. L8 — harness adapter ids: current nemo-fabric registers bare `nvidia.fabric.codex` / `nvidia.fabric.hermes` (transport moved to runtime.transport). Swept the retired `.cli`/`.sdk` suffixes across the SDK runtime docstring, examples, notebook, skill-eval, and hermetic tests, plus the nemo-evaluator plugin's FabricRunnerTarget docstring/tests and regenerated spec. Guard against recurrence: add test_fabric_surface.py, a real-nemo_fabric contract test (importorskip elsewhere) that exercises the runtime's own call sites — enable_relay's observability shape, the composed ATIF/ATOF values, and bare-id resolution via the planner — and wire the Linux fabric-wheel-smoke CI job to run it (a bare import could not catch any of it). A new `fabric` path filter triggers that job on runtime/test edits as well as wheel bumps. Verified against real nemo-fabric 0.1.0rc2 installed from the lock (macOS arm64): both surface tests pass, and the pre-fix ATOF form fails the new sink assertion. Stacks on the Fabric-packaging work (#778) and shares its non-prerelease Fabric gate. Live ATIF relay capture still needs a real Fabric + nemo-relay gateway and was not CI-verified (the smoke job runs no agent). Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
changed the title
fabric extrafabric extra
Contributor
Author
|
Superseded by #844, which now targets `main` and contains this branch's commits plus the runtime migration. Collapsing the stack: Fabric 0.1.0rc2 removes API that this branch's own code depends on ( Closing in favour of #844. |
SandyChapman
added a commit
that referenced
this pull request
Jul 28, 2026
… API (enable_relay, adapter ids) Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis) and were masked because every fabric unit test fakes nemo_fabric. L7 — enable_relay: the dropped `config=` keyword now raises, and Fabric wants its own typed relay models. `_relay_config` now builds `RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))` from nemo_fabric's own types and passes it via `enable_relay(observability=...)`, mirroring Fabric's Harbor integration (Fabric owns the schema). ATOF's destination lives on a typed sink list (`RelayAtofFileSinkConfig`), not flat on the config — and because Fabric's models are `extra="allow"`, the stale flat form is accepted *silently* and simply never exports, so only asserting the composed values catches it. L8 — harness adapter ids: current nemo-fabric registers bare `nvidia.fabric.codex` / `nvidia.fabric.hermes` (transport moved to runtime.transport). Swept the retired `.cli`/`.sdk` suffixes across the SDK runtime docstring, examples, notebook, skill-eval, and hermetic tests, plus the nemo-evaluator plugin's FabricRunnerTarget docstring/tests and regenerated spec. Guard against recurrence: add test_fabric_surface.py, a real-nemo_fabric contract test (importorskip elsewhere) that exercises the runtime's own call sites — enable_relay's observability shape, the composed ATIF/ATOF values, and bare-id resolution via the planner — and wire the Linux fabric-wheel-smoke CI job to run it (a bare import could not catch any of it). A new `fabric` path filter triggers that job on runtime/test edits as well as wheel bumps. Verified against real nemo-fabric 0.1.0rc2 installed from the lock (macOS arm64): both surface tests pass, and the pre-fix ATOF form fails the new sink assertion. Stacks on the Fabric-packaging work (#778) and shares its non-prerelease Fabric gate. Live ATIF relay capture still needs a real Fabric + nemo-relay gateway and was not CI-verified (the smoke job runs no agent). Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Jul 29, 2026
… API (enable_relay, adapter ids) Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis) and were masked because every fabric unit test fakes nemo_fabric. L7 — enable_relay: the dropped `config=` keyword now raises, and Fabric wants its own typed relay models. `_relay_config` now builds `RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))` from nemo_fabric's own types and passes it via `enable_relay(observability=...)`, mirroring Fabric's Harbor integration (Fabric owns the schema). ATOF's destination lives on a typed sink list (`RelayAtofFileSinkConfig`), not flat on the config — and because Fabric's models are `extra="allow"`, the stale flat form is accepted *silently* and simply never exports, so only asserting the composed values catches it. L8 — harness adapter ids: current nemo-fabric registers bare `nvidia.fabric.codex` / `nvidia.fabric.hermes` (transport moved to runtime.transport). Swept the retired `.cli`/`.sdk` suffixes across the SDK runtime docstring, examples, notebook, skill-eval, and hermetic tests, plus the nemo-evaluator plugin's FabricRunnerTarget docstring/tests and regenerated spec. Guard against recurrence: add test_fabric_surface.py, a real-nemo_fabric contract test (importorskip elsewhere) that exercises the runtime's own call sites — enable_relay's observability shape, the composed ATIF/ATOF values, and bare-id resolution via the planner — and wire the Linux fabric-wheel-smoke CI job to run it (a bare import could not catch any of it). A new `fabric` path filter triggers that job on runtime/test edits as well as wheel bumps. Verified against real nemo-fabric 0.1.0rc2 installed from the lock (macOS arm64): both surface tests pass, and the pre-fix ATOF form fails the new sink assertion. Stacks on the Fabric-packaging work (#778) and shares its non-prerelease Fabric gate. Live ATIF relay capture still needs a real Fabric + nemo-relay gateway and was not CI-verified (the smoke job runs no agent). Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Jul 29, 2026
… API (enable_relay, adapter ids) Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis) and were masked because every fabric unit test fakes nemo_fabric. L7 — enable_relay: the dropped `config=` keyword now raises, and Fabric wants its own typed relay models. `_relay_config` now builds `RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))` from nemo_fabric's own types and passes it via `enable_relay(observability=...)`, mirroring Fabric's Harbor integration (Fabric owns the schema). ATOF's destination lives on a typed sink list (`RelayAtofFileSinkConfig`), not flat on the config — and because Fabric's models are `extra="allow"`, the stale flat form is accepted *silently* and simply never exports, so only asserting the composed values catches it. L8 — harness adapter ids: current nemo-fabric registers bare `nvidia.fabric.codex` / `nvidia.fabric.hermes` (transport moved to runtime.transport). Swept the retired `.cli`/`.sdk` suffixes across the SDK runtime docstring, examples, notebook, skill-eval, and hermetic tests, plus the nemo-evaluator plugin's FabricRunnerTarget docstring/tests and regenerated spec. Guard against recurrence: add test_fabric_surface.py, a real-nemo_fabric contract test (importorskip elsewhere) that exercises the runtime's own call sites — enable_relay's observability shape, the composed ATIF/ATOF values, and bare-id resolution via the planner — and wire the Linux fabric-wheel-smoke CI job to run it (a bare import could not catch any of it). A new `fabric` path filter triggers that job on runtime/test edits as well as wheel bumps. Verified against real nemo-fabric 0.1.0rc2 installed from the lock (macOS arm64): both surface tests pass, and the pre-fix ATOF form fails the new sink assertion. Stacks on the Fabric-packaging work (#778) and shares its non-prerelease Fabric gate. Live ATIF relay capture still needs a real Fabric + nemo-relay gateway and was not CI-verified (the smoke job runs no agent). Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Jul 30, 2026
…els and align the runtime Installs NeMo Fabric from published wheels and migrates the Fabric agent-eval runtimes onto the API that release actually ships. Supersedes #778, which was the packaging half of this before rc-series Fabric removed API that half's own code depended on. PACKAGING The Fabric SDK (~2 MB / 3 packages) becomes a base dependency of nemo-evaluator-sdk; the harness adapters stay behind the `fabric` extra because they are the entire weight — codex ~299 MB and claude ~231 MB each ship an agent CLI binary. Without the extra the SDK still imports, composes a full Fabric config and type-checks against Fabric's real models; only `Fabric.plan`/`run` fail. Adapters are pinned directly rather than through `nemo-fabric[claude,...]`: rc4's metapackage adapter extras require `nemo-fabric-adapters-*==0.1.0`, a version that was never published, so those extras are unresolvable. plugins/nemo-agents pins directly for the same reason and shares this lock, so going through the extras makes the workspace unsatisfiable. hermes omits `[harness]` — that sub-extra pulls `hermes-agent`, whose exact `requests==2.33.0` pin conflicts with the workspace's `requests>=2.33.1` floor — and deepagents is omitted to match nemo-agents (AIRCORE-952). No platform marker: rc2+ publishes a macOS arm64 wheel alongside manylinux, so this resolves on every environment in [tool.uv] environments. `nemo-relay` is floored at >=0.6.0,<0.7, which is load-bearing — the runtime builds `AtofConfig(sinks=[...])`, a 0.6+ shape that 0.5.x accepts at install time and fails on at runtime. RUNTIME ALIGNMENT Five independent API drifts, all of the same kind — Fabric's models are `extra="allow"`, so each was accepted silently and simply did nothing: * `enable_relay` lost `config=`; now takes `observability=` built from Fabric's typed relay models. * ATOF's destination moved onto a typed sink list; the old flat form lands in the extras bag, leaves `sinks=None` and exports nothing. * Profiles were deleted outright — `FabricProfileConfig` is unexported, `Fabric.run`/`plan` take no `profiles`, and `FabricConfig.from_mapping` raises on a `profiles` key. Everything composes into one config; `_eval_lock_profiles` is gone, skills use `add_skill_path` (append + de-duplicate, which makes "treated arm silently drops preconfigured skills" structurally impossible), and the container runtime merges into the single `/in/agent.yaml` since `fabric run` no longer takes `--profile`. * `FabricConfig.models` is `dict[str, ModelConfig]`; a raw dict skipped validation and warned on serialization. * `RuntimeConfig` does not declare `mode`/`transport`; they reach the native core through extras, so examples build them with `from_mapping`. nemo-relay 0.6 made the same ATOF change, so both relay call sites migrated to `AtofConfig(sinks=[AtofFileSinkConfig(...)])`. BREAKING: `FabricRunnerTarget.profiles` is removed from the job spec and the OpenAPI schema, as is the `profiles=` argument to `FabricAgentRuntime` and `FabricContainerRuntime`. The upstream concept no longer exists — fold any overlay into `config`. GUARDING AGAINST RECURRENCE Every hermetic fabric test fakes `nemo_fabric`, so the fakes encoded whatever API their author wrote and kept passing as the real one moved; the profiles removal reached us as a CI failure rather than a test failure. `test_fabric_surface.py` drives the runtime's own call sites against the real wheels — `enable_relay`'s shape, the composed ATIF/ATOF values (not merely `is not None`, which passes with a stale field name), adapter resolution, and that the composed config is complete by Fabric's own rules. It also pins skill routing for both harnesses: the shipped codex adapter now declares `skills`, so it plans `harness_native` like hermes, leaving `SKILL_MODE_CODEX_SKILLS_DIR` a fallback for adapters that route skills `unsupported`. The `fabric-wheel-smoke` job runs those tests and is the only real-Fabric coverage in CI, which the workflow now records so it does not read as redundant. TOOLING `script/dev-install-fabric.sh` drops from 165 to 100 lines, scoped to the one thing wheels cannot provide: the `nemo-relay` gateway binary. The pip package is bindings-only, so the daemon exists solely as a GitHub release asset. Its version derives from the installed bindings, so daemon and bindings cannot drift. Verified end to end against real codex 0.142.5 + relay gateway 0.6.0 + nemo-fabric: COMPLETED trial, evidence {relay_atif, relay_atof, relay_config, result, stdout, trace, workspace}, a 1.0 MB ATIF containing `steps`, and has-trajectory scoring True. `relay_atof` being present is live proof the ATOF sink migration actually exports. Not verified: the containerized runtime's `fabric run` path — the Fabric CLI is not published as a wheel, so the image builds it from source. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Jul 30, 2026
…els and align the runtime Installs NeMo Fabric from published wheels and migrates the Fabric agent-eval runtimes onto the API that release actually ships. Supersedes #778, which was the packaging half of this before rc-series Fabric removed API that half's own code depended on. PACKAGING The Fabric SDK (~2 MB / 3 packages) becomes a base dependency of nemo-evaluator-sdk; the harness adapters stay behind the `fabric` extra because they are the entire weight — codex ~299 MB and claude ~231 MB each ship an agent CLI binary. Without the extra the SDK still imports, composes a full Fabric config and type-checks against Fabric's real models; only `Fabric.plan`/`run` fail. Adapters are pinned directly rather than through `nemo-fabric[claude,...]`: rc4's metapackage adapter extras require `nemo-fabric-adapters-*==0.1.0`, a version that was never published, so those extras are unresolvable. plugins/nemo-agents pins directly for the same reason and shares this lock, so going through the extras makes the workspace unsatisfiable. hermes omits `[harness]` — that sub-extra pulls `hermes-agent`, whose exact `requests==2.33.0` pin conflicts with the workspace's `requests>=2.33.1` floor — and deepagents is omitted to match nemo-agents (AIRCORE-952). No platform marker: rc2+ publishes a macOS arm64 wheel alongside manylinux, so this resolves on every environment in [tool.uv] environments. `nemo-relay` is floored at >=0.6.0,<0.7, which is load-bearing — the runtime builds `AtofConfig(sinks=[...])`, a 0.6+ shape that 0.5.x accepts at install time and fails on at runtime. RUNTIME ALIGNMENT Five independent API drifts, all of the same kind — Fabric's models are `extra="allow"`, so each was accepted silently and simply did nothing: * `enable_relay` lost `config=`; now takes `observability=` built from Fabric's typed relay models. * ATOF's destination moved onto a typed sink list; the old flat form lands in the extras bag, leaves `sinks=None` and exports nothing. * Profiles were deleted outright — `FabricProfileConfig` is unexported, `Fabric.run`/`plan` take no `profiles`, and `FabricConfig.from_mapping` raises on a `profiles` key. Everything composes into one config; `_eval_lock_profiles` is gone, skills use `add_skill_path` (append + de-duplicate, which makes "treated arm silently drops preconfigured skills" structurally impossible), and the container runtime merges into the single `/in/agent.yaml` since `fabric run` no longer takes `--profile`. * `FabricConfig.models` is `dict[str, ModelConfig]`; a raw dict skipped validation and warned on serialization. * `RuntimeConfig` does not declare `mode`/`transport`; they reach the native core through extras, so examples build them with `from_mapping`. nemo-relay 0.6 made the same ATOF change, so both relay call sites migrated to `AtofConfig(sinks=[AtofFileSinkConfig(...)])`. BREAKING: `FabricRunnerTarget.profiles` is removed from the job spec and the OpenAPI schema, as is the `profiles=` argument to `FabricAgentRuntime` and `FabricContainerRuntime`. The upstream concept no longer exists — fold any overlay into `config`. GUARDING AGAINST RECURRENCE Every hermetic fabric test fakes `nemo_fabric`, so the fakes encoded whatever API their author wrote and kept passing as the real one moved; the profiles removal reached us as a CI failure rather than a test failure. `test_fabric_surface.py` drives the runtime's own call sites against the real wheels — `enable_relay`'s shape, the composed ATIF/ATOF values (not merely `is not None`, which passes with a stale field name), adapter resolution, and that the composed config is complete by Fabric's own rules. It also pins skill routing for both harnesses: the shipped codex adapter now declares `skills`, so it plans `harness_native` like hermes, leaving `SKILL_MODE_CODEX_SKILLS_DIR` a fallback for adapters that route skills `unsupported`. The `fabric-wheel-smoke` job runs those tests and is the only real-Fabric coverage in CI, which the workflow now records so it does not read as redundant. TOOLING `script/dev-install-fabric.sh` drops from 165 to 100 lines, scoped to the one thing wheels cannot provide: the `nemo-relay` gateway binary. The pip package is bindings-only, so the daemon exists solely as a GitHub release asset. Its version derives from the installed bindings, so daemon and bindings cannot drift. Verified end to end against real codex 0.142.5 + relay gateway 0.6.0 + nemo-fabric: COMPLETED trial, evidence {relay_atif, relay_atof, relay_config, result, stdout, trace, workspace}, a 1.0 MB ATIF containing `steps`, and has-trajectory scoring True. `relay_atof` being present is live proof the ATOF sink migration actually exports. Not verified: the containerized runtime's `fabric run` path — the Fabric CLI is not published as a wheel, so the image builds it from source. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Jul 30, 2026
…els and align the runtime Installs NeMo Fabric from published wheels and migrates the Fabric agent-eval runtimes onto the API that release actually ships. Supersedes #778, which was the packaging half of this before rc-series Fabric removed API that half's own code depended on. PACKAGING The Fabric SDK (~2 MB / 3 packages) becomes a base dependency of nemo-evaluator-sdk; the harness adapters stay behind the `fabric` extra because they are the entire weight — codex ~299 MB and claude ~231 MB each ship an agent CLI binary. Without the extra the SDK still imports, composes a full Fabric config and type-checks against Fabric's real models; only `Fabric.plan`/`run` fail. claude and codex ride the metapackage extras (`nemo-fabric[claude,codex]`), which pin `nemo-fabric-adapters-*` to the metapackage's own version — so the adapters and the SDK we type against stay on one release by construction. They were pinned directly through rc5 because rc4's extras required `nemo-fabric-adapters-*==0.1.0`, a version that was never published; rc6 fixed that. hermes stays direct because it must omit `[harness]`, which the metapackage's `hermes-agent` extra applies: `[harness]` pulls `hermes-agent`, whose exact `requests==2.33.0` pin conflicts with the workspace's `requests>=2.33.1` floor. It cannot drift regardless — every adapter pins `nemo-fabric-adapters-common==<its own version>`, so one shared common forces one shared rc. deepagents is omitted to match nemo-agents (AIRCORE-952). No platform marker: rc2+ publishes a macOS arm64 wheel alongside manylinux, so this resolves on every environment in [tool.uv] environments. `nemo-relay` is floored at >=0.6.0,<0.7, which is load-bearing — the runtime builds `AtofConfig(sinks=[...])`, a 0.6+ shape that 0.5.x accepts at install time and fails on at runtime. RUNTIME ALIGNMENT Five independent API drifts, all of the same kind — Fabric's models are `extra="allow"`, so each was accepted silently and simply did nothing: * `enable_relay` lost `config=`; now takes `observability=` built from Fabric's typed relay models. * ATOF's destination moved onto a typed sink list; the old flat form lands in the extras bag, leaves `sinks=None` and exports nothing. * Profiles were deleted outright — `FabricProfileConfig` is unexported, `Fabric.run`/`plan` take no `profiles`, and `FabricConfig.from_mapping` raises on a `profiles` key. Everything composes into one config; `_eval_lock_profiles` is gone, skills use `add_skill_path` (append + de-duplicate, which makes "treated arm silently drops preconfigured skills" structurally impossible), and the container runtime merges into the single `/in/agent.yaml` since `fabric run` no longer takes `--profile`. * `FabricConfig.models` is `dict[str, ModelConfig]`; a raw dict skipped validation and warned on serialization. * `RuntimeConfig` does not declare `mode`/`transport`; they reach the native core through extras, so examples build them with `from_mapping`. nemo-relay 0.6 made the same ATOF change, so both relay call sites migrated to `AtofConfig(sinks=[AtofFileSinkConfig(...)])`. Because these all failed silently, `tests/agent_eval/test_fabric_surface.py` asserts the composed *values* against the real published wheels rather than mocks, and the `fabric-wheel-smoke` CI job runs it. rc6 passes it unchanged — no further drift in this release. BREAKING: `FabricRunnerTarget.profiles` is removed from the job spec and the OpenAPI schema, as is the `profiles=` argument to `FabricAgentRuntime` and `FabricContainerRuntime`. The upstream concept no longer exists — fold any overlay into `config`. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
added a commit
that referenced
this pull request
Jul 30, 2026
…els and align the runtime Installs NeMo Fabric from published wheels and migrates the Fabric agent-eval runtimes onto the API that release actually ships. Supersedes #778, which was the packaging half of this before rc-series Fabric removed API that half's own code depended on. PACKAGING The Fabric SDK (~2 MB / 3 packages) becomes a base dependency of nemo-evaluator-sdk; the harness adapters stay behind the `fabric` extra because they are the entire weight — codex ~299 MB and claude ~231 MB each ship an agent CLI binary. Without the extra the SDK still imports, composes a full Fabric config and type-checks against Fabric's real models; only `Fabric.plan`/`run` fail. claude and codex ride the metapackage extras (`nemo-fabric[claude,codex]`), which pin `nemo-fabric-adapters-*` to the metapackage's own version — so the adapters and the SDK we type against stay on one release by construction. They were pinned directly through rc5 because rc4's extras required `nemo-fabric-adapters-*==0.1.0`, a version that was never published; rc6 fixed that. hermes stays direct because it must omit `[harness]`, which the metapackage's `hermes-agent` extra applies: `[harness]` pulls `hermes-agent`, whose exact `requests==2.33.0` pin conflicts with the workspace's `requests>=2.33.1` floor. It cannot drift regardless — every adapter pins `nemo-fabric-adapters-common==<its own version>`, so one shared common forces one shared rc. deepagents is omitted to match nemo-agents (AIRCORE-952). nemo-fabric and nemo-fabric-runtime get license overrides. Both declare Apache-2.0 via PEP 639 `license_expression` on PyPI, and deps.dev agrees, but osv-scanner resolves them to UNKNOWN for the 0.1.x rc series — the same way it already does for safetensors 0.8.0rc1. Without the override `license find-missing` fails the lint job. No platform marker: rc2+ publishes a macOS arm64 wheel alongside manylinux, so this resolves on every environment in [tool.uv] environments. `nemo-relay` is floored at >=0.6.0,<0.7, which is load-bearing — the runtime builds `AtofConfig(sinks=[...])`, a 0.6+ shape that 0.5.x accepts at install time and fails on at runtime. RUNTIME ALIGNMENT Five independent API drifts, all of the same kind — Fabric's models are `extra="allow"`, so each was accepted silently and simply did nothing: * `enable_relay` lost `config=`; now takes `observability=` built from Fabric's typed relay models. * ATOF's destination moved onto a typed sink list; the old flat form lands in the extras bag, leaves `sinks=None` and exports nothing. * Profiles were deleted outright — `FabricProfileConfig` is unexported, `Fabric.run`/`plan` take no `profiles`, and `FabricConfig.from_mapping` raises on a `profiles` key. Everything composes into one config; `_eval_lock_profiles` is gone, skills use `add_skill_path` (append + de-duplicate, which makes "treated arm silently drops preconfigured skills" structurally impossible), and the container runtime merges into the single `/in/agent.yaml` since `fabric run` no longer takes `--profile`. * `FabricConfig.models` is `dict[str, ModelConfig]`; a raw dict skipped validation and warned on serialization. * `RuntimeConfig` does not declare `mode`/`transport`; they reach the native core through extras, so examples build them with `from_mapping`. nemo-relay 0.6 made the same ATOF change, so both relay call sites migrated to `AtofConfig(sinks=[AtofFileSinkConfig(...)])`. Because these all failed silently, `tests/agent_eval/test_fabric_surface.py` asserts the composed *values* against the real published wheels rather than mocks, and the `fabric-wheel-smoke` CI job runs it. rc6 passes it unchanged — no further drift in this release. BREAKING: `FabricRunnerTarget.profiles` is removed from the job spec and the OpenAPI schema, as is the `profiles=` argument to `FabricAgentRuntime` and `FabricContainerRuntime`. The upstream concept no longer exists — fold any overlay into `config`. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
NeMo Fabric now publishes wheels to PyPI, so the Fabric agent-eval runtime no longer needs a source build from a local checkout on Linux. This adds a Linux-gated
fabricextra onnemo-evaluator-sdkthat installs the publishednemo-fabric[claude,codex,deepagents,runtime]from the lock.Changes
fabricextra (Linux-gated) onnemo-evaluator-sdk. The nativenemo-fabric-runtimewheel (which provides thenemo_fabricmodule) ships manylinux only — no macOS wheel yet, so thesys_platform == 'linux'marker keeps macOS lock resolution intact.nemo-relay0.4.x → 0.5.x(stable, all-platform wheels) to align with Fabric's relay line. Mirrored in the bundle-package block (packages/nemo_platform) and the vendored-SDK block (sdk/python/nemo-platform). Observability imports (AtifConfig,AtofConfig,ComponentSpec,ObservabilityConfig) verified on 0.5.0.openai-codex,openai-codex-cli-bin) — same pattern as the existingsafetensors>=0.8.0rc0line; no global--prerelease=allow. (sqlite-vecwas initially constrained too, but it's a plain LangGraph transitive dep —langgraph-checkpoint-sqlite>=0.1.6— so it resolves to stable 0.1.9.)script/dev-install-fabric.shreworked (macOS-native dev path):nemo-fabric[codex,relay,runtime]— theruntimeextra provides the importablenemo_fabricafter Fabric's metapackage split ([codex,relay]alone no longer does).nemo-relaygateway from the NeMo-Relay GitHub releases (checksum-verified, pinned viaNEMO_RELAY_VERSION) instead of acargo installfrom a local NeMo-Relay checkout — no NeMo-Relay clone or Rust build needed. Source-build fallback for platforms with no prebuilt asset (Intel macOS) orNEMO_RELAY_REPO.fabric-wheel-smokeCI job (gated on thedepspath filter, registered inci-status): installs the extra from the lock on Linux and imports thenemo_fabricsurfaceruntime.pyuses. This is the only CI coverage that actually resolves/installs/imports the real fabric tree (unit tests use a hermeticnemo_fabricfake) and validates themanylinux_2_39runtime wheel installs on the runner glibc.Verification
uv lock --checkclean; branch rebased onto current main.uv syncpulls zero fabric packages (all Linux-marker-gated) — only bumps relay0.4.0 → 0.5.0.requires_live_fabric.fabric-wheel-smokegreen on Linux (realimport nemo_fabric).Known limitations / follow-ups
nemo-fabricships a non-prerelease and itsadapters/codexdrops theopenai-codexbeta pin → then drop the prerelease constraints and re-lock.hermesomitted:hermes-agentpinsrequests==2.33.0exactly, conflicting with the workspace'srequests>=2.33.1floor. Re-add once upstream loosens it.nemo-relaygateway daemon (required for live ATIF capture on out-of-process agents like Codex) is published as prebuilt GitHub-release binaries for all platforms incl. macOS arm64, but not as a pip wheel — the Relay team confirmed they don't plan to wheel it, so the dev script / containers install it as a per-platform binary.nemo-fabric-runtimewheel yet (upstream ask on Fabric).make update-sdkregen can follow at merge time.🤖 Generated with Claude Code
Summary by CodeRabbit
nemo_fabricimports successfully.