feat(evaluator-sdk): allow a prebuilt image for FabricContainerRuntime

[SandyChapman]

What

FabricContainerRuntime always provisioned the harness-agnostic Fabric image (build-if-missing). This adds an optional keyword-only image parameter so a caller can supply a prebuilt image tag when the sandbox needs extra tooling — e.g. a document-processing image (pandoc/libreoffice), a pinned base, or an image from an air-gapped registry.

Behavior

• Backward-compatible. image defaults to None; when unset, behavior is unchanged (ensure_fabric_image() build-if-missing on first run).
• When provided, the tag is used verbatim and the build step is skipped. Image building remains a client concern, as before.

Tests

Two new cases in test_fabric_container_runtime.py (existing fake-provider harness):

• a supplied image reaches the SandboxSpec and short-circuits ensure_fabric_image;
• the default path still provisions the build-if-missing image.

pytest green (17/17 in the file); ruff check and ruff format --check clean.

Summary by CodeRabbit

• New Features
  • Added an optional custom Fabric container image override for evaluation runs.
  • Caller-provided image tags are used as-is and stored in trial metadata.
  • When no image is provided, the runtime provisions the default image automatically.
• Bug Fixes
  • Failed trials now consistently include the selected image in trial metadata.
• Tests
  • Added coverage for supplied-image usage, default image provisioning, and early-failure metadata behavior.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5fc5288d-7a88-4d39-8a69-652c68824d4f

📥 Commits

Reviewing files that changed from the base of the PR and between 79cde46 and 908a2b5.

⛔ Files ignored due to path filters (1)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/container_runtime.py is excluded by !sdk/**

📒 Files selected for processing (2)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py

🚧 Files skipped from review as they are similar to previous changes (2)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py

---

📝 Walkthrough

Walkthrough

FabricContainerRuntime accepts an optional image override, conditionally provisions the default image, and records the selected image in sandbox and trial metadata. Tests cover override, default provisioning, and early-failure paths.

Changes

Fabric image override

Layer / File(s)	Summary
Runtime image selection packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py	The runtime accepts a keyword-only image tag, conditionally provisions the default image, passes the selected tag to sandbox creation, and includes it in failed-trial metadata.
Image selection validation packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py	Fake provider tracking and tests validate supplied-image, default-provisioning, trial metadata, and early-failure behavior.

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant FabricContainerRuntime
  participant ensure_fabric_image
  participant SandboxProvider
  Caller->>FabricContainerRuntime: provide image override or none
  alt image override supplied
    FabricContainerRuntime->>SandboxProvider: create SandboxSpec with supplied image
  else no override
    FabricContainerRuntime->>ensure_fabric_image: provision default image
    FabricContainerRuntime->>SandboxProvider: create SandboxSpec with default image
  end
  SandboxProvider-->>FabricContainerRuntime: return sandbox result
  FabricContainerRuntime->>FabricContainerRuntime: _to_trial adds selected image metadata

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#604: Modifies the same Fabric image-provisioning and trial-metadata flow.

Suggested reviewers: ngoncharenko

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	Title clearly summarizes the main change: adding optional prebuilt image support for FabricContainerRuntime.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fabric-container-image-override/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py`:
- Around line 109-112: The _run_task failure path currently calls _failed_trial
without the selected image metadata. Reuse the base metadata containing
self._image when constructing failed trials, including failures before _to_trial
during startup, seeding, or download, and add a regression test covering an
early failure that verifies trial.metadata["image"].

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: ad991084-7d3d-488a-9cae-6455c3b11237

📥 Commits

Reviewing files that changed from the base of the PR and between 321736d and 760d84b.

📒 Files selected for processing (2)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/container_runtime.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_container_runtime.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	26382/34033	77.5%	61.8%
Integration Tests	15171/32658	46.5%	18.7%