feat(evaluator-sdk): per-task workspace + held-out overlay verifier for Fabric agent-eval

[SandyChapman]

Two related changes that let Fabric run workspace-based coding-agent evaluations and grade them against held-out ground truth.

1. Per-task workspace in FabricAgentRuntime

FabricAgentRuntime was trajectory-only — it never seeded inputs['files'] and exposed no workspace filesystem evidence, so the workspace-scoring metrics that work against the Codex runtime couldn't score a Fabric trial. Since Fabric drives the harness on the host filesystem in a dir we choose (no container), the agent's edited files are recoverable directly.

Now every task runs in its own fresh per-task workspace: inputs['files'] are seeded into it (a no-op when there are none), the harness runs in it via an environment.workspace profile overlay, and the final tree is exposed as a workspace filesystem EvidenceDescriptor — identical key/kind to the Codex runtime, so existing workspace-reading metrics work unchanged. Staging is unconditional (not a flag): a uniform per-task workspace matches the Codex runtime and maps cleanly onto a per-task container volume later. A config-supplied environment.workspace is overridden per task.

2. overlay_files on LocalFilesystemEvidence.run_verifier

A {relative_path: contents} map of trusted files written over the throwaway verifier copy before the command runs — the reusable "load workspace + overlay held-out files + run verifier" primitive. This is how a grader supplies held-out artifacts (a canonical test suite, a reference implementation) that must not live in, and cannot be edited through, the agent's own workspace. Paths escaping the copy are rejected; stored evidence is never mutated.

Together: seed only agent-facing files into the workspace, keep graders' ground truth in the task's reference, overlay it at scoring time.

Testing

• pytest packages/nemo_evaluator_sdk/tests/agent_eval/ → 109 passed, 1 skipped (gated live test).
• Fabric runtime: workspace seeded + agent edit captured + evidence exposed; a no-files task still gets a workspace; a bad seed fails only that task.
• run_verifier overlay: held-out file appears in the copy, overlay wins over the agent's version, nested paths work, evidence untouched, path-escape rejected.
• ruff / ruff format / CI ty clean; make vendor synced.

Stack

main ← #561 (Taskset) ← #573 ← #565 (notebook). Base retargets to main as #561 merges.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Fabric task execution now stages each task in an isolated workspace, seeds provided input files, and includes the resulting workspace filesystem as evidence.
  • Evidence verification now supports optional overlay files applied to the verifier copy.
• Bug Fixes
  • Trial metadata now consistently reports agent_ok for both successful and failed trials.
  • Added stronger path containment protections for both workspace seeding and verifier overlays.
• Tests
  • Expanded coverage for overlay behavior, security validation, per-task workspace evidence/seeding (including empty inputs), traversal failures, and metric correctness.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7aa78275-0f87-47d2-9997-0bc6b70d60b4

📥 Commits

Reviewing files that changed from the base of the PR and between d72f36b and ba2761e.

⛔ Files ignored due to path filters (2)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/evidence.py is excluded by !sdk/**

📒 Files selected for processing (4)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/evidence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_evidence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py

🚧 Files skipped from review as they are similar to previous changes (4)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/evidence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_evidence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py

---

📝 Walkthrough

Walkthrough

Fabric now stages per-task workspaces, seeds them from inputs['files'], exports workspace evidence, and records agent_ok. run_verifier also supports overlaying files into a sandbox copy before verification. Tests cover both flows.

Changes

Fabric Runtime Workspace Staging

Layer / File(s)	Summary
Workspace staging and prompt packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py	Adds per-task workspace setup, seeding, workspace overlay wiring, and updated prompt handling for seeded files.
Trial evidence and metadata packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py	Passes workspace_dir into trial construction, adds workspace filesystem evidence on success, and sets agent_ok on success and failure.
Fabric runtime tests packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py	Adds workspace helper and tests for seeding, workspace evidence, empty inputs, traversal failure, agent_ok, and AgentPhaseSuccessMetric.

Verifier Overlay Files Support

Layer / File(s)	Summary
Sandbox overlay staging packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/evidence.py	Adds overlay_files support, sandbox copy logic, symlink filtering, and path containment checks.
Overlay tests packages/nemo_evaluator_sdk/tests/agent_eval/test_evidence.py	Adds tests for overlay staging, override behavior, stored evidence preservation, and escape rejection.

Sequence Diagram(s)

sequenceDiagram
  participant RunTask
  participant Seeder
  participant Profile
  participant Harness
  participant Trial

  RunTask->>Seeder: seed inputs['files'] into workspace_dir
  Seeder-->>RunTask: seeded_files
  RunTask->>Profile: build environment.workspace overlay
  RunTask->>Harness: run with _fabric_input(task, seeded_files)
  Harness-->>RunTask: result
  RunTask->>Trial: build trial(result, workspace_dir)

Loading

sequenceDiagram
  participant Caller
  participant RunVerifier
  participant Sandbox
  participant WriteOverlay

  Caller->>RunVerifier: run_verifier(overlay_files, cwd)
  RunVerifier->>Sandbox: copy evidence snapshot
  RunVerifier->>WriteOverlay: write overlay_files into sandbox
  WriteOverlay-->>RunVerifier: ValueError if path escapes root
  RunVerifier->>Sandbox: execute verifier command

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#339: Adds the LocalFilesystemEvidence surface this PR extends with overlay file staging.
• NVIDIA-NeMo/nemo-platform#563: Adds the workspace seeding machinery and escape handling used by the Fabric runtime change.
• NVIDIA-NeMo/nemo-platform#564: Changes the same Fabric runtime path that now stages workspaces and exports workspace evidence.

Suggested reviewers: ngoncharenko, arpitsardhana

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately captures both main changes: per-task workspaces and the verifier overlay support for Fabric agent-eval.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fabric-runtime-workspace/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py (1)

168-183: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Seed handling is outside the task timeout. asyncio.wait_for(...) only bounds client.run; seed_workspace(...) can still block first, so a slow handler can push this task past self._timeout_s. Wrap seeding in the timeout too, or enforce timeouts in every handler.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py`
around lines 168 - 183, The task timeout currently only wraps client.run in
runtime.py’s fabric executor, so seed_workspace can still run unbounded before
the timeout starts. Move the seeding work into the same timeout window as the
rest of the task execution by applying asyncio.wait_for around the
seed_workspace call (or otherwise ensuring the handler path is timed), then
continue building seeded_files and calling _fabric_input and client.run as
before.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py`:
- Around line 168-183: The task timeout currently only wraps client.run in
runtime.py’s fabric executor, so seed_workspace can still run unbounded before
the timeout starts. Move the seeding work into the same timeout window as the
rest of the task execution by applying asyncio.wait_for around the
seed_workspace call (or otherwise ensuring the handler path is timed), then
continue building seeded_files and calling _fabric_input and client.run as
before.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a543acb2-f790-4123-ba7d-01d20b676e21

📥 Commits

Reviewing files that changed from the base of the PR and between 016ba5d and 19724e2.

⛔ Files ignored due to path filters (2)

• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/fabric/runtime.py is excluded by !sdk/**
• sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/values/evidence.py is excluded by !sdk/**

📒 Files selected for processing (4)

• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/fabric/runtime.py
• packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/values/evidence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_evidence.py
• packages/nemo_evaluator_sdk/tests/agent_eval/test_fabric_runtime.py

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	23324/30499	76.5%	61.3%
Integration Tests	13630/29179	46.7%	19.9%

[SandyChapman]

@coderabbitai Re: the "seed handling is outside the task timeout" note in _run_task — intentionally leaving this as-is (won't-fix for this PR):

• Parity, not a regression. The Codex runtime (already shipped) does the identical thing: seed_workspace via asyncio.to_thread before asyncio.wait_for(client.run(...), timeout=self._timeout_s). This runtime matches that established pattern rather than diverging from it.
• The suggested fix doesn't actually bound anything. Wrapping the to_thread seed call in wait_for can't cancel a thread running in the default executor — on timeout the task is marked failed while the seed thread keeps running in the background. That's a misleading guarantee, not real cancellation.
• Seeding is off the event loop. Because it's offloaded via to_thread, a slow fileset handler doesn't stall other concurrent tasks; only that one task's wall-clock is unbounded.

If we ever need to bound seed/download time, the right place is a timeout inside the fileset handler itself (your secondary suggestion), not a wait_for here.

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

@SandyChapman Fair enough — parity with Codex and the fact that wait_for can't actually cancel a thread-pool task both make sense. Agreed the real fix belongs in the fileset handler if/when needed.