docs(evaluator-sdk): agentic-eval tutorial notebook (Codex via NeMo Fabric)

[SandyChapman]

End-to-end agentic-eval walkthrough: a small suite of coding tasks (fix a bug, write tests, write docs) run against Codex driven by NeMo Fabric, scored with metrics that inspect the agent's real workspace, rolled up into one comparable correctness view.

Reworked from the earlier Codex-CLI draft to reflect current main and the workspace support in #573:

• Target is FabricAgentRuntime: Fabric drives the Codex CLI, runs each task in its own per-task workspace, exposes the final tree as workspace evidence, and captures the ATIF trace (capture_trajectory=True). Depends on feat(evaluator-sdk): per-task workspace + held-out overlay verifier for Fabric agent-eval #573.
• Held-out grading, two shapes:
  • fix-bug — the test suite lives in the task's grader-only reference and is overlaid into a throwaway copy at scoring time via run_verifier(overlay_files=...) (also from feat(evaluator-sdk): per-task workspace + held-out overlay verifier for Fabric agent-eval #573). The agent only ever sees the buggy source; it can't edit the tests that grade it.
  • write-tests — the deliverable is the tests, so instead the authoritative module lives in reference and a WorkspaceFileUnchanged metric checksums the agent's copy against it (impl_unchanged); editing the code under test fails the correctness view.
  • A callout explains why you must not grade with artifacts the agent can edit.
• ViewSignals reference metric.type instead of string literals.
• Groups the suite into an AgentEvalTaskset.
• Renamed to agentic_eval_with_fabric.ipynb; prereqs document the native Fabric install.

Verification

Fabric target + run cells need the native stack (documented). Everything else validated locally against the SDK:

• all 36 cells' code parses; imports resolve; ruff format clean.
• fix-bug held-out overlay grading: a test suite never present in the workspace passes a fixed solution, fails a buggy one.
• write-tests checksum: impl_unchanged is True for an untouched module, False when tampered; the agent's own tests run and pass against the verified module.
• all three tasks + the taskset build and pass view-signal validation.

Stack

main ← #561 (Taskset) ← #573 (workspace support) ← #565 (this). Base retargets to main as parents merge.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added a tutorial notebook that demonstrates an end-to-end evaluation workflow for coding agents using NeMo Fabric.
  • Covers three scenarios: fixing a bug, writing tests, and writing documentation.
  • Includes automated scoring that checks workspace correctness, runs pytest-based verification, and uses an AI judge for doc quality.
  • Captures evaluation traces and produces an HTML dashboard plus persisted run artifacts for review.

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	23324/30499	76.5%	61.3%
Integration Tests	13631/29179	46.7%	20.0%

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bacfba20-e8bd-4a36-9bf4-76fde5ef8670

📥 Commits

Reviewing files that changed from the base of the PR and between 7eebacf and 1ff0b2a.

📒 Files selected for processing (1)

• packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb

🚧 Files skipped from review as they are similar to previous changes (1)

• packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb

---

📝 Walkthrough

Walkthrough

Adds a notebook that demonstrates agentic evaluation with NeMo Fabric, including setup, custom scoring, task definitions, Fabric execution, and result inspection.

Changes

Fabric Agentic Eval Notebook

Layer / File(s)	Summary
Environment setup and workspace helpers packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb	Prompts for NVIDIA_BUILD_API_KEY and NEMO_FABRIC_REPO, creates a temp output directory, defines inline(), and introduces the grading approach.
Custom scoring helpers packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb	Implements substring checks, SHA-256 verification, SHA parsing, and the pytest metric with optional held-out overlays.
Task definitions and taskset packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb	Defines the fix-bug, write-tests, and write-docs tasks, wires metrics, adds LlmDocReview, and bundles the taskset.
Fabric runtime and evaluation run packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb	Configures FabricAgentRuntime, enables trajectory capture, and runs AgentEvaluator.run_sync with dashboard output.
Result aggregation and notebook wrap-up packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb	Extracts summary metrics, flattens scores into a dataframe, prints trial evidence keys, adds next-step guidance, and closes notebook metadata.

Sequence Diagram(s)

sequenceDiagram
  participant Notebook
  participant AgentEvaluator
  participant FabricAgentRuntime
  participant Dashboard
  Notebook->>AgentEvaluator: run_sync(taskset, target)
  AgentEvaluator->>FabricAgentRuntime: launch trials with Codex CLI harness
  FabricAgentRuntime->>Dashboard: write run bundle and trajectory evidence
  AgentEvaluator->>Notebook: return result.summary and result.scores

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#369: Uses AgentEvaluator.run_sync and dashboard artifacts from that evaluator flow.
• NVIDIA-NeMo/nemo-platform#506: Uses SecretRef for the judge API key in the LLM-based doc metric.
• NVIDIA-NeMo/nemo-platform#564: Exercises FabricAgentRuntime and trajectory-capture evidence.

Suggested reviewers: ngoncharenko, arpitsardhana

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: a tutorial notebook for agentic evaluation using Codex via NeMo Fabric.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch agent-eval-tutorial/schapman

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb (1)

118-237: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicated evidence→workspace guard across all four metrics.

evidence = input.candidate.evidence; if evidence is not None and evidence.get("workspace") is not None: ws = await evidence.filesystem("workspace") is repeated verbatim in WorkspaceFileContains, WorkspaceFileUnchanged, PytestResults (lines 118-237), and again in LlmDocReview (lines 401-409). Extract a small helper.

♻️ Suggested helper

+async def _open_workspace(evidence):
+    """Return the workspace filesystem from evidence, or None if absent."""
+    if evidence is not None and evidence.get("workspace") is not None:
+        return await evidence.filesystem("workspace")
+    return None

Then each compute_scores becomes:

-        evidence = input.candidate.evidence
-        if evidence is not None and evidence.get("workspace") is not None:
-            ws = await evidence.filesystem("workspace")
-            if await ws.exists(self._path):
+        ws = await _open_workspace(input.candidate.evidence)
+        if ws is not None:
+            if await ws.exists(self._path):

Also applies to: 401-409

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb` around
lines 118 - 237, Repeated evidence-to-workspace access logic is duplicated
across WorkspaceFileContains, WorkspaceFileUnchanged, PytestResults, and
LlmDocReview. Extract a small shared helper that checks input.candidate.evidence
for a workspace and returns the filesystem object, then update each
compute_scores method to call it instead of repeating the guard and await
evidence.filesystem("workspace") sequence.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb`:
- Around line 118-237: Repeated evidence-to-workspace access logic is duplicated
across WorkspaceFileContains, WorkspaceFileUnchanged, PytestResults, and
LlmDocReview. Extract a small shared helper that checks input.candidate.evidence
for a workspace and returns the filesystem object, then update each
compute_scores method to call it instead of repeating the guard and await
evidence.filesystem("workspace") sequence.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1edbe513-9af8-4ae3-b8f4-314cf934c108

📥 Commits

Reviewing files that changed from the base of the PR and between ccb9d26 and 7eebacf.

📒 Files selected for processing (1)

• packages/nemo_evaluator_sdk/examples/agentic_eval_with_fabric.ipynb