refactor(evaluator-sdk): make the Codex runtime a general coding-agent runtime - #562
Merged
Merged
Conversation
Contributor
📝 WalkthroughWalkthroughCodex runtime resolution now returns the effective runtime and accepts prompt customization. The Codex CLI runtime seeds workspace files from task input, records seeding metadata, and validates seed paths. ProfBench uses the new resolver with its own prompt builder and runtime-to-score mapping. Tests cover the new runtime flow. ChangesCodex Runtime and ProfBench Integration
Sequence Diagram(s)sequenceDiagram
participant Caller
participant CodexCliAgentRuntime
participant Workspace
participant Process
Caller->>CodexCliAgentRuntime: _run_task(task)
CodexCliAgentRuntime->>Workspace: _seed_workspace(task)
Workspace-->>CodexCliAgentRuntime: seeded_files
CodexCliAgentRuntime->>CodexCliAgentRuntime: prompt_builder(task)
CodexCliAgentRuntime->>Process: send prompt via stdin
Process-->>CodexCliAgentRuntime: result
CodexCliAgentRuntime-->>Caller: AgentEvalTrial(metadata: agent_ok, seeded_files)
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
packages/nemo_evaluator_sdk/examples/profbench/runner.py (2)
255-263: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winPrompt dumps full
task.inputs, likely duplicating seeded file content.Runtime now seeds the workspace from
inputs["files"](per PR summary). Embeddingtask.inputsraw viaf"Inputs: {task.inputs}"risks re-dumping that same file content into the prompt text — bloating context/cost and producing an ugly dict repr instead of a clean task description. Consider excluding the seed-files key (or any large payload) from what's interpolated into the prompt, since seeded files are already on disk for Codex to read.♻️ Suggested filtering
def profbench_codex_prompt(task: AgentEvalTask) -> str: """Frame a task as a ProfBench candidate: return only the final answer text, no tooling chatter.""" + prompt_inputs = {k: v for k, v in task.inputs.items() if k != SEED_FILES_INPUT_KEY} return ( "Answer the ProfBench task below. Return only the final answer text; do not include " "analysis, markdown fences, tool logs, or commentary.\n\n" f"Task id: {task.id}\n" f"Intent: {task.intent}\n" - f"Inputs: {task.inputs}\n" + f"Inputs: {prompt_inputs}\n" )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/profbench/runner.py` around lines 255 - 263, The profbench_codex_prompt function is interpolating the entire task.inputs object, which can re-embed seeded file content and inflate the prompt with an unreadable dict repr. Update profbench_codex_prompt to build the prompt from a filtered view of task.inputs, excluding large payloads such as the files seed data (and any similarly bulky keys), while keeping only the concise task metadata needed for the model. Keep the change localized to profbench_codex_prompt so the prompt remains short and the workspace-seeded files are read from disk instead of duplicated in text.
248-252: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDict lookup can
KeyErrorifEffectiveCodexRuntimegrows.
PROFBENCH_SCORE_SOURCE[effective_runtime](line 298) will raise if the SDK enum adds a member this map doesn't cover. Since the enum lives in a separate file (runtime.py), nothing enforces the two stay in sync.♻️ Safer lookup with explicit error
- return target, None, PROFBENCH_SCORE_SOURCE[effective_runtime], effective_runtime + try: + score_source = PROFBENCH_SCORE_SOURCE[effective_runtime] + except KeyError: + raise ValueError(f"unmapped Codex runtime for score_source: {effective_runtime!r}") from None + return target, None, score_source, effective_runtimeAlso applies to: 290-298
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/examples/profbench/runner.py` around lines 248 - 252, The PROFBENCH score source mapping is not future-proof and can raise a KeyError when EffectiveCodexRuntime gains new members. Update the lookup used in runner.py around PROFBENCH_SCORE_SOURCE and the code that reads it in the profile bench flow to handle unknown runtimes explicitly, ideally by using a safe mapping access with a clear fallback or by raising a controlled error that names the unsupported effective_runtime. Keep the mapping and the lookup logic aligned so the behavior remains stable if runtime.py adds new enum values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py`:
- Around line 341-342: The Docker sandbox path in runtime selection currently
returns DockerSandboxAgentRuntime without handling inputs['files'], so seed
files are silently ignored. Update the Codex runtime flow around the
effective_runtime check to either stage the provided files before creating
DockerSandboxAgentRuntime or explicitly reject non-empty inputs['files'] with a
clear failure. Keep the fix localized to the runtime selection logic and use the
existing DockerSandboxAgentRuntime and EffectiveCodexRuntime.DOCKER_SANDBOX
symbols to wire it in.
---
Nitpick comments:
In `@packages/nemo_evaluator_sdk/examples/profbench/runner.py`:
- Around line 255-263: The profbench_codex_prompt function is interpolating the
entire task.inputs object, which can re-embed seeded file content and inflate
the prompt with an unreadable dict repr. Update profbench_codex_prompt to build
the prompt from a filtered view of task.inputs, excluding large payloads such as
the files seed data (and any similarly bulky keys), while keeping only the
concise task metadata needed for the model. Keep the change localized to
profbench_codex_prompt so the prompt remains short and the workspace-seeded
files are read from disk instead of duplicated in text.
- Around line 248-252: The PROFBENCH score source mapping is not future-proof
and can raise a KeyError when EffectiveCodexRuntime gains new members. Update
the lookup used in runner.py around PROFBENCH_SCORE_SOURCE and the code that
reads it in the profile bench flow to handle unknown runtimes explicitly,
ideally by using a safe mapping access with a clear fallback or by raising a
controlled error that names the unsupported effective_runtime. Keep the mapping
and the lookup logic aligned so the behavior remains stable if runtime.py adds
new enum values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ea73ed43-b8bf-44ab-af8a-3d3b1ce67d90
📒 Files selected for processing (3)
packages/nemo_evaluator_sdk/examples/profbench/runner.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py
Contributor
|
arpitsardhana
approved these changes
Jul 3, 2026
…t runtime
The shipped CodexCliAgentRuntime was ProfBench-shaped: a hardcoded "return only
the final answer text" prompt, no way to seed a workspace, and no success signal
for the standard AgentPhaseSuccessMetric. That made it unusable for general
coding tasks (write docs, write tests, fix a bug) and leaked a benchmark's
opinions into src/.
Generalize the runtime:
- Neutral, injectable prompt: default_codex_prompt presents the task and invites
workspace edits; both runtimes take a prompt_builder to override framing.
- Workspace seeding: inputs["files"] = {path: contents} is staged into the
agent's workspace before it runs (paths escaping the workspace are rejected),
so a task can hand the agent starter code.
- Stamp agent_ok from the exit code so AgentPhaseSuccessMetric works over Codex
trials.
Keep runtime selection generic and in the SDK (resolve_codex_runtime + the
RuntimeChoice/EffectiveCodexRuntime enums); move only ProfBench's own policy —
its candidate prompt and the candidate+judge score_source labels — into the
ProfBench example, inlined at the point of use in runner.py.
Backward compatible: the plugin's CodexRunnerTarget path is unchanged (the new
constructor args are optional) and now gets the neutral prompt for free.
Signed-off-by: Sandy Chapman <schapman@nvidia.com>
SandyChapman
force-pushed
the
codex-runtime-general-coding/schapman
branch
from
July 3, 2026 17:09
cab7311 to
0928ad6
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py (1)
102-117: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winKeep prompt generation inside the per-task failure guard.
Line 102 can raise from an injected
prompt_builder; today that escapesasyncio.gatherand can abort the run instead of producing one failed trial.Proposed fix
- prompt = self._prompt_builder(task) prompt_path = evidence_dir / "prompt.txt" task_path = evidence_dir / "task.json" stdout_path = evidence_dir / "stdout.jsonl" stderr_path = evidence_dir / "stderr.txt" final_output_path = evidence_dir / "final_output.txt" - prompt_path.write_text(prompt, encoding="utf-8") - task_path.write_text(task.model_dump_json(indent=2), encoding="utf-8") - command = self._command(workspace_dir=workspace_dir, final_output_path=final_output_path) process: Any | None = None try: + prompt = self._prompt_builder(task) + prompt_path.write_text(prompt, encoding="utf-8") + task_path.write_text(task.model_dump_json(indent=2), encoding="utf-8") # Seed inside the guarded block so a bad seed (e.g. a path escaping the workspace) fails # just this task rather than aborting the whole run. seeded_files = _seed_workspace(workspace_dir, task)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py` around lines 102 - 117, Move the prompt creation for each trial into the existing per-task try/except guard in the codex runtime so failures from an injected prompt_builder are treated as task-level errors instead of escaping and aborting asyncio.gather. In the runtime method that currently assigns prompt before the guarded block, wrap the self._prompt_builder(task) call together with prompt_path.write_text and the other per-task setup inside the same guarded section as _seed_workspace and command execution, so a bad prompt only produces one failed trial.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py`:
- Around line 421-431: The seed file handling in the runtime silently ignores
malformed payloads and coerces non-string file contents, which can hide bad
input and corrupt starter files. In the seeding path that reads
SEED_FILES_INPUT_KEY in the codex runtime, reject any present but non-Mapping
value with an error instead of returning an empty list, and reject any seed
entry whose contents are not already a string rather than converting it. Keep
the existing workspace escape check in place while updating the code around
workspace_root, written, and the per-file loop to fail fast on invalid seed
payloads.
---
Outside diff comments:
In
`@packages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.py`:
- Around line 102-117: Move the prompt creation for each trial into the existing
per-task try/except guard in the codex runtime so failures from an injected
prompt_builder are treated as task-level errors instead of escaping and aborting
asyncio.gather. In the runtime method that currently assigns prompt before the
guarded block, wrap the self._prompt_builder(task) call together with
prompt_path.write_text and the other per-task setup inside the same guarded
section as _seed_workspace and command execution, so a bad prompt only produces
one failed trial.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 7f6b46c1-7b69-4738-8377-a04faf870488
⛔ Files ignored due to path filters (1)
sdk/python/nemo-platform/src/nemo_platform/beta/evaluator/agent_eval/runtimes/codex/runtime.pyis excluded by!sdk/**
📒 Files selected for processing (3)
packages/nemo_evaluator_sdk/examples/profbench/runner.pypackages/nemo_evaluator_sdk/src/nemo_evaluator_sdk/agent_eval/runtimes/codex/runtime.pypackages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/nemo_evaluator_sdk/examples/profbench/runner.py
- packages/nemo_evaluator_sdk/tests/agent_eval/test_codex_runtime.py
SandyChapman
enabled auto-merge
July 3, 2026 17:24
This was referenced Jul 7, 2026
Merged
arpitsardhana
pushed a commit
that referenced
this pull request
Jul 9, 2026
…t runtime (#562) The shipped CodexCliAgentRuntime was ProfBench-shaped: a hardcoded "return only the final answer text" prompt, no way to seed a workspace, and no success signal for the standard AgentPhaseSuccessMetric. That made it unusable for general coding tasks (write docs, write tests, fix a bug) and leaked a benchmark's opinions into src/. Generalize the runtime: - Neutral, injectable prompt: default_codex_prompt presents the task and invites workspace edits; both runtimes take a prompt_builder to override framing. - Workspace seeding: inputs["files"] = {path: contents} is staged into the agent's workspace before it runs (paths escaping the workspace are rejected), so a task can hand the agent starter code. - Stamp agent_ok from the exit code so AgentPhaseSuccessMetric works over Codex trials. Keep runtime selection generic and in the SDK (resolve_codex_runtime + the RuntimeChoice/EffectiveCodexRuntime enums); move only ProfBench's own policy — its candidate prompt and the candidate+judge score_source labels — into the ProfBench example, inlined at the point of use in runner.py. Backward compatible: the plugin's CodexRunnerTarget path is unchanged (the new constructor args are optional) and now gets the neutral prompt for free. Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The shipped
CodexCliAgentRuntimewas ProfBench-shaped, which leaked a benchmark's opinions intosrc/and made it unusable for general coding-agent evaluation:agent_okwas never stamped, so the standardAgentPhaseSuccessMetricread False even on success.What changed
Generalize the runtime (
runtime.py):default_codex_promptstates the task's intent/inputs and invites the agent to read/create/edit files. Both runtimes accept aprompt_builderto override framing for a specific benchmark.inputs["files"] = {path: contents}convention (SEED_FILES_INPUT_KEY) is staged into the agent's workspace before it runs; paths escaping the workspace are rejected (→ failed trial for that task only). Seeded files are listed by name in the prompt.agent_okfrom the exit code soAgentPhaseSuccessMetricworks over Codex trials.Keep the ProfBench/generic boundary clean:
resolve_codex_runtime(...) -> (runtime, effective_runtime)plus theRuntimeChoice/EffectiveCodexRuntimeenums.score_sourcelabels — moves to the ProfBench example, inlined at the point of use inrunner.py(matching how the sibling model branch already builds its ownscore_source).Compatibility
Backward compatible: the plugin's
CodexRunnerTargetpath is unchanged (new constructor args are optional) and now gets the neutral prompt for free.Testing
agent_evalsuite + plugintest_agent_evaluate.py: 110 pass (added seeding /agent_ok/ custom-prompt-builder / generic-resolver coverage).ruff check/formatclean; CIlint-python-types.shexit 0.Independent of #561 (Taskset entity); branched off
main.Summary by CodeRabbit
New Features
Bug Fixes
Tests