feat(judge): trace-aware LLM judge via includeCommandTrace - #175
feat(judge): trace-aware LLM judge via includeCommandTrace#175sanchitmehtagit wants to merge 2 commits into
Conversation
judge() gains an opt-in { includeCommandTrace }. When set, the agent's
successful shell commands are appended to the judge input (under a
// COMMAND TRACE header) alongside workspace files, so a file-less CLI
eval has a judgeable artifact. Errored commands are dropped so the judge
sees only what took effect. Off by default — every file-based judge is
unchanged. ctx.toolCalls is already plumbed to the judge executor.
Ported from #82 (plumbing) in isolation — no mock CLI / routes / loader
changes. Adapted to the current evals-* package naming.
- eval-graders: GraderDef.includeCommandTrace; judge() JudgeOptions arg
- eval-core: formatCommandTrace() + append-when-opted-in in llm-judge
- tests: judge option defaults/opt-in; formatCommandTrace behaviour
- docs: AGENTS.md trace-aware judge note; ADDING_EVALS.md judge row
|
Warning Review limit reached
Next review available in: 43 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe judge grader now accepts an optional ChangesTrace-aware judge
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant JudgeOptions
participant LLMJudgeExecutor
participant EventToolCalls
JudgeOptions->>LLMJudgeExecutor: includeCommandTrace=true
LLMJudgeExecutor->>EventToolCalls: read successful run_command and bash calls
EventToolCalls-->>LLMJudgeExecutor: command trace
LLMJudgeExecutor-->>LLMJudgeExecutor: append trace to file content
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/ADDING_EVALS.md`:
- Line 98: The documentation describes an outdated judge() options contract. In
docs/ADDING_EVALS.md lines 98-98, update the adjacent options description to
state that options are primitive-specific and that judge() supports
includeCommandTrace; in AGENTS.md lines 121-123, change the grader-primitives
signature from judge(question, framework?) to judge(question, level?, options?).
In `@packages/evals-core/src/graders/executors/llm-judge.ts`:
- Around line 90-94: Add Vitest executor coverage for the includeCommandTrace
gate in JudgeExecutor.execute: with includeCommandTrace false, assert the
llmJudge input excludes the formatted command trace; with it true, assert the
input includes the formatted trace. Exercise the integration path through
execute rather than testing formatCommandTrace directly.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: eb2bab3e-3808-4309-b53d-66e8c561e6b0
📒 Files selected for processing (7)
AGENTS.mddocs/ADDING_EVALS.mdpackages/evals-core/src/graders/executors/llm-judge.tspackages/evals-core/tests/graders/executors.test.tspackages/evals-graders/src/primitives.tspackages/evals-graders/src/types.tspackages/evals-graders/tests/primitives.test.ts
4 participants
✏️ Changes
This ports the trace-aware LLM judge from #82 (the
plumbingbranch) in isolation, rebased onto latestmain. None of #82's mock-CLI, route-file, loader, or docker machinery comes with it. This is purely the judge change, so it can land independently of that larger infrastructure.judge()gains an opt-in third argument:judge(question, level?, { includeCommandTrace }). WhenincludeCommandTraceis true, the agent's successful shell commands are appended to the judge input under a// COMMAND TRACEheader, alongside the workspace files. This gives a judgeable artifact to evals whose work is entirely CLI invocations with no files to inspect, such as tenant configuration via the Auth0 CLI. Errored commands are dropped so the judge sees only what actually took effect.The flag defaults to
false, so every existing file-based judge is unchanged.ctx.toolCallsis already plumbed throughGraderContextto the judge executor onmain(event-based graders rely on it), so no wiring changes were needed.What changed, by package:
evals-graders: addedGraderDef.includeCommandTrace?: booleanand aJudgeOptionsargument onjudge()that defaults the flag tofalse.evals-core: addedformatCommandTrace(), which renders successfulrun_command/bashcalls under the header; the judge executor appends the trace to the file corpus only when the grader opts in.Tests:
judge()option default and opt-in;formatCommandTracebehaviour covering the header, thebashalias, dropping errored calls, ignoring non-shell tool calls, and empty input.Docs: a trace-aware judge note in
AGENTS.mdand an updatedjudgeprimitive row indocs/ADDING_EVALS.md.I described the changes on this PR.
🔮 Type of Change
🔗 References
Source PR this is carved out of: feat(plumbing): hermetic mock Auth0 CLI + trace-aware judge for tenant-config evals #82
I added at least one link (task, Slack thread, etc.) to explain why this change is needed.
📖 Documentation
Updated
AGENTS.md(trace-aware judge subsection under the judge exclusion prose) and thejudgeprimitive row indocs/ADDING_EVALS.mdto document the new optional argument.🎯 Testing
npm run build,npm test,npm run lint, andnpm run formatall pass locally. The full suite is green (503 tests inevals-core).New unit coverage:
formatCommandTracebehaviour inpackages/evals-core/tests/graders/executors.test.ts, andjudge()option handling inpackages/evals-graders/tests/primitives.test.ts.🚀 Deployment
Additive and backward-compatible: the new field is optional and off by default, so existing judges keep their current behaviour. No ordering constraints.
🔥 Rollback
Revert the single commit on this branch. The change is isolated to the judge primitive and executor, so a revert restores prior behaviour with no data or migration concerns.