feat(evals): add Relay SDK eval harness#1092
Closed
khaliqgant wants to merge 16 commits into
Closed
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Folds in the op argument and check-key conventions surfaced by the first authoring round so the relaunched workers start from a concrete spec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
More reviews will be available in 4 minutes and 10 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more credits in the billing tab to continue. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (65)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive, deterministic evaluation suite for the Relay SDK, adding multiple test suites, rubrics, and scripts to compile and execute the evaluations in memory. The review feedback highlights several critical areas for improvement in the newly added scripts: a potential crash in parseScalar when handling invalid JSON-like strings, a missing key property in the workspace initialization, unassigned cleanup handlers for DeliveryRunner instances, loss of descriptive messages for plain object errors in normalizeError, inconsistent exit code behavior for skipped tests in ci-summary.mjs, and a potential filtering bug when passing an empty tag set.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Comment on lines
+207
to
+216
| function parseScalar(raw) { | ||
| const value = String(raw).trim(); | ||
| if (value === "true") return true; | ||
| if (value === "false") return false; | ||
| if (/^-?\d+$/.test(value)) return Number(value); | ||
| if ((value.startsWith("{") && value.endsWith("}")) || (value.startsWith("[") && value.endsWith("]"))) { | ||
| return JSON.parse(value); | ||
| } | ||
| return value; | ||
| } |
There was a problem hiding this comment.
In parseScalar, if a string value starts with { and ends with } (or [ and ]) but is not valid JSON (for example, a template placeholder like {placeholder} or some descriptive text), JSON.parse(value) will throw a SyntaxError and crash the entire compilation process. Wrapping JSON.parse in a try-catch block and falling back to the raw string on failure makes the parser much more robust against non-JSON brace-enclosed strings.
function parseScalar(raw) {
const value = String(raw).trim();
if (value === "true") return true;
if (value === "false") return false;
if (/^-?\d+$/.test(value)) return Number(value);
if ((value.startsWith("{") && value.endsWith("}")) || (value.startsWith("[") && value.endsWith("]"))) {
try {
return JSON.parse(value);
} catch {
return value;
}
}
return value;
}| channels: new Map(), | ||
| messages: new Map(), | ||
| dmConversations: new Map(), | ||
| workspace: { id: mock.workspace?.id ?? "ws_eval", name: mock.workspace?.name ?? "relay-eval" }, |
There was a problem hiding this comment.
The workspace object initialized on line 52 does not include the key property from mock.workspace. As a result, state.workspace.key on line 67 is always undefined, and the default workspace key can never be customized via mock.workspace.key or mock.workspace.workspaceKey. Including the key in the initial workspace object resolves this.
Suggested change
| workspace: { id: mock.workspace?.id ?? "ws_eval", name: mock.workspace?.name ?? "relay-eval" }, | |
| workspace: { id: mock.workspace?.id ?? "ws_eval", name: mock.workspace?.name ?? "relay-eval", key: mock.workspace?.key ?? mock.workspace?.workspaceKey ?? "rk_live_default" }, |
| const invocationId = `inv_${++state.counters.invocation}`; | ||
| state.actionInvocations.set(invocationId, { invocationId, actionName: name, callerName: "sdk", input, status: "invoked" }); | ||
| emit(state, "actionInvoked", { type: "actionInvoked", invocationId, actionName: name, callerName: "sdk", handlerAgentId: "handler" }); | ||
| return { invocationId, actionName: name, input, status: "invoked" }; |
There was a problem hiding this comment.
The state.stopDelivery function is initialized to undefined but is never assigned a value when a DeliveryRunner is started in deliver(). This means state.stopDelivery?.() in the finally block of relayExecute does nothing, potentially leaving active timers or polling intervals from the DeliveryRunner hanging. Chaining the stop function of the runner to state.stopDelivery ensures proper cleanup of all started runners.
Suggested change
| return { invocationId, actionName: name, input, status: "invoked" }; | |
| const prevStop = state.stopDelivery; | |
| state.stopDelivery = () => { | |
| prevStop?.(); | |
| runner.stop?.(); | |
| }; | |
| await runner.start(); |
| } | ||
| return { | ||
| code: error?.code ?? error?.name ?? "error", | ||
| message: error instanceof Error ? error.message : String(error), |
There was a problem hiding this comment.
In normalizeError, if error is a plain object (which is common for some custom error representations or promise rejections) rather than an instance of Error, error instanceof Error will be false, and String(error) will return "[object Object]", losing the actual error message. Checking for error?.message before falling back to String(error) preserves the descriptive message.
Suggested change
| message: error instanceof Error ? error.message : String(error), | |
| message: error instanceof Error ? error.message : (error?.message ?? String(error)), |
| console.log(summary); | ||
|
|
||
| if (process.env.GITHUB_STEP_SUMMARY) writeFileSync(process.env.GITHUB_STEP_SUMMARY, summary, { flag: "a" }); | ||
| if (failed.length > 0 || skipped.length > 0) process.exitCode = 1; |
There was a problem hiding this comment.
In ci-summary.mjs, the script always sets process.exitCode = 1 if there are any skipped tests. However, in run-relay-evals.mjs, skipped tests only fail the run if failOnSkipped is explicitly enabled. This discrepancy means CI builds will unexpectedly fail on skipped tests even when not configured to do so. Aligning the exit code logic with the environment variables RELAY_EVAL_FAIL_ON_SKIPPED and HUMAN_EVAL_FAIL_ON_SKIPPED ensures consistent behavior.
Suggested change
| if (failed.length > 0 || skipped.length > 0) process.exitCode = 1; | |
| const failOnSkipped = process.env.RELAY_EVAL_FAIL_ON_SKIPPED === "1" || process.env.HUMAN_EVAL_FAIL_ON_SKIPPED === "1"; | |
| if (failed.length > 0 || (failOnSkipped && skipped.length > 0)) process.exitCode = 1; |
| const selectedCases = allCases.filter((testCase) => matchesHumanEvalFilters(testCase, { | ||
| suite: args.suite, | ||
| caseId: args.caseId, | ||
| tags: args.tags, |
There was a problem hiding this comment.
Passing an empty Set as the tags filter to matchesHumanEvalFilters can cause the filter function to treat it as an active filter (since the Set object itself is truthy), potentially resulting in zero cases being matched when no tags are specified. It is safer to pass undefined if the set is empty so that the tag filter is ignored.
tags: args.tags.size > 0 ? args.tags : undefined,
Contributor
|
Fixed the reproducible eval breakage I found after formatting the PR files. Changed:
Addressed comments
Advisory Notes
Verification run locally:
I did not print |
Contributor
|
Implemented scoped fixes for PR #1092’s eval harness. Changed:
Verification:
Addressed comments
Advisory Notes
|
Contributor
|
Reviewed PR #1092 and made scoped fixes in the PR files. Changes made:
Addressed comments
Advisory Notes
Verification passed locally:
|
Contributor
|
Blocked files: Reviewed PR #1092 and made scoped fixes. Changes made:
Validation:
Addressed comments
Advisory Notes
|
4 tasks
khaliqgant
pushed a commit
that referenced
this pull request
Jun 13, 2026
khaliqgant
pushed a commit
that referenced
this pull request
Jun 14, 2026
khaliqgant
added a commit
that referenced
this pull request
Jun 14, 2026
#1109) * docs(evals): add master plan for relay SDK eval suite Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(evals): add op-arg reference + executor result contract Folds in the op argument and check-key conventions surfaced by the first authoring round so the relaunched workers start from a concrete spec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(evals): mark op vocabulary LOCKED per W1 confirmation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add messaging relay eval suites * Add session listener facade eval suites * Add channel workspace agent directory eval suites * Add delivery actions eval suites * Relax W3 eval content expectations * Align W3 eval duplicate error codes * Relax delivery action eval expectations * Align W5 eval cases with runner output * Align workspace eval assertions with runner output * Mark capability eval hooks pending * Relax unknown agent removal eval assertion * feat(evals): add relay eval harness * chore: apply pr-reviewer fixes for #1092 * Add local workflow run commands * Avoid log read file race * Use Relayflows for local workflow runs * Add agent messaging eval suite and reports Introduce a new integration eval suite that exercises agent-to-agent messaging via the broker and scores protocol adherence (message-sent rate, phantom messages, ACK/DONE protocol, wrong-channel replies). Adds a full eval runner, scenarios, deterministic scoring, reporters (JSON + self-contained HTML viewer), a matrix roll-up, unit tests for scoring, and CLI helpers under tests/integration/broker/evals. Adds npm scripts (eval:build, eval:unit, eval:selftest, eval:toolcheck, eval:html, eval, eval:claude, eval:matrix) and gitignore entry for evals-reports. Also adds a Fleet Delivery design doc (specs/fleet-delivery.md), updates CHANGELOG.md, and adjusts integration test config/files (tsconfig, vitest, and broker harness utilities) to align the broker-harness with the current SDK/harness-driver APIs so the evals build/run cleanly. * feat(evals): add spawn/release reliability eval suite with onboarding variants Adds three new scenarios (s01-spawn-worker, s02-release-worker, s03-lifecycle) that test whether agents reliably call add_agent and remove_agent across four onboarding variants (bare, one-liner, brief, skill) — lightest to heaviest — to find the minimum text that achieves 10/10 reliability. Ground truth uses broker events (agent_spawned/agent_released) not text parsing. Adds opencode free-model support via --harness=opencode:mimo-v2-flash-free. Adds npm scripts: eval:lifecycle, eval:lifecycle:free, eval:lifecycle:matrix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore: apply pr-reviewer fixes for #1109 * feat(evals): multi-model lifecycle evals, HTML report redesign, 4 new scenarios Scoring fix: remove parent===leadAgent filter from scoreSpawn and waitForEvent — the broker HTTP API never emits parent in agent_spawned events, causing all spawn scores to return FAIL. Any agent_spawned event is now trusted as ground truth (eval scenarios have exactly one lead). Model threading: ScenarioContext.model + BrokerHarness.spawnAgent model option propagate claude --model flags through the eval harness. Runner parses claude:haiku/sonnet/opus into full model IDs; opencode model set via OPENCODE_MODEL env var. New eval scripts: eval:lifecycle:claude-models (haiku/sonnet/opus) and eval:lifecycle:full (all models + opencode + codex). 4 new realistic messaging scenarios: t01-thread-reply, r05-check-inbox, r06-group-dm, r07-list-agents — bringing the total to 12 lifecycle + 12 messaging = 24 scenarios. HTML report redesigned: CI dashboard dark theme, per-model lifecycle variant breakdown table, spawn/release rate bars, transcripts collapsed by default, scenario groups separated. SKILL.md: fix all mcp__relaycast__ → mcp__agent-relay__ and hierarchical tool names to registered flat names (eval:toolcheck now passes). Onboarding variants: one-liner explicitly names mcp__agent-relay__ prefix; skill variant cleaned up (removed confusing dual-form syntax). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): s04 native-subagent detection scenario Adds s04-no-native-subagents (4 onboarding variants) — a lifecycle scenario that specifically detects whether Claude falls back to its built-in Task tool instead of mcp__agent-relay__add_agent when asked to spawn parallel workers. Ground truth: agent_spawned broker event = relay tool used (PASS). Detection: worker_stream contains "Task(" with no agent_spawned = native subagent confirmed (FAIL, notes distinguish native vs phantom vs no-spawn). New scoring/native-subagent.ts exposes detectNativeSubagent() which scans cleanStreamOutput for the Task( invocation pattern Claude Code emits. HTML report: FAIL cards for s04 show a red "NATIVE TASK" pill in the header and a "tool: NATIVE TASK (not add_agent)" stat entry when detected. ScenarioResult gains nativeSubagentDetected?: boolean used by both the report renderer and the scenario notes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(broker): model-aware relay skill injection for small-tier models Eval data showed haiku achieves 0/5 spawn reliability without onboarding guidance and 5/5 with the full skill variant. Sonnet/Opus pass bare (0-shot). When an agent is spawned via the HTTP API (add_agent MCP path) with a small-tier model (haiku, gpt-*-mini, gemini-*-flash), the broker now automatically prepends a concise relay skill block to the initial task: ## Agent Relay — Worker Management ### Spawn a worker mcp__agent-relay__add_agent(name, cli, task) ### Release a worker mcp__agent-relay__remove_agent(name) This happens in api.rs after workers.spawn() returns the effective spec (normalized model), so the prefix uses the resolved model ID. The prefix is appended before the task is stored in initial_tasks and delivered as the agent's first message. Large models (sonnet, opus, pro, gpt-4o) receive no prefix — they reliably call the relay tools from context alone and the extra text is unnecessary. 4 unit tests cover haiku/mini/flash (prefix), sonnet/opus/pro (no prefix), and None model (no prefix). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): comprehensive opencode model lifecycle eval scripts Add 5 new eval scripts covering the full opencode model registry: eval:lifecycle:opencode-native — opencode-specific models (mimo-v2-flash-free, minimax-m2.5-free, big-pickle, gpt-5-nano) eval:lifecycle:opencode-gpt4 — GPT-4 family via opencode (gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano) eval:lifecycle:opencode-gpt5 — GPT-5 family (gpt-5, gpt-5-mini, gpt-5-nano, gpt-5.2, gpt-5.4) eval:lifecycle:opencode-reasoning — o-series reasoning (o1-mini, o3-mini, o4-mini, o3, codex-mini-latest) eval:lifecycle:opencode-all — all 18 opencode models in one sweep Each runs 5 repeats × 16 lifecycle scenarios (s01–s04, 4 onboarding variants). Models are passed via OPENCODE_MODEL env var; report label uses the short model name (e.g. opencode:gpt-4o) via the existing .split('/').pop() path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): s05 phrasing variants, auto-routing spec, skill text fix, opus timeout fix - Add s05-phrasing-variants.ts: 6 scenarios testing relay vocabulary in task prompts (neutral-worker/agent, relay-worker/agent, arw-worker/agent) with bare onboarding to isolate the pure phrasing effect on spawn reliability - Wire s05 into runner as --group=phrasing; add eval:phrasing and eval:phrasing:claude-models scripts; add eval:phrasing:all-harnesses - Add dedicated lifecycle eval scripts for all installed harnesses: codex, gemini, grok, cursor, droid; add eval:lifecycle:all-harnesses - Fix onboarding.ts skill text: replace "do it yourself for quick lookups" heuristic with explicit rule that honours direct delegation instructions; this was causing sonnet s01:skill=0% and opus s03:skill=0% - Make s03 response window model-aware: opus gets 120s per phase (up from 60s) via responseMs(model) helper; overall scenario timeout bumped to 300s - Add specs/auto-routing.md: task classifier + team composer design grounded in lifecycle eval results (sonnet+one-liner=100% lead, haiku=worker-only, opus timeout-limited on s03) - Record lifecycle eval b3oqx02zv findings in trail trajectory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(auto): add auto-routing Phase 1–3 — classifier, composer, Director prompt - packages/cli/src/auto/classifier.ts: heuristic task classifier (no extra LLM call); returns complexity/parallelizable/domains/estimatedWorkers in <1ms - packages/cli/src/auto/composer.ts: routing table → TeamSpec; lead is always sonnet (one-liner) or opus (bare); haiku is worker-only per eval data - packages/cli/src/auto/director-prompt.ts: builds pre-composed Director meta-prompt so lead only coordinates — uses "relay worker" noun from s05 phrasing eval (early data shows relay-worker = 60% vs neutral-worker = 0% for haiku with bare onboarding) - packages/cli/src/auto/index.ts: barrel export - 13/13 unit tests passing for classifier and composer Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): add s06 auto-routing Director scenario and auto-routing eval group - s06-auto-routing.ts: validates Director multi-worker spawn using pre-composed meta-prompt; PASS requires ≥2 relay agent_spawned + no native Task tool usage - scenarios/index.ts, runner.ts: register s06 under --group=auto-routing - package.json: add eval:auto-routing and eval:auto-routing:claude-models scripts - Runner header updated to document auto-routing group Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update auto-routing spec with confirmed timeout fix and s05 early phrasing data - Opus s03 bare=67% (up from 40%) after responseMs() 120s/phase fix, confirmed - s05 phrasing: haiku relay-worker=60% vs neutral-worker=0%, worker noun outperforms agent noun across all vocabulary tiers for haiku with bare onboarding - arw-worker (agent-relay worker) matching relay-worker performance at 100% so far - Updated open questions table with confirmed answers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update auto-routing spec with skill text analysis and phrasing finding The task vocabulary is the bigger spawn-reliability lever, not onboarding text: - Skill text heuristic fix: sonnet s01:skill 0%→33% (partial improvement) - Root cause: task uses neutral 'worker agent' vocabulary; relay-worker phrasing yields 60% vs neutral-agent 20% for haiku with bare onboarding (s05 confirms) - Production fix already in place: Director meta-prompt uses 'relay worker' Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): strengthen skill text with explicit relay-tool disambiguation Add explicit note in the skill onboarding variant to prevent models from confusing 'assign to a worker agent' in the task with their built-in Task capability. Uses 'relay worker' vocabulary in section headings for consistency with s05 phrasing eval findings (relay-worker=60% vs neutral-agent=20%). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): add routing table rationale — conditional guidance hurts capable models opus s03 brief=0% vs bare=67%, one-liner=67% confirms that conditional spawn guidance ('Spawn when... dedicated focus') gives capable models permission to skip delegation. Routing table correctly uses bare/one-liner for leads only. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): revise brief onboarding to remove conditional spawn guidance 'Spawn when a task needs dedicated focus' gave capable models (opus) permission to skip delegation. Now uses directive language: 'When the task says to delegate or assign work, call add_agent.' Also uses relay-worker vocabulary for consistency with s05 phrasing findings. Validated: opus s03 brief was 0% with old conditional text. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update opus s03 results table with confirmed timeout fix findings - bare=67%, one-liner=67%, brief=0% (fixed), skill=67% with 120s/phase timeout fix - All variants at 67% except brief which used a conditional spawn clause (now fixed) - brief onboarding reverted to directive language in codebase Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(broker): fix grok mcp add — NAME-first ordering and embed flag-args in command grok v0.2.x has two parser quirks vs standard CLI conventions: 1. The positional <NAME> argument must come before any options (--env, --command, --args) 2. Flag-shaped --args values like `-y` are rejected as unknown options Fix both in `grok_mcp_add_args`: move AGENT_RELAY_MCP_SERVER to position [2] (immediately after "mcp add"), and embed flag-shaped args into the --command string ("npx -y") rather than passing them via --args. Also: wire Phase 4 auto-routing into local agent spawn command (--model auto triggers classifyTask → composeTeam → buildDirectorPrompt and spawns a Director with the right model tier), and update eval scripts to add all-harnesses phrasing script with current opencode free model names. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): add multi-harness eval findings to auto-routing spec - Add non-Claude lifecycle table: codex/gemini 100%, droid 80%+, grok/cursor 0% - Add phrasing matrix (early data): non-Claude vocabulary-agnostic, Claude vocab-dependent - Key insight: relay-anchored nouns matter only for Claude; codex/droid/gemini/opencode achieve high pass rates with neutral vocabulary - Update open-questions table with cross-harness status Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update multi-harness phrasing table with 5-run results Codex: 100% all 6 variants (one 80% outlier on relay-agent) OpenCode: 100% on 5/6 variants, 80% neutral-agent Droid: 80-100% across all tested variants Gemini: 60-100% across tested variants Grok/Cursor: 0% all — behavioral non-starters Claude haiku: 0-60% vocabulary-dependent Claude sonnet: 0-40% even with relay-anchored vocabulary Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): clarify grok failure mode post MCP fix + confirm opencode phrasing complete - Grok: MCP config errors gone (exit 2 fixed); failure is now behavioral (model ignores relay tools despite MCP being available) - OpenCode phrasing complete (29/30 = 97%): perfect across 5 variants, one neutral-agent miss; relay-native with no vocabulary dependency - Mark opencode arw-agent column as complete in phrasing table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): add s01/s02/s03 lifecycle breakdown tables for non-Claude harnesses Key findings: - Codex: 100% s01+s02 all onboardings; s03 bare=80%, one-liner=100% - OpenCode: s03 bare=100% (best full-lifecycle result); brief onboarding weak on s01/s02 - Gemini: perfect spawn (s01=100%) but release degrades without onboarding - Droid: good spawn but nearly 0% release (s02) without skill onboarding - Release issue for droid/gemini compensated by explicit Director prompt Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update phrasing table with droid/gemini/grok/cursor data - Droid: 80-100% across all variants, relay-worker=100% outperforms neutral - Gemini: relay-agent=25% outlier (relay+agent confused model); relay-worker=80% - Grok: 0% all variants confirmed - Cursor: 0% all variants confirmed - Add universal recommendation: use 'relay worker' noun — best across both Claude and non-Claude models; avoid 'relay agent' (hurts Gemini badly) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update s03 lifecycle table with codex/opencode confirmed results codex s03: bare=80%, one-liner=100% opencode s03: bare=100% (best), one-liner=80%, brief=100% (so far) Still running: codex s03 brief/skill, opencode s03 skill, gemini/droid s03 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): emphasize parallel spawn in s06 Director prompt Models were calling add_agent for first worker then processing ACK DMs before spawning the second worker, causing 0% multi-spawn pass rate. Adding explicit CRITICAL instruction to execute all spawn calls back-to-back without waiting between them, and to ignore ACK DMs until all workers are spawned. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update auto-routing spec with complete s03/s05 non-Claude results - s05 phrasing: all non-Claude harnesses complete (5x5 runs each) - codex: 80-100% on all vocabulary variants (relay-native) - opencode: 80-100% across all variants - droid: 80-100%; relay-worker=100%, arw-agent=100% - gemini: neutral-agent/arw-agent=100%; relay-agent=40% (relay+agent suffix confuses gemini) - s03 full lifecycle: codex brief/skill=100% (confirmed); droid bare=100% (surprising given s02 bare=20%; directive task phrasing drives release); gemini one-liner=100% - Eval coverage table updated: s05 complete, s03 non-Claude answers resolved - s06 status: 0% with original sequential prompt; re-running with parallel-spawn fix Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update s03 with droid/gemini confirmed results droid: bare/one-liner/brief all 100% (surprising given s02 bare=20%; directive task phrasing "report DONE when complete" drives release even without explicit onboarding) gemini: one-liner/brief/skill all 100%; bare=60% is the only gap. Implication: Director prompt's explicit remove_agent calls + directive task phrasing substitute for skill onboarding on all viable harnesses. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): inject fake worker ACK in s06 to unblock Director second spawn Root cause: models spawn first worker then wait indefinitely for an ACK that never comes (worker task prompt doesn't include relay ACK instruction). Director never spawns the second worker within phaseMs. Fix: after the first agent_spawned event, inject an ACK message from that worker to the Director. This unblocks the Director's wait state and triggers the second spawn call. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): extend s06 spawn window to 180s and simplify ACK text Extended spawn detection from phaseMs (60s) to 180s to accommodate sequential spawning: Director spawns first worker, waits for ACK, then spawns second worker within the extended window. Simplified injected ACK text — "report DONE when complete" in the previous text was confusing models into thinking the worker was done. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): change s06 injection to Orchestrator nudge after first spawn Previous approach (Worker ACK injection) didn't trigger the second spawn because Director processed the ACK passively. Changed to an Orchestrator message that explicitly instructs the Director to spawn Worker-Frontend. This reflects the finding that models require external stimulus to chain multiple add_agent calls — they don't spontaneously spawn the second worker after the first add_agent returns. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): document s06 multi-spawn finding and add to open questions Key finding: models make one add_agent call per response then stop/wait. No prompt instruction (CRITICAL warnings, numbered mandatory actions) reliably chains two consecutive add_agent calls in PTY mode. External Orchestrator nudge after first spawn is required. This is a Phase 5 concern but should inform Phase 3 Director prompt design — consider pre-wiring workers before Director starts, or using sequential hand-off patterns instead of parallel spawn. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): s06 v6 — Director executes from meta-prompt alone, no trigger message All previous approaches (CRITICAL instruction, ACK injection, Orchestrator nudge) failed because an incoming message triggers only a partial response (single add_agent call). Switching to production-matching approach: Director spawns workers autonomously from its meta-prompt without any trigger message. Extends wait window to 180s from STARTUP_MS offset to cover Director boot + sequential spawn pattern. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(specs): update s04 table with codex/gemini/droid results codex: 100% all variants — never routes to native subagents droid: 0% on bare/one-liner (uses native Task without disambiguation); skill text that fixes s04 for Claude breaks droid s03 (0%) gemini: 80% bare, 60% one-liner, 80%+ brief/skill — mostly relay-native Key insight: droid needs skill onboarding to avoid native subagents (s04), but the SAME skill text that helps s04 breaks s03 — needs investigation. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): s06 v7 — count startup spawns before clearEvents The Director processes its task immediately after boot. clearEvents() after STARTUP_MS was discarding agent_spawned events fired during startup. Now count spawns that happened during startup before clearing, then continue watching for more spawns with the 180s window. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(evals): complete s05 phrasing data for all 7 harnesses; fix s06 scoring; document PTY multi-spawn limitation - s05 phrasing eval complete across all harnesses: opus achieves 100% on relay-worker/relay-agent/arw-worker/arw-agent (vs haiku 60%/20%/60%/40%), confirming relay-anchored vocabulary is required for Claude with opus showing the strongest native tool knowledge - s06 scoring fix: remove clearEvents() before scoring so startup spawns are counted by scoreSpawn; use plain sleep() instead of waitForEvent (which resolves immediately on buffered events) - s06 architectural conclusion: PTY-mode agents make exactly one tool call per turn — 0% multi-worker spawn across all harnesses and 8+ prompt approaches. Production fix: pre-spawn workers from CLI layer in Phase 4, not via Director - Update specs/auto-routing.md with opus phrasing table, s06 final answer, and revised open questions - Update CHANGELOG with complete phrasing findings and s06 architectural decision Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(cursor): resolve cursor CLI to cursor-agent binary, not grok's agent binary The broker's command_parse was mapping cli="cursor" to "agent", but the `agent` binary on PATH is /Users/khaliqgant/.grok/bin/agent (the Grok CLI). This meant all cursor eval runs were actually running Grok, explaining the 0% scores. Fix: map cursor → cursor-agent explicitly in command_parse. Update harness command to match. Update unit tests accordingly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(evals): cursor-agent confirmed 0% — not viable relay worker Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): add codex model tier comparison results and fix parseHarnessSpec for codex models - Fix parseHarnessSpec: codex models (gpt-5.4-mini, o3, etc.) are raw OpenAI model names and must not be qualified with a "codex/" prefix — add early-return for cli=codex - Add codex tier comparison section to eval-master-summary.html: gpt-5.5 recommended (16/16, 0% phantom), gpt-5.4-mini viable budget (15/16, 31% phantom), gpt-5.4 avoid (52% phantom despite 100% majority-vote), spark not viable (6/16) - Update action item #8 in HTML from "pending" to completed findings with tier table Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): add opencode batch eval script and codex tier constants to composer - Add run-opencode-models.sh: two-phase deterministic batch eval over 39 opencode model tiers. Phase 1 screens with s03 repeat=3; Phase 2 runs s01-s04 repeat=5 for models that score ≥67% in Phase 1. No LLM coordinator — pure shell loop. - Add CODEX_MODEL_TIERS, WorkerCli type, and per-harness HARNESS_ONBOARDING defaults to composer.ts based on lifecycle eval findings (2026-06-12) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): harden opencode batch script — tolerate per-model failures, add missing claude variants - run_eval: add || true so one model failure doesn't abort the entire batch - Expand model list from 41 to 45: add claude-sonnet-4, claude-opus-4-1/4-5/4-6/4-7 to match actual opencode models output Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): scope opencode batch to alternative/Chinese models only Drop Claude and OpenAI variants — those are already covered by native CLI evals. Keep: DeepSeek, Kimi, Qwen, Minimax, GLM, MiMo, Grok-via-opencode, Gemini-via-opencode, Nemotron, North-mini, big-pickle (19 models total). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(auto): add role-fit classification layer to composer and auto-routing spec - Add AgentRole type (lead, coordinator, worker, planner, reviewer, critic, verifier, judge, mapper, reducer, supervisor, debater) matching the choosing-swarm-patterns skill's role taxonomy - Add HarnessRoleMap + HARNESS_ROLE_MAP: eval-backed table mapping each harness to the roles it can fill (confirmed/provisional/not-viable/untested) - Add harnessesForRole() helper for pattern-aware slot selection - Expand WorkerSpec to carry cli, codexModel, opencodeModel - Add §5 Role-Fit Classification to specs/auto-routing.md: role definitions, eval signal → role mapping, provisional role-fit table, pattern → role → harness assignment table, and s07-s11 scenario roadmap for full coverage opencode Chinese/alternative model rows marked 'untested'; will update from Phase 1+2 batch eval results. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): add Phase 1 opencode batch results to role-fit map 19 alternative/Chinese models evaluated (s01-s04, repeat=3). 17/19 advance. Top tier (16/16, 0-2 phantoms): deepseek-v4-flash, deepseek-v4-flash-free, qwen3.6-plus, qwen3.5-plus, minimax-m2.5, minimax-m2.7, glm-5.1, big-pickle Confirmed (16/16, 3-9 phantoms): glm-5, gemini-3.1-pro, grok-build-0.1, gemini-3-flash Provisional (12-15/16): kimi-k2.5, kimi-k2.6, mimo-v2.5-free, gemini-3.5-flash, north-mini-code-free Eliminated: deepseek-v4-pro (11/16), nemotron-3-ultra-free (10/16) Key findings: - opencode normalizes flaky native CLIs: gemini bare 60% native → 16/16 via opencode; grok 0/48 native → 16/16 via opencode (model capable, native CLI MCP was broken) - All top-tier Chinese models are relay-native across all 4 onboarding variants - HARNESS_ROLE_MAP updated with per-model entries for all 19 tested models Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): add role-fit rankings and communicator sections to master summary Adds §7 "Role Rankings — Top 5 per Role" to eval-master-summary.html: - Lead/Coordinator: claude:sonnet > opus > codex:gpt-5.5 > opencode:deepseek-v4-flash > opencode:qwen3.6-plus - Worker: codex:gpt-5.5 > opencode:deepseek-v4-flash > minimax-m2.5 > qwen3.5/3.6-plus - Reviewer/Critic: claude:opus > sonnet > codex:gpt-5.5 > opencode:gemini-3.1-pro - Mapper/Reducer: codex:gpt-5.5 > opencode:deepseek-v4-flash > minimax-m2.5 > qwen3.6 > glm-5.1 - Judge: claude:opus > sonnet > codex:gpt-5.5 - Communicator: claude:opus > sonnet > codex:gpt-5.5 > opencode:gemini-3.1-pro > opencode:deepseek-v4-flash - Best lead callout: claude:sonnet + one-liner is the clear #1 across all criteria Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(evals): add @agent-relay/evals package with publish pipeline Creates packages/evals/ as the shared eval harness package: - package.json with subpath exports for types, runner, harness, scoring, and scenarios - tsconfig.json matching the monorepo TypeScript config - src/types.ts (canonical copy of EvalScenario/ScenarioResult/etc. types) - src/harness.ts (BrokerHarness interface for downstream type-checking) - src/index.ts (barrel re-export) Adds build:evals to root build:core chain (after harness-driver, before cli). Updates publish.yml: converts publish-harnesses to a matrix job covering both harnesses and evals — same dependency pattern (needs publish-packages so harness-driver is on the registry before evals can be installed). Also adds specs/agent-relay-evals-package.md scoping the full migration. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: auto-format Rust code with cargo fmt * feat(evals): add s07-lead-delegation scenario group Three new eval scenarios that test lead role discipline — the gap that existing s03/s04 don't cover (they only check spawn mechanics, not whether the lead actually delegates vs. self-implements): l01 — Unconditional delegation: coding task with no "don't do it yourself" hint. Good lead spawns; bad lead writes code itself. l02 — Temptation resistance: task explicitly says "you'd be faster doing it yourself". Lead must still delegate. l03 — Post-delegation synthesis: after workers report DONE, lead must send a synthesis message referencing their results. New scoring fields on ScenarioResult: - selfImplemented: lead's PTY stream contained code blocks / impl tool calls - synthesisOk: lead sent a synthesis message after receiving DONE All three run across bare/one-liner/brief/skill onboarding variants (12 total). Runner: --group=lead-delegation, npm scripts eval:lead-delegation, eval:lead-delegation:claude-models, eval:lead-delegation:all-harnesses. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(mcp): pass model via metadata in add_agent spawn call SpawnAgentRequest has no top-level model field; model must be passed through metadata so the broker can forward --model to the launched CLI. Also auto-format prettier fixes in auto/ and local-agent.ts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): bump @agent-relay/evals to 8.6.0 and regenerate lock file Package was pinned at 8.3.1 causing npm ci to fail — lock file was missing @agent-relay/evals and several stale version entries. Updated to match monorepo version 8.6.0 and regenerated package-lock.json. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: auto-format all files with prettier; add broker gitignore for eval artifacts Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): use rk_live_ substring check instead of glob in workspaces eval contentIncludes does literal substring matching; rk*live* was treated as the literal string, not a glob. The mock key rk_live_ws_eval_create matches rk_live_ as a prefix check. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: auto-format with Prettier * fix(ci): resolve three CI failures on PR #1109 1. composer.test.ts: check for role === 'reducer' not 'Synthesiser'; the AgentRole type uses 'reducer' for aggregator workers. 2. packages/cli/package.json: bump all @agent-relay/* workspace deps from 8.2.0 → 8.7.2 so npm ci resolves them to the local workspace symlinks instead of installing stale registry copies. This was shadowing the local @agent-relay/cloud build and hiding cloudWorkerStateDir. 3. package-lock.json: regenerated to remove the nested packages/cli/node_modules/@agent-relay/{cloud,config,sdk,...}@8.2.0 entries that were installed from npm. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(evals): re-apply rk_live_ substring fix after prettier bot revert The prettier bot ran on the pre-fix commit and its auto-format push overwrote the rk*live* → rk_live_ fix. Reapplied: contentIncludes does literal substring matching, not glob, so rk_live_ is correct. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: auto-format with Prettier * fix(evals): use exact mock key in workspace contentIncludes check Replace the glob-style rk*live* (reverted by prettier bot) with the full exact key rk_live_ws_eval_create that the mock actually returns. This is a precise substring check, passes prettier with no diff, and won't be overwritten by the prettier auto-format workflow. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lock): remove stale broker@8.2.0 entries from packages/cli/node_modules When CLI's @agent-relay/* deps were bumped from 8.2.0 → 8.7.2, the old @agent-relay/broker-*@8.2.0 optional entries left by the previous npm install were not cleaned by --package-lock-only. These shadowed the root workspace broker binaries (8.7.2), causing 'unexpected argument --instance-name' from the old CLI init interface. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Proactive Runtime Bot <agent@agent-relay.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: agent-relay-code[bot] <agent-relay-code[bot]@users.noreply.github.com> Co-authored-by: Will Washburn <will.washburn@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
willwashburn
added a commit
that referenced
this pull request
Jun 21, 2026
…cal dir A trajectory (`traj_o61z0ze6kvla`, "Review and fix PR #1092") was written to a stray root `.trajectories/` instead of the canonical `.agentworkforce/relay` data dir the tool now defaults to, so it never showed up in `trail list`. Move it into `.agentworkforce/trajectories/completed/2026-06/` and drop the empty root `.trajectories/`. Default `trail list` now counts it (257 -> 258); `trail doctor` is clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
willwashburn
added a commit
that referenced
this pull request
Jun 21, 2026
* ci(publish): drop the removed @agent-relay/telemetry package #1181 deleted the `packages/telemetry` placeholder, but the Publish Package workflow still listed `telemetry` in two publish matrices, so the spawned job failed at `cd packages/telemetry` ("No such file or directory"). Remove `telemetry` from the publish-all and publish-main-runtime-deps matrices. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(trajectories): consolidate stray root .trajectories into canonical dir A trajectory (`traj_o61z0ze6kvla`, "Review and fix PR #1092") was written to a stray root `.trajectories/` instead of the canonical `.agentworkforce/relay` data dir the tool now defaults to, so it never showed up in `trail list`. Move it into `.agentworkforce/trajectories/completed/2026-06/` and drop the empty root `.trajectories/`. Default `trail list` now counts it (257 -> 258); `trail doctor` is clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/andscripts/evals/.cases.jsonlartifacts.Validation
npm run evals:offlineNotes