Skip to content

Add Relaycast attribution and Agent Relay MCP tool events#1041

Merged
willwashburn merged 6 commits into
mainfrom
feat/telemetry-relaycast-mcp-actions
Jun 3, 2026
Merged

Add Relaycast attribution and Agent Relay MCP tool events#1041
willwashburn merged 6 commits into
mainfrom
feat/telemetry-relaycast-mcp-actions

Conversation

@willwashburn

@willwashburn willwashburn commented Jun 3, 2026

Copy link
Copy Markdown
Member

Summary

  • Pass Relaycast harness and Agent Relay distinct ID through CLI MCP and SDK Relaycast clients, preserving UA-like harness values such as claude/opus-48.
  • Add cli_install, cli_update, and agent_relay_tool_call telemetry; tool-call payloads use tool_type / tool_category for spawn, release, action, workspace, message, inbox, and related flows.
  • Standardize MCP/tool-call references on Agent Relay MCP naming: agent-relay, mcp__agent-relay__..., and mcp_agent_relay_...; legacy relaycast server keys are only consumed for config migration/compatibility.
  • Cover install/update telemetry, Agent Relay MCP tool-call tracking, and Relaycast attribution in focused tests.

Verification

  • npx vitest run packages/cli/src/cli/agent-relay-mcp.startup.test.ts packages/cli/src/cli/telemetry/client.test.ts packages/cli/src/cli/commands/core.test.ts
  • npm --prefix packages/cli run build
  • cargo test -p agent-relay-broker snippets
  • cargo test -p agent-relay-broker injection_format
  • cargo test -p agent-relay-broker cli_mcp_args
  • cargo test -p agent-relay-broker pty_worker
  • uv run --with pytest --with pytest-asyncio --with aiohttp --with pyyaml pytest packages/sdk-py/tests/communicate/adapters/test_claude_sdk.py

@willwashburn willwashburn requested a review from khaliqgant as a code owner June 3, 2026 16:47
@codeant-ai

codeant-ai Bot commented Jun 3, 2026

Copy link
Copy Markdown

CodeAnt AI is reviewing your PR.

@coderabbitai

coderabbitai Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Adds telemetry event types and CLI telemetry client behavior, derives Relaycast telemetry options from explicit values or environment, threads telemetry into SDK and CLI RelayCast/WsClient construction, instruments MCP tool calls with telemetry, and updates agent-relay naming/config across broker, runtime, plugins, tests, and docs.

Changes

Telemetry Instrumentation for CLI and SDK

Layer / File(s) Summary
Telemetry event schema and type contracts
packages/cli/src/cli/telemetry/events.ts, packages/cli/src/cli/telemetry/index.ts
Adds CliInstallEvent, CliUpdateEvent, AgentRelayToolCallEvent and updates TelemetryEventName/TelemetryEventMap and exports.
Relaycast telemetry option builders (CLI)
packages/cli/src/cli/lib/relaycast-telemetry.ts
Derives harness and agentRelayDistinctId from explicit options or ordered env keys; exposes workspace helper and merger.
Relaycast telemetry option builders (SDK)
packages/sdk/src/relaycast-telemetry.ts
Env-safe defaultEnv(), nonEmpty normalization, relaycastTelemetryOptions, relaycastWorkspaceTelemetryOptions, and withRelaycastTelemetry for SDK callers.
CLI telemetry client first-run initialization
packages/cli/src/cli/telemetry/client.ts, packages/cli/src/cli/telemetry/client.test.ts
Tracks first-run notice, emits one-time cli_install with version fallback, clears flag when telemetry disabled; tests verify single emission.
CLI update command telemetry tracking
packages/cli/src/cli/lib/core-maintenance.ts, packages/cli/src/cli/commands/core.test.ts
Computes toVersion, emits cli_update on install result with from_version/to_version/success and optional error_class; tests assert correct/no telemetry in --check, success, and safe failure cases.
MCP tool invocation telemetry and relay client integration
packages/cli/src/cli/agent-relay-mcp.ts, packages/cli/src/cli/agent-relay-mcp.startup.test.ts
Adds MCP action-type mapping, tracks agent_relay_tool_call events (tool_name, tool_type, tool_category, transport, success, duration, error_class), integrates telemetry into RelayCast/WsClient construction/inbox piggyback, and adds tests for propagation and tracking.
SDK workspace and messaging telemetry propagation
packages/sdk/src/agent-relay.ts, packages/sdk/src/messaging/relaycast.ts, packages/sdk/src/__tests__/agent-relay.test.ts
Extends AgentRelayCreateWorkspaceInput and RelaycastMessagingOptions to accept telemetry options, computes telemetry for workspace creation, forwards to RelayCast; tests cover explicit, workspace, and env-derived telemetry and process-global safety.
CLI bootstrap telemetry shutdown behavior
packages/cli/src/cli/bootstrap.ts
Detects stdio server invocations and conditionally skips telemetry shutdown during stdio-server runs; entrypoint error path now awaits telemetry shutdown when applicable.
Agent Relay broker/config migration and wiring
crates/broker/src/snippets.rs, crates/broker/src/cli_mcp_args.rs, crates/broker/src/relaycast/*, crates/broker/src/pty_worker.rs
Renames and reworks relaycast MCP helpers to agent-relay, introduces command override env, handles legacy relaycast migration, updates CLI injection builders and tests, and adjusts PTY startup markers and readiness gating.
Tests, plugins, docs, scripts updates
plugins/*, tests/*, web/*, packages/sdk-py/*, CHANGELOG.md
Replaces relaycast naming with agent-relay across prompts, examples, plugin configs, setup scripts, tests, and changelog entries.
Trajectory metadata
.agentworkforce/trajectories/active/traj_b1jrutolckfb/trajectory.json
Adds a decision event object to the trajectory's chapter events array (non-code metadata change).

Sequence Diagram(s)

sequenceDiagram
  participant Client as MCP Client
  participant Wrapper as InboxPiggybackWrapper
  participant Relay as RelayCast/WsClient
  participant Telemetry as Telemetry.track
  Client->>Wrapper: invoke tool (tool_name, args)
  Wrapper->>Wrapper: record startedAt, derive tool_type/category
  Wrapper->>Relay: execute tool call
  alt Success
    Relay-->>Wrapper: result
    Wrapper->>Telemetry: agent_relay_tool_call(success: true, duration_ms, tool_name, tool_type, tool_category, transport)
  else Failure
    Relay-->>Wrapper: throws / error result
    Wrapper->>Telemetry: agent_relay_tool_call(success: false, error_class, duration_ms, tool_name, tool_type, tool_category, transport)
  end
  Wrapper-->>Client: return result or recovery payload
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

Suggested reviewers

  • khaliqgant

🐰 I hopped through code with a tiny cheer,
First-run bells rung once, then clear,
Tool calls counted, durations told,
Errors named but secrets hold,
Relay stitched with telemetry gold.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/telemetry-relaycast-mcp-actions

@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@codeant-ai codeant-ai Bot added the size:XL This PR changes 500-999 lines, ignoring generated files label Jun 3, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45f94321c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/sdk/src/relaycast-telemetry.ts Outdated

export function relaycastTelemetryOptions(
explicit: RelaycastTelemetryOptions = {},
env: NodeJS.ProcessEnv = process.env

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid requiring process.env in SDK clients

In browser or edge runtimes that do not polyfill Node's global process, constructing an AgentRelay with a workspace key now calls relaycastTelemetryOptions() through createRelaycastClient, which evaluates this default parameter and throws before the RelayCast client is created. The SDK is documented for browser-worker delivery and other SDK code avoids static Node globals for this reason, so this should use a safe globalThis.process?.env fallback or only read env when process exists.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Fixed in 06743bf by removing the SDK helper default parameter that evaluated process.env. The SDK now reads env through a safe globalThis.process?.env fallback, and the regression test covers constructing AgentRelay when process is absent.

Comment thread packages/cli/src/cli/agent-relay-mcp.ts Outdated
Comment on lines +1008 to +1015
track('mcp_action_call', {
tool_name: input.toolName,
action_type: input.actionType,
transport: input.transport ?? 'unknown',
success: input.success,
duration_ms: Date.now() - input.startedAt,
...(input.errorClass ? { error_class: input.errorClass } : {}),
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Architect Review — HIGH

mcp_action_call is instrumented in the MCP stdio server, but this process never initializes the telemetry client in normal usage (agent-relay mcp or the agent-relay-mcp.js entrypoint), so all track() calls from this file are no-ops and the events are dropped at runtime.

Suggestion: Initialize telemetry in the MCP entrypoint/startup path (or a shared MCP bootstrap) before creating the server, using initTelemetry with an appropriate app/surface and showNotice: false, and ensure shutdown() is called on process exit so mcp_action_call events can flush.

Fix in Cursor | Fix in VSCode Claude

(Use Cmd/Ctrl + Click for best experience)

Prompt for AI Agent 🤖
This is an **Architect / Logical Review** comment left during a code review. These reviews are first-class, important findings — not optional suggestions. Do NOT dismiss this as a 'big architectural change' just because the title says architect review; most of these can be resolved with a small, localized fix once the intent is understood.

**Path:** packages/cli/src/cli/agent-relay-mcp.ts
**Line:** 1008:1015
**Comment:**
	*HIGH: `mcp_action_call` is instrumented in the MCP stdio server, but this process never initializes the telemetry client in normal usage (`agent-relay mcp` or the `agent-relay-mcp.js` entrypoint), so all `track()` calls from this file are no-ops and the events are dropped at runtime.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
If a suggested approach is provided above, use it as the authoritative instruction. If no explicit code suggestion is given, you MUST still draft and apply your own minimal, localized fix — do not punt back with 'no suggestion provided, review manually'. Keep the change as small as possible: add a guard clause, gate on a loading state, reorder an await, wrap in a conditional, etc. Do not refactor surrounding code or expand scope beyond the finding.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid duplicate of the MCP init issue. Fixed in 06743bf by initializing telemetry in the MCP stdio startup path and avoiding immediate shutdown from the parent CLI mcp command flow.

Comment thread packages/cli/src/cli/agent-relay-mcp.ts Outdated
Comment on lines +1008 to +1015
track('mcp_action_call', {
tool_name: input.toolName,
action_type: input.actionType,
transport: input.transport ?? 'unknown',
success: input.success,
duration_ms: Date.now() - input.startedAt,
...(input.errorClass ? { error_class: input.errorClass } : {}),
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: This new MCP telemetry path assumes the telemetry client is initialized, but the mcp command flow explicitly skips initTelemetry in CLI bootstrap, so this track call is a no-op at runtime and mcp_action_call events are never emitted. Initialize telemetry for MCP server runs (for example with notice disabled) before tool handling starts, or explicitly wire a telemetry transport that does not depend on the global CLI init path. [api mismatch]

Severity Level: Major ⚠️
- ⚠️ MCP stdio server emits no mcp_action_call telemetry events.
- ⚠️ Dashboards cannot track MCP tool usage or error rates.
- ⚠️ MCP action grouping by action_type is never populated.
- ⚠️ Production MCP issues lack per-call telemetry traces.
Steps of Reproduction ✅
1. Invoke the CLI MCP server via the normal entrypoint, e.g. run `agent-relay mcp` which
executes `runCli()` from `packages/cli/src/cli/index.ts:7-22` and dispatches to the `mcp`
command defined in `packages/cli/src/cli/bootstrap.ts:48-54`.

2. In `runCli()` (`packages/cli/src/cli/bootstrap.ts:89-102`), observe that
`shouldSkipTelemetryInit(argv)` (`bootstrap.ts:65-69`) returns `true` for the `mcp`
command because `STDIO_SERVER_COMMANDS` is defined as `new Set(['mcp'])` at
`bootstrap.ts:147-148`, so the guarded `initTelemetry({ ... })` call at
`bootstrap.ts:93-101` is never executed for MCP runs.

3. The `mcp` command action in `createProgram()` (`bootstrap.ts:48-54`) imports
`./agent-relay-mcp.js` and calls `startAgentRelayMcpStdio(mod.optionsFromEnv())`, which in
turn creates the MCP server (`createAgentRelayMcpServer`) and wraps tool handlers with
`enableInboxPiggyback` in `packages/cli/src/cli/agent-relay-mcp.ts:83-66 (wrapper starting
at 95-105)`.

4. When an MCP client invokes any registered tool (for example `invoke_action` or
`add_agent`), the wrapped handler in `enableInboxPiggyback` executes and calls
`trackMcpActionCall` at `agent-relay-mcp.ts:113-120, 18-25, 32-40`, which internally calls
`track('mcp_action_call', { ... })` at `agent-relay-mcp.ts:65-80`; however, since
`initTelemetry()` was never called, the telemetry client `client/commonProps/distinctId`
remain `null` (see `packages/cli/src/cli/telemetry/client.ts:22-27` and `initTelemetry` at
`client.ts:148-187`), so `track()` immediately returns without emitting anything
(`client.ts:189-203`), meaning `mcp_action_call` events are never actually sent for any
MCP tool execution.

Fix in Cursor | Fix in VSCode Claude

(Use Cmd/Ctrl + Click for best experience)

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** packages/cli/src/cli/agent-relay-mcp.ts
**Line:** 1008:1015
**Comment:**
	*Api Mismatch: This new MCP telemetry path assumes the telemetry client is initialized, but the `mcp` command flow explicitly skips `initTelemetry` in CLI bootstrap, so this `track` call is a no-op at runtime and `mcp_action_call` events are never emitted. Initialize telemetry for MCP server runs (for example with notice disabled) before tool handling starts, or explicitly wire a telemetry transport that does not depend on the global CLI init path.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Fixed in 06743bf: startAgentRelayMcpStdio now initializes telemetry with showNotice: false and surface: mcp, installs a beforeExit flush hook for the stdio entrypoint, and the CLI success path no longer immediately shuts telemetry down for stdio server commands.

@codeant-ai

codeant-ai Bot commented Jun 3, 2026

Copy link
Copy Markdown

CodeAnt AI finished reviewing your PR.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/cli/src/cli/agent-relay-mcp.startup.test.ts (1)

401-451: ⚡ Quick win

Add one assertion for the stdio bootstrap RelayCast path.

These cases cover websocket, direct client, and create_workspace attribution, but the new resolveStdioBootstrapOptions() constructor wrapping in packages/cli/src/cli/agent-relay-mcp.ts:1997-2002 is still untested. One small assertion there would keep harness/distinct-id propagation from regressing on the bootstrap path too.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/cli/src/cli/agent-relay-mcp.startup.test.ts` around lines 401 - 451,
Add one assertion that verifies the stdio bootstrap RelayCast path receives the
Relaycast telemetry context: after calling createAgentRelayMcpServer(...) with
telemetryTransport: 'stdio', assert that the options produced by
resolveStdioBootstrapOptions (the stdio bootstrap constructor path) include
harness and agentRelayDistinctId propagation (e.g., inspect the stdio/bootstrap
client config in the test mocks similar to how websocket/direct clients are
checked). Reference resolveStdioBootstrapOptions and createAgentRelayMcpServer
and assert the stdio/bootstrap mock's config contains harness: 'claude/opus-48'
and agentRelayDistinctId: 'distinct_test'.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/cli/src/cli/agent-relay-mcp.ts`:
- Around line 1086-1094: The tracking call using trackMcpActionCall is emitted
too early (before the opportunistic inbox() piggyback completes), so move the
success-path tracking to just before the final return result to ensure
duration_ms covers the full wrapped handler; specifically, after you compute
result and await any inbox() fetch, call trackMcpActionCall({ toolName: name,
actionType, transport: telemetryTransport, startedAt, success:
!isErrorToolResult(result), ...(isErrorToolResult(result) ? { errorClass:
'ToolResultError' } : {}) }) so duration_ms is measured correctly; make the same
adjustment for the other identical tracking invocation near the other tool path
that currently mirrors this block.

---

Nitpick comments:
In `@packages/cli/src/cli/agent-relay-mcp.startup.test.ts`:
- Around line 401-451: Add one assertion that verifies the stdio bootstrap
RelayCast path receives the Relaycast telemetry context: after calling
createAgentRelayMcpServer(...) with telemetryTransport: 'stdio', assert that the
options produced by resolveStdioBootstrapOptions (the stdio bootstrap
constructor path) include harness and agentRelayDistinctId propagation (e.g.,
inspect the stdio/bootstrap client config in the test mocks similar to how
websocket/direct clients are checked). Reference resolveStdioBootstrapOptions
and createAgentRelayMcpServer and assert the stdio/bootstrap mock's config
contains harness: 'claude/opus-48' and agentRelayDistinctId: 'distinct_test'.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 87e95e2d-af92-4a35-a587-b920d05d5988

📥 Commits

Reviewing files that changed from the base of the PR and between 0a65127 and c3a74a8.

📒 Files selected for processing (14)
  • .agentworkforce/trajectories/active/traj_b1jrutolckfb/trajectory.json
  • packages/cli/src/cli/agent-relay-mcp.startup.test.ts
  • packages/cli/src/cli/agent-relay-mcp.ts
  • packages/cli/src/cli/commands/core.test.ts
  • packages/cli/src/cli/lib/core-maintenance.ts
  • packages/cli/src/cli/lib/relaycast-telemetry.ts
  • packages/cli/src/cli/telemetry/client.test.ts
  • packages/cli/src/cli/telemetry/client.ts
  • packages/cli/src/cli/telemetry/events.ts
  • packages/cli/src/cli/telemetry/index.ts
  • packages/sdk/src/__tests__/agent-relay.test.ts
  • packages/sdk/src/agent-relay.ts
  • packages/sdk/src/messaging/relaycast.ts
  • packages/sdk/src/relaycast-telemetry.ts

Comment thread packages/cli/src/cli/agent-relay-mcp.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 14 files

Re-trigger cubic

@willwashburn

willwashburn commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

Handled the remaining review/user feedback across 06743bf9, 205a7ccc, e7ce801d, and 157a721e:

  • Added stdio bootstrap Relaycast attribution coverage for resolveStdioBootstrapOptions().
  • Initialized telemetry in MCP stdio startup and kept the parent agent-relay mcp command from immediately shutting it down.
  • Moved success-path tool-call timing after inbox piggyback so duration_ms matches caller-visible latency.
  • Made SDK Relaycast telemetry env lookup safe when process is absent.
  • Renamed MCP tool-call telemetry to agent_relay_tool_call; payloads now use tool_name, tool_type, tool_category, transport, success, duration_ms, and optional error_class.
  • Agent Relay MCP prompts/config/tests now use agent-relay, mcp__agent-relay__..., and mcp_agent_relay_...; generated Codex/OpenCode/Cursor configs use agent-relay, with old config keys kept only where needed for migration/cleanup.
  • invoke_action reads the requested Agent Relay action/tool name for telemetry while still avoiding action input values.

Validated with focused CLI/SDK tests, broker MCP/snippets/PTY filters, and package builds.

@codeant-ai

codeant-ai Bot commented Jun 3, 2026

Copy link
Copy Markdown

CodeAnt AI is running Incremental review

@willwashburn

willwashburn commented Jun 3, 2026

Copy link
Copy Markdown
Member Author

Follow-up on Agent Relay tool kind naming in e7ce801d:

  • Telemetry now emits agent_relay_tool_call, not mcp_action_call.
  • The payload uses tool_name, tool_type, and tool_category; tool_type carries the Agent Relay action/tool name such as agent.create, agent.release, or github.open_pr.
  • tool_category provides coarse rollups (spawn, release, action, etc.) so dashboards can track lifecycle buckets without losing the durable tool/action name.
  • invoke_action reads the requested action name for telemetry while still avoiding action input values.
  • Agent Relay MCP prompts/config/tests now use agent-relay / mcp__agent-relay__... / mcp_agent_relay_... naming.

Validated with focused CLI telemetry tests, broker MCP/snippets/PTY filters, the Python adapter test, and npm --prefix packages/cli run build.

@codeant-ai codeant-ai Bot added size:XL This PR changes 500-999 lines, ignoring generated files and removed size:XL This PR changes 500-999 lines, ignoring generated files labels Jun 3, 2026
@codeant-ai

codeant-ai Bot commented Jun 3, 2026

Copy link
Copy Markdown

CodeAnt AI Incremental review completed.

@codeant-ai

codeant-ai Bot commented Jun 3, 2026

Copy link
Copy Markdown

CodeAnt AI is running Incremental review

@willwashburn willwashburn changed the title Add Relaycast telemetry attribution and MCP action events Add Relaycast attribution and Agent Relay MCP tool events Jun 3, 2026
@codeant-ai codeant-ai Bot added size:XXL This PR changes 1000+ lines, ignoring generated files and removed size:XL This PR changes 500-999 lines, ignoring generated files labels Jun 3, 2026
@codeant-ai

codeant-ai Bot commented Jun 3, 2026

Copy link
Copy Markdown

CodeAnt AI Incremental review completed.

@github-actions

github-actions Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Preview deployed!

Environment URL
Web https://d31xlb3iiqwhi.cloudfront.net

This preview will be cleaned up when the PR is merged or closed.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 51 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="plugins/codex-relay-skill/scripts/setup.sh">

<violation number="1" location="plugins/codex-relay-skill/scripts/setup.sh:175">
P2: Installer dropped legacy `relaycast` key migration; existing configs get a second MCP block instead of reusing prior values. This breaks compatibility for upgraded users with customized relaycast settings.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

}
{
if (!in_block && $0 ~ /^[[:space:]]*mcp_servers[.]relaycast[.]command[[:space:]]*=/) {
if (!in_block && $0 ~ /^[[:space:]]*mcp_servers[.]agent-relay[.]command[[:space:]]*=/) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Installer dropped legacy relaycast key migration; existing configs get a second MCP block instead of reusing prior values. This breaks compatibility for upgraded users with customized relaycast settings.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At plugins/codex-relay-skill/scripts/setup.sh, line 175:

<comment>Installer dropped legacy `relaycast` key migration; existing configs get a second MCP block instead of reusing prior values. This breaks compatibility for upgraded users with customized relaycast settings.</comment>

<file context>
@@ -172,30 +172,30 @@ ensure_relaycast_mcp_block() {
     }
     {
-      if (!in_block && $0 ~ /^[[:space:]]*mcp_servers[.]relaycast[.]command[[:space:]]*=/) {
+      if (!in_block && $0 ~ /^[[:space:]]*mcp_servers[.]agent-relay[.]command[[:space:]]*=/) {
         block_seen = 1
         dotted_seen = 1
</file context>

@willwashburn willwashburn merged commit 1cb41cf into main Jun 3, 2026
47 of 48 checks passed
@willwashburn willwashburn deleted the feat/telemetry-relaycast-mcp-actions branch June 3, 2026 19:15
khaliqgant pushed a commit that referenced this pull request Jun 16, 2026
My earlier fix only re-instrumented add_agent/remove_agent inline, which
satisfied the failing test but silently dropped agent_relay_tool_call
telemetry for the ~28 other owned tools that PR #1041 deliberately
instrumented (post_message, send_dm, channels, reactions, inbox, etc.).

Restore the original wrapper-based tracking in enableInboxPiggyback so
every statically-registered owned tool emits telemetry again, and add a
skip for the action-routed path (invoke_action + the dynamic per-action
tools in actionToolNames) to honor the fleet SDK commit's intent that
registered actions do not route through per-tool telemetry. Reverts the
inline handler boilerplate and the registerAgentRelayTools param.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
khaliqgant pushed a commit that referenced this pull request Jun 16, 2026
…metry entries

Fresh-eyes review follow-ups:
- gitignore /opencode.json: the opencode harness writes it to the repo
  root during eval runs with a live rk_live_ workspace key + at_live_
  agent token; it was untracked but not ignored, one `git add .` from a
  secret leak.
- Drop 3 stale AGENT_RELAY_TOOL_CALL_METADATA entries (get_message,
  get_thread, list_channel_members) — no such tools are registered, so
  the entries were dead lookups carried over from PR #1041.
- Fix run-task-exit-evals.sh header: the s08 scenario self-releases via
  remove_agent(self), not `/exit` output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
khaliqgant added a commit that referenced this pull request Jun 16, 2026
* feat(spawn): add task-exit mode

* chore: apply pr-reviewer fixes for #1129

* style: auto-format with Prettier

* chore: apply pr-reviewer fixes for #1129

* style: auto-format with Prettier

* feat(evals): add s08-task-exit scenario and multi-model runner script

Adds eval coverage for the task_exit spawn mode introduced in #1129:
- s08-task-exit.ts: directly spawns a worker with EXIT_AFTER_TASK_INSTRUCTION
  and checks for a clean agent_exited (code 0) event — no lead required
- ScenarioResult gains exitConfirmed and exitCode optional fields
- Runner gains task-exit group (--group=task-exit) and imports TASK_EXIT_EVAL_SCENARIOS
- run-task-exit-evals.sh drives the full matrix: claude (haiku/sonnet/opus),
  codex (gpt-4.5/gpt-4.1/o3/o4-mini), and opencode in batches of 3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: auto-format with Prettier

* fix(evals): rewrite s08 to test remove_agent(self) rather than /exit output

The original approach relied on the model outputting `/exit` as bare text,
which the broker PTY watches for but Claude models never emit reliably.
The correct task-exit pattern is for the worker to call
mcp__agent-relay__remove_agent with its own name, which triggers the broker
to kill the PTY via workers.release().

Changes:
- s08-task-exit: spawn worker with explicit name in task, wait for
  agent_released event, score pass = message sent + self-released
- broker-harness: thread exitAfterTask through spawnAgent options
- fleet.rs: add missing exit_after_task: false to ListenApiRequest::Spawn

Verified results (1 run each):
  claude:haiku   PASS  claude:sonnet  PASS  claude:opus  PASS
  codex          PASS  opencode       FAIL (sent=0, no self-release)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(evals): s08 — use remove_agent(self) for all harnesses including opencode

/exit output was evaluated for opencode but found unreliable: the model sends
the task message but does not produce a bare /exit line the PTY watcher can
detect. remove_agent(self) works across all harnesses.

Also fixes opencode sent=0 regression: use relay message trigger after startup
(matching all other scenario patterns) so the action fires after MCP is up.

Verified results:
  claude:haiku PASS  opencode PASS  (all previous Claude/codex results unchanged)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(cli): restore add_agent/remove_agent telemetry lost in fleet SDK merge

The fleet SDK commit (491d9a0) reorganized agent-relay-mcp.ts and
inadvertently removed the agent_relay_tool_call telemetry tracking that
was added in 1cb41cf. The fleet commit also changed the invoke_action
test expectation to NOT fire telemetry (actions API has its own path),
so the blanket enableInboxPiggyback wrapper approach no longer fits.

Restores tracking by adding inline trackAgentRelayToolCall calls directly
to the add_agent and remove_agent handlers, threaded via a new
telemetryTransport param on registerAgentRelayTools. invoke_action via
the actions API continues to skip agent_relay_tool_call as the fleet
commit intended.

Also runs npm install to link @agent-relay/fleet into node_modules,
fixing the fleet.test.ts module-not-found failure.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(cli): remove dead metadata lookup code from telemetry fix

The generic AGENT_RELAY_TOOL_CALL_METADATA table, readInvokedActionName,
agentRelayActionNameCategory, and agentRelayToolCallMetadata were added
when restoring telemetry via the enableInboxPiggyback blanket wrapper,
but the final approach uses inline calls in add_agent/remove_agent
instead. Remove the unused code.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(cli): restore full owned-tool telemetry via wrapper, exclude actions

My earlier fix only re-instrumented add_agent/remove_agent inline, which
satisfied the failing test but silently dropped agent_relay_tool_call
telemetry for the ~28 other owned tools that PR #1041 deliberately
instrumented (post_message, send_dm, channels, reactions, inbox, etc.).

Restore the original wrapper-based tracking in enableInboxPiggyback so
every statically-registered owned tool emits telemetry again, and add a
skip for the action-routed path (invoke_action + the dynamic per-action
tools in actionToolNames) to honor the fleet SDK commit's intent that
registered actions do not route through per-tool telemetry. Reverts the
inline handler boilerplate and the registerAgentRelayTools param.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore: review fixes — gitignore opencode.json secret, trim stale telemetry entries

Fresh-eyes review follow-ups:
- gitignore /opencode.json: the opencode harness writes it to the repo
  root during eval runs with a live rk_live_ workspace key + at_live_
  agent token; it was untracked but not ignored, one `git add .` from a
  secret leak.
- Drop 3 stale AGENT_RELAY_TOOL_CALL_METADATA entries (get_message,
  get_thread, list_channel_members) — no such tools are registered, so
  the entries were dead lookups carried over from PR #1041.
- Fix run-task-exit-evals.sh header: the s08 scenario self-releases via
  remove_agent(self), not `/exit` output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(cli): exclude action-routed spawn tool from per-tool telemetry

The `spawn` tool delegates to the actions surface (actions.invoke('spawn'))
exactly like `invoke_action` and the dynamic per-action tools, which are
deliberately excluded from agent_relay_tool_call telemetry. Tracking `spawn`
but not `invoke_action({name:'spawn'})` counted the same underlying action
differently depending on which MCP surface the caller used.

Introduce ACTION_ROUTED_TOOL_NAMES ({invoke_action, spawn}) and skip per-tool
telemetry for those names alongside the actionToolNames check. Adds a test
asserting spawn still runs but emits no telemetry.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Proactive Runtime Bot <agent@agent-relay.com>
Co-authored-by: agent-relay-code[bot] <agent-relay-code[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant