feat(agent): V8 inspector (CDP) tools for debugging/profiling workers (#626) - #1547
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces V8 inspector (CDP) tools for the built-in agent, enabling operators to attach to worker threads, evaluate expressions, set breakpoints/logpoints, and record CPU profiles. The review feedback highlights several critical security and reliability improvements: marking inspector_set_logpoint as destructive to prevent unauthorized arbitrary code execution, caching connection promises to avoid concurrent connection race conditions, fixing a memory leak in the sleep function's abort listener, and aligning the THREADS_DEBUG default check with the server's behavior.
| } | ||
|
|
||
| // One live CDP connection per worker debug port, reused across tool calls. | ||
| const sessions = new Map<number, CdpSession>(); |
There was a problem hiding this comment.
To prevent concurrent connection race conditions where multiple tool calls concurrently attempt to connect to the same port, store the Promise<CdpSession> in the sessions map instead of the resolved CdpSession.
| const sessions = new Map<number, CdpSession>(); | |
| const sessions = new Map<number, Promise<CdpSession>>(); |
| async function sessionFor(port: number, host: string, signal?: AbortSignal): Promise<CdpSession> { | ||
| const existing = sessions.get(port); | ||
| if (existing) return existing; | ||
| const { wsUrl, title } = await fetchWebSocketUrl(host, port, signal); | ||
| const session = await openCdp(wsUrl, () => sessions.delete(port)); | ||
| session.title = title; | ||
| sessions.set(port, session); | ||
| return session; | ||
| } |
There was a problem hiding this comment.
Refactor sessionFor to use and cache the connection promise. This prevents duplicate WebSocket connections from being opened if multiple tool calls are made concurrently before the connection is established.
async function sessionFor(port: number, host: string, signal?: AbortSignal): Promise<CdpSession> {
let sessionPromise = sessions.get(port);
if (!sessionPromise) {
sessionPromise = (async () => {
try {
const { wsUrl, title } = await fetchWebSocketUrl(host, port, signal);
const session = await openCdp(wsUrl, () => sessions.delete(port));
session.title = title;
return session;
} catch (err) {
sessions.delete(port);
throw err;
}
})();
sessions.set(port, sessionPromise);
}
return sessionPromise;
}| export function _closeInspectorSessions(): void { | ||
| for (const session of sessions.values()) session.close(); | ||
| sessions.clear(); | ||
| } |
There was a problem hiding this comment.
Update _closeInspectorSessions to handle the promise-based session cache synchronously by calling .then() on the cached promises.
| export function _closeInspectorSessions(): void { | |
| for (const session of sessions.values()) session.close(); | |
| sessions.clear(); | |
| } | |
| export function _closeInspectorSessions(): void { | |
| for (const sessionPromise of sessions.values()) { | |
| sessionPromise.then( | |
| (session) => session.close(), | |
| () => {} | |
| ); | |
| } | |
| sessions.clear(); | |
| } |
| }, | ||
| handler: async (args: any, ctx: AgentToolContext) => { | ||
| const port = resolvePort(deps, Number(args.workerIndex)); |
There was a problem hiding this comment.
Security Vulnerability: inspector_set_logpoint allows arbitrary JavaScript code execution in the worker thread via the logExpression parameter (which is evaluated as part of the breakpoint condition). This can be used to bypass the approval gate for destructive actions (like inspector_evaluate). It must be marked as destructive: true to ensure it is routed through the operator approval gate.
},
},
destructive: true, // arbitrary in-worker code execution via logExpression
handler: async (args: any, ctx: AgentToolContext) => {| const inspectorTools = buildInspectorTools({ | ||
| debugEnabled: env.get(CONFIG_PARAMS.THREADS_DEBUG) !== false, | ||
| startingPort: (env.get(CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT) as number | undefined) ?? undefined, | ||
| host: (env.get(CONFIG_PARAMS.THREADS_DEBUG_HOST) as string | undefined) ?? '127.0.0.1', | ||
| getWorkerCount: () => workers.length, | ||
| }); |
There was a problem hiding this comment.
The default value for THREADS_DEBUG is inconsistent with server/threads/threadServer.js. In threadServer.js, debugging is disabled by default if not set (falsy). However, using !== false here causes debugEnabled to default to true when THREADS_DEBUG is undefined. This should be updated to explicitly check for true or truthiness to match the server's behavior.
| const inspectorTools = buildInspectorTools({ | |
| debugEnabled: env.get(CONFIG_PARAMS.THREADS_DEBUG) !== false, | |
| startingPort: (env.get(CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT) as number | undefined) ?? undefined, | |
| host: (env.get(CONFIG_PARAMS.THREADS_DEBUG_HOST) as string | undefined) ?? '127.0.0.1', | |
| getWorkerCount: () => workers.length, | |
| }); | |
| const inspectorTools = buildInspectorTools({ | |
| debugEnabled: env.get(CONFIG_PARAMS.THREADS_DEBUG) === true, | |
| startingPort: (env.get(CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT) as number | undefined) ?? undefined, | |
| host: (env.get(CONFIG_PARAMS.THREADS_DEBUG_HOST) as string | undefined) ?? '127.0.0.1', | |
| getWorkerCount: () => workers.length, | |
| }); |
| function sleep(ms: number, signal?: AbortSignal): Promise<void> { | ||
| return new Promise((resolve, reject) => { | ||
| if (signal?.aborted) return reject(new Error('aborted')); | ||
| const timer = setTimeout(resolve, ms); | ||
| signal?.addEventListener( | ||
| 'abort', | ||
| () => { | ||
| clearTimeout(timer); | ||
| reject(new Error('aborted')); | ||
| }, | ||
| { once: true } | ||
| ); | ||
| }); | ||
| } |
There was a problem hiding this comment.
There is a memory leak in the sleep function. If the sleep timer fires and resolves normally, the abort event listener remains attached to the signal. For long-lived signals, this prevents the promise and its closures from being garbage collected. The event listener must be removed when the promise settles.
| function sleep(ms: number, signal?: AbortSignal): Promise<void> { | |
| return new Promise((resolve, reject) => { | |
| if (signal?.aborted) return reject(new Error('aborted')); | |
| const timer = setTimeout(resolve, ms); | |
| signal?.addEventListener( | |
| 'abort', | |
| () => { | |
| clearTimeout(timer); | |
| reject(new Error('aborted')); | |
| }, | |
| { once: true } | |
| ); | |
| }); | |
| } | |
| function sleep(ms: number, signal?: AbortSignal): Promise<void> { | |
| return new Promise((resolve, reject) => { | |
| if (signal?.aborted) return reject(new Error('aborted')); | |
| let timer: NodeJS.Timeout; | |
| const onAbort = () => { | |
| clearTimeout(timer); | |
| reject(new Error('aborted')); | |
| }; | |
| timer = setTimeout(() => { | |
| signal?.removeEventListener('abort', onAbort); | |
| resolve(); | |
| }, ms); | |
| signal?.addEventListener('abort', onAbort, { once: true }); | |
| }); | |
| } |
| it('marks evaluate and set_breakpoint destructive, others not', () => { | ||
| const tools = toolMap(baseDeps); | ||
| assert.equal(tools.get('inspector_evaluate').destructive, true); | ||
| assert.equal(tools.get('inspector_set_breakpoint').destructive, true); | ||
| assert.ok(!tools.get('inspector_attach').destructive); | ||
| assert.ok(!tools.get('inspector_set_logpoint').destructive); | ||
| assert.ok(!tools.get('inspector_profile_cpu').destructive); | ||
| }); |
There was a problem hiding this comment.
Update the destructiveness assertions to verify that inspector_set_logpoint is correctly marked as destructive.
| it('marks evaluate and set_breakpoint destructive, others not', () => { | |
| const tools = toolMap(baseDeps); | |
| assert.equal(tools.get('inspector_evaluate').destructive, true); | |
| assert.equal(tools.get('inspector_set_breakpoint').destructive, true); | |
| assert.ok(!tools.get('inspector_attach').destructive); | |
| assert.ok(!tools.get('inspector_set_logpoint').destructive); | |
| assert.ok(!tools.get('inspector_profile_cpu').destructive); | |
| }); | |
| it('marks evaluate, set_breakpoint, and set_logpoint destructive, others not', () => { | |
| const tools = toolMap(baseDeps); | |
| assert.equal(tools.get('inspector_evaluate').destructive, true); | |
| assert.equal(tools.get('inspector_set_breakpoint').destructive, true); | |
| assert.equal(tools.get('inspector_set_logpoint').destructive, true); | |
| assert.ok(!tools.get('inspector_attach').destructive); | |
| assert.ok(!tools.get('inspector_profile_cpu').destructive); | |
| }); |
|
Reviewed; no blockers found. |
|
|
Cross-model review (Gemini + Codex + Harper-domain) run before marking ready. Gemini surfaced real CDP lifecycle/concurrency hazards — all fixed in the latest commit (see
Codex leg returned no structured findings. 61 agent unit tests green (incl. a live CDP round-trip). |
| * Two layers: | ||
| * 1. The safety envelope (`resolvePort`, exercised through the tool handlers) — | ||
| * pure, no network. This is the security-relevant part: debug must be | ||
| * enabled, a starting port configured, and workerIndex in range and NOT the |
There was a problem hiding this comment.
Suggestion (non-blocking): use plain assert instead of node:assert/strict — AGENTS.md explicitly calls out that strict mode's deep-equality and coercion rules cause more friction and surprising failures than they prevent, and that plain assert is the house style.
| * enabled, a starting port configured, and workerIndex in range and NOT the | |
| const assert = require('node:assert'); |
…#626) Adds the operator-only inspector tool surface from the #626 design — the last of the operator tools not yet built. The agent (main thread) attaches over the Chrome DevTools Protocol to *worker* thread inspector ports to evaluate, set breakpoints/logpoints, and CPU-profile a worker without stalling the thread it runs on. - agent/tools/inspectorTool.ts (new): buildInspectorTools(deps) → five tools: inspector_attach, inspector_evaluate, inspector_set_breakpoint, inspector_set_logpoint, inspector_profile_cpu. A minimal CDP client over `ws` (id-correlated request/response + event listeners), one reused connection per worker debug port. Deps (debug config + live worker count) are injected so the tool is unit-testable without a server boot. - Safety envelope (resolvePort): requires threads_debug + threads_debug_startingPort, range-checks workerIndex against the live worker pool, and REJECTS workerIndex < 0 (the main thread — a self-attach/breakpoint would deadlock the agent). - Worker debug port = threads_debug_startingPort + workerIndex (mirrors server/threads/threadServer.js). Logpoints are non-pausing (breakpoint whose condition console.logs and returns false). CPU profiles are summarized to the hottest functions by self time so the observation stays small. - evaluate + set_breakpoint are marked destructive (arbitrary in-worker code / can pause a live worker) → gated by the loop's approval flow. - agent/toolset.ts + agent/agent.ts: compose inspector tools into the operator-only set; deps read from env (threads_debug*) and the live `workers` pool. - unitTests/agent/inspectorTool.test.js: safety-envelope guards, summarizeProfile, and a LIVE CDP round-trip (opens a real inspector on an ephemeral port; drives attach + evaluate + profile). 60 agent unit tests green. Stacked on #1545 (registry tools). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Cross-model review (Gemini) flagged real lifecycle/concurrency hazards in the CDP tooling; all addressed: - openCdp never settled if the socket closed/aborted before 'open' → sessionFor hung the agent. Now a pre-open close/error/abort rejects the connect promise. - Session-cache races: cache the connection *promise* (dedupes concurrent opens); evict by identity so a stale close from a replaced connection can't drop a live session; drop a failed connect so the next call retries. - Breakpoints could wedge a worker (paused, no resume path). Every connection now registers a Debugger.paused handler that logs a stack snapshot and auto-resumes, so a hit is observable via the Harper log but never leaves the worker paused. - Logpoint injection: logExpression is now JSON-encoded and eval'd (not spliced as raw code), so it can't break out of the wrapper to force a pause. set_logpoint is also marked destructive (it runs an expression in the worker on every hit), alongside evaluate and set_breakpoint. - CDP calls now carry a per-call abort signal + 30s timeout, so an unresponsive worker can't hang the agent loop. The connect signal guards only the handshake (aborting one caller no longer tears down the shared connection). - Strict workerIndex parsing: reject ""/null/false/[] instead of Number()-coercing them to worker 0. - Profiler.disable on cleanup; resolvePort rejects a port past 65535. - _closeInspectorSessions handles in-flight opens. Also fixes a test-teardown deadlock: inspector.close() blocks until CDP clients drop, so the live-round-trip after() hook only closes our client and lets --exit tear down the inspector. Full agent unit suite: 61 passing, build clean. Codex leg returned no structured findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
525cdd3 to
84cf845
Compare
b8d5e63 to
5d2df52
Compare
…st (#626) The dev-mode install CI uses for unit tests sets threads.debug: true, so threadServer opens the V8 inspector on the main process during the suite. The live-CDP test's before hook then called inspector.open() unconditionally and threw ERR_INSPECTOR_ALREADY_ACTIVATED. Reuse the live inspector when one is already open; only open our own ephemeral port otherwise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…tered (#626) (#1545) * feat(agent): consume MCP tool registry (Operations profile), RBAC-filtered (#626) Fold the unified MCP tool registry's Operations profile (#617) into the built-in agent's toolset, fulfilling #626's tool-composition design: the agent consumes the same registry the MCP server exposes, RBAC-filtered for its configured user. Re-does the intent of the abandoned #893 against current main (that branch was ~465 commits stale and used the pre-merge registry API). - agent/registryTools.ts (new): ensureOperationsToolsRegistered() populates the Operations profile on the main thread (idempotent); composeRegistryTools() snapshots the profile, filters by visibleTo(agentUser), and adapts each ToolDef -> AgentTool (inputSchema->parameters, destructiveHint->approval gate, ToolResult->return value with isError->throw so the loop records a recoverable failure rather than aborting). - agent/agent.ts: resolve the configured agent user via server.getUser (falls back to a super_user identity with a warning when unresolved) and thread the registry tools through both composeToolset calls. - agent/toolset.ts: merge operator-only + registry tools; operator-only tools win on a name collision. - unitTests/agent/registryTools.test.js: 10 tests over RBAC filtering, shape adaptation, destructive gating, result unwrapping, and the collision rule. No server boot / no LLM credits. Enforcement note: visibleTo controls listing only; real RBAC runs per-call in the operation handler via hdb_user set to the agent's identity. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): fail closed + per-call identity for registry tools (#626) Cross-model review (Codex + Gemini) flagged two authorization defects in the registry integration; both fixed at root: - Fail-open super_user: resolveAgentUser previously fabricated a { super_user: true } identity whenever agent.user couldn't be resolved, silently escalating a misconfigured or transient restricted service account to admin. Now resolveAgentIdentity only falls back to super_user for the *default* hdb_agent bootstrap user (whose provisioning #626 defers); an explicitly configured user that won't resolve throws (fail closed), and the agent runs with only its operator-only tools until the operator fixes agent.user. - Stale cached identity: the agent user was resolved once at startup and closed over in every tool handler, so a role revocation/change wasn't honored until restart. The enforcement identity is now resolved *per call* (composeRegistryTools takes a resolveIdentity thunk), mirroring how the MCP HTTP path re-auths per request. The startup snapshot is used only for visibleTo listing (not a boundary). Adjudicated out: Gemini's "setConfig ignores agent.user changes" — set_agent_config does not accept `user` (not in its patch keys), so it can't change at runtime. Adds 2 unit tests: per-call re-resolution honors a live role change; a fail-closed rejection propagates and the operation never dispatches. 51 agent unit tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(agent): V8 inspector (CDP) tools for debugging/profiling workers (#626) (#1547) * feat(agent): V8 inspector (CDP) tools for debugging/profiling workers (#626) Adds the operator-only inspector tool surface from the #626 design — the last of the operator tools not yet built. The agent (main thread) attaches over the Chrome DevTools Protocol to *worker* thread inspector ports to evaluate, set breakpoints/logpoints, and CPU-profile a worker without stalling the thread it runs on. - agent/tools/inspectorTool.ts (new): buildInspectorTools(deps) → five tools: inspector_attach, inspector_evaluate, inspector_set_breakpoint, inspector_set_logpoint, inspector_profile_cpu. A minimal CDP client over `ws` (id-correlated request/response + event listeners), one reused connection per worker debug port. Deps (debug config + live worker count) are injected so the tool is unit-testable without a server boot. - Safety envelope (resolvePort): requires threads_debug + threads_debug_startingPort, range-checks workerIndex against the live worker pool, and REJECTS workerIndex < 0 (the main thread — a self-attach/breakpoint would deadlock the agent). - Worker debug port = threads_debug_startingPort + workerIndex (mirrors server/threads/threadServer.js). Logpoints are non-pausing (breakpoint whose condition console.logs and returns false). CPU profiles are summarized to the hottest functions by self time so the observation stays small. - evaluate + set_breakpoint are marked destructive (arbitrary in-worker code / can pause a live worker) → gated by the loop's approval flow. - agent/toolset.ts + agent/agent.ts: compose inspector tools into the operator-only set; deps read from env (threads_debug*) and the live `workers` pool. - unitTests/agent/inspectorTool.test.js: safety-envelope guards, summarizeProfile, and a LIVE CDP round-trip (opens a real inspector on an ephemeral port; drives attach + evaluate + profile). 60 agent unit tests green. Stacked on #1545 (registry tools). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): harden inspector CDP client after cross-model review (#626) Cross-model review (Gemini) flagged real lifecycle/concurrency hazards in the CDP tooling; all addressed: - openCdp never settled if the socket closed/aborted before 'open' → sessionFor hung the agent. Now a pre-open close/error/abort rejects the connect promise. - Session-cache races: cache the connection *promise* (dedupes concurrent opens); evict by identity so a stale close from a replaced connection can't drop a live session; drop a failed connect so the next call retries. - Breakpoints could wedge a worker (paused, no resume path). Every connection now registers a Debugger.paused handler that logs a stack snapshot and auto-resumes, so a hit is observable via the Harper log but never leaves the worker paused. - Logpoint injection: logExpression is now JSON-encoded and eval'd (not spliced as raw code), so it can't break out of the wrapper to force a pause. set_logpoint is also marked destructive (it runs an expression in the worker on every hit), alongside evaluate and set_breakpoint. - CDP calls now carry a per-call abort signal + 30s timeout, so an unresponsive worker can't hang the agent loop. The connect signal guards only the handshake (aborting one caller no longer tears down the shared connection). - Strict workerIndex parsing: reject ""/null/false/[] instead of Number()-coercing them to worker 0. - Profiler.disable on cleanup; resolvePort rejects a port past 65535. - _closeInspectorSessions handles in-flight opens. Also fixes a test-teardown deadlock: inspector.close() blocks until CDP clients drop, so the live-round-trip after() hook only closes our client and lets --exit tear down the inspector. Full agent unit suite: 61 passing, build clean. Codex leg returned no structured findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): tolerate an already-active inspector in CDP round-trip test (#626) The dev-mode install CI uses for unit tests sets threads.debug: true, so threadServer opens the V8 inspector on the main process during the suite. The live-CDP test's before hook then called inspector.open() unconditionally and threw ERR_INSPECTOR_ALREADY_ACTIVATED. Reuse the live inspector when one is already open; only open our own ephemeral port otherwise. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Kris Zyp <kris@harperdb.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Kris Zyp <kris@harperdb.io> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Summary
Adds the operator-only V8 inspector (CDP) tools from the #626 design — the last operator tool not yet built. The built-in agent runs on the main thread; these tools attach over the Chrome DevTools Protocol to worker thread inspector ports so it can evaluate expressions, set breakpoints/logpoints, and CPU-profile a worker without stalling the thread it runs on.
Tools (
agent/tools/inspectorTool.ts)inspector_attachinspector_evaluateinspector_set_breakpointinspector_set_logpointconsole.logs and returnsfalse)inspector_profile_cpudurationMs, return hottest functions by self timews(already a direct dep), one reused connection per worker debug port, evicted on socket close.threads_debug_startingPort + workerIndex, mirroringserver/threads/threadServer.js. Attaching requiresthreads_debug: trueand a configuredthreads_debug_startingPort.summarizeProfilereduces it to the top-N functions by self time so the LLM observation stays small.Safety envelope (
resolvePort)The security-relevant part. Every tool call must clear:
threads_debugenabled andthreads_debug_startingPortconfigured (else a clear error telling the operator what to set);workerIndexan integer, in range against the live worker pool;workerIndex >= 0— the main thread is where the agent itself runs; a self-attach + breakpoint would deadlock it. This is the operator-agent counterpart to the app-developer agent (Add agent-loop orchestration /toolMode: 'auto'toscope.models#612), which runs on a worker and must never attach to itself.evaluateandset_breakpointare marked destructive, so they route through the loop's approval gate unlessautoApproveis set.Tests (
unitTests/agent/inspectorTool.test.js)summarizeProfile— self-time ranking + topN cap on a synthetic profile.attach+evaluate(40+2 → 42, exception → thrown) +profile_cpuagainst it. This process stands in for a worker (same protocol); it's never breakpointed (that would pause the test — the very reason main-thread attach is banned).Full agent unit suite: 60 passing.
npm run buildclean.Operator setup note (for docs)
To let the agent debug workers, operators set
threads_debug: trueand a sensiblethreads_debug_startingPortbefore asking the agent to attach.🤖 Generated with Claude Code