bug: child processes inherit agent-server stdout pipe → reader thread stuck, false '0 tool calls' failures

[vybe]

Summary

When Claude Code exits, any child process it spawned that still holds a copy of the stdout pipe prevents Trinity's reader thread from seeing EOF. The reader thread waits, times out after 30s, force-closes, and loses the final result block. Trinity reports the execution as failed with "0 tool calls" even though tools were actually called. This has been causing persistent false failures across multiple agents.

Component

Agent Runtime / agent-server subprocess management

Priority

P1

Error

WARNING:agent_server.utils.subprocess_pgroup:[Subprocess] Reader thread(s) still busy after process exit (pid=XXXXX, stuck_count=1) — killing process group, then waiting 30s for natural drain
ERROR:agent_server.utils.subprocess_pgroup:[Subprocess] Reader thread(s) still stuck after 30s post-kill grace — force-closing pipes; some buffered data may be lost (pid=XXXXX, stuck_count=1)
ERROR:agent_server.utils.subprocess_pgroup:[Subprocess] 1 reader thread(s) leaked for pid=XXXXX after force-close; continuing anyway
ERROR:agent_server.services.claude_code:[Headless Task] Execution completed without a result message after 0 tool calls / N turns (raw_messages=M). Likely cause: a tool or child subprocess inherited stdout and prevented the claude reader thread from capturing the final result block.
ERROR:agent_server.services.claude_code:[Headless Task] Error reading stdout: I/O operation on closed file.

Root Cause

When the agent-server spawns claude --print --output-format stream-json, it creates a stdout pipe to read the streaming result. Two categories of child processes can inherit this pipe fd and keep it alive after Claude exits:

1. Async Stop hooks ("async": true in ~/.claude/settings.json)

Claude Code spawns Stop hook scripts as background child processes before exiting. If these inherit fd 1 (the Trinity pipe), they hold the pipe open for their entire run. Example: a git-sync hook that runs git push in the background after the session ends keeps the pipe open until the push completes (or forever if it hangs).

2. Process-based MCP servers (entries with "command" in .mcp.json)

Claude Code spawns MCP servers as child processes (npx, uv tool run, etc.). Even though their stdout is redirected to a dedicated JSON-RPC pipe, they may inherit the parent Claude process's original stdout fd (the Trinity pipe) if FD_CLOEXEC is not set on it before the exec. These processes run for the duration of the session and may not release inherited fds promptly on shutdown.

In both cases, the fix is to ensure the Trinity stdout pipe fd has FD_CLOEXEC set before Claude Code spawns any children, so inherited file descriptors are automatically closed on exec.

Reproduction Steps

Via async Stop hook (most reproducible):

1. Configure an agent with ~/.claude/settings.json:

   { "hooks": { "Stop": [{ "hooks": [{ "type": "command", "command": "sleep 60", "async": true }] }] } }

2. Trigger a scheduled execution on that agent
3. Observe: Reader thread(s) still busy after process exit in agent container logs
4. Observe: execution recorded as failed with 0 tool calls in schedule_executions table, despite tools being used

Via process-based MCP server:

1. Configure an agent with .mcp.json:

   { "mcpServers": { "my-server": { "command": "npx", "args": ["-y", "some-mcp-server"] } } }

2. Trigger an execution that uses MCP tools
3. Observe intermittent Reader thread stuck failures after execution completes

Suggested Fix

In agent_server/utils/subprocess_pgroup.py (or wherever claude --print is spawned):

Before spawning the Claude subprocess, mark the write end of the stdout pipe with FD_CLOEXEC so it is automatically closed in any child Claude spawns:

import fcntl, os

# When creating the pipe for Claude's stdout:
read_fd, write_fd = os.pipe()

# Mark write end close-on-exec so Claude's children don't inherit it
flags = fcntl.fcntl(write_fd, fcntl.F_GETFD)
fcntl.fcntl(write_fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)

# Then spawn claude with stdout=write_fd

Or equivalently, if using Python's subprocess.Popen:

proc = subprocess.Popen(
    ["claude", "--print", "--output-format", "stream-json", ...],
    stdout=subprocess.PIPE,
    # pass_fds=() ensures no extra fds are inherited beyond 0/1/2
    # close_fds=True (default on POSIX) handles fds >= 3
    # The issue is that fd 1 of the *parent* may be inherited if not handled
)

The key invariant: the Trinity reader pipe fd must not be inheritable by anything Claude Code launches.

Agent-level workaround (for async Stop hooks):

Add exec >/dev/null at the start of the hook script body (after reading stdin). This immediately releases the inherited pipe fd in the background process.

Impact

• Executions recorded as failed in schedule_executions with tool_calls = 0 even when tools ran
• message column shows the error text; no execution log captured
• Agents with frequent changes (triggering git-sync hooks on every session) or process-based MCP servers fail the majority of their scheduled runs
• Retry logic re-runs the full task, doubling API cost and execution time

Environment

• Trinity version: f199383
• Docker: 24.x
• OS: Ubuntu 22.04

Related

• agent_server/utils/subprocess_pgroup.py — reader thread + process group management
• agent_server/services/claude_code.py — headless task execution, result parsing
• The error message in claude_code.py already correctly diagnoses the cause; this issue is to fix it at the source

[vybe]

Research findings

Analyzed the codebase and traced through the fix history (#407 → #520/#521 → #531/#532) to understand what remains unsolved.

Why the suggested FD_CLOEXEC fix won't work

The issue's suggested fix sets FD_CLOEXEC on write_fd before calling Popen. This doesn't prevent the inheritance problem because of how dup2 works:

1. Python's subprocess.Popen does dup2(write_fd, 1) in the child before exec
2. dup2 clears FD_CLOEXEC on the destination (fd 1)
3. Claude starts with fd 1 = Trinity pipe write end, no FD_CLOEXEC
4. Claude's children inherit fd 1 normally

Setting FD_CLOEXEC on write_fd before Popen is redundant with close_fds=True (which already closes fds ≥ 3) and does nothing to prevent fd 1 inheritance in grandchildren.

Actual root cause

The real reason the group kill (killpg(pgid, SIGKILL)) fails to unblock the reader is that async Stop hooks escape the process group via setsid().

When Claude Code (Node.js) spawns an async Stop hook, it uses Node.js's child_process.spawn with {detached: true} internally. detached: true calls setsid() in the child, putting it into a new session and new process group — completely invisible to os.killpg(pgid, SIGKILL). The hook inherits fd 1 (Trinity pipe write end), and even after the group kill, the hook keeps the pipe open for its entire run (e.g. git push over SSH).

This explains why the 30s post_kill_grace in drain_reader_threads still fails: the orphan is alive in a separate session, and we time out waiting for an EOF that never comes.

Context from the operator: the initial triggering agents were those with git-sync Claude Code hooks (Stop hooks that run git push at session end). Agents without async Stop hooks rarely see this. Agents calling process-based MCP servers may also be affected if those launchers call setsid().

Correct fix

After terminate_process_group(), scan /proc/PID/fd to find any process holding the write end of the specific pipe open, and SIGKILL them. This catches escaped setsid processes regardless of their session or group.

def _kill_pipe_write_holders(pipe_read_end) -> int:
    """Kill processes holding this pipe's write end open (Linux /proc only)."""
    pipe_inode = os.fstat(pipe_read_end.fileno()).st_ino
    inode_str = f'pipe:[{pipe_inode}]'
    our_pid = os.getpid()
    killed = 0
    for pid_str in os.listdir('/proc'):
        if not pid_str.isdigit():
            continue
        pid = int(pid_str)
        if pid == our_pid:
            continue
        for fd_name in os.listdir(f'/proc/{pid}/fd'):
            if os.readlink(f'/proc/{pid}/fd/{fd_name}') != inode_str:
                continue
            # Check fdinfo flags: O_WRONLY (1) = write end
            with open(f'/proc/{pid}/fdinfo/{fd_name}') as f:
                for line in f:
                    if line.startswith('flags:'):
                        if int(line.split()[1], 8) & 3:  # writable
                            os.kill(pid, signal.SIGKILL)
                            killed += 1
                        break
            break
    return killed

Called from drain_reader_threads immediately after terminate_process_group(), before the post_kill_grace wait.

Safety with parallel executions

Each Popen call creates an independent pipe with its own unique inode. The scan is keyed on pipe_inode from the specific stuck execution's read fd. A concurrent Claude execution in the same container has a different pipe inode; none of its subprocesses will match.

Implementation plan

• docker/base-image/agent_server/utils/subprocess_pgroup.py: add _kill_pipe_write_holders() + call from drain_reader_threads() after the group kill
• tests/unit/test_subprocess_pgroup.py: add a Linux-only test using os.setsid() in a subprocess to verify drain_reader_threads catches escaped processes that the group kill misses

[dolho]

/claim

[github-actions]

Claimed by @dolho. Label status-in-progress added.

Next steps:

1. Create branch: feature/548-<short-slug>
2. Implement the feature
3. Open a PR with Fixes #548