fix(agent-runtime): reap claude subprocess tree via process groups (#407)

[vybe]

Summary

Fixes #407 — agent-server spinning at ~83% CPU after claude CLI subprocess becomes <defunct> and all HTTP endpoints hang until the container is restarted.

• Launch claude with start_new_session=True and kill the entire process group on every exit path, so hook grandchildren (bash-guardrail, file-guardrail, output-scanner, …) can't outlive claude and wedge readline() via inherited pipe FDs.
• Restructure the task-path reader so both stdout and stderr read in threads; the main executor waits on the subprocess with a bounded timeout and drains readers with a grace period. If readers are still stuck, drain helper kills the group and force-closes pipes so readline() unwinds.
• Capture the pgid at spawn time (before wait() reaps the parent) and thread it through every cleanup callsite — after reap, os.getpgid(pid) raises, but the process group itself persists as long as any member does (the grandchildren we need to kill).
• ProcessRegistry.terminate() now signals the full tree via the stored pgid, preserving termination semantics even after the parent has been reaped.
• Helpers extracted to docker/base-image/agent_server/utils/subprocess_pgroup.py so they're testable without loading the rest of agent_server.

Root cause

From the issue: parent python3 /app/agent-server.py pinned at ~83% CPU with a [claude] <defunct> zombie beneath it. Last log line before the hang: [ProcessRegistry] Registered execution <id>. Claude CLI had already exited, but a hook grandchild kept the inherited stderr FD open — so our readline() never saw EOF, the executor thread wedged, and FastAPI stopped serving.

Analysis and code read both point at the same fix shape: process-group lifecycle + bounded waits + thread drain. No code change is a regression from the last release; exposure grew after max_turns_task was raised 20→50 (#361, 2d before the report) because tasks now do more hook-invoking tool calls before cap.

Test plan

• New regression test tests/unit/test_subprocess_pgroup.py (10 cases, all passing). Covers the production pattern end-to-end: harness parent forks a grandchild that keeps stderr open, exits, and the helper must kill the grandchild via the captured pgid and unwind the stuck reader within a bounded time. Also exercises idempotency, already-exited safety, and signal_process_tree fallback paths.
• Adjacent subprocess/orchestration unit tests still pass (test_error_classification, test_credential_sanitizer_agent, test_otel_trace_logging, test_orphaned_execution_recovery).
• Base image rebuilds cleanly (./scripts/deploy/build-base-image.sh).
• Imports resolve inside the base image (python3 -c "from agent_server.utils.subprocess_pgroup import ..." and from agent_server.services.claude_code import execute_headless_task).
• Deploy + validate against a scheduled task that hits the 50-turn cap (the original repro trigger).

Broader architectural question

The symptom class (#285 false-positive auth detection on stderr regex, #407 subprocess zombie/wedge) is downstream of running claude as a CLI subprocess and hand-parsing stream-json with threads + asyncio bridges. Whether to migrate this module to the Claude Agent SDK instead is tracked separately in #409 — not in scope here.

Closes #407
Related #409

🤖 Generated with Claude Code