fix(orphan-killer): SCHED_IDLE + scan deadline to prevent health-probe starvation (#808)

[vybe]

Summary

• The orphan-killer daemon thread could spin at 100% CPU for ~15 minutes when a D-state process delayed os.readlink() in the /proc scan. On 1-CPU containers this starved the uvicorn event loop, causing health probes to time out and the circuit breaker to open.
• Fix 1 — _set_idle_priority(): sets the daemon thread to SCHED_IDLE (Linux) or nice(19) (macOS/POSIX) so it only runs when no higher-priority thread wants the CPU — uvicorn health probes win even while the scan is looping.
• Fix 2 — _scan_deadline: per-iteration time.monotonic() check with an 8-second budget (_ORPHAN_SCAN_WALL_SECONDS) aborts the /proc loop once the budget expires, bounding total scan time regardless of D-state delays. 8s is inside the existing 11s asyncio.wait_for ceiling.

Root cause

Previous PRs (#650, #730, #747) bounded waiting behavior (how long drain_reader_threads blocks the caller) but the daemon thread itself was never stopped — it continued spinning at default priority for up to ~917 seconds. On a 1-CPU container, a thread at 100% CPU prevents all other runnable threads (including uvicorn's event loop) from running, causing health probe timeouts and circuit breaker activation (#631).

Changes

• docker/base-image/agent_server/utils/subprocess_pgroup.py:

  • _ORPHAN_SCAN_WALL_SECONDS = 8 — budget constant
  • _set_idle_priority() — SCHED_IDLE/nice(19), no-op on unsupported platforms
  • _kill_orphan_pipe_writers() — added _scan_deadline: Optional[float] = None kwarg + per-iteration deadline check
  • _run_orphan_killer() — calls _set_idle_priority() and passes scan_deadline

• tests/unit/test_subprocess_pgroup.py:

  • Fixed mock signature: _slow_orphan_killer(fd, our_pgid, _scan_deadline=None)
  • New: TestKillOrphanPipeWriters.test_scan_deadline_stops_scan_early (Linux-only, verifies expired deadline aborts scan immediately)
  • New: TestSetIdlePriority (does_not_raise, idempotent — runs on all platforms)

Test Plan

• pytest tests/unit/test_subprocess_pgroup.py -v — 14 passed, 5 skipped (Linux-only on macOS)
• All pre-existing tests pass without modification to test logic
• No changes to production call sites — _scan_deadline is keyword-only with a default of None (backward compatible)

Fixes #808

🤖 Generated with Claude Code