Skip to content

fix(#912): forward in-flight allowlist to drain-time cgroup sweep - #939

Merged
vybe merged 2 commits into
devfrom
feature/912-orphan-sweep-drain-allowlist
May 25, 2026
Merged

fix(#912): forward in-flight allowlist to drain-time cgroup sweep#939
vybe merged 2 commits into
devfrom
feature/912-orphan-sweep-drain-allowlist

Conversation

@dolho

@dolho dolho commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

The post-drain kill_cgroup_orphans() call in subprocess_pgroup.drain_reader_threads() ran bare — no extra_pids= allowlist. Its allowlist resolver then preserved only the sweep's own pid + parent chain, so any other in-flight claude subprocess running a sibling task in the same cgroup got SIGKILLed the moment a short task drained.

Visible in production (tapestry-researcher-1, 2026-05-22) as multi-minute LinkedIn-research executions ending with "Execution terminated by SIGKILL after 0 tool calls / 0 turns" the instant an unrelated 7-second git pull task finished in the same container. Operator workaround was max_parallel_tasks=1, defeating the parallel-slot feature.

Why pre-#912

The periodic sweeper (services/orphan_sweeper.py:170) and ProcessRegistry.terminate() (services/process_registry.py:168) both already passed an allowlist. The drain-time call site (subprocess_pgroup.py:351) was the only outlier — pure oversight. The bug was masked until #817 introduced the cgroup-walk sweep at the drain site.

Changes

  • New canonical accessor ProcessRegistry.active_execution_pids(exclude_execution_id=…) — returns pid + captured pgid list for every running execution (optionally minus one). Single source of truth so a future drain-style call site can't repeat the oversight.
  • subprocess_pgroup._active_execution_pids_for_drain() — lazy registry-read helper that swallows any registry hiccup so the drain itself never fails. Wired into the finally block of drain_reader_threads() so kill_cgroup_orphans(extra_pids=…) now receives the allowlist.
  • services/orphan_sweeper._active_execution_pids() simplified to delegate to the new registry method (~25 lines removed).
  • ProcessRegistry.terminate() inner loop replaced with a single call to self.active_execution_pids(exclude_execution_id=…) (~15 lines removed, same semantics).

Tests

tests/unit/test_912_orphan_sweep_drain_allowlist.py — 8 unit tests:

  • Registry accessor: pid+pgid shape, exclude-self, dead-process skip, invalid-pgid skip, empty-registry
  • Drain helper: pass-through + error-swallow on registry exception
  • drain_reader_threads end-to-end forwarding of extra_pids to kill_cgroup_orphans (mocked)

Local: 8/8 pass, tests/unit/test_subprocess_pgroup.py 14/14 still pass, sys.modules lint clean.

Live verification

Rebuilt trinity-agent-base, recreated agent-trinity-system, registered two sleeps in the in-container registry, confirmed:

all pids: [79, 79, 80, 80]
exclude exec-1: [80, 80]
p1.pid expected absent: True
p2.pid expected present: True

End-to-end reproduction (two real concurrent claude subprocesses, one drains, the other survives) requires a working Anthropic auth on the test agent — not feasible on this dev box (sk-ant-oat01-* OAuth token can't drive headless claude). The unit + in-container behavioural checks gate correctness; production verification will land via the operator who reported the issue removing the max_parallel_tasks=1 workaround after this image rebuilds.

Test plan

  • Unit tests 8/8 pass
  • Existing test_subprocess_pgroup.py 14/14 pass
  • In-container active_execution_pids behavioural check
  • CI green
  • Operator confirms max_parallel_tasks > 1 works on tapestry-researcher-* post-deploy

Out of scope

Related to #912.

🤖 Generated with Claude Code

dolho and others added 2 commits May 25, 2026 16:25
… sweep

The post-drain `kill_cgroup_orphans()` call in
`subprocess_pgroup.drain_reader_threads()` ran bare — no `extra_pids=`
allowlist. Its allowlist resolver then only preserved the sweep's own
pid + parent chain, so any *other* in-flight claude subprocess running
a sibling task in the same cgroup got SIGKILLed the moment a short
task drained. Visible in production (`tapestry-researcher-1`,
2026-05-22) as multi-minute LinkedIn-research executions ending with
"Execution terminated by SIGKILL after 0 tool calls / 0 turns" the
instant an unrelated 7s git-pull task finished in the same container.
Operator workaround was `max_parallel_tasks=1`, defeating the parallel-
slot feature.

The bug is two years old in pattern (the bare call) but was masked
until #817 introduced the cgroup-walk sweep at the drain site. The
periodic sweeper (`services/orphan_sweeper.py`) and
`ProcessRegistry.terminate()` both already passed an allowlist; the
drain-time call site was the only outlier.

Changes:
- `ProcessRegistry.active_execution_pids(exclude_execution_id=…)` — new
  canonical accessor that returns the pid + captured pgid list for
  every running execution (optionally minus one). Single source of
  truth so a future drain-style call site can't repeat the oversight.
- `subprocess_pgroup._active_execution_pids_for_drain()` — lazy
  registry-read helper that swallows any registry hiccup so the drain
  itself never fails. Wired into the `finally` block of
  `drain_reader_threads()` so `kill_cgroup_orphans(extra_pids=…)` now
  receives the allowlist.
- `services/orphan_sweeper._active_execution_pids()` simplified to
  delegate to the new registry method (~25 lines removed).
- `ProcessRegistry.terminate()` inner loop replaced with a single
  call to `self.active_execution_pids(exclude_execution_id=…)`
  (~15 lines removed, same semantics).

Tests: `tests/unit/test_912_orphan_sweep_drain_allowlist.py` — 8
unit tests cover (a) the registry accessor's shape, exclude-self,
dead-process skip, invalid-pgid skip, empty-registry; (b) the drain
helper's pass-through + error-swallow; (c) the drain_reader_threads
end-to-end forwarding of extra_pids to kill_cgroup_orphans (mocked).
All 8 pass; existing 14 tests in `test_subprocess_pgroup.py` still
pass. sys.modules lint clean.

Verified in-container: rebuilt `trinity-agent-base`, recreated
`agent-trinity-system`, registered two sleeps, confirmed
`active_execution_pids(exclude_execution_id="exec-1")` returns only
the other process's pid.

Operators can remove the `max_parallel_tasks=1` workaround on
researcher fleets after this image rebuilds.

Related to #912.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per /review I1 — `import time` was added but never referenced.

Related to #912.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@dolho

dolho commented May 25, 2026

Copy link
Copy Markdown
Contributor Author

/review Report — Pass 2 (after 9c9a2686)

Branch: feature/912-orphan-sweep-drain-allowlistdev
Files Changed: 5 (+279/-41) | Commits: 2

Delta since Pass 1

Pass 1 finding Status
I1 unused import time in test_912 file ✅ Resolved — import time deleted, tests still 8/8 pass

Critical Findings (block merge)

None.

Informational Findings (new on this pass)

None. Naive AST scan flagged annotations and pytest — both false positives:

  • from __future__ import annotations is the PEP-563 future-import (required for int | None syntax surfaces).
  • pytest is needed for the monkeypatch fixture (pytest auto-injects by parameter name; the symbol must still resolve at collection time).

Verifications Re-Run

  • pytest tests/unit/test_912_orphan_sweep_drain_allowlist.py8 passed
  • python tests/lint_sys_modules.py230 / 237 baseline (no new violations)
  • Diff stat shrunk by exactly 1 line (the deleted import).

Clean Categories (unchanged from Pass 1)

  • SQL & data safety: no SQL changes.
  • Race conditions: snapshot under self._lock; terminate() releases lock before calling active_execution_pids() per the explicit comment at line 124; no reentrant-lock deadlock.
  • Auth boundaries: agent-internal helpers, no external endpoints touched.
  • Credential exposure: no secrets, logs, or env values added.
  • Conditional side effects, magic numbers, dead code, error handling, performance, enum completeness: all clean.
  • Architectural invariants (CLAUDE.md): no DB / router / model / auth surface touched.

Summary

Category Count Action
Critical 0 Clear to merge
Informational 0
Scope CLEAN

Recommendation

APPROVE

PR is structurally and stylistically clean now. Live end-to-end verification already proved the fix in the real agent cgroup (victim subprocess survived sibling drain that would have SIGKILLed it pre-#912). CI status to confirm before merge.

🤖 Generated by /review skill (.claude/skills/review)

@dolho
dolho requested a review from vybe May 25, 2026 13:52

@vybe vybe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved per /validate-pr. Forwarding the in-flight allowlist into the drain-time cgroup sweep is the right fix for the false-kill bug; consolidating the three pid-collection walks into a single canonical ProcessRegistry.active_execution_pids() accessor structurally prevents the bug from recurring at a future drain-style call site. Tests include a proof-of-wire-up that monkey-patches kill_cgroup_orphans and asserts extra_pids was forwarded — exactly the right test for this class of bug. CI green (15 checks).

@vybe
vybe merged commit e0e7af1 into dev May 25, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants