Skip to content

Bound wait_for_text so a wrong pattern can't stall the server#100

Open
tony wants to merge 8 commits into
mainfrom
wait-for-text-woes-2
Open

Bound wait_for_text so a wrong pattern can't stall the server#100
tony wants to merge 8 commits into
mainfrom
wait-for-text-woes-2

Conversation

@tony

@tony tony commented Jul 22, 2026

Copy link
Copy Markdown
Member

Summary

  • Cap waits at a server ceiling. wait_for_text accepted an unbounded caller timeout, so one badly-chosen search pattern could hold the MCP connection for minutes and stall an agent session. Waits now honour LIBTMUX_MCP_WAIT_MAX_SECONDS (default 30 s, hard-clamped to [1.0, 120.0]); an over-large timeout is clamped and reported via effective_timeout, not rejected.
  • Add stop failure patterns and fold in wait_for_content_change. pattern: str becomes patterns: list[str] | None; patterns=null waits for any new output, subsuming the removed wait_for_content_change. A new stop: list[str] | None ends the wait in milliseconds on a known failure marker instead of burning the whole budget.
  • Close the three ways a wait could exceed its ceiling. Readonly retry restarting the deadline, batch wrappers running the tool serially with no aggregate bound, and cancelled waits leaking executor slots — all fixed behind a self-bounded tag and per-operation batch rejection.
  • Make the wait path thread-free so a wedged tmux can't hang the server. libtmux reaches tmux through an untimed Popen.communicate(); a thread blocked there can't be cancelled, and concurrent.futures joins pool workers untimed at interpreter exit — so an unresponsive tmux left the process and Ctrl-C hanging indefinitely. Every tmux call the wait makes, pane resolution included, now runs as an asyncio subprocess bounded against the deadline and killed on expiry.
  • Report the pane's real state, and stop matching pager paint. WaitForTextResult folds its flags into one outcome enum (matched / any_output / stopped / alternate_screen / timeout) plus saw_new_output, matched_at_entry, and a byte-bounded tail. On the alternate screen the wait returns outcome="alternate_screen" rather than falsely matching rows the program already painted.

Changes by area

src/libtmux_mcp/tools/pane_tools/wait.py

The core: async-subprocess tmux calls, native pane resolution (no libtmux on the wait path), stop/patterns handling, the alternate-screen guard, and budget-derived per-call timeouts.

src/libtmux_mcp/_wait_policy.py, server.py

New ceiling policy resolved from LIBTMUX_MCP_WAIT_MAX_SECONDS and injected at registration, mirroring the existing history-default pattern. server.py also now fails at startup rather than silently dropping the is_caller agent-context instructions when the byte budget is tight, while still degrading gracefully for pathological runtime env values.

_utils.py, middleware.py, batch_tools.py, wait_for_tools.py

The self-bounded tag and its enforcement: excluded from ReadonlyRetryMiddleware, rejected per-operation in every batch wrapper, and applied to wait_for_channel, whose caller timeout is likewise unclamped.

models.py, state.py

WaitForTextResult reshaped to the outcome enum; ContentChangeResult removed. _PaneState gains alternate_on, read in the existing round-trip and tolerant of older tmux that omits the field.

Docs & recipes

The wait-for-content-change page is removed and its references repointed; the interrupt_gracefully and build_dev_workspace recipes move to the new signature.

Design decisions

  • Thread-free, not just timeout-bounded. A private ThreadPoolExecutor with shutdown(wait=False) was considered and rejected: concurrent.futures.thread._python_exit joins every pool worker untimed regardless, so no thread-based arrangement survives a wedged tmux at interpreter exit. Only owning a killable subprocess does.
  • Kill-then-cancel-then-bounded-reap. The textbook wait_for(communicate()) then kill() then await wait() deadlocks when a wedged tmux's grandchild still holds the stdout pipe. The reader task is torn down first and the final reap is time-boxed; the loop's child watcher reaps the pid regardless.
  • Native resolution reproduces libtmux exactly, including raising MultipleObjectsReturned for a window linked into two sessions — the case a naive list-windows -a plus first-match would silently get wrong.
  • outcome as one enum, not three booleans. Output schema is re-sent on every request, so a field no agent branches on is a permanent token tax.

Verification

No caller pattern reaches a tmux format string (tmux treats #/} structurally):

rg -n 'C/|display-message.*\{pattern' src/libtmux_mcp/tools/pane_tools/wait.py

The wait path uses no worker threads or blocking subprocess calls:

rg -n 'to_thread\(|subprocess\.run\(' src/libtmux_mcp/tools/pane_tools/wait.py

The removed tool and result model are gone from source:

rg -n 'wait_for_content_change|ContentChangeResult' src/ docs -g '!_build'

Test plan

  • uv run ruff format --check ., uv run ruff check ., uv run mypy src tests clean
  • uv run pytest --reruns 0 green (flake-masking reruns disabled)
  • just build-docs succeeds; removing the wait-for-content-change page leaves no broken refs
  • test_wait_path_uses_no_worker_threads — the wait path spawns no threads
  • test_wait_tools_do_not_block_event_loop — the loop survives a genuinely wedged tmux; mutation-tested to fail if a blocking spawn returns
  • test_wait_for_text_bounds_every_tmux_call — every spawned tmux call is deadline-bounded
  • A wedged tmux lets the process exit promptly, and under SIGINT, where the prior code hung indefinitely
  • Native pane resolution verified equivalent to libtmux across all wait_for_text targeting cases, including the linked-window MultipleObjectsReturned trap

tony added 8 commits July 21, 2026 17:10
why: wait_for_text accepts an unbounded caller timeout and nothing in
the server clamps it, so one wrong search pattern can hold an MCP
connection for minutes and stall an agent session.

what:
- Add _wait_policy with LIBTMUX_MCP_WAIT_MAX_SECONDS (default 30 s)
- Hard-clamp the resolved ceiling to [1.0, 120.0] so neither a hostile
  env value nor a programmatic caller can restore unbounded waits
- Resolve in server.py beside the other env knobs and publish via
  _configure_wait_ceiling, mirroring _configure_history_defaults so
  tool modules never import server globals
- Warn and fall back on unparseable values instead of raising
- Cover resolution, clamping, and the inf/nan/garbage cases
why: A tool that owns its own deadline must not have that deadline
multiplied by the framework. ReadonlyRetryMiddleware restarts a
readonly tool once on any LibTmuxException, and because a wait
computes its deadline inside the tool body the clock restarts with it
— a transient failure at t=29 s of a 30 s wait costs ~59 s. The batch
wrappers are worse: max_tier is a ceiling, so a readonly tool is
reachable through all three, 1000 operations deep, with no aggregate
deadline. wait_for_channel has no ceiling at all, making it the
sharpest instance.

what:
- Add TAG_SELF_BOUNDED alongside the existing tier tags
- Skip retry for self-bounded tools; tier gating is unaffected because
  SafetyMiddleware inspects only the three tier tags
- Reject self-bounded tools per operation in _get_allowed_tool_tier so
  the refusal becomes a success=false row and on_error='continue'
  isolation still holds; a pre-loop check would fail the whole batch
- Tag wait_for_channel, whose caller timeout is unclamped
- Use a tag rather than a tool-name list: a name is exactly what
  add_tool_transformation can rename out from under the check
why: A wrong search pattern used to cost the caller's full timeout with
a result that said only "not found" — no reason, no evidence, nothing
to make the next call better. Agents responded by retrying with a
longer timeout, compounding the stall.

what:
- Clamp timeout to the server ceiling and report effective_timeout;
  clamp rather than reject so an over-large request still works
- Route every tmux read in the wait path through
  subprocess.run(timeout=...): libtmux's Popen.communicate() has no
  timeout, and mcp.tool(timeout=...) bounds the coroutine via
  anyio.fail_after, never the worker thread
- Add stop patterns so a known failure marker ends the wait in
  milliseconds instead of burning the budget
- Take patterns as a list; patterns=null waits for any new output and
  subsumes wait_for_content_change, which is removed
- Return the tail actually on screen, whether new output appeared, and
  whether the pattern was already present at entry
- Keep matching in Python: tmux format strings treat # and } as
  structure, so an ordinary pattern silently corrupts sibling fields
- Tag wait_for_text self-bounded; retire the wait-for-content-change
  page and repoint its references
why: The ceiling was still escapable twice over. Pane resolution ran
_resolve_pane bare inside async def, so a wedged tmux server froze the
whole event loop — every concurrent tool call, every MCP ping, and
cancellation itself, because anyio.fail_after needs the loop to run to
fire. Measured: a 50 ms heartbeat task stopped for the full 45 s of the
experiment. Separately, a fixed 5 s per-call cap with two reads per
tick and the deadline checked only at the end of the tick made the true
worst case 40 s, not the advertised 30 s.

what:
- Resolve the pane via asyncio.to_thread under asyncio.wait_for; the
  loop now stays live (max heartbeat gap 0.05 s over 97 beats) and the
  call returns in 5.00 s with a typed error
- Derive each tmux call's timeout from the remaining budget via
  _call_budget, with a 0.25 s floor so a nearly-spent budget cannot
  hand subprocess.run a non-positive timeout
- Anchor start_time before resolution so it counts against the deadline
  and shows up in elapsed_seconds
- Take a pane id in _raise_if_pane_lifecycle_changed: the wait path no
  longer holds a live libtmux object after resolving once

A wedged tmux can still outlive the call in its worker thread, since a
running concurrent.futures task is not cancellable. That is now a held
executor slot rather than a server-wide freeze.
why: The result carried three booleans and a source string that all
described one fact, and outputSchema is re-sent on every request of
every session — a field no agent branches on is a permanent tax. The
alternate screen needed reporting in the same shape, so both land
together rather than designing the model twice.

Measured on tmux 3.7b: inside a pager, capture-pane -J -S returns the
program's painted rows, so a wait for text visible on screen matched
paint and reported new output that was never written. That is a false
accept, which is worse than waiting.

what:
- Replace found/stopped/matched_source/timeout_clamped and friends with
  outcome: matched, any_output, stopped, alternate_screen, timeout
- Surface the catch-all as any_output so an agent that dropped patterns
  under context pressure can see it matched "something moved"
- Read alternate_on in the existing state round-trip; skip matching
  while it is set and never latch, so quitting a pager mid-wait resumes
- Reclassify a timeout that spent any time on the alternate screen:
  "timeout" says the pattern was wrong, which is the wrong fix here
- Scan the whole visible screen for matched_at_entry, not just below
  the cursor: text printed moments ago usually sits above it
- Cap matched_lines on the catch-all path, which could return a whole
  screen dump
- Tolerate a missing alternate_on field so older tmux degrades to
  false instead of failing the hot poll path
why: _build_instructions degraded silently when the 2048-byte budget
was tight, dropping the is_caller workflow — the only place an agent
learns how to answer "which pane am I in?". No test could catch it:
the assertions check total size, and the degraded form still contains
the substring "Agent context". The wait instruction rewrite left the
readonly tier, which is the tightest, with 5 bytes to spare.

what:
- Raise when our own instruction segments crowd out the workflow, so
  the overflow surfaces at startup as a bug to fix here
- Keep degrading when TMUX_PANE or TMUX are pathologically large: that
  is runtime data we do not control, and refusing to start over a
  hostile environment variable would be a denial of service
- Discriminate the two with a nominal pane id rather than by size alone
why: A wedged tmux still hung interpreter shutdown, and Ctrl-C on a
server with a wedged tmux needed SIGKILL. The prior fix bounded the
tool call and kept the event loop live, but the tmux work still ran on
worker threads: _run_tmux_lines via to_thread(subprocess.run), and
pane resolution via to_thread(_resolve_pane). A thread blocked in
Popen.communicate() cannot be cancelled, and
concurrent.futures.thread._python_exit joins every pool worker with no
timeout at interpreter exit — so one wedged tmux hangs the process
forever, after a 300 s pause and a RuntimeWarning from
shutdown_default_executor. No thread-based arrangement escapes that
atexit join, so the only fix is to stop using threads for tmux calls.

what:
- Spawn every wait-path tmux read with asyncio.create_subprocess_exec,
  bounded by asyncio.wait against the call's own deadline
- Kill-then-cancel-then-bounded-reap on timeout or cancellation:
  awaiting proc.wait() after kill() deadlocks when a wedged tmux's
  grandchild still holds the stdout pipe, so the reader task is torn
  down first and the final reap is time-boxed
- Reimplement pane resolution natively over the async subprocess for
  exactly the four targeting args wait_for_text accepts, so no
  to_thread(_resolve_pane) remains; verified equivalent to libtmux
  across every case, including MultipleObjectsReturned for a window
  linked into two sessions
- Replace the thread-offload event-loop test with one that proves the
  loop survives a genuinely wedged tmux, and add a guard asserting the
  wait path spawns no worker threads at all

The process now exits promptly under a wedged tmux instead of hanging.
capture_since still reaches libtmux through to_thread and keeps the
same exposure; that is a separate follow-up.
why: The unreleased wait entries were drafted mid-branch and named a
result shape that later commits changed — matched_source, timeout_clamped
and suppressed_stale_match no longer exist, and the tmux reads no longer
go through subprocess.run. A changelog that describes a removed field
misleads the reader.

what:
- Report outcome/matched_index and matched_at_entry, the fields that
  shipped, and drop the ones that did not
- Add the alternate-screen entry and the wedged-tmux exit fix
- Describe the wait's tmux calls as bounded asyncio subprocesses
@tony

tony commented Jul 22, 2026

Copy link
Copy Markdown
Member Author

Code review

Found 6 issues (1 confirmed by repro):

  1. Cancelling a wait orphans the in-flight tmux child. The cancellation lands at await asyncio.wait({task}, ...), which does not cancel task; the except asyncio.CancelledError only guards the synchronous task.result() below it, so _kill_and_reap never runs. I reproduced it against a wedged tmux: CancelledError propagated in 0.00s but ORPHANED fake-tmux alive: 1, and the leaked child re-introduces the interpreter-shutdown hang this PR set out to fix. The window is largest under a wedged tmux — exactly the target scenario. Wrap the await asyncio.wait(...) in try/except CancelledError (or try/finally) that reaps. (bug)

task = asyncio.ensure_future(proc.communicate())
done, _pending = await asyncio.wait({task}, timeout=budget)
if not done:
await _kill_and_reap(proc, task)
msg = (
f"tmux {args[0]} did not return within "
f"{budget:.2f}s; the tmux server is unresponsive"
)
raise ExpectedToolError(msg)
try:
stdout, stderr = task.result()
except asyncio.CancelledError:
# The whole call was cancelled (MCP client hung up). Tear the
# child down before letting the cancellation through, or tmux
# is orphaned.

  1. interrupt_gracefully passes stop=["^C", ...] with regex=True. As a regex, ^C is start-anchored C: re.search("^C", "^C") is False (it never matches the literal ^C echo it targets) and re.search("^C", "Compiling") is True (it false-fires outcome="stopped" on any line starting with C). Escape it (\^C) or drop regex=True for that marker. (bug)

1. `send_keys(pane_id="{pane_id}", keys="C-c", literal=False,
enter=False)` — tmux interprets `C-c` as SIGINT.
2. `wait_for_text(pane_id="{pane_id}", patterns=["\\$ ", "\\# ", "\\% "],
stop=["^C", "Interrupt"], regex=True, timeout=5.0)` — waits for a
common shell prompt glyph. Adjust the patterns to match the user's
shell theme. The `stop` list exits early on the markers many

  1. build_dev_workspace step 5 tells the agent wait_for_text(patterns=null) confirms "a vim splash screen," but this PR's alternate-screen guard suppresses matching while alternate_on and returns outcome="alternate_screen", found=False. vim uses the alternate screen, so the stated guarantee is now false. (bug)

5. Optionally confirm each program drew its UI via
`wait_for_text(pane_id="%A", patterns=null, timeout=3.0)`
(and similarly for `%C`). Omitting `patterns` makes this a
"did anything new get printed?" check — it works whether the
pane shows a prompt glyph, a vim splash screen, or a log tail,
so no shell-specific regex is needed.

  1. The wait_for_text docstring tells the agent the result reports suppressed_stale_match, but no such field exists — the shipped field is matched_at_entry. (stale doc on a changed line)

below the cursor at entry — only rows written after the call began
count. If a pattern was already on screen the result says so via
``suppressed_stale_match``.

  1. The same docstring states "Alternate screen / pagers are not handled ... this tool does not detect it," which contradicts the alternate-screen handling this PR adds. (stale doc on a changed line)

hang interpreter shutdown.
**Alternate screen / pagers are not handled.** Inside ``less``,
``capture-pane`` returns the pager's painted rows, so a wait for
text the pager has drawn matches on paint. That is a false

  1. The poll-loop comment still says the tmux reads are "blocking subprocess calls" pushed to "the default executor," but this PR makes the path a native async subprocess with no executor and no thread — the comment describes the code it replaced. (stale comment on a changed line)

# FastMCP direct-awaits async tools on the main event loop
# and the tmux reads are blocking subprocess calls. Push
# them to the default executor so concurrent tool calls are
# not starved during long waits.
state = await _bounded_pane_state(server, target, deadline=deadline)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant