Skip to content

Fix TUI API recovery races#240

Merged
TraderSamwise merged 4 commits into
masterfrom
fix/tui-connection-resilience-next
Jun 24, 2026
Merged

Fix TUI API recovery races#240
TraderSamwise merged 4 commits into
masterfrom
fix/tui-connection-resilience-next

Conversation

@TraderSamwise

@TraderSamwise TraderSamwise commented Jun 24, 2026

Copy link
Copy Markdown
Owner

Summary

  • prevent stale TUI API failures from degrading a newer healthy connection
  • retry guarded auto-repair after cooldown instead of suppressing the same state permanently
  • validate runtime guard health against the current project state dir and classify wrong-project endpoints as repairable stale state
  • stop successful dashboard model refreshes from triggering guard probes unless a guard is currently active

Verification

  • yarn typecheck
  • yarn lint
  • yarn build
  • yarn vitest --run src/multiplexer/tui-api-runtime.test.ts src/multiplexer/dashboard-control.test.ts src/multiplexer/dashboard-model-service.test.ts src/multiplexer/runtime-guard.test.ts
  • yarn vitest --run

Summary by CodeRabbit

  • Bug Fixes
    • Improved runtime guard detection so services with mismatched identity are now marked as stale instead of healthy.
    • Made dashboard recovery and repair more resilient, including retry behavior after a cooldown and better handling when status changes back to normal.
    • Prevented older failed requests from overwriting newer successful ones, reducing unnecessary connection-state downgrades.

@vercel

vercel Bot commented Jun 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
app Ready Ready Preview, Comment Jun 24, 2026 7:49am

@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@TraderSamwise, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 39 minutes and 15 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 309d6ad4-edb6-4ca0-bd87-33b659c4bb12

📥 Commits

Reviewing files that changed from the base of the PR and between 892e595 and 2a56b78.

📒 Files selected for processing (4)
  • src/multiplexer/dashboard-control.test.ts
  • src/multiplexer/dashboard-control.ts
  • src/multiplexer/tui-api-runtime.test.ts
  • src/multiplexer/tui-api-runtime.ts
📝 Walkthrough

Walkthrough

The PR adds three coordinated improvements to the runtime guard and TUI API runtime: a projectStateDir-based service identity mismatch check in probeRuntimeGuard; a 5-second timed cooldown gate before retrying a failed startRuntimeGuardRepair; request-generation counters in TuiApiRuntime to prevent stale out-of-order failures from degrading a newer success; and conditional TUI recovery scheduling gated on runtime guard state.

Changes

Runtime guard identity, repair cooldown, and out-of-order gating

Layer / File(s) Summary
Service identity mismatch detection in runtime guard
src/multiplexer/runtime-guard.ts, src/multiplexer/runtime-guard.test.ts
RuntimeGuardInput gains optional serviceIdentityMismatch?: boolean; evaluateRuntimeGuard returns { kind: "stale", reason: "service-mismatch" } when set; probeRuntimeGuard compares json.projectStateDir from /health against getProjectStateDirFor(projectRoot) and passes the result through. Tests add the mismatch case for both evaluateRuntimeGuard and probeRuntimeGuard, and update all existing mocked health payloads to include projectStateDir.
Timed retry gate for runtime guard repair
src/multiplexer/dashboard-control.ts, src/multiplexer/dashboard-control.test.ts
Adds RUNTIME_GUARD_REPAIR_RETRY_MS = 5_000 and a runtimeGuardRepairRetryReady helper; startRuntimeGuardRepair exits early within the cooldown window, records runtimeGuardRepairRetryAt on failure, and clears it on success and when refreshRuntimeGuard resolves to ok. Tests replace the old "no re-spawn after failure" scenario with a fake-timer cooldown assertion and update mocked health payloads with projectStateDir.
Conditional TUI recovery scheduling
src/multiplexer/dashboard-model.ts, src/multiplexer/dashboard-model-service.test.ts
scheduleTuiApiRecovery is now called after applyDashboardModel only when host.runtimeGuardState.kind is present and not "ok". Tests assert refreshRuntimeGuard is not invoked when the host is not currently guarded and update the forced-refresh expectation accordingly.
Out-of-order request generation tracking in TuiApiRuntime
src/multiplexer/tui-api-runtime.ts, src/multiplexer/tui-api-runtime.test.ts
Adds requestGeneration and lastSuccessfulRequestGeneration counters; requestJson, mutateJson, and refresh each capture a per-call generation, set lastSuccessfulRequestGeneration on success, and only degrade state or invoke onRequestFailure when the failing generation is at or after the last successful one. runScheduledTuiApiRecovery now catches refreshRuntimeGuard exceptions and stores them on tuiApiRecoveryLastError while re-flagging recovery as pending. Tests verify the out-of-order safety and the recovery retry behavior.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

  • TraderSamwise/aimux#218: Touches the same tui-api-runtime.ts and dashboard-control.ts runtime guard/repair machinery that this PR extends with generation tracking and repair cooldown.
  • TraderSamwise/aimux#221: Modifies the same startRuntimeGuardRepair success/failure lifecycle in dashboard-control.ts that this PR augments with the timed retry gate.
  • TraderSamwise/aimux#233: Overlaps with the same scheduleTuiApiRecovery/refreshRuntimeGuard integration in dashboard-model.ts and tui-api-runtime.ts that this PR makes conditional and hardens.

Poem

🐇 A stale guard once fooled me with a mismatched dir,
Now projectStateDir tells me who's really in there!
Old failures cool down — five seconds, no sooner,
And out-of-order flops can't wreck a newer.
Recovery retries, generation in hand —
This bunny's runtime guard stands firm on the land! 🌿

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main theme: fixing race conditions in TUI API recovery and related guarded retry behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/tui-connection-resilience-next

Comment @coderabbitai help to get the list of available commands.

@TraderSamwise

Copy link
Copy Markdown
Owner Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
src/multiplexer/tui-api-runtime.test.ts (1)

93-156: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Add a regression for late older successes rewinding the generation marker.

The current cases miss this interleaving: newer success resolves, then an older success resolves, then another older failure resolves. That would catch the high-water mark regression in lastSuccessfulRequestGeneration.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/multiplexer/tui-api-runtime.test.ts` around lines 93 - 156, Add a
regression test in TuiApiRuntime that covers the missing interleaving where a
newer request succeeds, then an older request succeeds, then an even older
request fails, and verify the runtime still stays connected. Use the existing
TuiApiRuntime helpers and patterns from requestJson/mutateJson/refreshJson plus
getConnectionState and onRequestFailure to assert that a late older success does
not rewind lastSuccessfulRequestGeneration or allow the later failure to degrade
state.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/multiplexer/runtime-guard.ts`:
- Around line 156-158: The health-check handling in runtime-guard.ts is treating
a missing projectStateDir as a mismatch because it compares the field
unconditionally in the serviceManifest assignment block. Update the logic in the
/health response handling so serviceIdentityMismatch is only computed when
json.projectStateDir is present, using the existing
getProjectStateDirFor(projectRoot) comparison only for newer services that
include that field.

In `@src/multiplexer/tui-api-runtime.ts`:
- Line 108: Keep lastSuccessfulRequestGeneration as a high-water mark in
tui-api-runtime so it never decreases on older late successes. Update the
assignments in the request success paths (the spots around the success handling
in the affected methods) to only set this field when the new generation is
greater than the current value, preserving monotonic behavior and preventing
older failures from passing the <= gate.

---

Nitpick comments:
In `@src/multiplexer/tui-api-runtime.test.ts`:
- Around line 93-156: Add a regression test in TuiApiRuntime that covers the
missing interleaving where a newer request succeeds, then an older request
succeeds, then an even older request fails, and verify the runtime still stays
connected. Use the existing TuiApiRuntime helpers and patterns from
requestJson/mutateJson/refreshJson plus getConnectionState and onRequestFailure
to assert that a late older success does not rewind
lastSuccessfulRequestGeneration or allow the later failure to degrade state.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bfd5aa8f-895d-45ae-885c-1ea1db86478b

📥 Commits

Reviewing files that changed from the base of the PR and between e51857e and 892e595.

📒 Files selected for processing (8)
  • src/multiplexer/dashboard-control.test.ts
  • src/multiplexer/dashboard-control.ts
  • src/multiplexer/dashboard-model-service.test.ts
  • src/multiplexer/dashboard-model.ts
  • src/multiplexer/runtime-guard.test.ts
  • src/multiplexer/runtime-guard.ts
  • src/multiplexer/tui-api-runtime.test.ts
  • src/multiplexer/tui-api-runtime.ts

Comment on lines +156 to +158
if (status >= 200 && status < 300 && json?.pid === endpoint.pid && json?.serviceInfo) {
serviceManifest = json.serviceInfo;
serviceIdentityMismatch = json.projectStateDir !== getProjectStateDirFor(projectRoot);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
rg -nP 'projectStateDir' src/metadata-server.ts src/paths.ts
ast-grep run --pattern 'send($_, 200, { $$$ })' --lang typescript src/metadata-server.ts

Repository: TraderSamwise/aimux

Length of output: 14005


🏁 Script executed:

#!/bin/bash
set -euo pipefail

ast-grep outline src/multiplexer/runtime-guard.ts --view expanded
printf '\n--- runtime-guard excerpt ---\n'
sed -n '130,180p' src/multiplexer/runtime-guard.ts
printf '\n--- health response type search ---\n'
rg -n "projectStateDir|serviceInfo|/health|health" src -g'*.ts'

Repository: TraderSamwise/aimux

Length of output: 50375


🏁 Script executed:

#!/bin/bash
set -euo pipefail

ast-grep outline src/multiplexer/runtime-guard.ts --view expanded
printf '\n--- runtime-guard excerpt ---\n'
sed -n '130,180p' src/multiplexer/runtime-guard.ts
printf '\n--- health response type search ---\n'
rg -n "projectStateDir|serviceInfo|/health|health" src -g'*.ts'

Repository: TraderSamwise/aimux

Length of output: 50375


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- health route ---'
sed -n '1788,1812p' src/metadata-server.ts
printf '\n%s\n' '--- runtime-guard tests around projectStateDir ---'
sed -n '200,240p' src/multiplexer/runtime-guard.test.ts
printf '\n%s\n' '--- runtime-guard state handling ---'
sed -n '1,120p' src/multiplexer/runtime-guard.ts
printf '\n%s\n' '--- health contract type ---'
sed -n '280,310p' src/project-api-contract.ts

Repository: TraderSamwise/aimux

Length of output: 8021


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- health route ---'
sed -n '1788,1812p' src/metadata-server.ts
printf '\n%s\n' '--- runtime-guard tests around projectStateDir ---'
sed -n '200,240p' src/multiplexer/runtime-guard.test.ts
printf '\n%s\n' '--- runtime-guard state handling ---'
sed -n '1,120p' src/multiplexer/runtime-guard.ts
printf '\n%s\n' '--- health contract type ---'
sed -n '280,310p' src/project-api-contract.ts

Repository: TraderSamwise/aimux

Length of output: 8021


Guard missing projectStateDir before comparing.
/health includes this field in the current build, but older services can omit it. As written, undefined !== getProjectStateDirFor(projectRoot) turns that into a false service-mismatch; only compare when the field is present.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/multiplexer/runtime-guard.ts` around lines 156 - 158, The health-check
handling in runtime-guard.ts is treating a missing projectStateDir as a mismatch
because it compares the field unconditionally in the serviceManifest assignment
block. Update the logic in the /health response handling so
serviceIdentityMismatch is only computed when json.projectStateDir is present,
using the existing getProjectStateDirFor(projectRoot) comparison only for newer
services that include that field.

Comment thread src/multiplexer/tui-api-runtime.ts Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant