perf: stop tracing every command-path probe and bound git spawns

[arubi9]

Problem

On a Windows host with several projects open, the desktop app became unusable for tens of seconds at a time: Timed out while checking Codex app-server provider status, Git command timed out warnings, and a client stuck in Failed to connect. Reconnecting… because /.well-known/t3/environment exceeded its 10s connection-setup budget.

Tracing pointed at two independent causes.

1. A span per command-path probe

resolveCommandPathForPlatform walks every PATH entry against every PATHEXT candidate, and isExecutableFile was an Effect.fn — one span per probe. With a 57-entry PATH and 14 PATHEXT entries that is ~800 probes (~1600 with the lower-case variants) per lookup.

One 13.7s trace window contained 28,698 spans, of which 28,577 were shell.isExecutableFile — produced by just 22 resolveCommandPath calls. The trace file rotated at 10 MB every ~40s. Serializing that volume starved the event loop; unrelated fibers then blew past their own timeouts (a discoverClaudeSkills span read 79s while the work it does measures ~93ms).

Editor detection and provider health checks re-resolve the same commands on every refresh, and uninstalled CLIs (cursor-agent, grok here) pay the full miss every time.

Fix: make the per-candidate probe untraced, and cache resolution outcomes — hits and misses — for 30s, keyed on platform, command, PATH and PATHEXT. The TTL rather than a permanent cache so a newly installed CLI is still discovered.

Resolving 21 commands, five times, on the real PATH:

	1	2	3	4	5
before	3809ms	3566	3700	3369	3401
after	3264ms	1	1	4	1

2. Unbounded git spawns

A workspace refresh requests refs and status for every project and worktree at once, and each listRefs spawns six git processes (branchNoColor, defaultRef, remoteBranches, remoteNames, worktreeList, readBranchRecency). VcsProcess has timeouts but no concurrency bound.

One observed refresh issued 1,055 git spawns. Commands that take ~0.9s in isolation (and stay under 500ms at 12-way concurrency) took 6.5s under that load; ws.rpc.vcs.listRefs spans ran 51s, and the environment endpoint took 14.2s — past the client's 10s budget, so it reconnected and re-triggered the storm.

Fix: gate the spawn behind a semaphore sized from availableParallelism() (4–16 permits). The permit is taken around the spawn only, so queue time never counts against the command's own timeout.

Verification

Both fixes were built into a desktop artifact and run against the same workload that produced the traces above:

	before	after
shell.isExecutableFile spans	28,577 per 13.7s	0
/.well-known/t3/environment	14,215ms	76–393ms typical
git spawn concurrency	unbounded	capped

Tests:

• packages/shared/src/shell.test.ts — resolve, delete the file, resolve again (still cached), clearCommandPathCache(), resolve fails.
• apps/server/src/vcs/VcsProcess.test.ts — 40 competing fibers; peak in-flight stays within the bound. Verified to fail without the semaphore (expected 40 to be 8).
• vp test apps/server/src/vcs — 71 passed.
• vp run typecheck and vp check clean.

Six pre-existing Windows test failures (relayClient ×4, logging, t3-sqlite-state) fail identically on an unmodified checkout and are unrelated.

Not addressed

The refresh still issues a lot of git work — 384 commands in one window, ~145 of them repeated metadata lookups (readConfigValue, resolveCurrentUpstream, listRemoteNames, resolveDefaultBranchName). Caching those, or fetching refs only for visible projects, would reduce the work rather than just bounding it. Happy to follow up if that direction seems right.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches shared spawn/path resolution and global VCS concurrency; mis-tuning could delay refreshes or briefly serve stale “command not found” until cache TTL expires.

Overview
Addresses event-loop starvation that caused provider health checks, HTTP endpoints, and git commands to time out under heavy refresh load.

Command path resolution (shell.ts) stops emitting a trace span on every PATH/PATHEXT file probe by making isExecutableFile and the caching wrapper untraced. Adds a 30s TTL cache (platform, command, PATH, PATHEXT) for both hits and misses, plus clearCommandPathCache() for tests.

VCS process spawning (VcsProcess.ts) wraps each processRunner.run in a global semaphore (4–16 permits from availableParallelism() / 4) so workspace refreshes cannot launch unbounded git processes; queue time does not count toward per-command timeouts.

Tests cover cache reuse after the binary is removed and that 40 concurrent runs stay at or below the spawn cap.

^{Reviewed by Cursor Bugbot for commit dc7f172. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Cache command path resolutions and bound concurrent git process spawns

• resolveCommandPathForPlatform in shell.ts now caches results (hit or not-found) for 30s per (platform, command, PATH, PATHEXT) tuple, avoiding repeated filesystem probes. A new clearCommandPathCache() export forces re-resolution.
• isExecutableFile is changed to use Effect.fnUntraced to avoid generating a trace span on every probe call.
• VcsProcess.run in VcsProcess.ts now acquires a semaphore permit before spawning a child process, capping concurrency at max(4, min(16, floor(parallelism/4))) based on NodeOS.availableParallelism().
• Behavioral Change: git spawns that exceed the concurrency cap now queue and wait for a permit rather than spawning immediately.

📊 Macroscope summarized dc7f172. 2 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e0d36352-1f7c-4abb-a2af-ad1e3ddf5615

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

🟡 Medium src/shell.ts:582

resolveCommandPathForPlatform caches results for commands containing / or \ without including the current working directory in the cache key. Those commands are resolved relative to process.cwd(), so if the process changes directories within the 30-second TTL, the cache returns a path from the previous directory (or reuses a cached miss), yielding an incorrect or unusable resolution. Include process.cwd() in the cache key so entries don't leak across directory changes.

🤖 Copy this AI Prompt to have your agent fix this:

In file @packages/shared/src/shell.ts around line 582:

`resolveCommandPathForPlatform` caches results for commands containing `/` or `\` without including the current working directory in the cache key. Those commands are resolved relative to `process.cwd()`, so if the process changes directories within the 30-second TTL, the cache returns a path from the previous directory (or reuses a cached miss), yielding an incorrect or unusable resolution. Include `process.cwd()` in the cache key so entries don't leak across directory changes.

[macroscopeapp]

Approvability

Verdict: Needs human review

1 blocking correctness issue found. This performance optimization adds process concurrency limiting and command path caching. An unresolved review comment identifies a potential caching bug where relative command paths may resolve incorrectly if the working directory changes within the 30-second cache TTL.

^{You can customize Macroscope's approvability policy. Learn more.}

[jdpt0]

I can confirm, 0.0.29 is currently completely unusable on Windows. Seemingly random disconnects from the local server, timeouts stemming from the t3 environment file.

[haye-go]

very severe, t3code on windows is virtually almost unusable due to this git call storm. these stats are for just working on 1 thread on 1 project.

Operation	Observed	Effect
orchestration.subscribeThread	10 per WebSocket generation	Each independently scans the global 1.8-million-row event stream
subscribeVcsStatus	21 per generation	Duplicate UI consumers; the server attempts to coalesce these by checkout
vcs.listRefs	4,741 RPCs in the sampled run	Main overload source
Git child spans	28,801	28,784 targeted one checkout
WebSocket generations	At least 5	Every reconnect recreates the subscription set

--

Each vcs.listRefs call expands into approximately six concurrent Git commands:
branch enumeration
remote branch enumeration
remote-name lookup
default-ref lookup
worktree listing
branch-recency lookup

21 polls / 5 seconds ≈ 4.2 RPCs/second
4.2 × 6 Git commands ≈ 25 Git commands/second

---

update:

#4727 successfully eliminated the runaway ref-polling/OOM mechanism.

However, the remaining run shows:

• 30,313 shell.isExecutableFile spans
• 328 Git commands spread across many repositories
• 63 VCS-status subscriptions over three connection generations
• Continued reconnects despite normal memory usage

That maps directly to [PR #4778](#4778), which:

• stops tracing every Windows PATH/PATHEXT executable probe;
• caches command-path resolutions for 30 seconds;
• globally limits concurrent Git processes to 4–16.

Therefore:

• Fix Git ref refresh resource storms #4727 fixes the original catastrophic Git-ref storm.
• perf: stop tracing every command-path probe and bound git spawns #4778 is still needed to address the remaining Windows backend starvation.
• They are complementary, not alternatives.
• perf: stop tracing every command-path probe and bound git spawns #4778 is currently open and was based on code from before Fix Git ref refresh resource storms #4727 merged, so it still needs integration/rebase testing.