Skip to content

fix(server): cloud MCP probe transport failures fail open, never manufacture an outage - #2823

Merged
benjaminshafii merged 1 commit into
devfrom
fix/cloud-probe-fail-open
Jul 15, 2026
Merged

fix(server): cloud MCP probe transport failures fail open, never manufacture an outage#2823
benjaminshafii merged 1 commit into
devfrom
fix/cloud-probe-fail-open

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Summary

Root-cause fix for the field incident where a healthy Cloud MCP appeared broken on a corporate Windows machine and the agent refused to use its tools.

The chain: the health check's direct tools/list probe uses Node fetch (undici), which lacks the OS trust store — behind corporate TLS interception it throws fetch failed. That transport error was misclassified as cloud_tools_missing, which (a) flipped usable to false, (b) kept delivery.appliedRevision at null forever, and (c) fed the extensions-preview steering plugin the degraded per-prompt instruction ('OpenWork Cloud agent access is not ready… Direct the user to Settings → Connect → Repair and test'). The model obeyed our own steering and refused MCP tools that the engine (Bun, OS trust store) had connected and could call the whole time — verified in the field by running the same engine standalone (no OPENWORK_SERVER_URL env → neutral steering → tool calls succeeded against the same endpoint, same bearer, same workspace).

Changes

  • cloud-mcp-health.ts: transport-thrown probes now produce probe_unreachable (new failure code), kept diagnostic-only when the engine reports the MCP connected — the engine is authoritative; usable/phase/delivery now reflect engine reality. Probe failures remain visible in the health JSON (tools.direct.failure) for support.
  • Probe fetch prefers Electron net.fetch (Chromium stack, OS trust) when embedded in the desktop app; falls back to global fetch standalone/tests.
  • extensions-preview-steering.ts: fail-open guard — engine connected + probe-side failure (probe_unreachable/cloud_tools_missing) returns the neutral instruction, never the degraded 'Repair and test' steering. A probe on a different network stack must never talk the model out of tools the engine owns.

Validation

  • pnpm --filter openwork-server exec bun test src/cloud-mcp-health.test.ts src/opencode-plugins/openwork-extensions-preview-connect-steering.test.ts — 20 pass / 0 fail (new: transport-throw → usable:true + phase:ready + delivery applied + probe_unreachable surfaced; 401 → invalid_mcp_token unchanged; genuine missing tools → cloud_tools_missing unchanged; steering fail-open matrix)
  • pnpm --filter openwork-server test — full suite 450 tests, 0 fail, 7 skip
  • pnpm --filter openwork-server typecheck — clean

Not run: end-to-end fraimz on a corporate-TLS Windows machine — reproducing the MITM trust condition needs the Daytona Windows cert sandbox (daytona-windows-cert skill) or the affected field machine; unit fixtures replicate the exact observed failure mode instead. Field verification plan: install on the affected machine → Advanced diagnostics should show usable with probe_unreachable noted under direct tools → the previously refused prompt ('call openwork-cloud_search_capabilities…') should execute without any reconnect/reset.

Reviewer repro

  1. Stub the direct probe fetch to throw + engine mcp status connected → health is usable, delivery applied, probe failure diagnostic-only (see new tests).
  2. composeOpenWorkExtensionDiscoveryInstruction with engine connected + cloud_tools_missing → neutral instruction (previously degraded).

…facture an outage

On corporate Windows machines with TLS interception, the health check's
direct tools/list probe (Node undici, no OS trust store) threw 'fetch
failed' and was misclassified as cloud_tools_missing — flipping usable
to false, blocking delivery from being marked applied, and worst,
feeding the extensions-preview steering plugin a per-prompt instruction
telling the model Cloud agent access is not ready. The model then
refused MCP tools that the engine (Bun, OS trust) had registered,
connected, and could call the whole time: a false-negative probe
manufactured a real outage.

- transport-thrown probes now report probe_unreachable (diagnostic
  only): when the engine says the MCP is connected they no longer join
  failures, so usable/phase/delivery reflect the engine's reality
- the probe now prefers Electron net.fetch (Chromium, OS trust store)
  when embedded, falling back to global fetch elsewhere
- steering fails open: engine connected + probe-side failure returns
  the neutral instruction, never 'Direct the user to Repair and test'
- direct probe failures stay visible in the health JSON for support
@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 15, 2026 11:05pm
openwork-den Ready Ready Preview, Comment Jul 15, 2026 11:05pm
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 15, 2026 11:05pm
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 15, 2026 11:05pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
openwork-diagnostics Skipped Skipped Jul 15, 2026 11:05pm

@benjaminshafii

Copy link
Copy Markdown
Member Author

Live before/after on a real stack (real OpenWork server via bun src/cli.ts + real managed opencode sidecar + a mock cloud MCP endpoint that serves the engine's MCP client normally but kills the health probe's connection at transport level — reproducing the corporate-TLS failure mode without TLS):

dev (pre-fix) this PR
usable false true
phase cloud_tools_missing ready
firstFailure cloud_tools_missing null
engine.status connected connected
delivery failed, appliedRevision: null ready, applied == desired
probe error misreported as missing tools probe_unreachable under tools.direct.failure, transport error preserved
/experimental/connect/state (steering input) drives the degraded 'not ready → Repair and test' instruction usable:true → 'verified ready' instruction

The pre-fix column matches the field diagnostic from the affected machine byte-for-byte. Mock endpoint distinguishes callers by MCP initialize clientInfo.name (openwork-server-cloud-mcp-health vs opencode); its log shows the split directly:

served initialize for opencode
served tools/list
KILLED probe connection (clientInfo: openwork-server-cloud-mcp-health)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant