Skip to content

fix(server): engine-attested Cloud MCP health; network probe becomes on-demand - #2834

Merged
benjaminshafii merged 3 commits into
devfrom
fix/engine-attested-health
Jul 16, 2026
Merged

fix(server): engine-attested Cloud MCP health; network probe becomes on-demand#2834
benjaminshafii merged 3 commits into
devfrom
fix/engine-attested-health

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Summary

Root-cause fix #2 from the corporate-TLS field incident (follow-up to #2823). The architecture problem: background health re-verified what the engine already attested, over a different network stack — two verifiers, guaranteed eventual disagreement. This PR makes the engine the single authority for background health and demotes the network probe to an on-demand diagnostic.

  • readOpenworkCloudMcpHealth gains probe?: boolean (default false)
  • Default path: no network probe at all; tools attested from engine status (connected ⇒ initial tools/list succeeded, opencode ≥1.17 semantics); tools.direct.checked: false
  • GET …/health?probe=1 → full direct-probe verification (for the Advanced diagnostics Refresh / support)
  • POST …/reconcile keeps probe: true (explicit repair wants full verification)
  • /experimental/connect/state (per-prompt, feeds steering) pinned to probe: false — cheap and engine-attested

Validation

  • pnpm --filter openwork-server exec bun test src/cloud-mcp-health.test.ts src/cloud-mcp-reconcile.e2e.test.ts — 27 pass / 0 fail (new: default path makes zero fetches to the cloud endpoint, tools attested, delivery applied; route-level default vs ?probe=1; existing probe tests opt in via probe:true)
  • pnpm --filter openwork-server test — 453 pass / 0 fail (one earlier run had 2 unrelated reload-watcher flakes that passed on rerun and reproduce on clean dev)
  • pnpm --filter openwork-server typecheck — clean

Note: touches cloud-mcp-health.ts alongside the parallel egress PR (different regions; whichever merges second takes a trivial rebase). App-side follow-up (optional): Advanced diagnostics Refresh can pass probe=1 to keep its deep-verification behavior explicit.

…on-demand

Background health re-verified the engine's MCP over the server's own
network stack on every check — a duplicate verifier that can disagree
with the engine (field incident: corporate TLS). The engine is now the
authority: by default health attests tools from the engine's mcp status
(connected implies initial tools/list succeeded on opencode >=1.17) and
never touches the network. The direct tools/list probe runs only when
explicitly requested (GET health?probe=1) and on reconcile (an explicit
repair action). /experimental/connect/state — consumed per prompt by
the steering plugin — is pinned to probe:false.
@vercel

vercel Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 16, 2026 6:55am
openwork-den Ready Ready Preview, Comment Jul 16, 2026 6:55am
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 16, 2026 6:55am
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 16, 2026 6:55am
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
openwork-diagnostics Skipped Skipped Jul 16, 2026 6:55am

@benjaminshafii

Copy link
Copy Markdown
Member Author

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-16T10-26-33-754Z

Full frame proof with validated screenshots: evals/results/2026-07-16T10-26-33-754Z/fraimz.html (re-run: pnpm fraimz --flow windows-enterprise-ca-rootcause)

✅ windows-enterprise-ca-rootcause — Windows packaged app uses OS trust and engine-attested Cloud MCP readiness

Internal demo

  1. The packaged Windows app reproduces CA trust on/off/on with a restored final state

    🎙 We start with the repeatable control: the same packaged app can prove when the corporate CA is trusted, when it is removed, and when it is restored. The evidence stays focused on the root-cause signals, not secret material or process details.

    • ✅ control artifact ok is true
    • ✅ control artifact recorded an Electron Windows user agent
    • ✅ repro CA count in the Windows system store is greater than zero
    • ✅ repro CA count in default roots is zero
    • ✅ phase A rootCount is 1
    • ✅ phase A healthy request returns 400 openwork_workspace_not_found
    • ✅ phase A broken request returns 502 cert authority invalid
    • ✅ phase C rootCount is 1
    • ✅ phase C healthy request returns 400 openwork_workspace_not_found
    • ✅ phase C broken request returns 502 cert authority invalid
    • ✅ phase B rootCount is 0
    • ✅ phase B formerly healthy request returns 502 cert authority invalid
    • ✅ final state restored exactly one root
    • ✅ final state reports CDP ready
  2. The installed Electron app reports a running OpenWork server and healthy managed OpenCode loopback

    🎙 Next we confirm the real installed app is the one under test. Its Electron bridge reports a running OpenWork server, the expected workspace, a healthy managed OpenCode engine, and a live server-connected event stream.

    • ✅ renderer user agent is Electron on Windows
    • ✅ OpenWork server is running and has a base URL
    • ✅ /workspaces contains ws_c0107cd3f4f6
    • ✅ proxied /workspace/.../opencode/global/health returns HTTP 200
    • ✅ proxied OpenCode health is healthy version 1.17.11
    • ✅ /opencode/event returns a text/event-stream
    • ✅ /opencode/event yields a server.connected frame
  3. Default Cloud MCP health and Connect state stay ready without touching the fixture

    🎙 The default Connect health check now trusts the engine attestation without opening a direct network probe. Fixture counts stay still while the health and Connect state both report ready and usable.

    • ✅ fixture task is running and listening before reconcile
    • ✅ reconcile returns HTTP 200
    • ✅ reconcile health phase is ready
    • ✅ reconcile health is usable
    • ✅ reconcile health engine status is connected
    • ✅ reconcile performs the direct readiness probe
    • ✅ reconcile desired and applied revisions match
    • ✅ default Cloud MCP health returns HTTP 200
    • ✅ default health phase is ready
    • ✅ default health is usable
    • ✅ default health engine status is connected
    • ✅ default health firstFailure is null
    • ✅ default health desired and applied revisions match
    • ✅ default health leaves direct.checked false
    • ✅ /experimental/connect/state returns HTTP 200
    • ✅ Connect state reports Cloud MCP present
    • ✅ Connect state cloudHealth phase is ready
    • ✅ Connect state cloudHealth is usable
    • ✅ Connect state cloudHealth engine status is connected
    • ✅ Connect state cloudHealth firstFailure is null
    • ✅ Connect state cloudHealth desired and applied revisions match
    • ✅ Connect state cloudHealth leaves direct.checked false
    • ✅ fixture JSON-RPC method counts are unchanged by default health and Connect state
  4. health?probe=1 performs exactly one MCP initialize/initialized/tools-list handshake

    🎙 When the reviewer asks for an explicit probe, the direct MCP handshake happens exactly once. The OS-trusted path lists both cloud tools and the fixture count advances by initialize, initialized notification, and tools/list.

    • ✅ probe Cloud MCP health returns HTTP 200
    • ✅ probe health phase is ready
    • ✅ probe health is usable
    • ✅ probe health engine status is connected
    • ✅ probe health firstFailure is null
    • ✅ probe health desired and applied revisions match
    • ✅ probe health sets direct.checked true
    • ✅ direct tools/list includes search_capabilities
    • ✅ direct tools/list includes execute_capability
    • ✅ probe health has no direct failure
    • ✅ fixture method delta for initialize is exactly 1
    • ✅ fixture method delta for notifications/initialized is exactly 1
    • ✅ fixture method delta for tools/list is exactly 1
  5. A direct probe outage reports probe_unreachable without overriding engine-connected readiness

    🎙 If that diagnostic probe cannot reach the fixture, the app still reports the engine truth as ready and usable. The direct failure is clearly diagnostic, and the flow restarts the fixture before it exits.

    • ✅ fixture listener is down after stopping only the scheduled task
    • ✅ probe failure health still returns HTTP 200
    • ✅ probe failure health phase is ready
    • ✅ probe failure health is usable
    • ✅ probe failure health engine status is connected
    • ✅ probe failure health firstFailure is null
    • ✅ probe failure health desired and applied revisions match
    • ✅ probe failure leaves direct.checked false
    • ✅ probe failure code is probe_unreachable
    • ✅ probe failure reports no missing direct tools
    • ✅ fixture listener is restored
    • ✅ fixture scheduled task is running after restart
  6. The installed packaged steering smoke keeps engine-ready truth authoritative across all cases

    🎙 Finally we verify packaged steering consumes the same engine truth. Ready, auth-needed, thrown, unknown, and missing-entry cases choose the expected guidance without unnecessary server fetches.

    • ✅ packaged steering smoke result ok is true
    • ✅ all five packaged steering cases are present
    • ✅ connected ready case has zero server fetches and no repair guidance
    • ✅ needs_auth case gives sign-in guidance with zero server fetches
    • ✅ thrown case stays neutral with zero server fetches
    • ✅ unknown case stays neutral with zero server fetches
    • ✅ missing entry case performs exactly one fallback
    • ✅ result includes an installed bundle hash or an installed resources artifact path
    • ✅ installed bundle hash is present
    • ✅ Script frame 1 narrated: "We start with the repeatable control: the same packaged app can prove when the corporate"
    • ✅ Script frame 2 narrated: "Next we confirm the real installed app is the one under test. Its Electron bridge report"
    • ✅ Script frame 3 narrated: "The default Connect health check now trusts the engine attestation without opening a dir"
    • ✅ Script frame 4 narrated: "When the reviewer asks for an explicit probe, the direct MCP handshake happens exactly o"
    • ✅ Script frame 5 narrated: "If that diagnostic probe cannot reach the fixture, the app still reports the engine trut"
    • ✅ Script frame 6 narrated: "Finally we verify packaged steering consumes the same engine truth. Ready, auth-needed, "

@benjaminshafii
benjaminshafii merged commit 073f094 into dev Jul 16, 2026
11 checks passed
@benjaminshafii
benjaminshafii deleted the fix/engine-attested-health branch July 16, 2026 14:55
@benjaminshafii

Copy link
Copy Markdown
Member Author

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-16T16-10-41-638Z

Full frame proof with validated screenshots: evals/results/2026-07-16T16-10-41-638Z/fraimz.html (re-run: pnpm fraimz --flow windows-enterprise-ca-error-state)

✅ windows-enterprise-ca-error-state — Windows enterprise CA error state: old build fabricates outage, fixed build trusts engine truth

Internal demo

  1. The old build reports cloud_tools_missing even though the engine and Schannel endpoint are healthy

    🎙 We begin on the same machine before the fix: the engine is connected and Schannel can list the tools, but OpenWork reports a fabricated Cloud tools outage. This frame captures the exact contradiction that caused the model-facing error state.

    • ✅ pre-fix artifact user agent contains OpenWork/0.17.29
    • ✅ engine openwork-cloud status is connected
    • ✅ direct Schannel tools/list contrast returned HTTP 200
    • ✅ direct Schannel tools/list includes both expected tools
    • ✅ pre-fix default health phase is cloud_tools_missing
    • ✅ pre-fix default health is not usable
    • ✅ pre-fix first failure is cloud_tools_missing
    • ✅ pre-fix direct error contains fetch failed
    • ✅ pre-fix delivery appliedRevision is null
    • ✅ connect/state cloudHealth also reports cloud_tools_missing
  2. The old steering converts the fabricated outage into repair/refusal guidance

    🎙 The old steering then turns that fabricated outage into user-facing refusal guidance. The model is told the Cloud connection is not ready and to send the user to Repair and test.

    • ✅ pre-fix instruction says the connection is not ready
    • ✅ pre-fix instruction sends the user to Repair and test
    • ✅ pre-fix instruction names cloud_tools_missing
  3. The fixed build reports ready by default and the live app agrees

    🎙 Now the fixed build runs on the same machine, same corporate CA, and same endpoint without reconciling a new config. Default health trusts the engine, the fixture stays untouched, and the live app reports the same ready state.

    • ✅ fixed artifact user agent contains OpenWork/0.17.30
    • ✅ fixed artifact did not perform reconcile
    • ✅ fixed default health phase is ready
    • ✅ fixed default health is usable
    • ✅ fixed default health engine status is connected
    • ✅ fixed default health does not run a direct probe
    • ✅ fixed default health firstFailure is null
    • ✅ fixed default health desired and applied revisions match
    • ✅ fixed default-health fixture delta is zero
    • ✅ both before and fixed exe SHA256 prefixes are recorded
    • ✅ live fixed app default health request returns HTTP 200
    • ✅ live fixed app reports ready usable engine-connected default health
  4. The fixed build probes only on demand and steers the model toward Cloud tools

    🎙 Finally, the fixed build still supports an explicit diagnostic probe on demand. The probe performs one handshake, Connect remains ready, and steering tells the model to use the Cloud tools instead of refusing.

    • ✅ fixed explicit probe sets direct.checked true
    • ✅ fixed explicit probe includes both direct tools
    • ✅ fixed explicit probe delta is exactly one initialize, initialized notification, and tools/list
    • ✅ fixed connect/state remains ready and usable
    • ✅ fixed steering reports ready Cloud tooling
    • ✅ fixed steering fetchCount is zero
    • ✅ fixed steering omits not-ready and Repair and test guidance
    • ✅ Script frame 1 narrated: "We begin on the same machine before the fix: the engine is connected and Schannel can li"
    • ✅ Script frame 2 narrated: "The old steering then turns that fabricated outage into user-facing refusal guidance. Th"
    • ✅ Script frame 3 narrated: "Now the fixed build runs on the same machine, same corporate CA, and same endpoint witho"
    • ✅ Script frame 4 narrated: "Finally, the fixed build still supports an explicit diagnostic probe on demand. The prob"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minor issue in README: audience description is an incorrect generalization

1 participant