Skip to content

fix(plugins): steer Cloud tool availability from the engine's own MCP state - #2832

Merged
benjaminshafii merged 3 commits into
devfrom
fix/steering-engine-truth
Jul 16, 2026
Merged

fix(plugins): steer Cloud tool availability from the engine's own MCP state#2832
benjaminshafii merged 3 commits into
devfrom
fix/steering-engine-truth

Conversation

@benjaminshafii

@benjaminshafii benjaminshafii commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

Root-cause fix #1 from the corporate-TLS field incident (follow-up to #2823) plus removal of degraded steering entirely (maintainer decision after a second live incident where a false server verdict suppressed Cloud tool use while the engine was connected).

Steering now has exactly four outcomes — ready / sign-in / disabled / neutral — and is derived from the engine's own MCP state, in-process (client.mcp.status, directory-scoped):

  • connected → ready instruction (the only instruction that tells the model to call openwork-cloud_search_capabilities / execute_capability)
  • needs_auth / needs_client_registration → sign-in
  • disabled → disabled
  • failed / unknown / engine error → neutral (fail open — never tell the model tools are unavailable based on a derived opinion)
  • Engine has no openwork-cloud entry → server /experimental/connect/state fallback, mapped with the same four outcomes; every non-ready/sign-in/disabled health state → neutral
  • OPENWORK_CONNECT_DEGRADED_INSTRUCTION ("not ready… Repair and test") is deleted; no steering output can ever instruct against tool use. The fix(server): cloud MCP probe transport failures fail open, never manufacture an outage #2823 probe_unreachable/cloud_tools_missing special-case is subsumed and removed.

Validation

  • pnpm --filter openwork-server exec bun test src/opencode-plugins/ — 26 pass / 0 fail (all four mappings; connected → ready with zero server fetches; failed/unknown/thrown → neutral; regression test iterates every composed output asserting no 'not ready' / anti-tool-use text)
  • pnpm --filter openwork-server test — 452 pass / 7 skip / 0 fail; typecheck clean
  • Windows enterprise-CA e2e proof (packaged app, real engine, fake corporate CA): see fraimz comment below — packaged steering smoke validated engine-first selection with zero server fetches for engine-known states

Parallel siblings: #2834 (engine-attested health), #2835 (OS-trust egress).

… state

The extensions-preview steering previously phoned home to the OpenWork
server and asserted the server's health verdict to the model. Field
incident: that verdict can disagree with the engine (corporate TLS broke
the server's probe while the engine's MCP was connected), and the model
obeyed the wrong claim — refusing tools it actually had.

Invariant now enforced: tool-availability steering is computed from the
same process that owns the tools. The plugin asks the engine's mcp
status in-process (connected → ready, disabled → disabled, needs_auth →
sign-in, failed → degraded), falls back to server connect-state only
when the engine has no entry at all, and fails open to the neutral
instruction on any engine error.
@vercel

vercel Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 16, 2026 2:53pm
openwork-den Ready Ready Preview, Comment Jul 16, 2026 2:53pm
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 16, 2026 2:53pm
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 16, 2026 2:53pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
openwork-diagnostics Skipped Skipped Jul 16, 2026 2:53pm

@benjaminshafii

Copy link
Copy Markdown
Member Author

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-16T10-25-58-114Z

Full frame proof with validated screenshots: evals/results/2026-07-16T10-25-58-114Z/fraimz.html (re-run: pnpm fraimz --flow windows-enterprise-ca-rootcause)

✅ windows-enterprise-ca-rootcause — Windows packaged app uses OS trust and engine-attested Cloud MCP readiness

Internal demo

  1. The packaged Windows app reproduces CA trust on/off/on with a restored final state

    🎙 We start with the repeatable control: the same packaged app can prove when the corporate CA is trusted, when it is removed, and when it is restored. The evidence stays focused on the root-cause signals, not secret material or process details.

    • ✅ control artifact ok is true
    • ✅ control artifact recorded an Electron Windows user agent
    • ✅ repro CA count in the Windows system store is greater than zero
    • ✅ repro CA count in default roots is zero
    • ✅ phase A rootCount is 1
    • ✅ phase A healthy request returns 400 openwork_workspace_not_found
    • ✅ phase A broken request returns 502 cert authority invalid
    • ✅ phase C rootCount is 1
    • ✅ phase C healthy request returns 400 openwork_workspace_not_found
    • ✅ phase C broken request returns 502 cert authority invalid
    • ✅ phase B rootCount is 0
    • ✅ phase B formerly healthy request returns 502 cert authority invalid
    • ✅ final state restored exactly one root
    • ✅ final state reports CDP ready
  2. The installed Electron app reports a running OpenWork server and healthy managed OpenCode loopback

    🎙 Next we confirm the real installed app is the one under test. Its Electron bridge reports a running OpenWork server, the expected workspace, a healthy managed OpenCode engine, and a live server-connected event stream.

    • ✅ renderer user agent is Electron on Windows
    • ✅ OpenWork server is running and has a base URL
    • ✅ /workspaces contains ws_c0107cd3f4f6
    • ✅ proxied /workspace/.../opencode/global/health returns HTTP 200
    • ✅ proxied OpenCode health is healthy version 1.17.11
    • ✅ /opencode/event returns a text/event-stream
    • ✅ /opencode/event yields a server.connected frame
  3. Default Cloud MCP health and Connect state stay ready without touching the fixture

    🎙 The default Connect health check now trusts the engine attestation without opening a direct network probe. Fixture counts stay still while the health and Connect state both report ready and usable.

    • ✅ fixture task is running and listening before reconcile
    • ✅ reconcile returns HTTP 200
    • ✅ reconcile health phase is ready
    • ✅ reconcile health is usable
    • ✅ reconcile health engine status is connected
    • ✅ reconcile performs the direct readiness probe
    • ✅ reconcile desired and applied revisions match
    • ✅ default Cloud MCP health returns HTTP 200
    • ✅ default health phase is ready
    • ✅ default health is usable
    • ✅ default health engine status is connected
    • ✅ default health firstFailure is null
    • ✅ default health desired and applied revisions match
    • ✅ default health leaves direct.checked false
    • ✅ /experimental/connect/state returns HTTP 200
    • ✅ Connect state reports Cloud MCP present
    • ✅ Connect state cloudHealth phase is ready
    • ✅ Connect state cloudHealth is usable
    • ✅ Connect state cloudHealth engine status is connected
    • ✅ Connect state cloudHealth firstFailure is null
    • ✅ Connect state cloudHealth desired and applied revisions match
    • ✅ Connect state cloudHealth leaves direct.checked false
    • ✅ fixture JSON-RPC method counts are unchanged by default health and Connect state
  4. health?probe=1 performs exactly one MCP initialize/initialized/tools-list handshake

    🎙 When the reviewer asks for an explicit probe, the direct MCP handshake happens exactly once. The OS-trusted path lists both cloud tools and the fixture count advances by initialize, initialized notification, and tools/list.

    • ✅ probe Cloud MCP health returns HTTP 200
    • ✅ probe health phase is ready
    • ✅ probe health is usable
    • ✅ probe health engine status is connected
    • ✅ probe health firstFailure is null
    • ✅ probe health desired and applied revisions match
    • ✅ probe health sets direct.checked true
    • ✅ direct tools/list includes search_capabilities
    • ✅ direct tools/list includes execute_capability
    • ✅ probe health has no direct failure
    • ✅ fixture method delta for initialize is exactly 1
    • ✅ fixture method delta for notifications/initialized is exactly 1
    • ✅ fixture method delta for tools/list is exactly 1
  5. A direct probe outage reports probe_unreachable without overriding engine-connected readiness

    🎙 If that diagnostic probe cannot reach the fixture, the app still reports the engine truth as ready and usable. The direct failure is clearly diagnostic, and the flow restarts the fixture before it exits.

    • ✅ fixture listener is down after stopping only the scheduled task
    • ✅ probe failure health still returns HTTP 200
    • ✅ probe failure health phase is ready
    • ✅ probe failure health is usable
    • ✅ probe failure health engine status is connected
    • ✅ probe failure health firstFailure is null
    • ✅ probe failure health desired and applied revisions match
    • ✅ probe failure leaves direct.checked false
    • ✅ probe failure code is probe_unreachable
    • ✅ probe failure reports no missing direct tools
    • ✅ fixture listener is restored
    • ✅ fixture scheduled task is running after restart
  6. The installed packaged steering smoke keeps engine-ready truth authoritative across all cases

    🎙 Finally we verify packaged steering consumes the same engine truth. Ready, auth-needed, thrown, unknown, and missing-entry cases choose the expected guidance without unnecessary server fetches.

    • ✅ packaged steering smoke result ok is true
    • ✅ all five packaged steering cases are present
    • ✅ connected ready case has zero server fetches and no repair guidance
    • ✅ needs_auth case gives sign-in guidance with zero server fetches
    • ✅ thrown case stays neutral with zero server fetches
    • ✅ unknown case stays neutral with zero server fetches
    • ✅ missing entry case performs exactly one fallback
    • ✅ result includes an installed bundle hash or an installed resources artifact path
    • ✅ installed bundle hash is present
    • ✅ Script frame 1 narrated: "We start with the repeatable control: the same packaged app can prove when the corporate"
    • ✅ Script frame 2 narrated: "Next we confirm the real installed app is the one under test. Its Electron bridge report"
    • ✅ Script frame 3 narrated: "The default Connect health check now trusts the engine attestation without opening a dir"
    • ✅ Script frame 4 narrated: "When the reviewer asks for an explicit probe, the direct MCP handshake happens exactly o"
    • ✅ Script frame 5 narrated: "If that diagnostic probe cannot reach the fixture, the app still reports the engine trut"
    • ✅ Script frame 6 narrated: "Finally we verify packaged steering consumes the same engine truth. Ready, auth-needed, "

@benjaminshafii

Copy link
Copy Markdown
Member Author

fraimz — ✅ PASSED

1 passed · 0 failed · 0 skipped — run 2026-07-16T16-10-33-837Z

Full frame proof with validated screenshots: evals/results/2026-07-16T16-10-33-837Z/fraimz.html (re-run: pnpm fraimz --flow windows-enterprise-ca-error-state)

✅ windows-enterprise-ca-error-state — Windows enterprise CA error state: old build fabricates outage, fixed build trusts engine truth

Internal demo

  1. The old build reports cloud_tools_missing even though the engine and Schannel endpoint are healthy

    🎙 We begin on the same machine before the fix: the engine is connected and Schannel can list the tools, but OpenWork reports a fabricated Cloud tools outage. This frame captures the exact contradiction that caused the model-facing error state.

    • ✅ pre-fix artifact user agent contains OpenWork/0.17.29
    • ✅ engine openwork-cloud status is connected
    • ✅ direct Schannel tools/list contrast returned HTTP 200
    • ✅ direct Schannel tools/list includes both expected tools
    • ✅ pre-fix default health phase is cloud_tools_missing
    • ✅ pre-fix default health is not usable
    • ✅ pre-fix first failure is cloud_tools_missing
    • ✅ pre-fix direct error contains fetch failed
    • ✅ pre-fix delivery appliedRevision is null
    • ✅ connect/state cloudHealth also reports cloud_tools_missing
  2. The old steering converts the fabricated outage into repair/refusal guidance

    🎙 The old steering then turns that fabricated outage into user-facing refusal guidance. The model is told the Cloud connection is not ready and to send the user to Repair and test.

    • ✅ pre-fix instruction says the connection is not ready
    • ✅ pre-fix instruction sends the user to Repair and test
    • ✅ pre-fix instruction names cloud_tools_missing
  3. The fixed build reports ready by default and the live app agrees

    🎙 Now the fixed build runs on the same machine, same corporate CA, and same endpoint without reconciling a new config. Default health trusts the engine, the fixture stays untouched, and the live app reports the same ready state.

    • ✅ fixed artifact user agent contains OpenWork/0.17.30
    • ✅ fixed artifact did not perform reconcile
    • ✅ fixed default health phase is ready
    • ✅ fixed default health is usable
    • ✅ fixed default health engine status is connected
    • ✅ fixed default health does not run a direct probe
    • ✅ fixed default health firstFailure is null
    • ✅ fixed default health desired and applied revisions match
    • ✅ fixed default-health fixture delta is zero
    • ✅ both before and fixed exe SHA256 prefixes are recorded
    • ✅ live fixed app default health request returns HTTP 200
    • ✅ live fixed app reports ready usable engine-connected default health
  4. The fixed build probes only on demand and steers the model toward Cloud tools

    🎙 Finally, the fixed build still supports an explicit diagnostic probe on demand. The probe performs one handshake, Connect remains ready, and steering tells the model to use the Cloud tools instead of refusing.

    • ✅ fixed explicit probe sets direct.checked true
    • ✅ fixed explicit probe includes both direct tools
    • ✅ fixed explicit probe delta is exactly one initialize, initialized notification, and tools/list
    • ✅ fixed connect/state remains ready and usable
    • ✅ fixed steering reports ready Cloud tooling
    • ✅ fixed steering fetchCount is zero
    • ✅ fixed steering omits not-ready and Repair and test guidance
    • ✅ Script frame 1 narrated: "We begin on the same machine before the fix: the engine is connected and Schannel can li"
    • ✅ Script frame 2 narrated: "The old steering then turns that fabricated outage into user-facing refusal guidance. Th"
    • ✅ Script frame 3 narrated: "Now the fixed build runs on the same machine, same corporate CA, and same endpoint witho"
    • ✅ Script frame 4 narrated: "Finally, the fixed build still supports an explicit diagnostic probe on demand. The prob"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Idea: Rename into Labori

1 participant