Skip to content

feat(diagnostics): add independent runtime MCP verification with detailed failure attribution - #3199

Merged
reachjalil merged 1 commit into
different-ai:devfrom
reachjalil:feature/runtime-mcp-verification
Jul 27, 2026
Merged

feat(diagnostics): add independent runtime MCP verification with detailed failure attribution#3199
reachjalil merged 1 commit into
different-ai:devfrom
reachjalil:feature/runtime-mcp-verification

Conversation

@reachjalil

@reachjalil reachjalil commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

What this does

Extends the Agent Context diagnostic so it can verify the configured openwork-cloud MCP endpoint independently of OpenCode, from the OpenWork runtime's own network stack, and then answer the differential question support actually needs:

  • Did the independent Node/Bun runtime path reach the endpoint?
  • What did OpenCode report for the same managed entry?
  • If they disagree, which runtime boundary is implicated?
  • At exactly which network or protocol stage did the independent probe fail?

The probe runs the complete bounded MCP handshake — initialize (with protocol-version validation), notifications/initialized, tools/list, and a bounded best-effort DELETE session cleanup. It never calls an MCP tool, never mutates configuration, never reconnects OpenCode, and never remints credentials.

Runtime transport is now explicit

apps/server/src/server-fetch.ts gains runtimeDiagnosticFetch and runtimeDiagnosticTransportInfo(). The probe reports runtimeFamily (electron-node | node | bun | unknown) and transport (node-undici | bun-fetch | test-seam | unknown), so a Bun standalone result is never claimed to have come from Node, and the deliberate Node/undici path inside Electron main is a named seam instead of an accidental ?? fetch fallback.

The bare-fetch guard test now flags bare fetch references, not just call sites, so input.fetchImpl ?? fetch can no longer bypass it. The sweep surfaced and fixed three more accidental fallbacks: opencode-models-url.ts (→ loopbackFetch), worker-activity-heartbeat.ts (→ externalFetch), and the engine-client default in server.ts (→ explicit globalThis.fetch).

Eligibility is decoupled from OpenCode state

Cached registration states (failed, needs-auth, needs-client-registration, not-recorded) and agent tool policy no longer gate the probe — the probe exists precisely to diagnose those situations. They are now comparison inputs and separate reported dimensions. A denied or unavailable tool policy is never reported as endpoint unavailability.

The probe still skips (as not-performed, stage eligibility) for: remote-workspace privacy, disabled MCP config, missing/malformed/duplicate credentials, invalid endpoints, untrusted origins, expired overall deadline, and probe-concurrency exhaustion.

Reviewer note (deliberate behavior change): the credentialed probe now runs in states where it previously did not (engine-failed registration, denied/unavailable tool policy). It remains bounded to explicit user-initiated diagnostics runs, local workspaces, the exact managed entry, and explicitly trusted origins only.

Trust boundary preserved; on-prem audit result

Only built-in OpenWork Cloud origins, exact administrator-configured origins (OPENWORK_AGENT_DIAGNOSTICS_TRUSTED_ORIGINS), and loopback may receive the credentialed request. Renderer/user-supplied URLs cannot widen this.

Audit: the enterprise/on-prem Den base origin (denBaseUrl) is provisioned desktop-side (desktop bootstrap / enterprise activation / connect-link claims in apps/desktop/electron/workspace-store.mjs) and is not visible to the server process, so the diagnostic cannot derive a trusted origin from it today. Per that result, untrusted_endpoint is reported as probe not performed (warning, stage eligibility) with an actionable admin-configuration explanation — explicitly not as a network/TLS/MCP failure. Plumbing the activation denBaseUrl into the server's trust list is left as a follow-up because the desktop runtime env wiring is owned by #3118's area (see overlap below).

Detailed failure attribution

Failures no longer collapse into network_error/invalid_response. The typed result includes performed, stage, code, networkCode, retryable, runtimeFamily, transport, httpStatus, durationMs, toolsListPerformed, sessionEstablished, cleanupAttempted, cleanupSucceeded, engineRegistrationStatus, engineEvidenceSource, engineEvidenceAgeMs, proxyConfigured/extraCaConfigured (booleans only), a strictly validated referenceId (x-request-id), aggregate catalog counts, and a bounded list of sanitized step summaries.

  • Closed stage enum: eligibility, dns, connect, tls, proxy, initialize_request|_http|_protocol, initialized_notification, tools_list_request|_http|_protocol, catalog_validation, session_cleanup, complete.
  • Closed network codes: ENOTFOUND, EAI_AGAIN, ECONNREFUSED, ECONNRESET, ETIMEDOUT, UND_ERR_CONNECT_TIMEOUT, four TLS-validation codes + ERR_TLS_CERT_ALTNAME_INVALID, PROXY_ERROR, UNKNOWN_NETWORK_ERROR.
  • HTTP classes: 401 credential, 403 membership/scope/policy, 404 MCP route/deployment mismatch, 429 rate limit, 502/503/504 gateway, redirects rejected, bounded generic otherwise.
  • Protocol classes: content-type, oversize, invalid UTF-8, invalid JSON, malformed JSON-RPC envelope, request-ID mismatch, JSON-RPC error, unsupported protocol version, invalid session header, invalid catalog, required tools missing, pagination unsupported.
  • Retryability is derived deterministically from the closed sets (transient DNS, timeouts, resets, 429, 502-504, probe-busy → retryable; TLS validation, 401/403/404, protocol/catalog defects → not), never from raw message text.

Raw exception messages, cause objects, errno numbers, certificate subjects, paths, proxy values, response bodies, headers, and stack traces are never exported. Catalog validation is forward-compatible: extra provider tools no longer fail the check, but only the allowlisted expected IDs and aggregate counts are exported — provider-controlled names are never reflected.

Differential verdict

A new cloud-endpoint-differential check compares the runtime observation with the engine's cached registration evidence for the same entry: runtime_and_engine_connected, runtime_connected_engine_failed (implicates the engine-side path/lifecycle — endpoint is reachable), runtime_failed_engine_connected (implicates the OpenWork runtime network path — proxy/trust-store/DNS as seen by Node or Bun), runtime_and_engine_failed (shared outage), runtime_probe_not_performed, and engine_evidence_stale_or_unavailable (record missing, or a failure record older than 60s while the engine is reachable and could have refreshed it — mirroring the existing mcp_registration_stale_failure semantics; aged connected records are the engine's standing state and stay trusted so healthy steady-state runs do not warn).

Schema changes (server + shared types kept in exact sync)

  • New check ID cloud-endpoint-differential (canonical position after cloud-tool-catalog); one new label key in the report page + en.ts.
  • New safety field cloudSessionCleanupRequested disclosing the DELETE.
  • The cloudCatalogToolsListPerformed cross-invariant no longer requires syncStatus === "connected" or a passed tool-policy check (those were the schema-level encoding of the removed gates); it still requires the retained runtime openwork-cloud entry with the /mcp/agent terminal path, proving which managed entry was probed.

The fail-closed redaction principles from #2997 are preserved: all new detail values pass the existing safe-text schema (no URLs, paths, or credential-shaped text), and eligibility/trust failures stay "no request occurred" states.

Overlap with open PRs (checked 2026-07-27)

Tests run

From apps/server (Bun 1.3.9; BUN_RUNTIME_TRANSPILER_CACHE_PATH=0 to avoid a pre-existing flaky node:sqlite transpiler-cache resolution on this machine):

  • bun test (full package): 555 pass, 10 skip, 1 fail — the single failure is workspace-kv-store.node.test.ts failing to resolve node:sqlite under Bun; verified identical on the pristine base commit (git stash → same failure), unrelated to this change.
  • bun test src/agent-context-cloud-probe.test.ts src/no-bare-fetch.test.ts — 29 pass.
  • bun test src/agent-context-diagnostics.test.ts — 44 pass.
  • bun test src/agent-context-diagnostics.schema.test.ts src/agent-context-diagnostics-local-schema.test.ts src/agent-context-diagnostics.plugin-label.test.ts src/agent-context-engine-inspection.test.ts src/opencode-models-url.test.ts — pass.
  • bun test src/worker-activity-heartbeat.test.ts src/cloud-mcp-health.test.ts — 26 pass.
  • npx tsc --noEmit in apps/server, packages/types, and apps/app — clean.

From apps/app: bun test tests/agent-context-diagnostics-report.test.tsx — 13 pass.

Validation limitations

No video/fraimz run is attached: this is a server-side diagnostic-evidence change whose only UI delta is one additional check row rendered by the existing generic check renderer (label added to en.ts). To reproduce end-to-end: run the desktop app with a connected OpenWork Cloud workspace, open Settings → Connect → Agent Context diagnostics, run diagnostics, and inspect the cloud-tool-catalog details (stage/networkCode/transport/steps) and the new Runtime probe vs engine verdict check; to see the differential in action, stop or break the engine registration (e.g. revoke Cloud auth) and rerun — the catalog check now still probes and the differential reports runtime_connected_engine_failed.

🤖 Generated with Claude Code

…iled failure attribution

Extend the Agent Context cloud catalog probe into a full independent
runtime MCP verification of the managed openwork-cloud endpoint:

- Name the runtime transport explicitly (runtimeDiagnosticFetch +
  runtimeDiagnosticTransportInfo in server-fetch.ts) so reports state
  whether Node/undici (Electron main or standalone Node), Bun's fetch,
  or a test seam performed the probe, and harden the bare-fetch guard so
  reference fallbacks such as `?? fetch` can no longer bypass it.
- Decouple probe eligibility from cached OpenCode registration state and
  from agent tool policy; both remain reported dimensions and become
  comparison inputs for a new differential verdict instead of gates.
- Run the complete bounded MCP handshake: initialize (with protocol
  version validation), notifications/initialized, tools/list, and a
  bounded best-effort DELETE session cleanup that never converts a
  successful availability check into a failure. No tool is ever called.
- Attribute failures precisely: closed stage enum (dns/connect/tls/
  proxy/initialize_*/initialized_notification/tools_list_*/
  catalog_validation/session_cleanup), closed sanitized network-cause
  codes, split HTTP classes (401/403/404/429/502-504/redirects), split
  protocol classes (utf8/json/envelope/request-id/protocol-version/
  session-header/catalog/pagination), and deterministic retryability
  derived only from closed code sets.
- Accept forward-compatible catalogs: extra provider tools no longer
  fail validation, but only the allowlisted expected tool IDs and
  aggregate counts are exported; both required tools must be present.
- Add a cloud-endpoint-differential check distinguishing endpoint
  availability from engine availability (runtime_and_engine_connected,
  runtime_connected_engine_failed, runtime_failed_engine_connected,
  runtime_and_engine_failed, runtime_probe_not_performed,
  engine_evidence_stale_or_unavailable) with engine evidence source and
  age surfaced.
- Keep the trust boundary: only built-in Cloud origins, exact
  administrator-configured origins, and loopback may receive the
  credentialed probe; untrusted_endpoint is reported as probe-not-
  performed trust configuration, never as a network or MCP failure.
  Boolean-only proxy/extra-CA posture and a strictly validated
  x-request-id reference are the only environment-derived exports.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 27, 2026 6:42pm

@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

@reachjalil is attempting to deploy a commit to the Different AI Team on Vercel.

A member of the Team first needs to authorize it.

@reachjalil
reachjalil marked this pull request as ready for review July 27, 2026 18:45
@reachjalil
reachjalil merged commit dda3325 into different-ai:dev Jul 27, 2026
11 of 15 checks passed
reachjalil added a commit that referenced this pull request Jul 27, 2026
…ine-evidence staleness (#3200)

* fix(diagnostics): probe activated on-prem Den origins and correct engine-evidence staleness

Follow-up to #3199, from a real enterprise diagnostics report where the
runtime probe could not run at all.

Trust the activated enterprise control plane. An on-prem deployment's
managed openwork-cloud entry points at the customer's own Den origin, so
the probe reported untrusted_endpoint and sent nothing — leaving exactly
the deployments that most need differential diagnosis with no evidence
beyond OpenCode's opaque "failed". The desktop writes
enterpriseActivation only after a signed activation claim verifies, so
that record is administrator-provisioned configuration rather than a
renderer- or request-supplied URL. The server now reads it (bounded,
read-only, local workspaces only) and admits its exact origin to the
diagnostics allowlist. Arbitrary URLs, non-https origins, and
credential-, query-, or fragment-bearing values are still rejected, and
a missing or malformed record fails closed.

Report trust provenance instead of the origin. trustSource
(builtin-cloud | loopback | administrator-env | enterprise-activation |
untrusted | not-evaluated) and enterpriseActivationPresent let an
administrator tell "this install is not activated" apart from "it is
activated, but against a different origin than the configured Cloud MCP"
— a real misconfiguration that previously looked identical — without the
report ever carrying a hostname.

Correct the differential staleness rule. Treating any engine record
older than 60s as stale made every healthy steady-state run report
engine_evidence_stale_or_unavailable, because a connected registration
is the engine's standing state and ages naturally between syncs. Only a
failure record the reachable engine could already have refreshed is
downgraded, mirroring mcp_registration_stale_failure; an aged failure on
an unreachable engine keeps implicating the engine.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(diagnostics): assert the probe only ever reaches the configured Den endpoint

An on-prem operator's probe must reach their own Den deployment, never
OpenWork-hosted Cloud. The built-in origins are a membership allowlist
consulted with .has() and are never a destination or fallback, but that
invariant was only enforced by inspection. Pin every request URL of the
handshake and the session cleanup to the exact configured endpoint, and
assert no OpenWork-hosted origin is contacted, at both the probe and
analyzer levels.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant