Skip to content

feat(diagnostics): capture MCP registration errors and cloud endpoint transport evidence - #3201

Merged
benjaminshafii merged 3 commits into
devfrom
fix/engine-mcp-failure-evidence
Jul 27, 2026
Merged

feat(diagnostics): capture MCP registration errors and cloud endpoint transport evidence#3201
benjaminshafii merged 3 commits into
devfrom
fix/engine-mcp-failure-evidence

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Problem

Self-hosted Den behind corporate TLS (Blue Yonder POC): openwork-cloud MCP registration fails on some machines while sign-in, Den API, models, and inference all work. Three weeks of debugging produced zero captured error cause, because:

  • server.ts parsed only entry.status from OpenCode's POST /mcp registration response and discarded entry.error — the exact TLS message from the failing runtime (OpenCode's failed status carries { status: "failed", error: string }).
  • cloud-tool-catalog fails closed (untrusted_endpoint) for self-hosted origins, so its transport classifier never runs.
  • mcps[].origin was hardcoded null and path a constant — endpoint drift between users was invisible.

Change

  1. Persist OpenCode's registration error (sanitized via sanitizeDiagnosticString, bounded to 400 chars) through the registration record into engine-mcp-sync.details.failedRegistrations[].errorSummary, plus a classified transportCause (tls_incomplete_chain, tls_untrusted_ca, dns_error, connection_refused, …) covering Node and Bun error shapes.
  2. Scrubbing instead of whole-string redaction: URL/path segments become [url]/[path] placeholders so endpoint-bearing TLS errors survive (failed to connect to [url] unable to verify the first certificate), while credential-bearing strings still fully redact.
  3. New cloud-endpoint-transport check — credential-free (bare node:tls handshake, no HTTP, no headers): exact verify error code, served chain length + subject/issuer CNs (the "is the proxy serving leaf-only?" question), DNS/TCP reachability, and local trust evidence (tls.getCACertificates("system").length, NODE_EXTRA_CA_CERTS presence + bundle cert count). Runs only when the managed cloud MCP is not connected.
  4. Real origin + path for managed openwork-cloud MCP entries (no query/hash/userinfo; strict schema validators added).
  5. Credentialed-probe trust gating (untrusted_endpoint contract, semi-airgapped-den-contract eval) untouched; report-safety claims stay truthful (rawEngineErrorsIncluded: false, new sanitizedEngineErrorSummariesIncluded / credentialFreeTransportProbePerformed flags).

Proof (real run, real TLS-broken endpoint)

runAgentContextDiagnostics against a live self-signed TLS server with the incident-shaped registration record — schema-valid report now carries:

"failedRegistrations": [{
  "name": "openwork-cloud", "status": "failed", "source": "engine_status", "recordAgeMs": 15982,
  "errorSummary": "failed to connect to [url] unable to verify the first certificate",
  "transportCause": "tls_incomplete_chain", "engineReachableNow": false
}],
"cloud-endpoint-transport": {
  "status": "failed", "code": "endpoint_tls_untrusted",
  "details": {
    "verifyErrorCode": "DEPTH_ZERO_SELF_SIGNED_CERT",
    "dnsResolved": true, "tcpConnected": true,
    "servedChainLength": 1,
    "servedChain": [{"subjectCN": "openwork-poc.blueyonder.example", "issuerCN": "openwork-poc.blueyonder.example", "selfIssued": true}],
    "systemCaCertificateCount": 0, "bundledCaCertificateCount": 144, "nodeExtraCaCertsSet": false
  }
},
"mcps": [{"name": "openwork-cloud", "source": "config.remote", "origin": "https://127.0.0.1:60695", "path": "/api/den/mcp/agent", "syncStatus": "failed"}]

Note systemCaCertificateCount: 0 — reproduced live on macOS, the same measurement as the control machine, now visible in every report.

Tests

  • bun test (apps/server): agent-context-diagnostics, agent-context-diagnostics-local-schema, agent-context-diagnostics.schema, mcp.engine-sync.e2e, agent-context-transport-probe (real openssl self-signed TLS server, ECONNREFUSED, ENOTFOUND), agent-context-cloud-probe, cloud-mcp-health, cloud-mcp-reconcile.e2e — 150 pass / 0 fail
  • bun test --isolate (apps/app): agent-context-diagnostics report component tests — 20 pass / 0 fail
  • pnpm --filter openwork-server typecheck ✅ · pnpm --filter @openwork/app typecheck

Validation honesty: verified at the server level with a real TLS endpoint plus component tests; no fraimz/Electron-UI run for the new check's wizard rendering (label wiring is enforced by the Record<AgentContextDiagnosticCheckId, string> typecheck). Repro steps for reviewers: cd apps/server && bun test src/agent-context-transport-probe.test.ts src/agent-context-diagnostics.test.ts.

… transport evidence

Self-hosted Den deployments behind corporate TLS failed openwork-cloud MCP
registration for three weeks with zero captured cause: the engine's error
string was discarded at parse time, the credentialed catalog probe fails
closed for self-hosted origins, and mcps[] hid the real endpoint.

- persist OpenCode's registration error (sanitized, bounded) through the
  registration record into engine-mcp-sync failedRegistrations, with a
  classified transportCause (tls_incomplete_chain, tls_untrusted_ca, ...)
- scrub URL/path segments to [url]/[path] placeholders instead of
  whole-string redaction so endpoint-bearing TLS errors survive sanitization
- new credential-free cloud-endpoint-transport check: bare TLS handshake
  captures the exact verify error code, served chain length/subjects, and
  local trust evidence (system CA count, NODE_EXTRA_CA_CERTS state)
- report real origin + path for managed openwork-cloud MCP entries
- keep the credentialed-probe trust gating and report-safety claims intact
@vercel

vercel Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 27, 2026 8:51pm
openwork-den Ready Ready Preview, Comment Jul 27, 2026 8:51pm
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 27, 2026 8:51pm
openwork-diagnostics Ready Ready Preview, Comment Jul 27, 2026 8:51pm
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 27, 2026 8:51pm

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying openwork with  Cloudflare Pages  Cloudflare Pages

Latest commit: 7a0010c
Status:⚡️  Build in progress...

View logs

@benjaminshafii

Copy link
Copy Markdown
Member Author

Rebased onto dev via merge commits after #3199/#3200/#3203 landed. Recomposition notes:

  • Kept dev's architecture: agent-context-cloud-probe.ts (differential verdict, feat(diagnostics): add independent runtime MCP verification with detailed failure attribution #3199) and the enterprise-activation trust gate (fix(diagnostics): probe activated on-prem Den origins and correct engine-evidence staleness #3200) are untouched — verified byte-identical to dev.
  • This PR's remaining unique value (all still absent on dev): OpenCode's own registration error persisted into engine-mcp-sync.failedRegistrations[].errorSummary + classified transportCause (dev still discards entry.error at parse time); URL/path scrubbing so endpoint-bearing TLS errors survive sanitization; real origin/path on managed cloud MCP entries; and the cloud-endpoint-transport check now positioned after cloud-endpoint-differential, contributing only the TLS-layer evidence the differential lacks: served certificate chain (length, subject/issuer CNs) and local trust-store counts (systemCaCertificateCount, NODE_EXTRA_CA_CERTS bundle cert count — dev has only a boolean).
  • Re-verified end to end after the merge: real runAgentContextDiagnostics against a live TLS-broken endpoint yields a schema-valid report with errorSummary: "failed to connect to [url] unable to verify the first certificate", transportCause: tls_incomplete_chain, verifyErrorCode: DEPTH_ZERO_SELF_SIGNED_CERT, servedChainLength: 1, origin + prefixed path.
  • Tests after merge: apps/server 165+47 pass / 0 fail across the diagnostics, probe, schema, engine-sync, health, and no-bare-fetch suites; apps/app 33 pass / 0 fail incl. the report component test from feat(diagnostics): add independent runtime MCP verification with detailed failure attribution #3199; both typechecks green.
  • Diff vs dev is now exactly 14 files, all diagnostics-scoped.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant