Skip to content

test(evals): pin the semi air-gapped Den deployment contract - #3136

Merged
benjaminshafii merged 1 commit into
devfrom
evals/semi-airgapped-den-contract
Jul 25, 2026
Merged

test(evals): pin the semi air-gapped Den deployment contract#3136
benjaminshafii merged 1 commit into
devfrom
evals/semi-airgapped-den-contract

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Why

Full air-gap is rare. The common enterprise shape is a split: laptops have internet and VPN, while Den web and Den API run inside the customer's private network.

That shape silently depends on four guarantees, and no test held any of them. A refactor could have changed on-prem behavior — or an enterprise firewall requirement — without anything turning red.

What this adds

evals/flows/semi-airgapped-den-contract.flow.mjs (internal, requiresApp: false) plus its voiceover. Four frames, each backed by real evidence rather than prose:

Frame Guarantee Why it matters behind a VPN
1 resolveDenBaseUrls ignores an explicitly configured API origin when a base URL is present Keeps the firewall request to one hostname. If this flips, every customer's allowlist silently becomes wrong.
2 url-guard refuses private/reserved MCP addresses by default; DEN_ALLOW_PRIVATE_MCP_URLS is the only documented way in If the default flips open, hosted multi-tenant Den becomes SSRF-exposed
3 The cloud catalog probe returns untrusted_endpoint and performs no request for an untrusted origin A self-hosted deployment leaks no organization MCP token, and the diagnostic is skipped, not failed
4 DEN_DIAGNOSTICS_ORIGIN selects the outbound self-check target, public host as explicit default; bearer token minimum enforced A VPN-only Den can point its self-check at an internal diagnostics service

Evidence comes from focused bun test filters plus direct imports of the pure functions — following the in-flow test pattern already used by installer-tls-trust.flow.ts, and structured on helm-custom-ca.flow.mjs.

Values verified rather than assumed: default origin is https://diagnostic.openworklabs.com; the bearer token minimum is 24 characters (23 rejected, 24 accepted).

Runs anywhere

No app, no database, no network. requiresApp: false, and it passes when handed a deliberately dead CDP URL:

```
$ pnpm evals --flow semi-airgapped-den-contract --cdp-url http://127.0.0.1:1
✓ Frame 1 ✓ Frame 2 ✓ Frame 3 ✓ Frame 4
Result: PASSED — 1 passed, 0 failed, 0 skipped
```

It can fail — verified on two different frames

A flow that cannot fail proves nothing, so this was checked in both directions, each time reverting the product source completely afterward.

Breaking frame 1 (make resolveDenBaseUrls prefer the explicit API origin):
```
✗ Frame 1 — resolveDenBaseUrls ignores an explicit API origin when baseUrl is present
Result: FAILED — 0 passed, 1 failed, 0 skipped
```

Breaking frame 2 (force allowPrivateMcpUrls = true):
```
✗ Frame 2 — env.allowPrivateMcpUrls is false by default
Result: FAILED — 0 passed, 1 failed, 0 skipped
```

git status is clean of both temporary edits; the final diff is only the two new eval files.

Tests run

```
pnpm evals --flow semi-airgapped-den-contract # PASSED (automation)
pnpm fraimz --flow semi-airgapped-den-contract # PASSED (demo) + Voice-over script coverage ✓
pnpm evals:typecheck # pass
pnpm evals:test # 4 tests, 4 pass, 0 fail
pnpm evals --list | grep semi-airgapped # discovered
```

Fraimz

Passed. fraimz.html produced (32,938 bytes), 34 recorded assertions, every claim bound to an observable assertion via witness(...) with raw evidence attached through ctx.output(...). Narration coverage passes with no drift against evals/voiceovers/semi-airgapped-den-contract.md.

Compatibility

No breaking changes. Eval flow and voiceover only — no product code, configuration, chart, or docs. Nothing ships to users.

Scope — what this is not

This proves the semi air-gapped contract at the code level. It does not stand up a VPN-isolated Den with a private CA and drive the real desktop app through the installer → sign-in → Cloud MCP chains. That remains worth doing; the existing cloud MCP flows are already parameterized by OPENWORK_EVAL_DEN_API_URL / OPENWORK_EVAL_DEN_WEB_URL, so pointing them at an internal Den with a private CA is the natural next step rather than new flows.

Related

Most enterprise installs are not fully air-gapped: laptops have internet plus
VPN while Den web and Den API sit inside the customer's private network. That
shape depends on four guarantees that no test held, so a refactor could change
on-prem behavior, or an enterprise firewall requirement, without anything
turning red.

Add an internal flow with four frames, each backed by real evidence rather
than prose:

1. resolveDenBaseUrls ignores an explicitly configured API origin when a base
   URL is present, so the desktop keeps needing exactly one Den hostname.
2. url-guard refuses private and reserved MCP addresses by default, and
   DEN_ALLOW_PRIVATE_MCP_URLS is the only documented way in.
3. The cloud catalog probe returns untrusted_endpoint and performs no request
   for an untrusted origin, so a self-hosted deployment leaks no organization
   MCP token and the diagnostic is skipped rather than failed.
4. DEN_DIAGNOSTICS_ORIGIN selects the outbound self-check target, with the
   public host as an explicit default, and the bearer token minimum of 24
   characters is enforced.

Runs with no app, no database, and no network: requiresApp is false and it
passes against a dead CDP URL. Evidence comes from focused Bun test filters
plus direct imports of the pure functions.

Verified it can fail: breaking resolveDenBaseUrls fails frame 1 and forcing
allowPrivateMcpUrls true fails frame 2, each naming the broken property.
@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 25, 2026 6:17pm
openwork-den Ready Ready Preview, Comment Jul 25, 2026 6:17pm
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 25, 2026 6:17pm
openwork-diagnostics Ready Ready Preview, Comment Jul 25, 2026 6:17pm
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 25, 2026 6:17pm

@benjaminshafii

Copy link
Copy Markdown
Member Author

Daytona clean-sandbox validation — Passed

Ran this internal contract flow on a fresh Daytona Linux Electron sandbox with the combined enterprise-network branch checked out. The app itself was running, but the flow's requiresApp: false path used no CDP, network, database, or Den stack.

✓ Frame 1 — single Den origin contract       187ms
✓ Frame 2 — private MCP default-deny         742ms
✓ Frame 3 — no token sent to untrusted host  203ms
✓ Frame 4 — diagnostics origin configurable 1464ms
Result: PASSED — 1 passed, 0 failed, 0 skipped

Also on that same clean sandbox:

Outbound access manifest: 35 covered, 31 observed, 0 stale
Desktop tests: 116 pass, 0 fail, 1 skipped
Installer tests: 53 pass, 0 fail

The flow therefore passes outside the developer's local machine and does not depend on ambient local services.

@benjaminshafii
benjaminshafii merged commit 4e675e0 into dev Jul 25, 2026
10 checks passed
@benjaminshafii
benjaminshafii deleted the evals/semi-airgapped-den-contract branch July 25, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant