test(evals): pin the semi air-gapped Den deployment contract - #3136
Merged
Conversation
Most enterprise installs are not fully air-gapped: laptops have internet plus VPN while Den web and Den API sit inside the customer's private network. That shape depends on four guarantees that no test held, so a refactor could change on-prem behavior, or an enterprise firewall requirement, without anything turning red. Add an internal flow with four frames, each backed by real evidence rather than prose: 1. resolveDenBaseUrls ignores an explicitly configured API origin when a base URL is present, so the desktop keeps needing exactly one Den hostname. 2. url-guard refuses private and reserved MCP addresses by default, and DEN_ALLOW_PRIVATE_MCP_URLS is the only documented way in. 3. The cloud catalog probe returns untrusted_endpoint and performs no request for an untrusted origin, so a self-hosted deployment leaks no organization MCP token and the diagnostic is skipped rather than failed. 4. DEN_DIAGNOSTICS_ORIGIN selects the outbound self-check target, with the public host as an explicit default, and the bearer token minimum of 24 characters is enforced. Runs with no app, no database, and no network: requiresApp is false and it passes against a dead CDP URL. Evidence comes from focused Bun test filters plus direct imports of the pure functions. Verified it can fail: breaking resolveDenBaseUrls fails frame 1 and forcing allowPrivateMcpUrls true fails frame 2, each naming the broken property.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Member
Author
Daytona clean-sandbox validation — PassedRan this internal contract flow on a fresh Daytona Linux Electron sandbox with the combined enterprise-network branch checked out. The app itself was running, but the flow's Also on that same clean sandbox: The flow therefore passes outside the developer's local machine and does not depend on ambient local services. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Full air-gap is rare. The common enterprise shape is a split: laptops have internet and VPN, while Den web and Den API run inside the customer's private network.
That shape silently depends on four guarantees, and no test held any of them. A refactor could have changed on-prem behavior — or an enterprise firewall requirement — without anything turning red.
What this adds
evals/flows/semi-airgapped-den-contract.flow.mjs(internal,requiresApp: false) plus its voiceover. Four frames, each backed by real evidence rather than prose:resolveDenBaseUrlsignores an explicitly configured API origin when a base URL is presenturl-guardrefuses private/reserved MCP addresses by default;DEN_ALLOW_PRIVATE_MCP_URLSis the only documented way inuntrusted_endpointand performs no request for an untrusted originDEN_DIAGNOSTICS_ORIGINselects the outbound self-check target, public host as explicit default; bearer token minimum enforcedEvidence comes from focused
bun testfilters plus direct imports of the pure functions — following the in-flow test pattern already used byinstaller-tls-trust.flow.ts, and structured onhelm-custom-ca.flow.mjs.Values verified rather than assumed: default origin is
https://diagnostic.openworklabs.com; the bearer token minimum is 24 characters (23 rejected, 24 accepted).Runs anywhere
No app, no database, no network.
requiresApp: false, and it passes when handed a deliberately dead CDP URL:```
$ pnpm evals --flow semi-airgapped-den-contract --cdp-url http://127.0.0.1:1
✓ Frame 1 ✓ Frame 2 ✓ Frame 3 ✓ Frame 4
Result: PASSED — 1 passed, 0 failed, 0 skipped
```
It can fail — verified on two different frames
A flow that cannot fail proves nothing, so this was checked in both directions, each time reverting the product source completely afterward.
Breaking frame 1 (make
resolveDenBaseUrlsprefer the explicit API origin):```
✗ Frame 1 — resolveDenBaseUrls ignores an explicit API origin when baseUrl is present
Result: FAILED — 0 passed, 1 failed, 0 skipped
```
Breaking frame 2 (force
allowPrivateMcpUrls = true):```
✗ Frame 2 — env.allowPrivateMcpUrls is false by default
Result: FAILED — 0 passed, 1 failed, 0 skipped
```
git statusis clean of both temporary edits; the final diff is only the two new eval files.Tests run
```
pnpm evals --flow semi-airgapped-den-contract # PASSED (automation)
pnpm fraimz --flow semi-airgapped-den-contract # PASSED (demo) + Voice-over script coverage ✓
pnpm evals:typecheck # pass
pnpm evals:test # 4 tests, 4 pass, 0 fail
pnpm evals --list | grep semi-airgapped # discovered
```
Fraimz
Passed.
fraimz.htmlproduced (32,938 bytes), 34 recorded assertions, every claim bound to an observable assertion viawitness(...)with raw evidence attached throughctx.output(...). Narration coverage passes with no drift againstevals/voiceovers/semi-airgapped-den-contract.md.Compatibility
No breaking changes. Eval flow and voiceover only — no product code, configuration, chart, or docs. Nothing ships to users.
Scope — what this is not
This proves the semi air-gapped contract at the code level. It does not stand up a VPN-isolated Den with a private CA and drive the real desktop app through the installer → sign-in → Cloud MCP chains. That remains worth doing; the existing cloud MCP flows are already parameterized by
OPENWORK_EVAL_DEN_API_URL/OPENWORK_EVAL_DEN_WEB_URL, so pointing them at an internal Den with a private CA is the natural next step rather than new flows.Related