Skip to content

docs(self-host): document the private-network (semi air-gapped) deployment - #3135

Merged
benjaminshafii merged 1 commit into
devfrom
docs/private-network-deployment
Jul 25, 2026
Merged

docs(self-host): document the private-network (semi air-gapped) deployment#3135
benjaminshafii merged 1 commit into
devfrom
docs/private-network-deployment

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Why

Full air-gap is rare. The common enterprise shape is a split: employee laptops have normal internet and VPN access, while Den web and Den API run inside the customer's private network.

That shape was undocumented, so every customer rediscovered its gotchas by hitting failures.

Adds packages/docs/start-here/private-network-deployment.mdx, registered in the existing Self-host docs group.

Structure

The two network paths are documented separately, because different teams approve them.

What the desktop must reach — one Den origin over the VPN, plus the internet destinations that need mirroring otherwise (github.com and both release-asset hosts, registry.npmjs.org, models.openworklabs.com), with the fully-internal alternative of mounting installer artifacts so the browser only ever talks to Den.

What Den must reach — only two hard requirements: its MySQL endpoint, and ghcr.io at chart/image-pull time unless mirrored. Everything else is a per-feature table that can stay blocked until that feature is approved. Includes the process-level NODE_USE_ENV_PROXY / HTTPS_PROXY / NO_PROXY / NODE_EXTRA_CA_CERTS settings a Den container needs behind a corporate proxy or TLS inspection.

Three surprises for operators — each with symptom → cause → fix:

Surprise Symptom Fix
Internal MCP servers refused MCP_URL_BLOCKED DEN_ALLOW_PRIVATE_MCP_URLS=1, with the SSRF tradeoff stated plainly
Cloud MCP diagnostic skipped Settings → Debug says the endpoint is outside the diagnostics trust policy OPENWORK_AGENT_DIAGNOSTICS_TRUSTED_ORIGINS, set before launch
Den's outbound self-check unreachable Cannot reach diagnostic.openworklabs.com Self-host ee/apps/diagnostics, point DEN_DIAGNOSTICS_ORIGIN at it

The second one carries the most important framing in the page: it affects the diagnostic only — Cloud MCP itself works without the variable. Operators must not read a skipped probe as a broken deployment. It also records that the variable cannot be set from the in-app environment editor, because that store strips OPENWORK_* keys.

Verification checklist — the three chains to confirm after deploying (installer activates, desktop signs in, Cloud MCP works in a real chat), each with what success looks like and where to look first when it fails.

A contradiction resolved — this decided whether IT opens one hostname or two

Two existing docs disagreed:

  • packages/docs/start-here/self-host.mdx:117-119"The desktop app no longer needs, reads, or writes a separate API URL."
  • docs/org-install-links.md:58-60"The desktop must be able to reach both origins."

Resolved from the code, not the prose:

  • apps/app/src/app/lib/den.ts:547-557resolveDenBaseUrls ignores an explicit apiBaseUrl when a baseUrl is present and derives <baseUrl>/api/den. den.ts:560-563 builds MCP from that derived value.
  • apps/app/tests/den-mcp-url.test.ts:19-24 — existing test "ignores an explicit API origin when a base URL is present" pins it.
  • DEN_API_PUBLIC_URL is still consumed, but for claims and identifiers, not desktop reachability: auth.ts:127-140, mcp/resource.ts:53-55, routes/org/install-links.ts:124-130, desktop-connect-grants.ts:99-115.
  • The exception: apps/desktop/electron/connect-link.mjs:332-344 builds the install-link exchange endpoint from the apiBaseUrl carried in the link, so a separate API origin must be reachable for that one-time exchange. External MCP clients also reach their configured origin directly and do not inherit the desktop's single-origin behavior.

So: steady-state desktop traffic needs one origin. docs/org-install-links.md was overbroad and is corrected; the self-host.mdx sentence is narrowed to say it applies to normal signed-in traffic. Both now state their context instead of disagreeing.

This matters concretely — it is the difference between a firewall request for one hostname and one for two.

Compatibility

No breaking changes. Docs only. The sole non-docs edits are the two corrected sentences above. No product code, chart, .env.example, or test is touched.

Verification

```
node -e "JSON.parse(require('fs').readFileSync('packages/docs/docs.json','utf8'))"

docs.json parses OK

```

Internal links resolved against real files:

Link Target
/cloud/run-in-the-cloud/cloud-mcp packages/docs/cloud/run-in-the-cloud/cloud-mcp.mdx OK
/start-here/github-connector-helm packages/docs/start-here/github-connector-helm.mdx OK

Both code claims underpinning the contradiction fix were re-verified independently by reading connect-link.mjs:332-344 and den-mcp-url.test.ts:19-24 directly.

Related

These are independent and can merge in any order; this page stands alone.

Fraimz

Skipped, stated explicitly per AGENTS.md: documentation only, no runtime path changed.

…yment

Most enterprise installs are not fully air-gapped. The common shape is laptops
with internet plus VPN, and Den web and Den API inside the customer's private
network. That split was undocumented, so every customer rediscovered its
gotchas by hitting failures.

Add a published page covering the two network paths separately, because
different teams approve them: what the desktop must reach, what Den must reach
(only MySQL and ghcr.io at pull time are hard requirements), and the three
things that surprise operators, each with symptom, cause, and fix:

- Internal MCP servers are refused by the SSRF guard until
  DEN_ALLOW_PRIVATE_MCP_URLS=1, with the security tradeoff stated.
- The Settings > Debug cloud catalog probe is skipped because only hosted
  origins are trusted. It states explicitly that this affects the diagnostic
  only and Cloud MCP still works, so a skipped probe is not read as an outage,
  and that the variable must be set before launch because the in-app
  environment store strips OPENWORK_* keys.
- Den's outbound self-check targets a public host by default and can be
  self-hosted with DEN_DIAGNOSTICS_ORIGIN.

Also resolve a contradiction that decided whether IT opens one hostname or
two. docs/org-install-links.md claimed "the desktop must be able to reach both
origins"; resolveDenBaseUrls ignores an explicit API origin when a base URL is
present and derives <baseUrl>/api/den, proven by the existing den-mcp-url test.
Steady-state desktop traffic needs one origin. A separate API origin is only
reachable-critical for the one-time install-link exchange, which builds its
endpoint from the apiBaseUrl carried in the link, and for external MCP clients.
Both docs now state their context instead of disagreeing.
@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 25, 2026 6:05pm
openwork-den Ready Ready Preview, Comment Jul 25, 2026 6:05pm
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 25, 2026 6:05pm
openwork-diagnostics Ready Ready Preview, Comment Jul 25, 2026 6:05pm
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 25, 2026 6:05pm

@mintlify

mintlify Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
differentai 🟢 Ready View Preview Jul 25, 2026, 6:05 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@mintlify

mintlify Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
differentai 🟡 Building Jul 25, 2026, 6:04 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@benjaminshafii
benjaminshafii merged commit 5fc4a4b into dev Jul 25, 2026
12 checks passed
benjaminshafii added a commit that referenced this pull request Jul 25, 2026
…ce (#3139)

* docs(enterprise): document outbound network access and guard it in CI

Enterprise networks block outbound connections by default and IT must approve
each destination, but nothing in this repo listed what OpenWork needs to reach.
Two destinations were named nowhere at all:

- objects.githubusercontent.com, the redirect target of every release-asset
  download. Approving only github.com lets a download start and then fail
  partway through.
- registry.npmjs.org, used by `npx -y openwork-ui-mcp` in packaged builds.
  Blocked, the UI-control MCP is silently unavailable while the app still opens.

Add docs/enterprise/outbound-access.md for customer IT, in plain language, with
a copy-pasteable minimum allowlist and a what-breaks-when-blocked column. Desktop
client access is documented separately from self-hosted Den access because
different teams approve them on different networks.

Back it with docs/enterprise/outbound-access.json and
scripts/check-outbound-access.mjs, wired into ci-tests.yml, so a feature that
introduces a new destination must document its enterprise impact. The check also
reports manifest entries that no longer exist in code, so the list cannot rot.
Hosts reached indirectly (redirect targets, subprocesses, links, schema strings)
are classified by kind and exempt from the staleness rule.

Docs, one check script, one CI step, one package script. No product behavior,
configuration, API, or chart surface changes.

* docs(self-host): make the cloud catalog diagnostic actionable on-prem

Settings > Debug runs a bounded initialize -> tools/list check against the
managed openwork-cloud MCP entry. It carries an organization MCP bearer token,
so the probe refuses to send that credential to an origin outside its trust
list. DEFAULT_TRUSTED_ORIGINS holds only the hosted origins, so on every
self-hosted deployment the check returns untrusted_endpoint and never runs.

The escape hatch, OPENWORK_AGENT_DIAGNOSTICS_TRUSTED_ORIGINS, appeared in
exactly two places in the repo: its own source file and its own test. The
guidance an operator actually saw was "have the server administrator explicitly
trust the development origin" -- which calls a production on-prem Den a
development origin and never names the variable, so nobody could act on it.

Name the variable in the operator action, and document it in the self-hosting
guide with the real format rules enforced by configuredTrustedOrigins: bare
origins, comma-separated, https except loopback, invalid entries ignored rather
than failing open, additive to the hosted defaults. The docs state explicitly
that this enables the diagnostic only and that Cloud MCP itself works without
it, so operators do not read a skipped probe as a broken deployment.

The trust model is deliberately unchanged: agent-context-cloud-probe.ts is
untouched, so which origins are trusted cannot differ from before. Trusting the
endpoint's own origin would be circular, since the probe exists to validate the
very config that names it. Tests pin that an unlisted self-hosted origin still
performs no fetch, and that the action keeps naming the variable.

* docs(helm): expose the private-network MCP opt-out to operators

Den's SSRF guard refuses to fetch private/reserved addresses on behalf of
users, which is correct on hosted multi-tenant deployments. But a common
enterprise topology puts Den inside the customer's private network with their
MCP servers on private addresses, and there the guard blocks every internal
server. The operator sees MCP_URL_BLOCKED and is told to "use a public HTTPS
MCP URL or change the deployment's private-network policy through security
review", with no indication that a supported opt-out exists.

env.ts has provided DEN_ALLOW_PRIVATE_MCP_URLS=1 all along, but it appeared
only in source, unit tests, and one eval flow: not in .env.example, not in the
chart, not in any doc. Surface it in all three places so the fix is reachable
without reading the source.

Scoped to den-api only. den-api, den-web, and inference all envFrom the same
ConfigMap, so adding the key there would inject an SSRF-bypass flag into two
workloads that never read it; the value is emitted directly on the den-api
container instead. Only den-api consumes allowPrivateMcpUrls (mcp-connections,
plugin-system/store, enterprise-mcp-client-adapter, external-mcp-client).

Nothing renders unless an operator opts in, so helm template with default
values is byte-identical to the previous chart: no ConfigMap checksum change
and no pod restart on upgrade. Blank, 0, and false render nothing; only "1"
enables it, matching the trimmed string comparison in env.ts; any other value
fails the render rather than guessing. url-guard.ts and env.ts are untouched.

* docs(enterprise): name the current GitHub release-asset host

Verified against a live OpenWork release asset: github.com now redirects
release downloads to release-assets.githubusercontent.com, not
objects.githubusercontent.com. An allowlist naming only the old host would
still let downloads start and then fail partway through, which is the exact
symptom this page exists to prevent.

List both. release-assets.githubusercontent.com is what GitHub serves today;
objects.githubusercontent.com stays documented for older clients and any
staged rollback.

* docs(self-host): document the private-network (semi air-gapped) deployment

Most enterprise installs are not fully air-gapped. The common shape is laptops
with internet plus VPN, and Den web and Den API inside the customer's private
network. That split was undocumented, so every customer rediscovered its
gotchas by hitting failures.

Add a published page covering the two network paths separately, because
different teams approve them: what the desktop must reach, what Den must reach
(only MySQL and ghcr.io at pull time are hard requirements), and the three
things that surprise operators, each with symptom, cause, and fix:

- Internal MCP servers are refused by the SSRF guard until
  DEN_ALLOW_PRIVATE_MCP_URLS=1, with the security tradeoff stated.
- The Settings > Debug cloud catalog probe is skipped because only hosted
  origins are trusted. It states explicitly that this affects the diagnostic
  only and Cloud MCP still works, so a skipped probe is not read as an outage,
  and that the variable must be set before launch because the in-app
  environment store strips OPENWORK_* keys.
- Den's outbound self-check targets a public host by default and can be
  self-hosted with DEN_DIAGNOSTICS_ORIGIN.

Also resolve a contradiction that decided whether IT opens one hostname or
two. docs/org-install-links.md claimed "the desktop must be able to reach both
origins"; resolveDenBaseUrls ignores an explicit API origin when a base URL is
present and derives <baseUrl>/api/den, proven by the existing den-mcp-url test.
Steady-state desktop traffic needs one origin. A separate API origin is only
reachable-critical for the one-time install-link exchange, which builds its
endpoint from the apiBaseUrl carried in the link, and for external MCP clients.
Both docs now state their context instead of disagreeing.

* docs(self-host): consolidate network, air-gap, and certificate guidance

Promote Self-host from a small Start here group to a dedicated docs tab while
preserving every existing start-here URL. Consolidate scattered network
material into five focused published pages: air-gapped deployment, outbound
network access, certificate trust and proxies, installer delivery, and network
diagnostics. Keep the private-network topology page as the high-level entry.

Make boundaries explicit instead of conflating them:

- mounted installers solve installer delivery, not full product isolation;
- desktop Chromium, spawned sidecars, Den containers, and MySQL use different
  trust surfaces;
- Linux Electron may require NSS/user trust even when /etc/ssl is correct;
- sslmode=require encrypts but is not certificate identity verification;
- a skipped Cloud catalog diagnostic does not mean Cloud MCP is broken.

Promote the machine-guarded outbound manifest into customer-facing docs,
retain repository operator pages and their existing anchors, and slim the
self-host/private-network overviews so there is one canonical owner per detail.
The Helm and diagnostics READMEs remain authoritative for low-level mechanics.

This branch is stacked on PRs #3131, #3132, #3133, and #3135. It changes docs
and navigation only; no product, Helm template, environment, skill, or eval
behavior changes.
@benjaminshafii
benjaminshafii deleted the docs/private-network-deployment branch July 25, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant