diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md new file mode 100644 index 0000000000..4059bedd32 --- /dev/null +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -0,0 +1,540 @@ +--- +authors: + - "@johntmyers" +state: review +links: + - https://github.com/NVIDIA/OpenShell/issues/1107 + - https://github.com/NVIDIA/OpenShell/pull/2155 + - https://github.com/NVIDIA/OpenShell/pull/1083 + - https://github.com/NVIDIA/OpenShell/pull/1151 + - https://github.com/NVIDIA/OpenShell/pull/1286 + - https://github.com/NVIDIA/OpenShell/pull/1511 + - https://github.com/NVIDIA/OpenShell/pull/1738 + - https://github.com/NVIDIA/OpenShell/pull/2027 + - https://github.com/NVIDIA/OpenShell/pull/1865 + - https://github.com/NVIDIA/OpenShell/pull/1938 +--- + +# RFC 0005 - Sandbox Proxy Egress Adapter Model + + + +## Summary + +Refactor sandbox egress around shared authorization, destination-validation, +and relay boundaries. CONNECT, forward HTTP, native TCP capture, policy DNS, +`inference.local`, `policy.local`, and metadata loopback become narrow adapters +that translate userland entry points into common runtime intents. Policy +evaluation, destination validation, supervisor middleware, credential +injection, request-body rewrite, WebSocket handling, protocol processing, and +upstream dialing happen behind shared boundaries. + +The RFC describes the complete forward-looking architecture. It is designed +to land incrementally across multiple pull requests. The first milestone only +restructures CONNECT and forward HTTP, extracts shared primitives, and +preserves every current user-facing feature and enforcement behavior. Later +milestones add policy DNS, transparent TCP capture, native protocol processors, +and optional deployment shapes on top of those boundaries. + +The codebase has already moved in this direction by splitting networking into +`openshell-supervisor-network` and process/netns work into +`openshell-supervisor-process`. This RFC proposes the next internal boundary: +make proxy entry mechanisms pluggable without duplicating authorization, +destination validation, or relay behavior. + +Supporting detail lives in: + +- [Current shape appendix](current-shape.md) +- [Technical design appendix](technical-design.md) +- [Implementation plan](implementation-plan.md) + +## Motivation + +The sandbox proxy supports several connection surfaces: explicit CONNECT, +forward HTTP, local inference and policy APIs, metadata loopback, TLS +termination, REST, GraphQL, JSON-RPC, MCP, and WebSocket inspection, +credential injection, supervisor middleware, and nftables-backed bypass +detection. These features are valuable, but changes to policy and enforcement +still tend to touch multiple entry paths. + +The risk is asymmetric enforcement. A security fix can be added to CONNECT and +missed in forward HTTP; endpoint metadata can be selected differently from the +logged policy; a credential path can gain request-body or WebSocket support +without the same behavior existing in another relay mode. + +The target shape separates three concerns: + +- **Adapters** describe how userland reached the networking component. +- **Authorization** decides whether the egress is allowed and what endpoint + behavior applies. +- **Relays** own bytes, credentials, protocol parsing, and upstream dialing. + +The first milestone targets the current embedded/network-only supervisor +runtime and preserves its existing user-facing behavior while the internal +seams move. The same boundaries then support policy DNS and transparent TCP, +native protocol processing, and future deployment modes without duplicating +authorization or relay logic. + +## Non-goals + +- Replace CONNECT with forward proxy as the only explicit proxy mode. +- Add SOCKS support. +- Add HTTP/2 L7 parsing in this refactor. Inspected HTTP paths should continue + to reject unsupported h2c upgrades instead of silently upgrading to raw + traffic. +- Redesign provider credential storage. +- Reintroduce iptables as the sandbox packet filtering backend. +- Use eBPF connect hooks for transparent capture. Native TCP capture needs a + userland proxy in the byte stream for TLS termination and protocol parsing. +- Add policy-declared supervisor-proxied host-local endpoints. Issue + [#1633](https://github.com/NVIDIA/OpenShell/issues/1633) can consume these + boundaries in separate feature work. +- Change the existing endpoint-only runtime's process-identity semantics during + the compatibility refactor. Future identity-less deployment modes require an + explicit capability contract and cannot inherit endpoint-only behavior by + accident. + +## Proposal + +### Migration Big Rocks + +1. **Transport and local-service adapters.** CONNECT, forward HTTP, + transparent TCP, policy DNS, `inference.local`, `policy.local`, and metadata + loopback become small adapters. They parse their surface and produce either + an egress intent, a local response, or a DNS answer. They do not duplicate + policy evaluation. +2. **Egress intent and decision.** Shared authorization evaluates L4 policy and + endpoint selection once per connection intent and returns one decision + containing the matched policy, matched endpoint, optional process identity + evidence used for evaluation, allowed IP metadata, TLS behavior, protocol + enforcement, and credential injection and middleware plans. +3. **Relays.** Relays receive an authorized destination connector, not an + already-open upstream socket. HTTP relays evaluate every request before + upstream write. TCP relays copy bytes for L4-only endpoints or hand the + stream to a protocol processor when endpoint policy requires native protocol + enforcement. + +The first implementation milestone populates a compatibility +`EgressDecision` through the existing separate queries so type extraction does +not change behavior. That transitional envelope is not the target single +authorization result. Generation-consistent materialization and deterministic +endpoint selection cut over separately after shadow comparison, before later +transport adapters depend on the result. + +### Unified Adapter Flow + +```mermaid +flowchart TD + User["Userland payload / harness"] + + subgraph ExplicitProxy["Explicit proxy listener"] + ProxyBytes["HTTP proxy bytes"] + IsConnect{"CONNECT request?"} + Connect["CONNECT adapter"] + Forward["Forward HTTP adapter"] + ProxyBytes --> IsConnect + IsConnect -- Yes --> Connect + IsConnect -- No --> Forward + end + + subgraph NativeTcp["Policy DNS + native TCP"] + NameLookup["Userland DNS lookup"] + PolicyDns["Policy DNS adapter"] + DnsAnswer["DNS answer + correlated active mapping"] + NativeConnect["Userland connect(ip:port)"] + TcpAdapter["Transparent TCP adapter"] + NameLookup --> PolicyDns + PolicyDns --> DnsAnswer + DnsAnswer --> NativeConnect + NativeConnect --> TcpAdapter + end + + subgraph LocalApis["Sandbox-local services"] + InferenceReq["Request to inference.local"] + PolicyReq["Request to policy.local"] + MetadataReq["Request to metadata loopback"] + InferenceAdapter["Inference local adapter"] + PolicyAdapter["Policy local adapter"] + MetadataAdapter["Metadata loopback adapter"] + InferenceReq --> InferenceAdapter + PolicyReq --> PolicyAdapter + MetadataReq --> MetadataAdapter + end + + subgraph Shared["Shared external egress pipeline"] + Intent["EgressIntent"] + Auth["Authorize and select endpoint"] + Decision["EgressDecision"] + Validate["Resolve and validate destination"] + Relay["Relay"] + Deny["Adapter-specific deny response"] + Intent --> Auth + Auth --> Allowed{"Allowed?"} + Allowed -- No --> Deny + Allowed -- Yes --> Decision + Decision --> Validate + Validate --> Relay + end + + User --> ProxyBytes + User --> NameLookup + User --> NativeConnect + User --> InferenceReq + User --> PolicyReq + User --> MetadataReq + + Connect --> Intent + Forward --> Intent + TcpAdapter --> Intent + InferenceAdapter --> InferenceResp["Local inference response"] + PolicyAdapter --> PolicyResp["Local policy response"] + MetadataAdapter --> MetadataResp["Local metadata credential response"] +``` + +Each adapter still owns its response shape. If authorization denies a CONNECT +intent, the CONNECT adapter returns a tunnel denial. If forward HTTP is denied, +the forward adapter returns an HTTP denial. If policy DNS refuses a name, it +returns the appropriate DNS response. The shared layer decides the outcome; +the adapter renders it for its protocol. + +### Relay Flow + +```mermaid +flowchart TD + Start["Authorized egress + destination connector"] + Start --> FirstReq{"Forward HTTP adapter
already has first request?"} + + FirstReq -- Yes --> ForwardEnforcement{"Endpoint enforcement"} + ForwardEnforcement -- "None or HTTP" --> HttpReq["Parsed HTTP request"] + ForwardEnforcement -- "Protocol processor" --> BadForward["Deny: HTTP request for native protocol endpoint"] + + FirstReq -- No --> Prepare["Prepare readable client stream"] + Prepare --> TlsPolicy{"TLS handling enabled?"} + TlsPolicy -- No --> Readable["Client stream"] + TlsPolicy -- Yes --> Peek["Peek client bytes"] + Peek --> Tls{"TLS ClientHello?"} + Tls -- Yes --> Terminate["Shared TLS terminator"] + Tls -- No --> Readable + Terminate --> Readable + + Readable --> Enforce{"Endpoint enforcement"} + Enforce -- "None" --> Sniff{"HTTP request detected?"} + Sniff -- Yes --> ParseHttp["Parse HTTP request"] + Sniff -- No --> TcpRelay["TcpRelay
connect upstream and copy bytes"] + ParseHttp --> HttpReq + + Enforce -- "HTTP" --> MustHttp{"HTTP request detected?"} + MustHttp -- Yes --> ParseHttp + MustHttp -- No --> DenyHttp["Deny: expected HTTP"] + + Enforce -- "Protocol processor" --> Processor["TcpRelay hands stream to protocol processor"] + Processor --> ProcessorOwns["Processor owns message loop
and calls connector when allowed"] + + subgraph HttpLoop["HTTP relay request loop"] + HttpReq --> HttpMode{"HTTP endpoint policy?"} + HttpMode -- "L4-only HTTP" --> ReqAllowed["Request admitted by connection decision"] + HttpMode -- "REST / GraphQL / JSON-RPC / MCP / WebSocket" --> ReqPolicy{"Request policy allowed?"} + ReqPolicy -- No --> ReqDeny["Local HTTP deny
no upstream write"] + ReqPolicy -- Yes --> ReqAllowed + ReqAllowed --> Middleware{"Supervisor middleware
configured?"} + Middleware -- Yes --> MwEval["Run HTTP_REQUEST / PRE_CREDENTIALS middleware"] + Middleware -- No --> Creds["Resolve static placeholders
and token grants"] + MwEval --> MwAllowed{"Middleware allowed?"} + MwAllowed -- No --> MwDeny["Local middleware deny
no credential injection"] + MwAllowed -- Yes --> Creds + Creds --> Rewrite["Inject credentials into configured slots"] + Rewrite --> HttpDial["Connect or reuse upstream"] + HttpDial --> HttpResponse["Write request and relay response"] + HttpResponse --> Upgrade{"101 WebSocket upgrade?"} + Upgrade -- No --> NextReq{"Another HTTP request
on this connection?"} + NextReq -- Yes --> HttpReq + NextReq -- No --> Done["HTTP relay done"] + Upgrade -- Yes --> WsInspect{"WebSocket inspection
or rewrite configured?"} + WsInspect -- No --> RawUpgrade["Raw upgraded stream"] + WsInspect -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] + end +``` + +Read this as two phases. The top half chooses the relay shape from the adapter +surface and endpoint enforcement. The `HTTP relay request loop` only receives a +parsed HTTP request. Supervisor middleware is not another policy funnel; it is +an optional request-path hook after HTTP policy allows the request and before +OpenShell-managed credential injection. + +Relay rules: + +- HTTP credential injection happens in both HTTP modes: L4-only HTTP and + HTTP-inspected. +- HTTP-inspected endpoints include `rest`, `graphql`, `json-rpc`, `mcp`, and + `websocket`. JSON-RPC and MCP are HTTP L7 protocols, not native TCP protocol + processors. +- Supervisor middleware is a typed relay hook. V1 middleware runs on parsed + HTTP requests at `HTTP_REQUEST / PRE_CREDENTIALS`, after network and request + policy admit the request and before OpenShell injects credentials. +- Middleware can allow, deny, replace the bounded request body, add approved + headers, and emit audit-safe findings/metadata. External middleware must not + receive OpenShell-managed credentials. +- Credential injection includes static placeholder rewrite and endpoint-bound + dynamic token grants. Token grants run after policy allow and before upstream + write; failures deny without forwarding the request. +- Middleware-transformed content must not create a new path for resolving + OpenShell credential placeholders unless the middleware hook is explicitly + trusted as credential-capable. The safe default is to fail closed on newly + introduced reserved placeholders before credential injection. +- Static credential rewrite covers request target, query, headers, opt-in REST + request bodies, and opt-in client-to-server WebSocket text frames. +- HTTP L7 policy is evaluated before upstream write for each request. JSON-RPC + and MCP evaluation parse bounded JSON-RPC-over-HTTP bodies; MCP adds + tool-aware selectors for `tools/call`. +- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101` + upgrade, the WebSocket relay owns frame parsing when text-frame credential + rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe + compression handling is configured. Other upgraded streams remain raw. +- Forward HTTP must stay in the shared HTTP relay loop or in an equivalent + guarded single-request relay. It must not evaluate one request and then + switch to raw bidirectional copy. +- `protocol: tcp` or an omitted protocol means L4 authorization plus byte copy, + except that HTTP-looking streams may still use HTTP credential injection. +- Future native protocol processors, such as Redis, Postgres, or MySQL, own the + full message loop and can parse multiple commands or queries on one TCP + session. A processor may be in-tree, middleware-backed, or a combination + where in-tree framing exposes typed middleware hooks. + +### Adapter Responsibilities + +CONNECT remains the generic explicit proxy mode for HTTPS and arbitrary TCP. +The CONNECT adapter parses `CONNECT host:port` into an `EgressIntent`, asks the +shared authorization boundary for an `EgressDecision`, returns the tunnel-ready +response only after the connection is allowed, and then hands the tunnel to the +relay. The upstream connection is opened by the HTTP relay or protocol +processor when payload policy allows it. The compatibility milestone preserves +the current raw-relay dial point until the processor boundary exists. + +Forward HTTP is compatibility for clients that send absolute-form HTTP +requests. The adapter parses the first request, rewrites proxy framing only at +the relay boundary, rejects `https://` absolute-form requests, rejects +unsupported h2c upgrades on inspected routes, and either stays in a shared HTTP +request loop or forces `Connection: close` for a guarded single request. + +Transparent TCP is for native clients that do not know they are using a proxy. +It depends on policy DNS and nftables capture: DNS answers create correlated +active endpoint mappings, userland later calls `connect(ip:port)`, nftables +redirects matching traffic to a userland listener, and the TCP adapter recovers +the original destination before building an intent. + +Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is +query-driven: check whether the name is policy-eligible, resolve through +trusted DNS, filter returned IPs, publish the active endpoint mapping, and +answer userland. The later `connect(ip:port)` still runs through normal +authorization and must correlate with the mapping created by that DNS answer. + +Local service adapters stay outside the normal external egress relay: +`inference.local` routes chat, completion, model discovery, embeddings, and +provider-specific inference traffic through the router with local limits; +`policy.local` exposes current policy, denial summaries, proposal submission, +and proposal wait routes; metadata loopback serves provider metadata +credentials to SDKs that bypass HTTP proxy variables. + +### Network Enforcement Substrate + +Current main uses nftables for sandbox bypass enforcement. It accepts +proxy-bound traffic, loopback, and established flows, then rejects and +optionally logs other TCP/UDP traffic for the bypass monitor. That is current +enforcement, not native TCP capture. + +```mermaid +flowchart TD + Packet["Userland packet"] --> ProxyDest{"Proxy destination?"} + ProxyDest -- Yes --> AcceptProxy["nftables accept"] + ProxyDest -- No --> Capture{"Active policy-DNS capture match?"} + Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"] + Capture -- No --> Reject["nftables log + reject bypass"] + Reject --> Monitor["Bypass monitor emits OCSF"] +``` + +Transparent TCP extends this nftables model with explicit capture rules that +run before the reject path and are scoped to unexpired, policy-correlated DNS +mappings. It does not add a parallel iptables path. The compatibility milestone +leaves the current table unchanged; capture arrives in a later feature phase. + +### Deployment Modes + +| Mode | Shape | Status | +|------|-------|--------| +| Embedded supervisor | `openshell-sandbox` orchestrates `openshell-supervisor-network` and `openshell-supervisor-process` | Current | +| Network-only supervisor | Networking, policy, proxy, local services, and background tasks run without a payload process leaf | Current runtime mode | +| Standalone proxy binary | Supervisor launches networking as a separate process with explicit APIs | Future packaging/API work | +| Sidecar proxy | Proxy runs outside the payload container but inside the sandbox boundary | Future isolation mode | + +A pluggable proxy must expose the right userland surfaces, implement the +gateway APIs it needs, and prove equivalent policy enforcement through tests. +If supervisor middleware is configured, the proxy runtime must also receive the +effective middleware service registry, validate and refresh bindings, enforce +`fail_open` and `fail_closed`, buffer within configured caps, invoke middleware +on the request path, and emit middleware OCSF events. + +Process identity is mode-dependent. Embedded supervisor mode normally requires +successful workload process, binary, and ancestor resolution; a lookup failure +continues to deny. A trusted runtime can explicitly select the existing +endpoint-only mode, in which identity is recorded as intentionally unavailable +and policy evaluation keeps its current endpoint-only semantics. The refactor +must not represent either case as a fabricated empty identity or accidentally +convert a lookup failure into endpoint-only evaluation. + +Future standalone and sidecar modes must advertise identity capability. A mode +without local identity needs an explicit unavailable-identity contract and +policy validation for binary/path predicates; it does not automatically inherit +endpoint-only semantics. The nftables rules that force, capture, or reject +userland traffic remain owned by the sandbox network boundary even if the proxy +process later moves into a standalone binary or sidecar. + +### Migration And Operational Contract + +Mechanical adapter, destination, and relay extractions ship as isolated, +revertible changes without a permanent feature flag. The deterministic +authorization result is different: it first runs beside the legacy queries in +shadow mode, reports audit-safe mismatches through internal telemetry, and +retains the legacy evaluator through the cutover observation window. + +Policy DNS, transparent TCP, native processors, and alternate deployment modes +land only after the shared authorization and relay contracts are authoritative. +Each is a separate, feature-bearing pull request or series with its own +capability gating, migration, telemetry, and rollback plan; they are not bundled +into the compatibility-only refactor. + +Adapter response bytes and OCSF event class, action, disposition, severity, +status, destination, actor, firewall rule, message, and status detail are +compatibility surfaces. Moving code does not justify changing them. Performance +is also measured at each phase; fewer OPA evaluations are a target to verify, +not an unmeasured claim. + +## Implementation plan + +The detailed migration plan lives in [implementation-plan.md](implementation-plan.md). +The intended order is: + +1. Lock down current responses, OCSF events, policy outcomes, credential + behavior, upstream-dial timing, and local-service behavior with regression + coverage. +2. Introduce compatibility `EgressIntent` and `EgressDecision` envelopes while + preserving current lookup precedence and failure defaults. +3. Centralize destination validation behind an unopened connector. +4. Materialize one generation-consistent authorization decision, compare it + against the legacy queries, then cut over deterministic endpoint selection + and fail-closed metadata handling as an independently reviewable step. +5. Consolidate HTTP request-loop, credential, WebSocket, and middleware relay + behavior in separately shippable subphases. +6. Consolidate TLS handling and existing raw TCP relay selection. +7. Preserve current local-service boundaries and remove compatibility plumbing. +8. Add the native protocol-processor dispatch contract, then add protocol + implementations as independently reviewed features. +9. Add policy DNS state and transparent TCP capture with mandatory + DNS-answer-to-connect correlation and separate mapping generations. +10. Define capability-checked standalone or sidecar runtime contracts and + complete cleanup after each boundary is in use. + +Steps 1 through 7 are the compatibility foundation and may themselves span +several pull requests. Steps 8 through 10 are later feature milestones; their +presence in this RFC defines the direction without adding them to the initial +refactor branch. + +## Risks + +- Tightening L7 and TLS metadata failures from fail-open to deny may expose + latent policy or Rego errors. `allowed_ips` and SSRF validation already fail + more conservatively; tests must cover each query independently. +- Deterministic endpoint selection may change ambiguous overlapping policies. + The new decision must shadow the legacy queries and report mismatches before + any semantic cutover. +- Token grants add a runtime dependency on SPIFFE Workload API and token + endpoints. Failures should remain fail-closed and sanitized. +- Transparent TCP capture adds network-namespace interception and mutable DNS + mapping state. It must coexist with nftables reject/log enforcement, prevent + unrelated bare-IP connections from inheriting DNS authorization, and fail + closed across policy or mapping generation changes. +- Sidecar or standalone modes may intentionally lack process identity. + Binary/path-scoped policy needs an advertised identity capability and policy + validation; missing identity cannot silently broaden an allow. +- Metadata loopback and `policy.local` expand sandbox-local control surfaces + and need strict route validation, body limits, redaction, and authentication + boundaries. +- Provider-composed policy rules use a reserved namespace. Decisions and logs + must distinguish provider-derived policy from user-authored policy without + exposing provider rules as editable sandbox proposals. +- Supervisor middleware adds a synchronous request-path dependency. Body caps, + timeout behavior, registry reloads, and `fail_open` choices must be visible + in telemetry so operators can diagnose whether content inspection ran. +- Moving OCSF emission sites can accidentally change event class, action, + disposition, severity, message, or actor/destination fields. Adapter response + shapes and OCSF schemas are compatibility requirements, not cleanup targets. +- New decision/context objects add per-connection allocations on a hot path. + Performance claims require before/after measurements of OPA evaluation count, + allocation volume, and connection/request latency. +- Structural phases back out by reverting their isolated commits. The + deterministic-decision cutover must retain the legacy evaluator long enough + for shadow comparison and immediate rollback; a permanent feature flag is not + required for the purely mechanical phases. + +## Alternatives + +### Keep patching each entry path + +This has the lowest short-term cost but keeps security behavior duplicated +across CONNECT, forward HTTP, and local services. It also makes future TCP +application protocol support harder because each parser must be wired through +multiple entry mechanisms. + +### Replace CONNECT with forward proxy + +Forward proxy only covers plaintext absolute-form HTTP requests. It is not a +replacement for HTTPS tunnels, WebSocket tunnels, or arbitrary TCP clients. +CONNECT should remain the generic explicit proxy mode. + +### Build only transparent TCP + +Transparent TCP helps native clients but does not replace explicit proxy +support used by common HTTP tooling. It also requires the shared authorization, +destination, and relay boundaries plus policy DNS and nftables capture before +it can safely preserve endpoint identity. For that reason it is a later phase +of this RFC, not the first implementation change. + +## Prior art + +The current `openshell-supervisor-network` split is the immediate prior step: +it already separates proxy, OPA, L7, inference routing, policy-local routes, +TLS, and token grants from process supervision. + +The current `openshell-supervisor-process` netns and bypass monitor are the +packet-enforcement substrate. Transparent TCP extends that nftables model in a +later phase rather than creating a second firewall path. + +The existing L7 relay is the behavioral prior art for this RFC. It already +proves per-request HTTP evaluation, GraphQL parsing, JSON-RPC/MCP body +inspection, WebSocket frame handling, request-body rewrite, and token-grant +injection can live behind relay boundaries. + +RFC 0009 supervisor middleware is the extension prior art. It defines +`HTTP_REQUEST / PRE_CREDENTIALS` as a supervisor-owned hook that can inspect, +deny, or transform admitted HTTP requests before credentials are injected. RFC +0005 should place that hook inside the shared relay rather than making each +adapter wire middleware separately. + +## Open questions + +1. Should overlapping endpoint metadata be rejected at policy load time, or + should one documented policy/endpoint precedence key select the complete + decision? The initial compatibility refactor does not choose between them. +2. What mismatch-free observation window is sufficient before the + deterministic decision replaces the legacy endpoint queries? +3. Should metadata loopback be modeled as an adapter inside + `openshell-supervisor-network`, or remain orchestrated by `openshell-sandbox` + with shared credential/provider helpers? +4. What TTL cap should policy DNS use, and should policy reload immediately + invalidate all active mappings or permit a bounded drain period that cannot + authorize new connections? +5. Which original-destination mechanism and nftables redirect mode should each + supported runtime use while keeping capture rules ahead of bypass rejection? +6. Which identity capabilities must standalone and sidecar runtimes advertise + before the gateway accepts binary/path-scoped policy for them? diff --git a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md new file mode 100644 index 0000000000..0a12c6d957 --- /dev/null +++ b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md @@ -0,0 +1,279 @@ +# Current Shape Appendix + +This appendix records the current proxy shape and the review findings that +motivate the adapter model. The main RFC intentionally keeps these details out +of the direction document. + +## Current Runtime Split + +The proxy is no longer only a large module inside `openshell-sandbox`. +Current main has three relevant runtime owners: + +```mermaid +flowchart TD + Sandbox["openshell-sandbox
orchestrator"] + Network["openshell-supervisor-network
proxy, OPA, L7, TLS, inference,
policy.local, token grants"] + Process["openshell-supervisor-process
process leaf, SSH, netns,
nftables, bypass monitor"] + Denials["Denial/activity aggregators"] + Gateway["Gateway policy/provider APIs"] + + Sandbox --> Network + Sandbox --> Process + Network --> Denials + Process --> Denials + Sandbox --> Gateway + Network --> Gateway +``` + +`openshell-sandbox` creates the shared network namespace, owns denial/activity +channels, starts the policy poll loop, starts networking, starts the metadata +loopback server when needed, and then optionally starts the process leaf. If +`process_enabled` is false, the supervisor can run in network-only mode and +keep networking/background tasks alive until shutdown. + +`openshell-supervisor-network` owns the explicit proxy listener, OPA engine +integration, L7 enforcement, TLS termination, inference routing, policy-local +routes, identity cache, provider credential injection, and token grants. + +`openshell-supervisor-process` owns process execution, SSH, network namespace +helpers, nftables bypass rules, and the bypass monitor that turns nftables LOG +entries into OCSF events. + +In embedded supervisor mode, the network leaf normally uses process metadata +resolved by the process/orchestrator side for binary-scoped policy and OCSF +context. Current runtime configuration can instead select endpoint-only policy +evaluation. The adapter model must preserve that intentional mode separately +from an identity lookup failure; the latter continues to deny when binary +identity is required. + +## Current Userland-Facing Surfaces + +The networking surface currently includes: + +- CONNECT proxy traffic for HTTPS and generic TCP tunnels. +- Forward HTTP proxy traffic for absolute-form HTTP requests. +- `inference.local` for local inference routing. +- `policy.local` for current policy, denial summaries, proposal submission, + and proposal wait routes. +- GCE metadata loopback for SDKs that bypass HTTP proxy variables. +- nftables bypass enforcement for direct TCP/UDP egress that does not enter + the proxy. +- OPA/Rego policy and endpoint metadata lookups. +- DNS resolution and endpoint validation for CONNECT and forward HTTP egress. +- Static provider credential injection and redaction. +- Endpoint-bound dynamic token grant injection. +- Opt-in REST request-body credential rewrite. +- L7 REST, GraphQL, JSON-RPC, MCP, WebSocket, and + GraphQL-over-WebSocket enforcement. + +The issue is not that these features exist. The issue is that entry mechanisms, +policy evaluation, endpoint metadata lookup, credential injection, and byte +relay decisions are still interleaved. + +## Current CONNECT Shape + +```mermaid +flowchart TD + Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["CONNECT denial"] + Allowed -- Yes --> Meta["Query endpoint metadata"] + Meta --> Config{"L7, TLS, or credential config?"} + Config -- No --> Tunnel["Return tunnel-ready response"] + Config -- Yes --> Tunnel + Tunnel --> Inspect["Inspect tunneled bytes when possible"] + Inspect --> Relay["HTTP/WebSocket/TCP relay selection"] + Relay --> Inject["Middleware, static credentials, and token grants if configured"] + Inject --> Upstream["Open upstream when relay policy allows"] +``` + +CONNECT is still the strongest entry shape because the tunnel relay can keep +parsing HTTP requests on long-lived connections and enforce request policy per +request. + +## Current Forward HTTP Shape + +```mermaid +flowchart TD + Client["Absolute-form HTTP request"] --> Parse["Parse first request"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["HTTP denial"] + Allowed -- Yes --> L7{"Matching L7 endpoint?"} + L7 -- Yes --> Eval["Evaluate REST/GraphQL/JSON-RPC/MCP/WebSocket policy"] + Eval --> Guard["Reject unsupported h2c upgrade when inspected"] + Guard --> Rewrite["Rewrite to origin-form + configured credentials"] + L7 -- No --> Rewrite + Rewrite --> Token["Apply token grant if endpoint-bound"] + Token --> Close["Force Connection: close except WebSocket upgrade"] + Close --> Upstream["Open upstream"] + Upstream --> Relay["Guarded HTTP relay / upgrade relay"] +``` + +Latest main no longer has the old raw-copy-after-first-request shape for +ordinary forward HTTP. It rewrites ordinary requests with `Connection: close`, +uses guarded HTTP relay helpers for body handling, rejects inspected h2c +upgrades, injects token grants, and sends allowed WebSocket upgrades through +the upgrade relay. That is a narrower surface than the historical bidirectional +copy, but it is still orchestrated separately from the CONNECT relay path. + +## Current Local Service Shape + +```mermaid +flowchart TD + Request["Request to local name"] --> Match{"Known local route?"} + Match -- "inference.local" --> Inference["Inference route adapter"] + Match -- "policy.local" --> Policy["Policy local adapter"] + Match -- "metadata loopback" --> Metadata["Metadata credential server"] + Match -- No --> External["Normal egress path"] + Inference --> InferenceResp["Local inference response"] + Policy --> PolicyResp["Local policy response"] + Metadata --> MetadataResp["Metadata response"] +``` + +`inference.local` now covers buffered and streaming inference shapes including +chat/completion routes, model discovery, embeddings, and provider-specific +routes. `policy.local` supports the agentic approval loop: agents can submit +narrow proposals and wait on approval/reload before retrying. Metadata +loopback exists for provider credentials consumed by SDKs that do not honor +HTTP proxy variables. + +These are userland-facing network surfaces. They should stay distinct from +external egress while still fitting the adapter model. + +## Adjacent In-Flight Supervisor Middleware Shape + +PRs #1738 and #2027 propose supervisor middleware as an HTTP request hook in +the proxy relay. That work is adjacent to this RFC rather than a separate entry +adapter. + +```mermaid +flowchart TD + Req["Parsed admitted HTTP request"] --> Policy["Network and request policy already allowed"] + Policy --> Hook["HTTP_REQUEST / PRE_CREDENTIALS middleware"] + Hook --> Outcome{"Middleware outcome"} + Outcome -- "deny" --> Deny["Local deny, no credential injection"] + Outcome -- "allow / mutate" --> Creds["Credential injection"] + Creds --> Upstream["Upstream write"] +``` + +The proposed middleware chain is selected by admitted destination host, runs in +deterministic order, buffers bounded request bodies, applies `fail_open` or +`fail_closed`, emits audit-safe findings, and runs before OpenShell-managed +credentials are injected. It can inspect WebSocket upgrade requests because +they are HTTP requests, but it does not inspect post-upgrade WebSocket frames +in v1. + +RFC 0005 should account for this by treating middleware as part of the shared +request processing plan. CONNECT and forward HTTP should not each learn how to +select and invoke middleware independently. + +## Current Network Namespace Enforcement + +```mermaid +flowchart TD + Start["Process in sandbox network namespace"] --> Dest{"Destination"} + Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"] + Dest -- "Loopback" --> Loopback["Accept loopback"] + Dest -- "Established/related" --> Established["Accept response packet"] + Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"] + Reject --> Monitor["Bypass monitor reads dmesg"] + Monitor --> OCSF["OCSF network + detection events"] +``` + +The process leaf installs an `inet` nftables filter table for bypass +enforcement. The table accepts proxy-bound traffic, loopback, and established +flows, then rejects and optionally logs other TCP/UDP traffic. It does not +currently redirect native TCP connections into the proxy. + +## Findings To Preserve + +### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes + +The historical forward path evaluated at most the first absolute-form request, +rewrote it, then switched to bidirectional copy. Bytes already buffered after +the first header block, or later pipelined requests on the same client/upstream +connection, could reach upstream without the CONNECT L7 relay's per-request +parser/evaluator. + +Latest main mitigates this by forcing ordinary forward HTTP to one request per +connection and by using guarded relay helpers. The adapter model should +preserve the invariant either by keeping forward HTTP single-request/close or +by passing the first parsed request into a shared HTTP relay loop. + +### Endpoint config is not tied to deterministic matched policy + +The policy name used for L4 authorization and logging is the lexicographically +smallest matching policy. L7 candidates are collected independently and later +selected by request-path specificity. TLS and `allowed_ips` use the first +extended endpoint config returned by a separate query. Exact-declared-host is +another independent existential query. With overlapping host, port, and binary +rules, those results can describe different endpoints and policies on the same +connection. + +The adapter model requires authorization to return one decision with one +deterministic matched endpoint. + +### Policy materialization can span generations + +The L4 decision, L7 route, TLS mode, `allowed_ips`, and exact-declared-host +signal are materialized through separate engine calls. The L7 route records a +generation for its tunnel evaluator, but the sibling metadata queries are not +all asserted against the L4 decision generation. A reload during setup can +therefore assemble one logical connection decision from different policy +generations. + +The target decision carries one top-level policy generation. Every +policy-derived field must be evaluated from that generation, and relay startup +must reject a stale decision before an upstream request is written. + +### Endpoint metadata query failures should not erase enforcement + +Failure behavior is not uniform today. L7 configuration failure becomes no L7 +configuration, and TLS configuration failure becomes automatic TLS handling; +either can erase intended enforcement. `allowed_ips` and the downstream SSRF +validation path are already more conservative. The migration must test each +query independently instead of describing all metadata failures as equivalent. + +The adapter model treats endpoint metadata as part of the authorization result. +Failure to materialize required metadata should deny rather than erase extended +configuration. + +### Destination validation must be shared + +Private address checks, `allowed_ips`, exact declared private endpoint trust, +trusted gateway aliases, SSRF checks, and control-plane port blocks have grown +over time. They should be centralized so CONNECT and forward HTTP use the same +resolved-destination rules. Existing local services remain outside normal +external destination validation. + +## Existing Feature Inventory + +The refactor should preserve: + +- CONNECT explicit proxy support. +- Forward HTTP explicit proxy support. +- Network-only supervisor mode. +- nftables bypass reject/log enforcement. +- Provider credential injection and redaction. +- Dynamic token grant injection through SPIFFE-backed provider credentials. +- Supervisor middleware `HTTP_REQUEST / PRE_CREDENTIALS` when it lands. +- REST request-body credential rewrite. +- WebSocket text-frame credential rewrite. +- REST endpoint method/path policy. +- GraphQL-over-HTTP policy. +- JSON-RPC-over-HTTP method policy. +- MCP Streamable HTTP method and tool policy. +- WebSocket transport and GraphQL-over-WebSocket policy. +- h2c rejection on inspected HTTP routes. +- Inference routing through `inference.local`, including embeddings. +- Agent-facing policy advisor routes through `policy.local`. +- GCE metadata loopback for supported provider credentials. +- Timeout and resource tracking for client, upstream, and local service work. +- Structured OCSF logging for network and HTTP policy outcomes. +- SSRF and internal address protections. +- Exact declared private endpoint handling. +- Control-plane port protection. +- `allowed_ips` endpoint restrictions. +- TLS auto-detection and termination for inspectable client connections. diff --git a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md new file mode 100644 index 0000000000..20a685c42e --- /dev/null +++ b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md @@ -0,0 +1,258 @@ +# Implementation Plan + +This plan is intentionally separate from the main RFC so the proposal can stay +direction-focused. The RFC is an incremental roadmap, not one pull request. +Phases 0 through 7 form the compatibility foundation: they restructure current +CONNECT, forward HTTP, raw TCP, and local-service behavior without adding a new +user-facing transport. Phases 8 and later add the forward-looking capabilities +after the shared contracts are authoritative. + +## Phase 0 - Compatibility Baseline + +- Cover CONNECT and forward HTTP allow/deny responses, including exact status, + headers, and adapter-specific error bodies. +- Cover forward HTTP pipelining, keep-alive follow-on requests, the current + `Connection: close` mitigation, `https://` absolute-form rejection, and h2c + rejection on inspected endpoints. +- Cover the current overlapping-policy outcomes separately for matched policy, + L7 route selection, TLS, `allowed_ips`, and exact-declared-host. +- Inject failures into L7, TLS, `allowed_ips`, and exact-declared-host queries. + Record the current fail-open or fail-closed result for each query rather than + treating all endpoint metadata errors as equivalent. +- Cover control-plane ports, cloud metadata, always-blocked addresses, exact + declared private endpoints, IP-literal synthesis, trusted gateway aliases, + and explicit `allowed_ips` through CONNECT and forward HTTP. +- Cover identity-required success/failure, unsupported-platform behavior where + possible, and intentional endpoint-only evaluation. Prove an empty + `exec.path` cannot satisfy binary-scoped policy while identity is required. +- Cover static credential injection, token grants, REST body rewrite, + WebSocket text-frame rewrite and policy, GraphQL, JSON-RPC, and MCP behavior. +- Cover `inference.local`, `policy.local`, metadata loopback, and unchanged + nftables bypass reject/log behavior. +- Capture stable OCSF event class, activity/action/disposition, severity, + status, destination, actor, firewall rule, message, and status detail for + representative allow and deny paths. +- Record a performance baseline for OPA evaluations, per-connection + allocations, and CONNECT/forward request latency. + +## Phase 1 - Adapters And Compatibility Decision Envelope + +- Introduce CONNECT and forward HTTP `EgressIntent` construction inside + `openshell-supervisor-network`. +- Introduce a transitional `EgressDecision` carrying L4 outcome, policy + generation, process evidence, and endpoint fields while preserving the + current query timing, precedence, and failure defaults. +- Keep `LookupFailed`/unsupported identity as a denial when identity is + required. Keep explicitly configured endpoint-only mode behavior unchanged. +- Keep adapter-specific responses and OCSF emission at the protocol boundary. +- Do not claim the transitional decision is one atomic OPA result; document its + compatibility hydration until Phase 3 cuts over. + +This phase is a mechanical extraction. It must be independently shippable and +revertible without changing user-visible policy or relay behavior. + +## Phase 2 - Shared Destination Validation + +- Move DNS resolution, explicit `allowed_ips`, exact declared endpoints, + implicit IP-literal handling, trusted gateway aliases, SSRF checks, + cloud-metadata blocks, and control-plane-port blocks into one validator. +- Represent the selected validation mode explicitly instead of passing an + ambiguous collection of booleans. +- Return an unopened `UpstreamConnector` so adapters and relays preserve the + current point at which upstream TCP is created. +- Prove CONNECT and forward HTTP retain their existing denial responses, OCSF + fields, and dial timing while using the shared validator. + +## Phase 3 - Generation-Consistent Authorization Cutover + +- Define either rejection of ambiguous overlapping endpoint metadata or one + documented policy/endpoint precedence key before changing enforcement. +- Add one OPA result that materializes matched policy/source, matched endpoint, + destination constraints, TLS, HTTP enforcement, credential plan, and + middleware selection from one policy generation. +- Attach a generation-pinned `TunnelPolicyEngine` to relay context for + per-request REST, GraphQL, JSON-RPC, MCP, and WebSocket evaluation. Relays do + not rematerialize connection-level endpoint policy. +- Run the new query in shadow mode beside legacy queries. Emit internal, + audit-safe mismatch telemetry without changing existing OCSF network/HTTP + events or enforcement. +- Add reload-race tests proving every materialized field matches the top-level + generation and stale decisions stop before upstream request write. +- Make deterministic selection and fail-closed L7/TLS metadata errors a + dedicated cutover only after mismatch cases are understood. Retain the + legacy evaluator temporarily for immediate rollback. + +This is the only phase that intentionally tightens ambiguous or error behavior; +it must not be hidden inside the structural refactor commits. + +## Phase 4 - Forward HTTP Adapter + +- Keep absolute-form parsing and adapter-specific errors at the forward HTTP + boundary. +- Pass the buffered first request into a shared HTTP relay, or retain the + guarded single-request/`Connection: close` path until Phase 5a is ready. +- Preserve `https://` absolute-form rejection and inspected h2c rejection. +- Preserve the invariant that no unevaluated follow-on request can reach raw + bidirectional copy. + +## Phase 5 - Relay Consolidation + +### Phase 5a - HTTP request loop + +- Centralize HTTP parsing and per-request REST, GraphQL, JSON-RPC, and MCP + evaluation behind the generation-pinned request-policy handle. +- Evaluate every request before upstream write and preserve the current rule + that a denied request does not create an upstream session. +- Preserve bounded JSON-RPC/MCP inspection and audit-safe logging that omits + params and tool arguments. + +### Phase 5b - Credential injection + +- Unify static target/query/header rewrite, endpoint-bound token grants, and + opt-in REST request-body rewrite after request allow and before upstream + write. +- Preserve buffering limits, supported content types, `Content-Length` + recomputation, redaction, token caching, and fail-closed unresolved secrets. + +### Phase 5c - WebSocket + +- Move allowed upgrades behind the shared relay while preserving raw upgraded + passthrough, opt-in text-frame credential rewrite, WebSocket transport policy, + GraphQL-over-WebSocket policy, and safe compression behavior. + +### Phase 5d - Supervisor middleware + +- Land only after the supervisor middleware dependency is available. +- Run `HTTP_REQUEST / PRE_CREDENTIALS` after request allow and before static or + dynamic credential injection. +- Preserve ordering, body caps, `fail_open`/`fail_closed`, safe headers, + findings, metadata, and rejection of middleware-introduced credential + placeholders. + +Each subphase must be independently testable and shippable; Phase 5 is not a +single flag-day cutover. + +## Phase 6 - Shared TLS And TCP Relay Boundary + +- Move client-side TLS detection and termination before the HTTP/raw-TCP relay + split without changing handshake, certificate, or upstream-connect timing. +- Keep endpoint TLS behavior on `EgressDecision` and preserve `tls: skip` as the + explicit raw-tunnel path. +- Use one existing raw `TcpRelay` byte-copy primitive for L4 traffic. +- Add a protocol-processor dispatch contract without enabling a concrete new + protocol in the compatibility milestone. +- Let processors own their message loop and call the validated connector only + when protocol state allows. Permit in-tree, middleware-backed, and hybrid + processors with typed middleware operations. + +## Phase 7 - Existing Local Services And Cleanup + +- Keep `inference.local` as a local adapter with its existing TLS, route, + provider-auth, streaming/buffered limit, and OCSF behavior. +- Keep `policy.local` as a local adapter for current policy, bounded denial + summaries, proposals, and proposal wait. +- Decide whether metadata loopback remains orchestrated by `openshell-sandbox` + or moves behind a local adapter boundary; preserve startup/failure behavior + either way. +- Keep the local-routing and destination contracts extensible for issue + [#1633](https://github.com/NVIDIA/OpenShell/issues/1633), while leaving its + policy surface and host-loopback authorization to separate feature work. +- Remove compatibility endpoint queries only after Phase 3 is authoritative. +- Remove duplicated destination/relay plumbing without centralizing + adapter-specific response rendering. +- Update the living architecture documentation once each implemented boundary + reflects current code. + +Completion of Phase 7 is the compatibility milestone: existing user-facing +features and capabilities are preserved on the new internal structure. The +following phases are feature-bearing work and land in separate pull requests or +series. + +## Phase 8 - Policy DNS And Transparent TCP + +- Add policy DNS registration for native TCP endpoint names. +- Replace static host-file mapping with query-driven DNS answers resolved + through trusted DNS and filtered through destination controls. +- Store normalized name, endpoint ID, IP/port, policy generation, distinct DNS + mapping generation, mapping ID, and expiration in active mapping state. +- Require every captured connect to correlate with the unexpired mapping + created by its policy-eligible DNS answer. Do not allow unrelated bare-IP + traffic to inherit a policy-DNS decision. +- Publish mapping and nftables capture updates atomically from the adapter's + perspective. +- Add nftables REDIRECT/TPROXY capture rules ahead of the bypass reject path; + do not add a parallel iptables path. +- Coordinate capture-rule ownership with + `openshell-supervisor-process::netns` and preserve reject/log fallback for + unmatched traffic. +- Recover the original destination, construct a transparent-TCP intent, and run + normal generation-consistent authorization and destination validation. +- Define TTL caps, policy-reload invalidation, stale-mapping behavior, and + rollback before enabling capture by default. + +## Phase 9 - Native Protocol Processors + +- Add concrete Redis, Postgres, MySQL, or other processors one protocol at a + time, each with a separately reviewed policy schema and operational limits. +- Keep omitted/`tcp` endpoints on raw L4 byte copy; never infer a native + processor from traffic alone. +- Test multi-message sessions, pre-dial denial, handshake-required dialing, + per-command/query evaluation, middleware hooks, timeouts, and redaction. +- Capability-gate policy that names a processor unavailable in the running + proxy build. + +## Phase 10 - Runtime Boundary + +- Keep embedded and network-only supervisor modes as the migration baseline. +- Define the proxy runtime API needed for a future standalone binary or + sidecar: configured listeners, policy updates, provider credentials, token + grants, middleware registry, gateway calls, telemetry, denial/activity + events, and shutdown. +- Advertise process-identity and protocol-processor capabilities. Reject policy + that requires unavailable binary/path identity or processor support. +- Represent intentional runtime identity unavailability separately from the + existing endpoint-only mode and from lookup failure. +- Add gateway capability negotiation if proxy and gateway versions can differ. + +## Phase 11 - Final Cleanup + +- Remove any compatibility query/evaluator retained for deterministic-decision + rollback after its observation window closes. +- Remove stale static `/etc/hosts`, iptables, or single-process assumptions from + proxy design and architecture documentation as the corresponding later phase + lands. +- Keep adapter-specific response rendering and OCSF contracts at their protocol + boundaries. + +## Testing And Operational Validation + +- Unit-test adapter intent construction, response rendering, explicit + destination modes, identity evidence, and authorization precedence. +- Integration-test destination validation across CONNECT and forward HTTP, + then reuse the same suite for transparent TCP when Phase 8 lands. +- Integration-test HTTP keep-alive/pipelining, REST, GraphQL, JSON-RPC, MCP, + WebSocket, credentials, token grants, middleware, and TLS/raw-TCP selection. +- Integration-test `inference.local`, `policy.local`, and metadata loopback body + limits, timeouts, redaction, and local denial responses. +- Compare OCSF fixtures before and after each migration subphase. +- Exercise policy reload between L4 decision, endpoint materialization, relay + startup, and long-lived per-request evaluation. +- Add protocol-processor harness tests before adding Redis, Postgres, MySQL, or + similar enforcement. Each concrete processor adds multi-message, handshake, + timeout, denial, redaction, and middleware coverage. +- Integration-test policy DNS filtering, TTL, distinct mapping generations, + policy-reload invalidation, atomic capture-rule updates, original-destination + recovery, captured-connect correlation, and rejection of unrelated bare-IP + connects. +- Test standalone/sidecar capability negotiation and prove missing identity or + processor support fails during policy validation rather than broadening an + allow at runtime. +- Re-run the performance baseline after the compatibility envelope, after the + single-decision query, and after relay consolidation. Treat reduced OPA calls + as a measured result rather than an assumed benefit. +- Back out structural phases by reverting their isolated commits. Keep shadow + comparison and the legacy evaluator available through the deterministic + cutover observation window. +- Gate later transport/runtime phases independently so disabling policy DNS or + transparent capture restores the existing explicit-proxy and bypass-reject + behavior without reverting the compatibility foundation. diff --git a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md new file mode 100644 index 0000000000..b03d07b36a --- /dev/null +++ b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md @@ -0,0 +1,563 @@ +# Technical Design Appendix + +This appendix carries implementation-level design details behind the main RFC. + +## Existing Runtime Boundary + +`openshell-supervisor-network::run::run_networking` is the current networking +startup boundary. It builds policy-local context, waits for policy binary +symlink resolution, creates the identity cache, writes the TLS CA, builds TLS +state, resolves inference routes, wires provider credentials and token grants, +and starts the proxy. The supervisor middleware work extends this boundary with +middleware registry construction and reload behavior. + +This is a useful outer boundary, but it is not yet the proxy adapter boundary. +The proxy still needs internal `EgressIntent` and `EgressDecision` boundaries +so CONNECT, forward HTTP, local routes, and future native TCP capture do not +duplicate policy and relay orchestration. The first implementation milestone +wires only current surfaces; later milestones add new adapters to the same +contract. + +## Shared Data Boundaries + +### EgressIntent + +`EgressIntent` is the normalized description of what userland is trying to do. + +It should carry: + +- entry transport: CONNECT, forward HTTP, transparent TCP, local HTTP, policy + DNS, or metadata loopback; +- requested destination host/port or captured original IP/port; +- optional process identity inputs collected by the adapter/runtime; +- optional first HTTP request for forward proxy traffic; +- optional local service route; +- policy generation and, for policy DNS/transparent TCP, a distinct DNS + mapping generation and correlation handle. + +Adapters build intents. They should not query endpoint metadata, select TLS +mode, or select relays. + +### EgressDecision + +`EgressDecision` is the policy result consumed by validation and relay code. + +It should carry: + +- allow or deny; +- one top-level policy generation used for every policy-derived field; +- deterministic matched policy identifier; +- whether the policy is user-authored, provider-derived, or local-service + internal; +- deterministic matched endpoint identifier and endpoint metadata; +- process identity availability and any identity fields used for evaluation; +- destination and allowed IP constraints; +- TLS behavior; +- protocol enforcement; +- credential injection plan; +- supervisor middleware plan; +- the request-policy selection needed to create a pinned per-request L7 + evaluator when HTTP inspection is configured; +- logging context and denial reason. + +Relay code should read this decision. It should not query OPA again for +endpoint metadata, TLS mode, allowed IPs, credential behavior, middleware +selection, or relay selection. Long-lived HTTP relays still evaluate each +request through the generation-pinned L7 evaluator carried in `RelayContext`; +that is request authorization, not endpoint rematerialization. Later native +protocol processors use the same pattern with a generation-pinned protocol +evaluator for per-command or per-query decisions. + +## Protocol Enforcement + +Use a protocol enforcement value derived from endpoint policy: + +| Policy protocol | Enforcement | Relay behavior | +|-----------------|-------------|----------------| +| omitted / `tcp` | None | L4 authorization plus byte relay, with optional HTTP sniff for credential injection | +| `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite | +| `graphql` | HTTP | HTTP request parser with GraphQL-over-HTTP rules | +| `json-rpc` | HTTP | HTTP request parser plus bounded JSON-RPC-over-HTTP method inspection | +| `mcp` | HTTP | HTTP request parser plus bounded MCP Streamable HTTP method/tool inspection | +| `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy | +| future `redis`, `postgres`, `mysql`, ... | Protocol processor | Protocol-specific processor owns framing, middleware hooks, and the message loop | + +`protocol: tcp` is effectively the default L4 mode. It should not run native +protocol processors. Avoid using the term "provider" for processor concepts +because providers are already a first-class credential and routing domain in +OpenShell. Concrete native processors land after the shared dispatch contract. + +## Suggested Types + +The exact Rust shape can evolve, but the boundaries should look like this: + +```rust +enum EgressTransport { + Connect, + ForwardHttp, + TransparentTcp, + PolicyDns, + LocalHttp, + MetadataLoopback, +} + +struct EgressIntent { + transport: EgressTransport, + destination: RequestedDestination, + process: ProcessIdentityEvidence, + first_request: Option, + local_route: Option, + correlation: Option, +} + +struct EgressDecision { + policy_generation: PolicyGeneration, + outcome: PolicyOutcome, + matched_policy: Option, + endpoint: Option, + process: EvaluatedProcessIdentity, + request_processing: RequestProcessingPlan, + log_context: EgressLogContext, +} + +enum ProcessIdentityEvidence { + Available(ProcessIdentity), + Unavailable(ProcessIdentityUnavailableReason), +} + +enum ProcessIdentityUnavailableReason { + EndpointOnlyMode, + DeclaredRuntimeMode(RuntimeMode), + UnsupportedPlatform, + LookupFailed, +} + +struct EvaluatedProcessIdentity { + evidence: ProcessIdentityEvidence, + fields_used: Vec, +} + +struct MatchedPolicy { + id: PolicyId, + source: PolicySource, +} + +enum PolicySource { + User, + ProviderDerived, + LocalService, +} + +struct MatchedEndpoint { + id: EndpointId, + destination: DestinationValidationPlan, + tls: TlsPolicy, + enforcement: ProtocolEnforcement, +} + +struct DestinationValidationPlan { + address_authorization: AddressAuthorization, +} + +enum AddressAuthorization { + DefaultPublicOnly, + ExplicitAllowedIps(Vec), + ExactDeclaredHost, + ImplicitIpLiteral(IpAddr), + TrustedGatewayAlias { expected_ip: IpAddr }, +} + +struct RequestProcessingPlan { + middleware: SupervisorMiddlewarePlan, + credentials: CredentialInjectionPlan, +} + +enum ProtocolEnforcement { + None, + Http(HttpL7Config), + ProtocolProcessor(ProtocolProcessorConfig), +} + +enum HttpL7Protocol { + Rest, + Graphql, + JsonRpc, + Mcp, + Websocket, +} + +struct HttpL7Config { + protocol: HttpL7Protocol, + path: EndpointPathScope, + allow_encoded_slash: bool, + enforcement_mode: L7EnforcementMode, + websocket_credential_rewrite: bool, + request_body_credential_rewrite: bool, + websocket_graphql_policy: bool, + graphql_max_body_bytes: usize, + json_rpc_max_body_bytes: usize, + mcp_strict_tool_names: bool, +} + +struct CredentialInjectionPlan { + static_placeholders: StaticPlaceholderPlan, + token_grant: Option, +} + +struct StaticPlaceholderPlan { + http_target_query_header: bool, + rest_request_body: bool, + websocket_text_frames: bool, +} + +struct TokenGrantPlan { + provider_key: String, + auth_style: TokenGrantAuthStyle, + token_endpoint: String, +} + +struct SupervisorMiddlewarePlan { + stages: Vec, + min_body_limit: Option, + registry_generation: PolicyGeneration, +} + +struct SupervisorMiddlewareStage { + policy_name: String, + binding_id: String, + operation: MiddlewareOperation, + phase: MiddlewarePhase, + order: i32, + on_error: MiddlewareOnError, + config: MiddlewareConfig, +} + +enum MiddlewareOperation { + HttpRequest, + Future(String), +} + +enum MiddlewarePhase { + PreCredentials, + Future(String), +} + +struct RelayContext { + decision: EgressDecision, + request_policy: Option, + protocol_policy: Option, + connector: UpstreamConnector, + deadlines: RelayDeadlines, + telemetry: RelayTelemetry, +} + +struct ResolvedEndpointCorrelation { + policy_generation: PolicyGeneration, + mapping_generation: DnsMappingGeneration, + mapping_id: DnsMappingId, +} + +struct PinnedRequestPolicy { + generation: PolicyGeneration, + evaluator: TunnelPolicyEngine, +} + +struct PinnedProtocolPolicy { + generation: PolicyGeneration, + evaluator: ProtocolPolicyEngine, +} +``` + +`UpstreamConnector` is the relay-owned dial boundary. It encapsulates the +validated destination and lets relays or processors open an upstream connection +only after current request or protocol policy allows it. + +`DestinationValidationPlan` selects one current validation mode. All modes +retain control-plane-port and cloud-metadata blocks. Default and explicit IP +paths retain the always-blocked loopback, link-local, and unspecified-address +checks. `ImplicitIpLiteral` is synthesized only for an explicitly declared IP +host. `TrustedGatewayAlias` may accept the one runtime-discovered gateway IP but +does not become a general private-address exemption. + +`policy_generation`, the optional pinned request/protocol evaluators, endpoint +metadata, and middleware selection must describe one policy snapshot. +Authorization asserts that every sub-materialization used that generation. If +the generation changes before relay startup, the adapter receives a +stale-policy denial rather than a mixed decision. + +## Process Identity Availability + +Process identity is evidence, not a string to fabricate when lookup fails. +Embedded mode normally populates binary, PID, ancestry, command-line path, and +binary hash data. When binary identity is required, `LookupFailed` and +`UnsupportedPlatform` remain denials. An explicitly configured endpoint-only +runtime records `Unavailable(EndpointOnlyMode)` and keeps the current +endpoint-only policy behavior. This RFC does not silently turn one state into +the other or change the endpoint-only trust contract. + +A future standalone or sidecar runtime that intentionally lacks local identity +uses `DeclaredRuntimeMode`, not `EndpointOnlyMode`, and advertises that +capability to policy validation. The runtime contract must define binary/path +predicates as unavailable and reject incompatible policy before traffic starts, +unless a later accepted policy design specifies a different fail-closed rule. + +The decision records identity availability and fields used so OCSF logs and +deny responses distinguish a binary policy denial, an identity lookup failure, +and intentional endpoint-only evaluation. Tests must prove an empty synthetic +`exec.path` cannot satisfy a binary-scoped rule while identity is required. +Adding new identity-less deployment modes, or changing how binary predicates +behave in endpoint-only mode, requires the capability work in the later runtime +phase and cannot be smuggled into the compatibility refactor. + +## Current Owners And Proposed Cleanup + +| Current owner | Current responsibility | Proposed cleanup | +|---------------|------------------------|------------------| +| `openshell-sandbox` | Orchestrator, policy poll loop, denial/activity channels, metadata loopback startup, network-only lifecycle | Keep as orchestration; avoid embedding per-entry proxy policy decisions | +| `openshell-supervisor-network::run` | Networking startup and handles | Become the stable runtime API for embedded and future standalone modes | +| `openshell-supervisor-network::proxy` | CONNECT, forward HTTP, local route dispatch, destination validation, denial rendering | Split into adapters, authorization, destination, relay selection, and adapter response rendering | +| `openshell-supervisor-network::opa` | Policy engine and Rego queries | Return deterministic `EgressDecision` data instead of separate policy and endpoint lookups | +| `openshell-supervisor-network::l7` | REST, GraphQL, JSON-RPC, MCP, WebSocket, inference helpers, TLS, token grants | Keep as protocol/relay implementation behind shared relay boundaries | +| `openshell-supervisor-network::policy_local` | `policy.local` state and routes | Model as a local adapter with explicit limits and proposal/wait behavior | +| `openshell-supervisor-middleware` | Middleware registry, built-ins, service contract, and chain execution | Treat as a relay hook dependency selected by `EgressDecision`, not as adapter-specific policy logic | +| `openshell-supervisor-process::netns` | nftables bypass rules and namespace helpers | Remain owner of bypass enforcement; coordinate future capture rules with network proxy mappings | +| `openshell-supervisor-process::bypass_monitor` | nftables LOG parsing and OCSF bypass telemetry | Remain telemetry producer for bypass violations | +| `openshell-core::secrets` and provider credential state | Static placeholder sources and dynamic credential metadata | Feed credential injection plans; do not leak secrets into decision logs | + +## Policy DNS And Resolved TCP State + +Policy DNS is query-driven rather than a static `/etc/hosts` snapshot. + +1. Policy load registers eligible native TCP endpoint names. +2. Userland performs a DNS lookup. +3. Policy DNS checks whether the name is eligible in the current policy + generation. +4. Policy DNS resolves through trusted upstream DNS and filters answers through + endpoint metadata and SSRF controls. +5. The adapter publishes the DNS answer plus an active mapping containing the + normalized name, endpoint identifier, IP/port, policy generation, distinct + DNS mapping generation, opaque mapping ID, and expiration. +6. Capture rules are updated atomically for the active mapping. +7. Userland later calls `connect(ip:port)`. +8. Transparent TCP recovers the original destination and requires an unexpired + mapping that correlates that connect with the policy-eligible DNS answer. +9. Normal egress authorization and relay selection run against a policy + generation consistent with the mapping contract. + +The resolved endpoint store is active state produced by policy-eligible lookups +and consumed by transparent TCP connects. Policy generation and DNS mapping +generation are separate values: a DNS refresh can replace mappings without a +policy reload, while a policy reload can invalidate mappings whose endpoint +contract is no longer current. A captured connection with no mapping, a stale +mapping, or a mismatched endpoint/port fails closed. An unrelated bare-IP +connection cannot inherit a policy-DNS authorization merely because it targets +an IP currently present in the mapping store. + +## nftables Boundary + +Current main uses nftables, not iptables, for sandbox network bypass +enforcement. The installed `inet` table accepts traffic to the sandbox proxy, +loopback, and established/related flows, then rejects and optionally logs other +TCP/UDP traffic. The bypass monitor reads those log lines and emits OCSF +network and detection events. + +Transparent TCP capture builds on this same nftables substrate in a later +feature phase: + +- capture rules run before the generic bypass reject rules; +- capture rules are scoped to active policy-DNS IP/port mappings; +- mapping and capture-rule updates are atomic from the adapter's perspective; +- reject/log rules remain the fallback for unmatched TCP/UDP egress; +- VM or Podman driver nftables rules are infrastructure NAT/isolation and are + not the proxy policy enforcement point. + +The initial CONNECT/forward refactor does not change the installed table. This +section defines the consumer contract that the shared adapter and decision +boundaries must support when transparent capture lands. + +## Endpoint Selection And OPA + +Today the matched policy name, L7 candidates, first TLS/`allowed_ips` endpoint, +and exact-declared-host signal are selected through independent rules. OPA/Rego +should return policy and endpoint metadata through one deterministic +authorization result. It should not let those fields describe different +matches. + +Two acceptable approaches: + +- Reject overlapping endpoint metadata at load or merge time. +- Define a single deterministic precedence key and use it for both policy name + and endpoint metadata. + +Endpoint metadata query failures should fail closed when metadata is required +for the selected endpoint. They should not silently downgrade to L4 behavior. +The top-level decision generation must also match every policy-derived field; +reload during materialization yields a stale decision instead of mixing +generations. + +This semantic cutover is separate from introducing the Rust types. The new +query first runs in shadow mode beside the legacy queries, records audit-safe +mismatches through internal telemetry, and preserves legacy enforcement. After +the precedence rule is accepted and mismatch cases are understood, a dedicated +change switches the authoritative result and retains the legacy evaluator long +enough for immediate rollback. + +Provider-derived policies use a reserved rule-name namespace. The gateway and +sandbox sync should prevent user-authored `_provider_*` rules, and +`policy.local` proposal surfaces should not expose provider-derived rules as +editable user policy. `EgressDecision` should still identify provider-derived +matches for logging and debugging. + +## Credential Injection Boundary + +Credential injection belongs in the HTTP/WebSocket relay after policy allow and +supervisor middleware, and before upstream write. + +1. Authorization selects the endpoint and computes a credential injection plan. +2. Supervisor middleware runs on the admitted request before credentials are + visible. +3. The HTTP relay resolves credentials only when it has an allowed request. +4. Static placeholder values are resolved and redacted from logs. +5. Endpoint-bound token grants obtain or reuse a dynamic access token. +6. The final upstream request or WebSocket frame is rewritten immediately + before write. + +Both L4-only HTTP and HTTP-inspected paths can inject credentials. The +difference is whether REST, GraphQL, or WebSocket policy is evaluated before +the rewrite. + +Credential rewrite slots should be explicit: + +- request target, query values, and headers for HTTP-family traffic; +- REST request bodies only when `request_body_credential_rewrite` is enabled; +- client-to-server WebSocket text frames only when + `websocket_credential_rewrite` is enabled; +- GraphQL-over-WebSocket connection/control messages when they are carried in + text frames and the endpoint enables the WebSocket rewrite path; +- token grant headers for endpoint-bound provider credentials. + +Request-body rewrite is REST-only. It should buffer bounded UTF-8 textual +bodies, including JSON, form-url-encoded, and `text/*`, recompute +`Content-Length`, preserve unsupported bodies that contain no reserved +credential markers, and fail closed when a reserved placeholder cannot be +resolved safely. Binary WebSocket frames are not rewritten. + +Token grants are dynamic credential injection. They use provider metadata to +request a SPIFFE JWT-SVID, exchange it for an OAuth2 access token, cache the +token, and inject either an `Authorization: Bearer` header or a configured +custom header. Token grant failures should return a local relay error and must +not forward the request upstream. + +Middleware-transformed content should be treated as untrusted input from a +credential perspective. External middleware must not receive OpenShell-managed +credentials, and it should not be able to synthesize new reserved credential +placeholders that OpenShell later resolves into secrets. Unless a future hook +is explicitly built-in-only and credential-capable, the relay should fail +closed or strip newly introduced reserved placeholders before static +placeholder rewrite and token grant injection. + +## Supervisor Middleware Boundary + +Supervisor middleware is a typed relay hook, not a replacement for protocol +framing. The relay or protocol processor must first parse enough structure to +construct the operation-specific middleware input. + +For v1, the operation is `HTTP_REQUEST / PRE_CREDENTIALS`: + +1. Network policy, destination validation, and request policy admit the + request. +2. The HTTP relay selects the middleware chain from the request processing + plan. +3. The relay buffers the request body within the smallest selected stage limit. +4. The chain evaluates in deterministic order. +5. A deny short-circuits before credential injection or upstream write. +6. An allow can replace the request body, add approved headers, emit findings, + and pass metadata forward. +7. The transformed request then enters credential injection and upstream write. + +Middleware selection is independent from the matched endpoint policy. It is a +request processing plan selected by admitted destination host, order, and +binding metadata. The decision boundary should materialize it with the same +policy generation used for endpoint selection so a long-lived tunnel cannot mix +old endpoint policy with a new middleware registry. + +V1 middleware can inspect WebSocket upgrade requests because those are HTTP +requests. It does not inspect post-upgrade WebSocket frames. A future frame +hook should be a separate operation such as `WEBSOCKET_MESSAGE / +BEFORE_FORWARD` owned by the WebSocket relay. + +## Protocol Processor Boundary + +Protocol processors operate on streams owned by the relay. + +- HTTP parsing converts bytes into request metadata, evaluates request policy, + runs the `HTTP_REQUEST / PRE_CREDENTIALS` middleware hook when configured, + and loops for keep-alive or pipelined requests. +- JSON-RPC and MCP processing are HTTP L7 processors: they parse bounded + JSON-RPC-over-HTTP request bodies after HTTP parsing and before upstream + forwarding. Generic JSON-RPC policy matches methods; MCP policy can also + match `tools/call` tool names. +- WebSocket parsing starts only after an allowed HTTP upgrade. It validates the + handshake/frame stream and owns client-to-server text-frame inspection when + credential rewrite, transport message policy, GraphQL-over-WebSocket policy, + or compression handling is configured. +- Native TCP protocol processors read client and upstream streams as needed and + own their message loop. +- A protocol processor can deny before dialing, dial for a server handshake, or + keep evaluating commands or queries throughout the session. +- A protocol processor may be in-tree, middleware-backed, or a hybrid where + in-tree framing exposes typed middleware operations for content evaluation. + +HTTP and WebSocket relays receive the generation-pinned request evaluator +because request policy must continue throughout long-lived sessions. No +processor rematerializes endpoint, TLS, allowed-IP, credential, or middleware +selection. This avoids a separate dial-strategy enum: each processor knows +which protocol milestone is sufficient to call the validated connector. + +## Local Service Adapter Boundary + +Local services are network surfaces but not normal external egress: + +- `inference.local` terminates local client traffic, validates known inference + routes, strips caller auth, injects provider routing/auth, and applies + streaming or buffered limits based on route type. +- `policy.local` serves policy snapshots, denial summaries, proposal + submission, and proposal wait. It should never expose secrets or provider + rules as editable policy. +- Metadata loopback serves provider metadata credentials for SDKs that bypass + HTTP proxy variables. It should use the same provider credential state and + redaction discipline as other credential paths. + +These adapters may call gateway APIs or local credential helpers, but they +should not bypass policy and credential invariants that apply to external +egress. + +Issue [#1633](https://github.com/NVIDIA/OpenShell/issues/1633) is a prospective +consumer of these boundaries, not a feature defined by this RFC. A +policy-declared host-local endpoint should use an explicit local-routing adapter +or destination mode; it must not become a general loopback exemption in the +external destination validator. Its feature design still needs to choose the +policy surface (reserved hostname versus endpoint flag), define authorization +before the supervisor connects to host loopback, and specify driver/runtime +capabilities. That work can reuse `EgressIntent`, adapter-specific responses, +and the unopened connector boundary without changing this RFC's compatibility +milestone. + +## Timeout And Resource Ownership + +| Owner | Resource | +|-------|----------| +| Adapter | Client-side parse timeout and adapter-specific deny response | +| Authorization | OPA deadline and policy evaluation telemetry | +| Destination validator | DNS timeout, allowed IP checks, SSRF checks, control-plane port checks | +| TLS terminator | Client TLS handshake timeout and certificate selection | +| HTTP relay | Per-request read/write deadlines, body caps, request-body rewrite caps, upstream reuse | +| WebSocket relay | Upgrade validation, frame limits, text-frame rewrite, compression limits, message policy | +| TCP relay | Byte-copy idle timeout and half-close handling | +| Protocol processor | Protocol message timeouts, middleware hook timeouts, and processor-specific limits | +| Local service adapter | Local route body limits, response caps, gateway call timeout | +| Token grant resolver | SPIFFE Workload API timeout, token endpoint timeout, cache TTL | +| Middleware runner | Service timeout, body cap, failure policy, registry generation | + +Timeouts should be recorded in telemetry at the owner boundary that can explain +the failure.