From 7d72efc27a949746c7186ad845d1dc3f14e68262 Mon Sep 17 00:00:00 2001 From: John Myers Date: Thu, 21 May 2026 16:10:58 -0700 Subject: [PATCH 1/8] docs(rfc): propose sandbox proxy egress adapter model Signed-off-by: John Myers --- .../README.md | 420 ++++++++++++++++++ .../current-shape.md | 167 +++++++ .../implementation-plan.md | 127 ++++++ .../technical-design.md | 259 +++++++++++ 4 files changed, 973 insertions(+) create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/README.md create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/current-shape.md create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/technical-design.md diff --git a/rfc/0004-sandbox-proxy-egress-adapter/README.md b/rfc/0004-sandbox-proxy-egress-adapter/README.md new file mode 100644 index 0000000000..01001ca15e --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/README.md @@ -0,0 +1,420 @@ +--- +authors: + - "@johntmyers" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/1107 + - https://github.com/NVIDIA/OpenShell/pull/1083 + - https://github.com/NVIDIA/OpenShell/pull/1151 +--- + +# RFC 0004 - Sandbox Proxy Egress Adapter Model + + + +## Summary + +Refactor sandbox egress around one shared authorization and relay pipeline. +CONNECT, forward HTTP proxy, transparent native TCP, policy DNS, +`inference.local`, and `policy.local` should become adapters that translate +userland entry points into a common egress intent. Policy evaluation, +destination validation, credential injection, request-body rewrite, +WebSocket upgrade handling, protocol parsing, and relay ownership should happen +behind shared boundaries. + +This RFC keeps the main direction in this document. Supporting detail lives in: + +- [Current shape appendix](current-shape.md) +- [Technical design appendix](technical-design.md) +- [Implementation plan](implementation-plan.md) + +## Motivation + +The sandbox proxy has accumulated separate egress paths for CONNECT, forward +HTTP, local services, inference routing, endpoint metadata, credential +injection, and L7 policy. That makes security changes easy to apply to one path +and miss in another. + +The target shape separates three concerns: + +- **Adapters** describe how userland reached the proxy. +- **Authorization** decides whether that egress is allowed and what endpoint + behavior applies. +- **Relays** own bytes, credentials, protocol parsing, and upstream dialing. + +## Non-goals + +- Replace CONNECT with forward proxy as the only explicit proxy mode. +- Add SOCKS support. +- Add HTTP/2 L7 parsing in this refactor. +- Redesign provider credential storage. +- Reintroduce iptables as the sandbox packet filtering backend. +- Use eBPF connect hooks for transparent capture. Native TCP capture needs a + userland proxy in the byte stream for TLS termination and protocol parsing. + +## Proposal + +### Migration Big Rocks + +1. **Transport adapters.** CONNECT, forward HTTP, transparent TCP, policy DNS, + and local service routes become small entry adapters. They parse their + surface and produce either an egress intent, a local response, or a DNS + answer. They do not duplicate policy evaluation. +2. **Egress intent and decision.** The shared authorization boundary evaluates + L4 policy once per connection intent and returns one decision containing the + matched policy, matched endpoint, process identity, allowed IP metadata, TLS + behavior, and protocol enforcement. +3. **Relays.** Relays receive an authorized destination connector, not an + already-open upstream socket. HTTP relays evaluate every request before + dialing, own REST request-body credential rewrite, and hand allowed + WebSocket upgrades to the WebSocket relay. TCP application parsers own their + protocol loop and decide when a validated upstream connection is needed. + +### Unified Adapter Flow + +```mermaid +flowchart TD + User["Userland payload / harness"] + + subgraph ExplicitProxy["Explicit proxy listener"] + ProxyBytes["HTTP proxy bytes"] + IsConnect{"CONNECT request?"} + Connect["CONNECT adapter"] + Forward["Forward HTTP adapter"] + ProxyBytes --> IsConnect + IsConnect -- Yes --> Connect + IsConnect -- No --> Forward + end + + subgraph NativeTcp["Policy DNS + native TCP"] + NameLookup["Userland DNS lookup"] + PolicyDns["Policy DNS adapter"] + DnsAnswer["DNS answer"] + NativeConnect["Userland connect(ip:port)"] + TcpAdapter["Transparent TCP adapter"] + NameLookup --> PolicyDns + PolicyDns --> DnsAnswer + DnsAnswer --> NativeConnect + NativeConnect --> TcpAdapter + end + + subgraph LocalApis["Sandbox-local services"] + InferenceReq["Request to inference.local"] + PolicyReq["Request to policy.local"] + InferenceAdapter["Inference local adapter"] + PolicyAdapter["Policy local adapter"] + InferenceReq --> InferenceAdapter + PolicyReq --> PolicyAdapter + end + + subgraph Shared["Shared egress pipeline"] + Intent["Egress intent"] + Auth["Authorize and select endpoint"] + Decision["Egress decision"] + Validate["Resolve and validate destination"] + Relay["Relay"] + Deny["Adapter-specific deny response"] + Intent --> Auth + Auth --> Allowed{"Allowed?"} + Allowed -- No --> Deny + Allowed -- Yes --> Decision + Decision --> Validate + Validate --> Relay + end + + User --> ProxyBytes + User --> NameLookup + User --> NativeConnect + User --> InferenceReq + User --> PolicyReq + + Connect --> Intent + Forward --> Intent + TcpAdapter --> Intent + InferenceAdapter --> InferenceResp["Local inference response"] + PolicyAdapter --> PolicyResp["Local policy response"] +``` + +### Relay Flow + +```mermaid +flowchart TD + Start["Authorized egress + destination connector"] + Start --> HasFirst{"First HTTP request already parsed?"} + + HasFirst -- Yes --> ForwardMode{"Selected enforcement"} + ForwardMode -- "L4 only" --> HttpCred["HTTP relay
credential injection only"] + ForwardMode -- "HTTP rules" --> HttpL7["HTTP relay
REST/GraphQL/WebSocket policy"] + ForwardMode -- "TCP app rules" --> BadForward["Deny: HTTP request for TCP app endpoint"] + + HasFirst -- No --> Inspect["Inspect tunnel or native stream bytes"] + Inspect --> SkipTls{"Endpoint says skip TLS handling?"} + SkipTls -- Yes --> TcpBytes["TCP relay
byte copy"] + SkipTls -- No --> Peek["Peek client bytes"] + Peek --> IsTls{"TLS ClientHello?"} + IsTls -- Yes --> Tls["Shared TLS terminator"] + IsTls -- No --> Readable["Readable client stream"] + Tls --> Readable + + Readable --> Mode{"Selected enforcement"} + Mode -- "L4 only" --> SniffHttp{"Looks like HTTP?"} + SniffHttp -- Yes --> HttpCred + SniffHttp -- No --> TcpBytes + + Mode -- "HTTP rules" --> MustHttp{"Looks like HTTP?"} + MustHttp -- Yes --> HttpL7 + MustHttp -- No --> DenyHttp["Deny: expected HTTP"] + + Mode -- "TCP app rules" --> TcpParser["TCP relay
application parser owns loop"] + + HttpCred --> Creds["Resolve and redact credentials"] + HttpL7 --> CredsL7["Resolve and redact credentials"] + CredsL7 --> ParseHttp["Parse and evaluate each HTTP request"] + ParseHttp --> HttpAllowed{"Request allowed?"} + HttpAllowed -- No --> HttpDeny["Local HTTP deny
no upstream connect"] + HttpAllowed -- Yes --> Rewrite["Rewrite configured credential slots"] + Creds --> Rewrite + Rewrite --> HttpDial["Connect or reuse upstream"] + HttpDial --> HttpResponse["Write request and relay response"] + HttpResponse --> Upgrade{"101 WebSocket upgrade?"} + Upgrade -- No --> NextHttp["Continue HTTP request loop"] + Upgrade -- Yes --> WsMode{"WebSocket inspection needed?"} + WsMode -- No --> RawUpgrade["Raw upgraded stream"] + WsMode -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] + NextHttp --> ParseHttp + + TcpParser --> ParserDial["Parser dials upstream when protocol allows"] + TcpBytes --> TcpDial["Connect upstream"] + TcpDial --> ByteCopy["Copy bytes"] +``` + +Relay rules: + +- HTTP credential injection happens in both HTTP modes: L4-only HTTP and + HTTP-inspected. +- Credential injection includes request target, query, headers, opt-in REST + request bodies, and opt-in client-to-server WebSocket text frames. +- HTTP L7 policy is evaluated before upstream dial for each request. +- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101` + upgrade, the WebSocket relay owns frame parsing when text-frame credential + rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe + compression handling is configured. Other upgraded streams remain raw. +- Forward HTTP must stay in the shared HTTP relay loop. It must not evaluate + one request and then switch to raw bidirectional copy. Keeping forward HTTP + single-request with `Connection: close` is also acceptable, but the invariant + is that no follow-on request bytes reach upstream unevaluated. +- `protocol: tcp` means L4 authorization plus byte copy unless HTTP is detected + for credential injection. +- Future TCP application parsers, such as Redis or Postgres, own the full + message loop and can parse multiple commands over one TCP session. + +### CONNECT Adapter + +CONNECT remains the standard explicit proxy tunnel for HTTPS and arbitrary TCP. +It parses the CONNECT line into an egress intent, then waits for the shared +relay to decide if and when an upstream connection should be opened. + +```mermaid +flowchart TD + Client["Client sends CONNECT host:port"] --> Parse["Parse target"] + Parse --> Intent["Build egress intent"] + Intent --> Auth["Shared authorization"] + Auth --> Allowed{"Destination allowed?"} + Allowed -- No --> Deny["CONNECT deny response"] + Allowed -- Yes --> Ready["Return tunnel-ready response"] + Ready --> Relay["Relay inspects tunneled bytes"] + Relay --> Dial["Relay or parser connects upstream when allowed"] +``` + +CONNECT should stay because forward proxy is only a plaintext HTTP request +format. CONNECT is still the generic explicit proxy mode for TLS and non-HTTP +TCP clients. + +### Forward HTTP Adapter + +Forward HTTP is compatibility for clients that send absolute-form HTTP requests. +The adapter parses the first request and hands it to the shared HTTP relay or +an equivalent guarded single-request relay. + +```mermaid +flowchart TD + Req["Absolute-form HTTP request"] --> Parse["Parse URI and first request"] + Parse --> Intent["Build egress intent"] + Intent --> Auth["Shared authorization"] + Auth --> Allowed{"Allowed?"} + Allowed -- No --> Deny["HTTP deny response"] + Allowed -- Yes --> Relay["Shared or guarded HTTP relay"] + Relay --> Mode{"Connection mode"} + Mode -- "Persistent" --> Loop["Evaluate every request on this connection"] + Mode -- "Single request" --> Close["Force Connection: close"] +``` + +### Transparent TCP Adapter + +Transparent TCP supports native clients that do not know they are using a +proxy. The capture mechanism should be network namespace interception into a +userland proxy listener. Since main now uses nftables for sandbox bypass +enforcement, transparent capture should be designed as nftables +REDIRECT/TPROXY state in the inner sandbox network namespace, not as an +iptables path. + +```mermaid +flowchart TD + Policy["Policy load / reload"] --> Register["Register native TCP names"] + Lookup["Userland DNS lookup"] --> Dns["Policy DNS adapter"] + Register --> Dns + Dns --> Answer["Return approved IPs"] + Answer --> Capture["Enable capture for active IP:port"] + Connect["Userland connect(ip:port)"] --> Capture + Capture --> Adapter["Transparent TCP adapter"] + Adapter --> Intent["Build egress intent from original destination"] + Intent --> Shared["Shared authorization and relay"] +``` + +### Policy DNS + +Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is +query-driven: check whether the name is policy-eligible, resolve through trusted +DNS, filter returned IPs, publish the active endpoint mapping, and answer +userland. + +```mermaid +flowchart TD + Query["DNS query from userland"] --> Adapter["Policy DNS adapter"] + Adapter --> Known{"Registered native TCP policy name?"} + Known -- No --> Refuse["NXDOMAIN / REFUSED / SERVFAIL"] + Known -- Yes --> Upstream["Trusted upstream DNS lookup"] + Upstream --> Filter["Filter answers against endpoint policy"] + Filter --> Publish["Publish active mapping and capture rule"] + Publish --> Answer["DNS answer"] +``` + +The later `connect(ip:port)` still creates the egress intent and runs through +normal authorization. + +### Network Enforcement Substrate + +Current main uses nftables for bypass enforcement. It accepts proxy-bound +traffic and loopback, accepts established flows, then rejects and optionally +logs other TCP/UDP traffic for the bypass monitor. That is enforcement, not +native TCP capture. + +```mermaid +flowchart TD + Conn["Userland packet"] --> ProxyDest{"Proxy destination?"} + ProxyDest -- Yes --> AcceptProxy["nftables accept"] + ProxyDest -- No --> Capture{"Future native TCP capture match?"} + Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"] + Capture -- No --> Reject["nftables log + reject bypass"] + Reject --> Monitor["Bypass monitor emits OCSF"] +``` + +The transparent TCP work should extend this nftables model with explicit +capture rules that run before the reject path and are scoped to active policy +DNS mappings. + +### Local Service Adapters + +`inference.local` and `policy.local` are sandbox-local APIs. They should use +the adapter model, but they do not represent normal external egress. + +```mermaid +flowchart TD + A["Request to inference.local"] --> B["Inference local adapter"] + B --> C{"TLS and inference context available?"} + C -- No --> D["Local denial and log"] + C -- Yes --> E["Terminate client TLS"] + E --> F["Parse HTTP request"] + F --> G{"Known inference route?"} + G -- Yes --> H["Route through openshell-router"] + H --> I["Strip caller auth and inject provider auth/model"] + I --> J["Stream response with limits"] + G -- No --> K["403 and close"] +``` + +```mermaid +flowchart TD + A["Request to policy.local"] --> B["Policy local adapter"] + B --> C{"Local route"} + C -- "Current policy" --> D["Policy snapshot response"] + C -- "Recent denials" --> E["Bounded denial summaries"] + C -- "Policy proposal" --> F["Validate and submit proposal"] + D --> G["HTTP response"] + E --> G + F --> G +``` + +### Deployment Modes + +The first implementation can remain embedded in `openshell-sandbox`, but the +proxy should be shaped around explicit runtime contracts. + +| Mode | Shape | Main concern | +|------|-------|--------------| +| Embedded | Current sandbox process owns proxy modules | Lowest migration cost | +| Standalone process | Sandbox supervisor launches a proxy binary | Clear process/API boundary | +| Sidecar | Proxy runs outside the payload container but inside the sandbox boundary | Reliable process identity across namespaces | + +A pluggable proxy must expose the configured userland surfaces, implement the +gateway APIs it needs, and prove equivalent policy enforcement through tests. +The nftables rules that force or reject userland traffic belong to the sandbox +network boundary even if the proxy process later moves into a standalone binary +or sidecar. + +## Implementation plan + +The migration plan lives in [implementation-plan.md](implementation-plan.md). +The intended order is: first add regression coverage, then introduce the shared +authorization result and destination validation, then preserve the current +forward HTTP single-request/guarded-relay invariant, then add shared TLS +handling, TCP parser boundaries, nftables-backed policy DNS capture, local +service adapters, and finally the runtime boundary cleanup. + +## Risks + +- Tightening endpoint metadata failures from fail-open to deny may expose + latent policy or Rego errors. +- Deterministic endpoint selection may reject policies that currently load but + only work by accident. +- Transparent TCP capture adds network namespace interception complexity. +- Transparent TCP capture must coexist with the current nftables bypass + reject/log table without creating gaps where direct egress bypasses the proxy. +- Sidecar mode needs a reliable identity source for binary/path scoped policy. +- `policy.local` expands the sandbox-local control surface and needs strict + route validation, body limits, redaction, and gateway authentication. + +## Alternatives + +- Keep patching each current proxy path separately. This has the lowest short + term cost but keeps the security surface duplicated. +- Replace CONNECT with forward proxy. This does not work for arbitrary TCP and + is not a replacement for HTTPS tunnels. +- Build only transparent TCP. This helps native clients but does not replace + explicit proxy support used by common HTTP tooling. + +## Open questions + +1. Should overlapping endpoint metadata be rejected at policy load time, or + should policy name plus endpoint index define precedence? +2. Should missing TLS state fail closed for credential-capable or inspected + endpoints? +3. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted, + or should DNS query correlation be required for stricter modes? +4. What TTL cap and stale-generation grace period should policy DNS use? +5. Which process identity source should sidecar mode use when it cannot inspect + payload process metadata through local `/proc`? +6. Which proxy capabilities should be negotiated with the gateway at startup? + +## Expected result + +Adding a new HTTP-family protocol parser should require parser code, policy +schema/Rego support, tests, and docs. It should not require new CONNECT and +forward-proxy branches. REST, GraphQL, WebSocket upgrade policy, request-body +credential rewrite, and WebSocket text-frame rewrite should all remain behind +the shared HTTP/WebSocket relay boundary. + +Adding a native TCP application parser should require policy DNS/capture +support, a TCP application parser, policy rules, tests, and docs. Plain +`protocol: tcp` remains L4 authorization plus byte relay. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md new file mode 100644 index 0000000000..b428fed145 --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md @@ -0,0 +1,167 @@ +# Current Shape Appendix + +This appendix records the current proxy shape and the review findings that +motivate the adapter model. The main RFC intentionally keeps these details out +of the direction document. + +## Current Entry Points + +The sandbox proxy currently handles multiple userland-facing paths in the same +large module: + +- CONNECT proxy traffic for HTTPS and generic TCP tunnels. +- Forward HTTP proxy traffic for absolute-form HTTP requests. +- Local service routes such as `inference.local`. +- Network namespace bypass enforcement through nftables reject/log rules. +- Policy and endpoint metadata lookups through OPA/Rego. +- DNS resolution and endpoint validation for CONNECT and forward HTTP egress. +- Credential injection and redaction for provider-backed HTTP egress. +- Opt-in REST request-body credential rewrite. +- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement. + +The issue is not that these features exist. The issue is that entry mechanisms, +policy evaluation, endpoint metadata lookup, credential injection, and byte +relay decisions are interleaved. + +## Current CONNECT Shape + +```mermaid +flowchart TD + Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["CONNECT denial"] + Allowed -- Yes --> Meta["Query endpoint metadata"] + Meta --> Config{"L7 or credential config?"} + Config -- No --> Raw["Open upstream and copy bytes"] + Config -- Yes --> Tunnel["Return tunnel-ready response"] + Tunnel --> Inspect["Parse tunneled HTTP when possible"] + Inspect --> L7["Evaluate HTTP policy"] + L7 --> Inject["Inject credentials if configured"] + Inject --> Upstream["Write upstream and relay response"] +``` + +This path has the strongest HTTP relay behavior because it can keep parsing +requests on a long-lived tunnel and enforce L7 rules per request. + +## Current Forward HTTP Shape + +```mermaid +flowchart TD + Client["Absolute-form HTTP request"] --> Parse["Parse first request"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["HTTP denial"] + Allowed -- Yes --> L7{"Matching L7 endpoint?"} + L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"] + Eval --> Rewrite["Rewrite to origin-form + configured credentials"] + L7 -- No --> Rewrite + Rewrite --> Close["Force Connection: close except WebSocket upgrade"] + Close --> Upstream["Open upstream"] + Upstream --> Relay["Guarded HTTP relay / upgrade relay"] +``` + +The latest main branch no longer has the old raw-copy-after-first-request shape +for ordinary forward HTTP. It rewrites ordinary requests with `Connection: +close`, uses guarded HTTP relay helpers for body handling, and sends allowed +WebSocket upgrades through the same upgrade relay. That is a narrower surface +than the historical bidirectional copy, but it is still implemented separately +from the CONNECT relay path. + +## Current Network Namespace Enforcement + +```mermaid +flowchart TD + Start["Process in sandbox network namespace"] --> Dest{"Destination"} + Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"] + Dest -- "Loopback" --> Loopback["Accept loopback"] + Dest -- "Established/related" --> Established["Accept response packet"] + Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"] + Reject --> Monitor["Bypass monitor reads dmesg"] + Monitor --> OCSF["OCSF network + detection events"] +``` + +The sandbox now installs an `inet` nftables filter table for bypass +enforcement. The table accepts proxy-bound traffic, loopback, and established +flows, then rejects and optionally logs other TCP/UDP traffic. It does not +currently redirect native TCP connections into the proxy. + +## Current Local Service Shape + +```mermaid +flowchart TD + Request["Request to local name"] --> Match{"Known local route?"} + Match -- "inference.local" --> Inference["Inference routing logic"] + Match -- "policy.local" --> Policy["Policy local logic"] + Match -- No --> External["Normal egress path"] + Inference --> LocalResponse["Local response"] + Policy --> LocalResponse +``` + +Local routes are userland-facing proxy surfaces. They should stay distinct from +external egress while still fitting the adapter model. + +## Findings To Preserve + +### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes + +The historical forward path evaluated at most the first absolute-form request, +rewrote it, then switched to bidirectional copy. Bytes already buffered after +the first header block, or later pipelined requests on the same client/upstream +connection, could reach upstream without the CONNECT L7 relay's per-request +parser/evaluator. + +Latest main mitigates this by forcing ordinary forward HTTP to one request per +connection and by using guarded relay helpers. The adapter model should +preserve the invariant either by keeping forward HTTP single-request/close or +by passing the first parsed request into a shared HTTP relay loop. + +### Endpoint config is not tied to deterministic matched policy + +The policy name used for L4 authorization and logging can be selected through a +different precedence rule than endpoint metadata. With overlapping host, port, +and binary rules, allowed IPs, TLS behavior, enforcement, and +`allow_encoded_slash` can come from a different endpoint than the policy name +logged and used for L4 allow. + +The adapter model requires authorization to return one decision with one +deterministic matched endpoint. + +### Endpoint metadata query failures fail open to L4 behavior + +If endpoint metadata lookup fails, callers can interpret the result as no L7 +configuration and downgrade to credential-only or raw L4 relay. + +The adapter model treats endpoint metadata as part of the authorization result. +Failure to materialize required metadata should deny rather than erase extended +configuration. + +### Control-plane port block only applies on one resolution path + +Blocked control-plane ports are enforced inside one allowed-IPs validation +path, while the normal host-based path uses a different validation route. + +The adapter model moves resolution, allowed IP checks, SSRF checks, and +control-plane port blocks into shared destination validation. + +## Existing Feature Inventory + +The refactor should preserve: + +- CONNECT explicit proxy support. +- Forward HTTP explicit proxy support. +- nftables bypass reject/log enforcement. +- Provider credential injection and redaction. +- REST request-body credential rewrite. +- WebSocket text-frame credential rewrite. +- REST endpoint method/path policy. +- GraphQL L7 policy. +- WebSocket transport and GraphQL-over-WebSocket policy. +- Inference routing through `inference.local`. +- Agent-facing policy routes through `policy.local`. +- Timeout and resource tracking for client, upstream, and local service work. +- Structured OCSF logging for network and HTTP policy outcomes. +- SSRF and internal address protections. +- Control-plane port protection. +- `allowed_ips` endpoint restrictions. +- TLS termination for inspectable client connections. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md new file mode 100644 index 0000000000..94ba53b7f0 --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md @@ -0,0 +1,127 @@ +# Implementation Plan + +This plan is intentionally separate from the main RFC so the proposal can stay +direction-focused. + +## Phase 0 - Regression Tests + +- Add tests for forward HTTP pipelining and keep-alive follow-on requests, + including the current `Connection: close` mitigation. +- Add tests for overlapping endpoint metadata selection. +- Add tests for endpoint metadata query failures. +- Add tests for control-plane port blocking through all destination validation + paths. +- Add nftables bypass enforcement tests that verify proxy-bound traffic is + accepted while direct TCP/UDP egress is rejected and logged when available. + +## Phase 1 - Authorization Result + +- Introduce `EgressIntent` and `EgressDecision`. +- Make authorization return matched policy and matched endpoint metadata + together. +- Fail closed when required endpoint metadata cannot be materialized. +- Emit consistent OCSF network denial events from the shared boundary. + +## Phase 2 - Shared Destination Validation + +- Move DNS resolution, allowed IP filtering, SSRF checks, and control-plane port + checks into one destination validation path. +- Return an `UpstreamConnector` rather than an opened upstream socket. +- Add tests proving CONNECT, forward HTTP, and transparent TCP use the same + validation behavior. + +## Phase 3 - Forward HTTP Adapter + +- Convert forward HTTP into an adapter that parses the first absolute-form + request and builds an egress intent. +- Route the parsed first request into the shared HTTP relay or preserve the + current guarded single-request relay behavior. +- Keep the no-raw-copy invariant after the first request. + +## Phase 4 - HTTP And WebSocket Relay Consolidation + +- Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket + upgrade policy, credential resolution, redaction, request rewrite, upstream + dial, and response relay. +- Evaluate every HTTP request before upstream write. +- Ensure denied HTTP requests do not create upstream TCP sessions. +- Preserve opt-in REST request-body credential rewrite behind the shared HTTP + relay, including bounded buffering, supported content-type handling, + `Content-Length` recomputation, and fail-closed unresolved placeholders. +- Preserve WebSocket upgrade handling behind the shared relay, including + opt-in client-to-server text-frame credential rewrite, WebSocket transport + message policy, GraphQL-over-WebSocket policy, and raw passthrough for other + upgraded protocols. + +## Phase 5 - Shared TLS Termination + +- Move client-side TLS detection and termination before the HTTP/TCP relay + split. +- Keep endpoint TLS behavior on `EgressDecision`. +- Remove duplicate HTTP-specific and TCP-specific TLS termination decisions. + +## Phase 6 - TCP Relay And Parser Boundary + +- Rename raw TCP relay concepts to `TcpRelay`. +- Add a TCP application parser dispatch point for future protocol enforcement. +- Keep `protocol: tcp` as L4 authorization plus byte copy. +- Let TCP application parsers own their message loop and call the connector + when protocol state allows. + +## Phase 7 - Policy DNS And Transparent TCP + +- Add policy DNS registration for native TCP endpoint names. +- Replace static host-file mapping with query-driven DNS answers. +- Publish active DNS answer state and capture rules. +- Implement nftables REDIRECT/TPROXY capture rules ahead of the bypass reject + path; do not add a parallel iptables path. +- Implement transparent TCP adapter lookup from captured original destination + to active endpoint generation. +- Decide TTL and stale-generation behavior. + +## Phase 8 - Local Service Adapters + +- Model `inference.local` as a local adapter with TLS termination, route + validation, provider auth injection, streaming limits, and OCSF logging. +- Model `policy.local` as a local adapter for current policy, bounded denial + summaries, and policy proposals. +- Keep both paths outside normal external egress relay. + +## Phase 9 - Runtime Boundary + +- Keep embedded mode for the first migration. +- Define the proxy runtime API needed for a future standalone binary: + configured listeners, policy updates, gateway calls, telemetry, and shutdown. +- Identify process identity requirements for standalone and sidecar modes. + +## Phase 10 - Cleanup + +- Remove duplicated endpoint metadata queries from relay paths. +- Remove duplicated deny rendering where adapters can own response shape. +- Remove any remaining forward HTTP raw-copy fallback. +- Update architecture docs once implementation lands. + +## Testing Plan + +- Unit-test each adapter's intent construction and deny response shape. +- Unit-test authorization precedence for overlapping policy and endpoint rules. +- Integration-test shared destination validation across CONNECT, forward HTTP, + and transparent TCP. +- Integration-test HTTP keep-alive and pipelined requests with REST, GraphQL, + and WebSocket upgrade enforcement. +- Integration-test credential injection in L4-only HTTP and HTTP-inspected + paths. +- Integration-test REST request-body credential rewrite for JSON, + form-url-encoded, `text/*`, unsupported content types, chunked framing, body + caps, and unresolved placeholders. +- Integration-test WebSocket text-frame credential rewrite, raw upgraded + passthrough, WebSocket message policy, GraphQL-over-WebSocket policy, and + safe compression negotiation. +- Integration-test TLS termination before HTTP/TCP relay split. +- Integration-test `protocol: tcp` byte-copy behavior. +- Add parser harness tests before adding Redis, Postgres, or similar TCP + application parsers. +- Integration-test policy DNS TTL, stale generation handling, and captured + connect correlation. +- Integration-test `inference.local` and `policy.local` body limits, timeout + behavior, redaction, and local denial responses. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md new file mode 100644 index 0000000000..b13e259f45 --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md @@ -0,0 +1,259 @@ +# Technical Design Appendix + +This appendix carries the implementation-level design details behind the main +RFC. + +## Shared Data Boundaries + +### EgressIntent + +`EgressIntent` is the normalized description of what userland is trying to do. + +It should carry: + +- entry transport: CONNECT, forward HTTP, transparent TCP, or local HTTP; +- requested destination host/port or captured original IP/port; +- process identity inputs collected by the adapter/runtime; +- optional first HTTP request for forward proxy traffic; +- optional local service route. + +Adapters build intents. They should not query endpoint metadata or select +relays. + +### EgressDecision + +`EgressDecision` is the policy result consumed by validation and relay code. + +It should carry: + +- allow or deny; +- deterministic matched policy identifier; +- deterministic matched endpoint identifier and endpoint metadata; +- process identity used for evaluation; +- destination and allowed IP constraints; +- TLS behavior; +- protocol enforcement; +- logging context and denial reason. + +Relay code should read this decision. It should not query OPA again for +endpoint metadata, TLS mode, allowed IPs, or parser selection. + +## Protocol Enforcement + +Use a protocol enforcement value derived from endpoint policy: + +| Policy protocol | Enforcement | Relay behavior | +|-----------------|-------------|----------------| +| omitted / `tcp` | None | L4 authorization plus byte relay, with optional HTTP sniff for credential injection | +| `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite | +| `graphql` | HTTP | HTTP request parser with GraphQL rules | +| `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy | +| future `redis`, `postgres`, `mysql`, ... | TCP application | Protocol-specific TCP parser owns the message loop | + +`protocol: tcp` is effectively the default L4 mode. It should not run TCP +application parsers. + +Avoid using the term "provider" for these parser concepts because providers +are already a first-class credential and routing domain in OpenShell. + +## Suggested Types + +The exact Rust shape can evolve, but the boundaries should look like this: + +```rust +enum EgressTransport { + Connect, + ForwardHttp, + TransparentTcp, + LocalHttp, +} + +struct EgressIntent { + transport: EgressTransport, + destination: RequestedDestination, + process: ProcessIdentity, + first_request: Option, + local_route: Option, +} + +struct EgressDecision { + outcome: PolicyOutcome, + matched_policy: Option, + endpoint: Option, + log_context: EgressLogContext, +} + +struct MatchedEndpoint { + id: EndpointId, + allowed_ips: AllowedIpPolicy, + tls: TlsPolicy, + enforcement: ProtocolEnforcement, +} + +enum ProtocolEnforcement { + None, + Http(HttpL7Config), + TcpApplication(TcpApplicationConfig), +} + +enum HttpL7Protocol { + Rest, + Graphql, + Websocket, +} + +struct HttpL7Config { + protocol: HttpL7Protocol, + allow_encoded_slash: bool, + websocket_credential_rewrite: bool, + request_body_credential_rewrite: bool, + websocket_graphql_policy: bool, +} + +struct RelayContext { + decision: EgressDecision, + connector: UpstreamConnector, + deadlines: RelayDeadlines, + telemetry: RelayTelemetry, +} +``` + +`UpstreamConnector` is the relay-owned dial boundary. It encapsulates the +validated destination and lets relays/parsers open an upstream connection only +after protocol policy allows it. + +## Module Layout + +A future split could look like: + +| Module | Responsibility | +|--------|----------------| +| `proxy::adapter::connect` | Parse CONNECT and render CONNECT responses | +| `proxy::adapter::forward_http` | Parse absolute-form HTTP and preserve first request | +| `proxy::adapter::transparent_tcp` | Recover captured original destination | +| `proxy::adapter::policy_dns` | Answer eligible DNS queries and publish active mappings | +| `proxy::adapter::local` | Implement `inference.local` and `policy.local` surfaces | +| `proxy::auth` | Build decisions from intents and OPA results | +| `proxy::destination` | Resolve, filter, and validate destinations | +| `proxy::netfilter` | Own nftables bypass and future transparent capture rules | +| `proxy::relay::http` | HTTP request loop, credentials, REST/GraphQL/WebSocket upgrade policy | +| `proxy::relay::websocket` | WebSocket frame validation, text-frame rewrite, and message policy | +| `proxy::relay::tcp` | TCP byte relay and TCP application parser dispatch | +| `proxy::relay::tls` | Shared client-side TLS termination | +| `proxy::parser` | HTTP, WebSocket, and TCP application parser traits/config | +| `proxy::telemetry` | OCSF and tracing helpers | + +## Policy DNS And Resolved TCP State + +Policy DNS should be query-driven rather than a static `/etc/hosts` snapshot. + +1. Policy load registers eligible native TCP endpoint names. +2. Userland performs DNS lookup. +3. Policy DNS checks whether the name is registered for native TCP. +4. Policy DNS resolves through trusted upstream DNS. +5. Answers are filtered against endpoint metadata and SSRF controls. +6. The adapter publishes the DNS answer, endpoint generation, and capture rule. +7. Userland later calls `connect(ip:port)`. +8. Transparent TCP recovers the original destination and maps it to the active + endpoint generation. +9. Normal egress authorization and relay selection run. + +The resolved endpoint store is therefore not a preemptive global DNS snapshot. +It is active state produced by policy-eligible lookups and consumed by +transparent TCP connects. + +## nftables Boundary + +Current main uses nftables, not iptables, for sandbox network bypass +enforcement. The installed `inet` table accepts traffic to the sandbox proxy, +loopback, and established/related flows, then rejects and optionally logs other +TCP/UDP traffic. The bypass monitor reads those log lines and emits OCSF +network and detection events. + +Transparent TCP capture should build on this same nftables substrate: + +- capture rules must run before the generic bypass reject rules; +- capture rules should be scoped to active policy DNS IP/port mappings; +- capture state should be updated atomically with endpoint generation changes; +- reject/log rules remain the fallback for unmatched TCP/UDP egress; +- VM or Podman driver nftables rules are infrastructure NAT/isolation and + should not be treated as the proxy policy enforcement point. + +## Endpoint Selection And OPA + +OPA/Rego should return policy and endpoint metadata through one deterministic +authorization result. It should not let policy name and endpoint config be +selected by different precedence rules. + +Two acceptable approaches: + +- Reject overlapping endpoint metadata at load or merge time. +- Define a single deterministic precedence key and use it for both policy name + and endpoint metadata. + +Endpoint metadata query failures should fail closed when metadata is required +for the selected endpoint. They should not silently downgrade to L4 behavior. + +## Credential Injection Boundary + +Credential injection belongs in the HTTP relay: + +1. Authorization selects the endpoint and confirms credentials may be used. +2. The HTTP relay resolves credentials only when it has an allowed HTTP request. +3. Secrets are redacted from logs and policy-visible metadata. +4. The final upstream request or frame is rewritten with real credentials + immediately before write. + +Both L4-only HTTP and HTTP-inspected paths can inject credentials. The +difference is whether REST, GraphQL, or WebSocket policy is evaluated before +the rewrite. + +Credential rewrite slots should be explicit: + +- request target, query values, and headers for HTTP-family traffic; +- REST request bodies only when `request_body_credential_rewrite` is enabled; +- client-to-server WebSocket text frames only when + `websocket_credential_rewrite` is enabled; +- GraphQL-over-WebSocket connection/control messages when they are carried in + text frames and the endpoint enables the WebSocket rewrite path. + +Request-body rewrite is REST-only. It should buffer bounded UTF-8 textual +bodies, including JSON, form-url-encoded, and `text/*`, recompute +`Content-Length`, preserve unsupported bodies that contain no reserved +credential markers, and fail closed when a reserved placeholder cannot be +resolved safely. Binary WebSocket frames are not rewritten. + +## Parser Boundary + +Protocol parsers operate on streams owned by the relay. + +- HTTP parsing converts bytes into request metadata, evaluates request policy, + and loops for keep-alive or pipelined requests. +- WebSocket parsing starts only after an allowed HTTP upgrade. It validates the + handshake/frame stream and owns client-to-server text-frame inspection when + credential rewrite, transport message policy, GraphQL-over-WebSocket policy, + or compression handling is configured. +- TCP application parsers read client and upstream streams as needed and own + their message loop. +- A TCP parser can deny before dialing, dial for a server handshake, or keep + evaluating commands/queries throughout the session. + +This avoids a separate dial strategy enum. The parser knows which protocol +milestone is sufficient to call the validated connector. + +## Timeout And Resource Ownership + +| Owner | Resource | +|-------|----------| +| Adapter | Client-side parse timeout and adapter-specific deny response | +| Authorization | OPA deadline and policy evaluation telemetry | +| Destination validator | DNS timeout, allowed IP checks, SSRF checks, control-plane port checks | +| TLS terminator | Client TLS handshake timeout and certificate selection | +| HTTP relay | Per-request read/write deadlines, body caps, request-body rewrite caps, upstream reuse | +| WebSocket relay | Upgrade validation, frame limits, text-frame rewrite, compression limits, message policy | +| TCP relay | Byte-copy idle timeout and half-close handling | +| TCP parser | Protocol message timeouts and parser-specific limits | +| Local service adapter | Local route body limits, response caps, gateway call timeout | + +Timeouts should be recorded in telemetry at the owner boundary that can explain +the failure. From f565618e1631db49c2f61f85a26ed4d0a09510d1 Mon Sep 17 00:00:00 2001 From: John Myers Date: Thu, 21 May 2026 16:10:58 -0700 Subject: [PATCH 2/8] docs(rfc): propose sandbox proxy egress adapter model Signed-off-by: John Myers --- .../README.md | 420 ++++++++++++++++++ .../current-shape.md | 167 +++++++ .../implementation-plan.md | 127 ++++++ .../technical-design.md | 259 +++++++++++ 4 files changed, 973 insertions(+) create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/README.md create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/current-shape.md create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md create mode 100644 rfc/0004-sandbox-proxy-egress-adapter/technical-design.md diff --git a/rfc/0004-sandbox-proxy-egress-adapter/README.md b/rfc/0004-sandbox-proxy-egress-adapter/README.md new file mode 100644 index 0000000000..01001ca15e --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/README.md @@ -0,0 +1,420 @@ +--- +authors: + - "@johntmyers" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/1107 + - https://github.com/NVIDIA/OpenShell/pull/1083 + - https://github.com/NVIDIA/OpenShell/pull/1151 +--- + +# RFC 0004 - Sandbox Proxy Egress Adapter Model + + + +## Summary + +Refactor sandbox egress around one shared authorization and relay pipeline. +CONNECT, forward HTTP proxy, transparent native TCP, policy DNS, +`inference.local`, and `policy.local` should become adapters that translate +userland entry points into a common egress intent. Policy evaluation, +destination validation, credential injection, request-body rewrite, +WebSocket upgrade handling, protocol parsing, and relay ownership should happen +behind shared boundaries. + +This RFC keeps the main direction in this document. Supporting detail lives in: + +- [Current shape appendix](current-shape.md) +- [Technical design appendix](technical-design.md) +- [Implementation plan](implementation-plan.md) + +## Motivation + +The sandbox proxy has accumulated separate egress paths for CONNECT, forward +HTTP, local services, inference routing, endpoint metadata, credential +injection, and L7 policy. That makes security changes easy to apply to one path +and miss in another. + +The target shape separates three concerns: + +- **Adapters** describe how userland reached the proxy. +- **Authorization** decides whether that egress is allowed and what endpoint + behavior applies. +- **Relays** own bytes, credentials, protocol parsing, and upstream dialing. + +## Non-goals + +- Replace CONNECT with forward proxy as the only explicit proxy mode. +- Add SOCKS support. +- Add HTTP/2 L7 parsing in this refactor. +- Redesign provider credential storage. +- Reintroduce iptables as the sandbox packet filtering backend. +- Use eBPF connect hooks for transparent capture. Native TCP capture needs a + userland proxy in the byte stream for TLS termination and protocol parsing. + +## Proposal + +### Migration Big Rocks + +1. **Transport adapters.** CONNECT, forward HTTP, transparent TCP, policy DNS, + and local service routes become small entry adapters. They parse their + surface and produce either an egress intent, a local response, or a DNS + answer. They do not duplicate policy evaluation. +2. **Egress intent and decision.** The shared authorization boundary evaluates + L4 policy once per connection intent and returns one decision containing the + matched policy, matched endpoint, process identity, allowed IP metadata, TLS + behavior, and protocol enforcement. +3. **Relays.** Relays receive an authorized destination connector, not an + already-open upstream socket. HTTP relays evaluate every request before + dialing, own REST request-body credential rewrite, and hand allowed + WebSocket upgrades to the WebSocket relay. TCP application parsers own their + protocol loop and decide when a validated upstream connection is needed. + +### Unified Adapter Flow + +```mermaid +flowchart TD + User["Userland payload / harness"] + + subgraph ExplicitProxy["Explicit proxy listener"] + ProxyBytes["HTTP proxy bytes"] + IsConnect{"CONNECT request?"} + Connect["CONNECT adapter"] + Forward["Forward HTTP adapter"] + ProxyBytes --> IsConnect + IsConnect -- Yes --> Connect + IsConnect -- No --> Forward + end + + subgraph NativeTcp["Policy DNS + native TCP"] + NameLookup["Userland DNS lookup"] + PolicyDns["Policy DNS adapter"] + DnsAnswer["DNS answer"] + NativeConnect["Userland connect(ip:port)"] + TcpAdapter["Transparent TCP adapter"] + NameLookup --> PolicyDns + PolicyDns --> DnsAnswer + DnsAnswer --> NativeConnect + NativeConnect --> TcpAdapter + end + + subgraph LocalApis["Sandbox-local services"] + InferenceReq["Request to inference.local"] + PolicyReq["Request to policy.local"] + InferenceAdapter["Inference local adapter"] + PolicyAdapter["Policy local adapter"] + InferenceReq --> InferenceAdapter + PolicyReq --> PolicyAdapter + end + + subgraph Shared["Shared egress pipeline"] + Intent["Egress intent"] + Auth["Authorize and select endpoint"] + Decision["Egress decision"] + Validate["Resolve and validate destination"] + Relay["Relay"] + Deny["Adapter-specific deny response"] + Intent --> Auth + Auth --> Allowed{"Allowed?"} + Allowed -- No --> Deny + Allowed -- Yes --> Decision + Decision --> Validate + Validate --> Relay + end + + User --> ProxyBytes + User --> NameLookup + User --> NativeConnect + User --> InferenceReq + User --> PolicyReq + + Connect --> Intent + Forward --> Intent + TcpAdapter --> Intent + InferenceAdapter --> InferenceResp["Local inference response"] + PolicyAdapter --> PolicyResp["Local policy response"] +``` + +### Relay Flow + +```mermaid +flowchart TD + Start["Authorized egress + destination connector"] + Start --> HasFirst{"First HTTP request already parsed?"} + + HasFirst -- Yes --> ForwardMode{"Selected enforcement"} + ForwardMode -- "L4 only" --> HttpCred["HTTP relay
credential injection only"] + ForwardMode -- "HTTP rules" --> HttpL7["HTTP relay
REST/GraphQL/WebSocket policy"] + ForwardMode -- "TCP app rules" --> BadForward["Deny: HTTP request for TCP app endpoint"] + + HasFirst -- No --> Inspect["Inspect tunnel or native stream bytes"] + Inspect --> SkipTls{"Endpoint says skip TLS handling?"} + SkipTls -- Yes --> TcpBytes["TCP relay
byte copy"] + SkipTls -- No --> Peek["Peek client bytes"] + Peek --> IsTls{"TLS ClientHello?"} + IsTls -- Yes --> Tls["Shared TLS terminator"] + IsTls -- No --> Readable["Readable client stream"] + Tls --> Readable + + Readable --> Mode{"Selected enforcement"} + Mode -- "L4 only" --> SniffHttp{"Looks like HTTP?"} + SniffHttp -- Yes --> HttpCred + SniffHttp -- No --> TcpBytes + + Mode -- "HTTP rules" --> MustHttp{"Looks like HTTP?"} + MustHttp -- Yes --> HttpL7 + MustHttp -- No --> DenyHttp["Deny: expected HTTP"] + + Mode -- "TCP app rules" --> TcpParser["TCP relay
application parser owns loop"] + + HttpCred --> Creds["Resolve and redact credentials"] + HttpL7 --> CredsL7["Resolve and redact credentials"] + CredsL7 --> ParseHttp["Parse and evaluate each HTTP request"] + ParseHttp --> HttpAllowed{"Request allowed?"} + HttpAllowed -- No --> HttpDeny["Local HTTP deny
no upstream connect"] + HttpAllowed -- Yes --> Rewrite["Rewrite configured credential slots"] + Creds --> Rewrite + Rewrite --> HttpDial["Connect or reuse upstream"] + HttpDial --> HttpResponse["Write request and relay response"] + HttpResponse --> Upgrade{"101 WebSocket upgrade?"} + Upgrade -- No --> NextHttp["Continue HTTP request loop"] + Upgrade -- Yes --> WsMode{"WebSocket inspection needed?"} + WsMode -- No --> RawUpgrade["Raw upgraded stream"] + WsMode -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] + NextHttp --> ParseHttp + + TcpParser --> ParserDial["Parser dials upstream when protocol allows"] + TcpBytes --> TcpDial["Connect upstream"] + TcpDial --> ByteCopy["Copy bytes"] +``` + +Relay rules: + +- HTTP credential injection happens in both HTTP modes: L4-only HTTP and + HTTP-inspected. +- Credential injection includes request target, query, headers, opt-in REST + request bodies, and opt-in client-to-server WebSocket text frames. +- HTTP L7 policy is evaluated before upstream dial for each request. +- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101` + upgrade, the WebSocket relay owns frame parsing when text-frame credential + rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe + compression handling is configured. Other upgraded streams remain raw. +- Forward HTTP must stay in the shared HTTP relay loop. It must not evaluate + one request and then switch to raw bidirectional copy. Keeping forward HTTP + single-request with `Connection: close` is also acceptable, but the invariant + is that no follow-on request bytes reach upstream unevaluated. +- `protocol: tcp` means L4 authorization plus byte copy unless HTTP is detected + for credential injection. +- Future TCP application parsers, such as Redis or Postgres, own the full + message loop and can parse multiple commands over one TCP session. + +### CONNECT Adapter + +CONNECT remains the standard explicit proxy tunnel for HTTPS and arbitrary TCP. +It parses the CONNECT line into an egress intent, then waits for the shared +relay to decide if and when an upstream connection should be opened. + +```mermaid +flowchart TD + Client["Client sends CONNECT host:port"] --> Parse["Parse target"] + Parse --> Intent["Build egress intent"] + Intent --> Auth["Shared authorization"] + Auth --> Allowed{"Destination allowed?"} + Allowed -- No --> Deny["CONNECT deny response"] + Allowed -- Yes --> Ready["Return tunnel-ready response"] + Ready --> Relay["Relay inspects tunneled bytes"] + Relay --> Dial["Relay or parser connects upstream when allowed"] +``` + +CONNECT should stay because forward proxy is only a plaintext HTTP request +format. CONNECT is still the generic explicit proxy mode for TLS and non-HTTP +TCP clients. + +### Forward HTTP Adapter + +Forward HTTP is compatibility for clients that send absolute-form HTTP requests. +The adapter parses the first request and hands it to the shared HTTP relay or +an equivalent guarded single-request relay. + +```mermaid +flowchart TD + Req["Absolute-form HTTP request"] --> Parse["Parse URI and first request"] + Parse --> Intent["Build egress intent"] + Intent --> Auth["Shared authorization"] + Auth --> Allowed{"Allowed?"} + Allowed -- No --> Deny["HTTP deny response"] + Allowed -- Yes --> Relay["Shared or guarded HTTP relay"] + Relay --> Mode{"Connection mode"} + Mode -- "Persistent" --> Loop["Evaluate every request on this connection"] + Mode -- "Single request" --> Close["Force Connection: close"] +``` + +### Transparent TCP Adapter + +Transparent TCP supports native clients that do not know they are using a +proxy. The capture mechanism should be network namespace interception into a +userland proxy listener. Since main now uses nftables for sandbox bypass +enforcement, transparent capture should be designed as nftables +REDIRECT/TPROXY state in the inner sandbox network namespace, not as an +iptables path. + +```mermaid +flowchart TD + Policy["Policy load / reload"] --> Register["Register native TCP names"] + Lookup["Userland DNS lookup"] --> Dns["Policy DNS adapter"] + Register --> Dns + Dns --> Answer["Return approved IPs"] + Answer --> Capture["Enable capture for active IP:port"] + Connect["Userland connect(ip:port)"] --> Capture + Capture --> Adapter["Transparent TCP adapter"] + Adapter --> Intent["Build egress intent from original destination"] + Intent --> Shared["Shared authorization and relay"] +``` + +### Policy DNS + +Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is +query-driven: check whether the name is policy-eligible, resolve through trusted +DNS, filter returned IPs, publish the active endpoint mapping, and answer +userland. + +```mermaid +flowchart TD + Query["DNS query from userland"] --> Adapter["Policy DNS adapter"] + Adapter --> Known{"Registered native TCP policy name?"} + Known -- No --> Refuse["NXDOMAIN / REFUSED / SERVFAIL"] + Known -- Yes --> Upstream["Trusted upstream DNS lookup"] + Upstream --> Filter["Filter answers against endpoint policy"] + Filter --> Publish["Publish active mapping and capture rule"] + Publish --> Answer["DNS answer"] +``` + +The later `connect(ip:port)` still creates the egress intent and runs through +normal authorization. + +### Network Enforcement Substrate + +Current main uses nftables for bypass enforcement. It accepts proxy-bound +traffic and loopback, accepts established flows, then rejects and optionally +logs other TCP/UDP traffic for the bypass monitor. That is enforcement, not +native TCP capture. + +```mermaid +flowchart TD + Conn["Userland packet"] --> ProxyDest{"Proxy destination?"} + ProxyDest -- Yes --> AcceptProxy["nftables accept"] + ProxyDest -- No --> Capture{"Future native TCP capture match?"} + Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"] + Capture -- No --> Reject["nftables log + reject bypass"] + Reject --> Monitor["Bypass monitor emits OCSF"] +``` + +The transparent TCP work should extend this nftables model with explicit +capture rules that run before the reject path and are scoped to active policy +DNS mappings. + +### Local Service Adapters + +`inference.local` and `policy.local` are sandbox-local APIs. They should use +the adapter model, but they do not represent normal external egress. + +```mermaid +flowchart TD + A["Request to inference.local"] --> B["Inference local adapter"] + B --> C{"TLS and inference context available?"} + C -- No --> D["Local denial and log"] + C -- Yes --> E["Terminate client TLS"] + E --> F["Parse HTTP request"] + F --> G{"Known inference route?"} + G -- Yes --> H["Route through openshell-router"] + H --> I["Strip caller auth and inject provider auth/model"] + I --> J["Stream response with limits"] + G -- No --> K["403 and close"] +``` + +```mermaid +flowchart TD + A["Request to policy.local"] --> B["Policy local adapter"] + B --> C{"Local route"} + C -- "Current policy" --> D["Policy snapshot response"] + C -- "Recent denials" --> E["Bounded denial summaries"] + C -- "Policy proposal" --> F["Validate and submit proposal"] + D --> G["HTTP response"] + E --> G + F --> G +``` + +### Deployment Modes + +The first implementation can remain embedded in `openshell-sandbox`, but the +proxy should be shaped around explicit runtime contracts. + +| Mode | Shape | Main concern | +|------|-------|--------------| +| Embedded | Current sandbox process owns proxy modules | Lowest migration cost | +| Standalone process | Sandbox supervisor launches a proxy binary | Clear process/API boundary | +| Sidecar | Proxy runs outside the payload container but inside the sandbox boundary | Reliable process identity across namespaces | + +A pluggable proxy must expose the configured userland surfaces, implement the +gateway APIs it needs, and prove equivalent policy enforcement through tests. +The nftables rules that force or reject userland traffic belong to the sandbox +network boundary even if the proxy process later moves into a standalone binary +or sidecar. + +## Implementation plan + +The migration plan lives in [implementation-plan.md](implementation-plan.md). +The intended order is: first add regression coverage, then introduce the shared +authorization result and destination validation, then preserve the current +forward HTTP single-request/guarded-relay invariant, then add shared TLS +handling, TCP parser boundaries, nftables-backed policy DNS capture, local +service adapters, and finally the runtime boundary cleanup. + +## Risks + +- Tightening endpoint metadata failures from fail-open to deny may expose + latent policy or Rego errors. +- Deterministic endpoint selection may reject policies that currently load but + only work by accident. +- Transparent TCP capture adds network namespace interception complexity. +- Transparent TCP capture must coexist with the current nftables bypass + reject/log table without creating gaps where direct egress bypasses the proxy. +- Sidecar mode needs a reliable identity source for binary/path scoped policy. +- `policy.local` expands the sandbox-local control surface and needs strict + route validation, body limits, redaction, and gateway authentication. + +## Alternatives + +- Keep patching each current proxy path separately. This has the lowest short + term cost but keeps the security surface duplicated. +- Replace CONNECT with forward proxy. This does not work for arbitrary TCP and + is not a replacement for HTTPS tunnels. +- Build only transparent TCP. This helps native clients but does not replace + explicit proxy support used by common HTTP tooling. + +## Open questions + +1. Should overlapping endpoint metadata be rejected at policy load time, or + should policy name plus endpoint index define precedence? +2. Should missing TLS state fail closed for credential-capable or inspected + endpoints? +3. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted, + or should DNS query correlation be required for stricter modes? +4. What TTL cap and stale-generation grace period should policy DNS use? +5. Which process identity source should sidecar mode use when it cannot inspect + payload process metadata through local `/proc`? +6. Which proxy capabilities should be negotiated with the gateway at startup? + +## Expected result + +Adding a new HTTP-family protocol parser should require parser code, policy +schema/Rego support, tests, and docs. It should not require new CONNECT and +forward-proxy branches. REST, GraphQL, WebSocket upgrade policy, request-body +credential rewrite, and WebSocket text-frame rewrite should all remain behind +the shared HTTP/WebSocket relay boundary. + +Adding a native TCP application parser should require policy DNS/capture +support, a TCP application parser, policy rules, tests, and docs. Plain +`protocol: tcp` remains L4 authorization plus byte relay. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md new file mode 100644 index 0000000000..b428fed145 --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md @@ -0,0 +1,167 @@ +# Current Shape Appendix + +This appendix records the current proxy shape and the review findings that +motivate the adapter model. The main RFC intentionally keeps these details out +of the direction document. + +## Current Entry Points + +The sandbox proxy currently handles multiple userland-facing paths in the same +large module: + +- CONNECT proxy traffic for HTTPS and generic TCP tunnels. +- Forward HTTP proxy traffic for absolute-form HTTP requests. +- Local service routes such as `inference.local`. +- Network namespace bypass enforcement through nftables reject/log rules. +- Policy and endpoint metadata lookups through OPA/Rego. +- DNS resolution and endpoint validation for CONNECT and forward HTTP egress. +- Credential injection and redaction for provider-backed HTTP egress. +- Opt-in REST request-body credential rewrite. +- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement. + +The issue is not that these features exist. The issue is that entry mechanisms, +policy evaluation, endpoint metadata lookup, credential injection, and byte +relay decisions are interleaved. + +## Current CONNECT Shape + +```mermaid +flowchart TD + Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["CONNECT denial"] + Allowed -- Yes --> Meta["Query endpoint metadata"] + Meta --> Config{"L7 or credential config?"} + Config -- No --> Raw["Open upstream and copy bytes"] + Config -- Yes --> Tunnel["Return tunnel-ready response"] + Tunnel --> Inspect["Parse tunneled HTTP when possible"] + Inspect --> L7["Evaluate HTTP policy"] + L7 --> Inject["Inject credentials if configured"] + Inject --> Upstream["Write upstream and relay response"] +``` + +This path has the strongest HTTP relay behavior because it can keep parsing +requests on a long-lived tunnel and enforce L7 rules per request. + +## Current Forward HTTP Shape + +```mermaid +flowchart TD + Client["Absolute-form HTTP request"] --> Parse["Parse first request"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["HTTP denial"] + Allowed -- Yes --> L7{"Matching L7 endpoint?"} + L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"] + Eval --> Rewrite["Rewrite to origin-form + configured credentials"] + L7 -- No --> Rewrite + Rewrite --> Close["Force Connection: close except WebSocket upgrade"] + Close --> Upstream["Open upstream"] + Upstream --> Relay["Guarded HTTP relay / upgrade relay"] +``` + +The latest main branch no longer has the old raw-copy-after-first-request shape +for ordinary forward HTTP. It rewrites ordinary requests with `Connection: +close`, uses guarded HTTP relay helpers for body handling, and sends allowed +WebSocket upgrades through the same upgrade relay. That is a narrower surface +than the historical bidirectional copy, but it is still implemented separately +from the CONNECT relay path. + +## Current Network Namespace Enforcement + +```mermaid +flowchart TD + Start["Process in sandbox network namespace"] --> Dest{"Destination"} + Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"] + Dest -- "Loopback" --> Loopback["Accept loopback"] + Dest -- "Established/related" --> Established["Accept response packet"] + Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"] + Reject --> Monitor["Bypass monitor reads dmesg"] + Monitor --> OCSF["OCSF network + detection events"] +``` + +The sandbox now installs an `inet` nftables filter table for bypass +enforcement. The table accepts proxy-bound traffic, loopback, and established +flows, then rejects and optionally logs other TCP/UDP traffic. It does not +currently redirect native TCP connections into the proxy. + +## Current Local Service Shape + +```mermaid +flowchart TD + Request["Request to local name"] --> Match{"Known local route?"} + Match -- "inference.local" --> Inference["Inference routing logic"] + Match -- "policy.local" --> Policy["Policy local logic"] + Match -- No --> External["Normal egress path"] + Inference --> LocalResponse["Local response"] + Policy --> LocalResponse +``` + +Local routes are userland-facing proxy surfaces. They should stay distinct from +external egress while still fitting the adapter model. + +## Findings To Preserve + +### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes + +The historical forward path evaluated at most the first absolute-form request, +rewrote it, then switched to bidirectional copy. Bytes already buffered after +the first header block, or later pipelined requests on the same client/upstream +connection, could reach upstream without the CONNECT L7 relay's per-request +parser/evaluator. + +Latest main mitigates this by forcing ordinary forward HTTP to one request per +connection and by using guarded relay helpers. The adapter model should +preserve the invariant either by keeping forward HTTP single-request/close or +by passing the first parsed request into a shared HTTP relay loop. + +### Endpoint config is not tied to deterministic matched policy + +The policy name used for L4 authorization and logging can be selected through a +different precedence rule than endpoint metadata. With overlapping host, port, +and binary rules, allowed IPs, TLS behavior, enforcement, and +`allow_encoded_slash` can come from a different endpoint than the policy name +logged and used for L4 allow. + +The adapter model requires authorization to return one decision with one +deterministic matched endpoint. + +### Endpoint metadata query failures fail open to L4 behavior + +If endpoint metadata lookup fails, callers can interpret the result as no L7 +configuration and downgrade to credential-only or raw L4 relay. + +The adapter model treats endpoint metadata as part of the authorization result. +Failure to materialize required metadata should deny rather than erase extended +configuration. + +### Control-plane port block only applies on one resolution path + +Blocked control-plane ports are enforced inside one allowed-IPs validation +path, while the normal host-based path uses a different validation route. + +The adapter model moves resolution, allowed IP checks, SSRF checks, and +control-plane port blocks into shared destination validation. + +## Existing Feature Inventory + +The refactor should preserve: + +- CONNECT explicit proxy support. +- Forward HTTP explicit proxy support. +- nftables bypass reject/log enforcement. +- Provider credential injection and redaction. +- REST request-body credential rewrite. +- WebSocket text-frame credential rewrite. +- REST endpoint method/path policy. +- GraphQL L7 policy. +- WebSocket transport and GraphQL-over-WebSocket policy. +- Inference routing through `inference.local`. +- Agent-facing policy routes through `policy.local`. +- Timeout and resource tracking for client, upstream, and local service work. +- Structured OCSF logging for network and HTTP policy outcomes. +- SSRF and internal address protections. +- Control-plane port protection. +- `allowed_ips` endpoint restrictions. +- TLS termination for inspectable client connections. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md new file mode 100644 index 0000000000..94ba53b7f0 --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md @@ -0,0 +1,127 @@ +# Implementation Plan + +This plan is intentionally separate from the main RFC so the proposal can stay +direction-focused. + +## Phase 0 - Regression Tests + +- Add tests for forward HTTP pipelining and keep-alive follow-on requests, + including the current `Connection: close` mitigation. +- Add tests for overlapping endpoint metadata selection. +- Add tests for endpoint metadata query failures. +- Add tests for control-plane port blocking through all destination validation + paths. +- Add nftables bypass enforcement tests that verify proxy-bound traffic is + accepted while direct TCP/UDP egress is rejected and logged when available. + +## Phase 1 - Authorization Result + +- Introduce `EgressIntent` and `EgressDecision`. +- Make authorization return matched policy and matched endpoint metadata + together. +- Fail closed when required endpoint metadata cannot be materialized. +- Emit consistent OCSF network denial events from the shared boundary. + +## Phase 2 - Shared Destination Validation + +- Move DNS resolution, allowed IP filtering, SSRF checks, and control-plane port + checks into one destination validation path. +- Return an `UpstreamConnector` rather than an opened upstream socket. +- Add tests proving CONNECT, forward HTTP, and transparent TCP use the same + validation behavior. + +## Phase 3 - Forward HTTP Adapter + +- Convert forward HTTP into an adapter that parses the first absolute-form + request and builds an egress intent. +- Route the parsed first request into the shared HTTP relay or preserve the + current guarded single-request relay behavior. +- Keep the no-raw-copy invariant after the first request. + +## Phase 4 - HTTP And WebSocket Relay Consolidation + +- Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket + upgrade policy, credential resolution, redaction, request rewrite, upstream + dial, and response relay. +- Evaluate every HTTP request before upstream write. +- Ensure denied HTTP requests do not create upstream TCP sessions. +- Preserve opt-in REST request-body credential rewrite behind the shared HTTP + relay, including bounded buffering, supported content-type handling, + `Content-Length` recomputation, and fail-closed unresolved placeholders. +- Preserve WebSocket upgrade handling behind the shared relay, including + opt-in client-to-server text-frame credential rewrite, WebSocket transport + message policy, GraphQL-over-WebSocket policy, and raw passthrough for other + upgraded protocols. + +## Phase 5 - Shared TLS Termination + +- Move client-side TLS detection and termination before the HTTP/TCP relay + split. +- Keep endpoint TLS behavior on `EgressDecision`. +- Remove duplicate HTTP-specific and TCP-specific TLS termination decisions. + +## Phase 6 - TCP Relay And Parser Boundary + +- Rename raw TCP relay concepts to `TcpRelay`. +- Add a TCP application parser dispatch point for future protocol enforcement. +- Keep `protocol: tcp` as L4 authorization plus byte copy. +- Let TCP application parsers own their message loop and call the connector + when protocol state allows. + +## Phase 7 - Policy DNS And Transparent TCP + +- Add policy DNS registration for native TCP endpoint names. +- Replace static host-file mapping with query-driven DNS answers. +- Publish active DNS answer state and capture rules. +- Implement nftables REDIRECT/TPROXY capture rules ahead of the bypass reject + path; do not add a parallel iptables path. +- Implement transparent TCP adapter lookup from captured original destination + to active endpoint generation. +- Decide TTL and stale-generation behavior. + +## Phase 8 - Local Service Adapters + +- Model `inference.local` as a local adapter with TLS termination, route + validation, provider auth injection, streaming limits, and OCSF logging. +- Model `policy.local` as a local adapter for current policy, bounded denial + summaries, and policy proposals. +- Keep both paths outside normal external egress relay. + +## Phase 9 - Runtime Boundary + +- Keep embedded mode for the first migration. +- Define the proxy runtime API needed for a future standalone binary: + configured listeners, policy updates, gateway calls, telemetry, and shutdown. +- Identify process identity requirements for standalone and sidecar modes. + +## Phase 10 - Cleanup + +- Remove duplicated endpoint metadata queries from relay paths. +- Remove duplicated deny rendering where adapters can own response shape. +- Remove any remaining forward HTTP raw-copy fallback. +- Update architecture docs once implementation lands. + +## Testing Plan + +- Unit-test each adapter's intent construction and deny response shape. +- Unit-test authorization precedence for overlapping policy and endpoint rules. +- Integration-test shared destination validation across CONNECT, forward HTTP, + and transparent TCP. +- Integration-test HTTP keep-alive and pipelined requests with REST, GraphQL, + and WebSocket upgrade enforcement. +- Integration-test credential injection in L4-only HTTP and HTTP-inspected + paths. +- Integration-test REST request-body credential rewrite for JSON, + form-url-encoded, `text/*`, unsupported content types, chunked framing, body + caps, and unresolved placeholders. +- Integration-test WebSocket text-frame credential rewrite, raw upgraded + passthrough, WebSocket message policy, GraphQL-over-WebSocket policy, and + safe compression negotiation. +- Integration-test TLS termination before HTTP/TCP relay split. +- Integration-test `protocol: tcp` byte-copy behavior. +- Add parser harness tests before adding Redis, Postgres, or similar TCP + application parsers. +- Integration-test policy DNS TTL, stale generation handling, and captured + connect correlation. +- Integration-test `inference.local` and `policy.local` body limits, timeout + behavior, redaction, and local denial responses. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md new file mode 100644 index 0000000000..b13e259f45 --- /dev/null +++ b/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md @@ -0,0 +1,259 @@ +# Technical Design Appendix + +This appendix carries the implementation-level design details behind the main +RFC. + +## Shared Data Boundaries + +### EgressIntent + +`EgressIntent` is the normalized description of what userland is trying to do. + +It should carry: + +- entry transport: CONNECT, forward HTTP, transparent TCP, or local HTTP; +- requested destination host/port or captured original IP/port; +- process identity inputs collected by the adapter/runtime; +- optional first HTTP request for forward proxy traffic; +- optional local service route. + +Adapters build intents. They should not query endpoint metadata or select +relays. + +### EgressDecision + +`EgressDecision` is the policy result consumed by validation and relay code. + +It should carry: + +- allow or deny; +- deterministic matched policy identifier; +- deterministic matched endpoint identifier and endpoint metadata; +- process identity used for evaluation; +- destination and allowed IP constraints; +- TLS behavior; +- protocol enforcement; +- logging context and denial reason. + +Relay code should read this decision. It should not query OPA again for +endpoint metadata, TLS mode, allowed IPs, or parser selection. + +## Protocol Enforcement + +Use a protocol enforcement value derived from endpoint policy: + +| Policy protocol | Enforcement | Relay behavior | +|-----------------|-------------|----------------| +| omitted / `tcp` | None | L4 authorization plus byte relay, with optional HTTP sniff for credential injection | +| `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite | +| `graphql` | HTTP | HTTP request parser with GraphQL rules | +| `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy | +| future `redis`, `postgres`, `mysql`, ... | TCP application | Protocol-specific TCP parser owns the message loop | + +`protocol: tcp` is effectively the default L4 mode. It should not run TCP +application parsers. + +Avoid using the term "provider" for these parser concepts because providers +are already a first-class credential and routing domain in OpenShell. + +## Suggested Types + +The exact Rust shape can evolve, but the boundaries should look like this: + +```rust +enum EgressTransport { + Connect, + ForwardHttp, + TransparentTcp, + LocalHttp, +} + +struct EgressIntent { + transport: EgressTransport, + destination: RequestedDestination, + process: ProcessIdentity, + first_request: Option, + local_route: Option, +} + +struct EgressDecision { + outcome: PolicyOutcome, + matched_policy: Option, + endpoint: Option, + log_context: EgressLogContext, +} + +struct MatchedEndpoint { + id: EndpointId, + allowed_ips: AllowedIpPolicy, + tls: TlsPolicy, + enforcement: ProtocolEnforcement, +} + +enum ProtocolEnforcement { + None, + Http(HttpL7Config), + TcpApplication(TcpApplicationConfig), +} + +enum HttpL7Protocol { + Rest, + Graphql, + Websocket, +} + +struct HttpL7Config { + protocol: HttpL7Protocol, + allow_encoded_slash: bool, + websocket_credential_rewrite: bool, + request_body_credential_rewrite: bool, + websocket_graphql_policy: bool, +} + +struct RelayContext { + decision: EgressDecision, + connector: UpstreamConnector, + deadlines: RelayDeadlines, + telemetry: RelayTelemetry, +} +``` + +`UpstreamConnector` is the relay-owned dial boundary. It encapsulates the +validated destination and lets relays/parsers open an upstream connection only +after protocol policy allows it. + +## Module Layout + +A future split could look like: + +| Module | Responsibility | +|--------|----------------| +| `proxy::adapter::connect` | Parse CONNECT and render CONNECT responses | +| `proxy::adapter::forward_http` | Parse absolute-form HTTP and preserve first request | +| `proxy::adapter::transparent_tcp` | Recover captured original destination | +| `proxy::adapter::policy_dns` | Answer eligible DNS queries and publish active mappings | +| `proxy::adapter::local` | Implement `inference.local` and `policy.local` surfaces | +| `proxy::auth` | Build decisions from intents and OPA results | +| `proxy::destination` | Resolve, filter, and validate destinations | +| `proxy::netfilter` | Own nftables bypass and future transparent capture rules | +| `proxy::relay::http` | HTTP request loop, credentials, REST/GraphQL/WebSocket upgrade policy | +| `proxy::relay::websocket` | WebSocket frame validation, text-frame rewrite, and message policy | +| `proxy::relay::tcp` | TCP byte relay and TCP application parser dispatch | +| `proxy::relay::tls` | Shared client-side TLS termination | +| `proxy::parser` | HTTP, WebSocket, and TCP application parser traits/config | +| `proxy::telemetry` | OCSF and tracing helpers | + +## Policy DNS And Resolved TCP State + +Policy DNS should be query-driven rather than a static `/etc/hosts` snapshot. + +1. Policy load registers eligible native TCP endpoint names. +2. Userland performs DNS lookup. +3. Policy DNS checks whether the name is registered for native TCP. +4. Policy DNS resolves through trusted upstream DNS. +5. Answers are filtered against endpoint metadata and SSRF controls. +6. The adapter publishes the DNS answer, endpoint generation, and capture rule. +7. Userland later calls `connect(ip:port)`. +8. Transparent TCP recovers the original destination and maps it to the active + endpoint generation. +9. Normal egress authorization and relay selection run. + +The resolved endpoint store is therefore not a preemptive global DNS snapshot. +It is active state produced by policy-eligible lookups and consumed by +transparent TCP connects. + +## nftables Boundary + +Current main uses nftables, not iptables, for sandbox network bypass +enforcement. The installed `inet` table accepts traffic to the sandbox proxy, +loopback, and established/related flows, then rejects and optionally logs other +TCP/UDP traffic. The bypass monitor reads those log lines and emits OCSF +network and detection events. + +Transparent TCP capture should build on this same nftables substrate: + +- capture rules must run before the generic bypass reject rules; +- capture rules should be scoped to active policy DNS IP/port mappings; +- capture state should be updated atomically with endpoint generation changes; +- reject/log rules remain the fallback for unmatched TCP/UDP egress; +- VM or Podman driver nftables rules are infrastructure NAT/isolation and + should not be treated as the proxy policy enforcement point. + +## Endpoint Selection And OPA + +OPA/Rego should return policy and endpoint metadata through one deterministic +authorization result. It should not let policy name and endpoint config be +selected by different precedence rules. + +Two acceptable approaches: + +- Reject overlapping endpoint metadata at load or merge time. +- Define a single deterministic precedence key and use it for both policy name + and endpoint metadata. + +Endpoint metadata query failures should fail closed when metadata is required +for the selected endpoint. They should not silently downgrade to L4 behavior. + +## Credential Injection Boundary + +Credential injection belongs in the HTTP relay: + +1. Authorization selects the endpoint and confirms credentials may be used. +2. The HTTP relay resolves credentials only when it has an allowed HTTP request. +3. Secrets are redacted from logs and policy-visible metadata. +4. The final upstream request or frame is rewritten with real credentials + immediately before write. + +Both L4-only HTTP and HTTP-inspected paths can inject credentials. The +difference is whether REST, GraphQL, or WebSocket policy is evaluated before +the rewrite. + +Credential rewrite slots should be explicit: + +- request target, query values, and headers for HTTP-family traffic; +- REST request bodies only when `request_body_credential_rewrite` is enabled; +- client-to-server WebSocket text frames only when + `websocket_credential_rewrite` is enabled; +- GraphQL-over-WebSocket connection/control messages when they are carried in + text frames and the endpoint enables the WebSocket rewrite path. + +Request-body rewrite is REST-only. It should buffer bounded UTF-8 textual +bodies, including JSON, form-url-encoded, and `text/*`, recompute +`Content-Length`, preserve unsupported bodies that contain no reserved +credential markers, and fail closed when a reserved placeholder cannot be +resolved safely. Binary WebSocket frames are not rewritten. + +## Parser Boundary + +Protocol parsers operate on streams owned by the relay. + +- HTTP parsing converts bytes into request metadata, evaluates request policy, + and loops for keep-alive or pipelined requests. +- WebSocket parsing starts only after an allowed HTTP upgrade. It validates the + handshake/frame stream and owns client-to-server text-frame inspection when + credential rewrite, transport message policy, GraphQL-over-WebSocket policy, + or compression handling is configured. +- TCP application parsers read client and upstream streams as needed and own + their message loop. +- A TCP parser can deny before dialing, dial for a server handshake, or keep + evaluating commands/queries throughout the session. + +This avoids a separate dial strategy enum. The parser knows which protocol +milestone is sufficient to call the validated connector. + +## Timeout And Resource Ownership + +| Owner | Resource | +|-------|----------| +| Adapter | Client-side parse timeout and adapter-specific deny response | +| Authorization | OPA deadline and policy evaluation telemetry | +| Destination validator | DNS timeout, allowed IP checks, SSRF checks, control-plane port checks | +| TLS terminator | Client TLS handshake timeout and certificate selection | +| HTTP relay | Per-request read/write deadlines, body caps, request-body rewrite caps, upstream reuse | +| WebSocket relay | Upgrade validation, frame limits, text-frame rewrite, compression limits, message policy | +| TCP relay | Byte-copy idle timeout and half-close handling | +| TCP parser | Protocol message timeouts and parser-specific limits | +| Local service adapter | Local route body limits, response caps, gateway call timeout | + +Timeouts should be recorded in telemetry at the owner boundary that can explain +the failure. From 03d25482b099cccd75644ec971fc760bde999bdb Mon Sep 17 00:00:00 2001 From: John Myers Date: Fri, 26 Jun 2026 09:46:49 -0700 Subject: [PATCH 3/8] docs(rfc): update sandbox proxy adapter proposal Signed-off-by: John Myers --- .../README.md | 420 ------------------ .../current-shape.md | 167 ------- .../README.md | 404 +++++++++++++++++ .../current-shape.md | 223 ++++++++++ .../implementation-plan.md | 73 ++- .../technical-design.md | 157 +++++-- 6 files changed, 802 insertions(+), 642 deletions(-) delete mode 100644 rfc/0004-sandbox-proxy-egress-adapter/README.md delete mode 100644 rfc/0004-sandbox-proxy-egress-adapter/current-shape.md create mode 100644 rfc/0005-sandbox-proxy-egress-adapter/README.md create mode 100644 rfc/0005-sandbox-proxy-egress-adapter/current-shape.md rename rfc/{0004-sandbox-proxy-egress-adapter => 0005-sandbox-proxy-egress-adapter}/implementation-plan.md (58%) rename rfc/{0004-sandbox-proxy-egress-adapter => 0005-sandbox-proxy-egress-adapter}/technical-design.md (57%) diff --git a/rfc/0004-sandbox-proxy-egress-adapter/README.md b/rfc/0004-sandbox-proxy-egress-adapter/README.md deleted file mode 100644 index 01001ca15e..0000000000 --- a/rfc/0004-sandbox-proxy-egress-adapter/README.md +++ /dev/null @@ -1,420 +0,0 @@ ---- -authors: - - "@johntmyers" -state: draft -links: - - https://github.com/NVIDIA/OpenShell/issues/1107 - - https://github.com/NVIDIA/OpenShell/pull/1083 - - https://github.com/NVIDIA/OpenShell/pull/1151 ---- - -# RFC 0004 - Sandbox Proxy Egress Adapter Model - - - -## Summary - -Refactor sandbox egress around one shared authorization and relay pipeline. -CONNECT, forward HTTP proxy, transparent native TCP, policy DNS, -`inference.local`, and `policy.local` should become adapters that translate -userland entry points into a common egress intent. Policy evaluation, -destination validation, credential injection, request-body rewrite, -WebSocket upgrade handling, protocol parsing, and relay ownership should happen -behind shared boundaries. - -This RFC keeps the main direction in this document. Supporting detail lives in: - -- [Current shape appendix](current-shape.md) -- [Technical design appendix](technical-design.md) -- [Implementation plan](implementation-plan.md) - -## Motivation - -The sandbox proxy has accumulated separate egress paths for CONNECT, forward -HTTP, local services, inference routing, endpoint metadata, credential -injection, and L7 policy. That makes security changes easy to apply to one path -and miss in another. - -The target shape separates three concerns: - -- **Adapters** describe how userland reached the proxy. -- **Authorization** decides whether that egress is allowed and what endpoint - behavior applies. -- **Relays** own bytes, credentials, protocol parsing, and upstream dialing. - -## Non-goals - -- Replace CONNECT with forward proxy as the only explicit proxy mode. -- Add SOCKS support. -- Add HTTP/2 L7 parsing in this refactor. -- Redesign provider credential storage. -- Reintroduce iptables as the sandbox packet filtering backend. -- Use eBPF connect hooks for transparent capture. Native TCP capture needs a - userland proxy in the byte stream for TLS termination and protocol parsing. - -## Proposal - -### Migration Big Rocks - -1. **Transport adapters.** CONNECT, forward HTTP, transparent TCP, policy DNS, - and local service routes become small entry adapters. They parse their - surface and produce either an egress intent, a local response, or a DNS - answer. They do not duplicate policy evaluation. -2. **Egress intent and decision.** The shared authorization boundary evaluates - L4 policy once per connection intent and returns one decision containing the - matched policy, matched endpoint, process identity, allowed IP metadata, TLS - behavior, and protocol enforcement. -3. **Relays.** Relays receive an authorized destination connector, not an - already-open upstream socket. HTTP relays evaluate every request before - dialing, own REST request-body credential rewrite, and hand allowed - WebSocket upgrades to the WebSocket relay. TCP application parsers own their - protocol loop and decide when a validated upstream connection is needed. - -### Unified Adapter Flow - -```mermaid -flowchart TD - User["Userland payload / harness"] - - subgraph ExplicitProxy["Explicit proxy listener"] - ProxyBytes["HTTP proxy bytes"] - IsConnect{"CONNECT request?"} - Connect["CONNECT adapter"] - Forward["Forward HTTP adapter"] - ProxyBytes --> IsConnect - IsConnect -- Yes --> Connect - IsConnect -- No --> Forward - end - - subgraph NativeTcp["Policy DNS + native TCP"] - NameLookup["Userland DNS lookup"] - PolicyDns["Policy DNS adapter"] - DnsAnswer["DNS answer"] - NativeConnect["Userland connect(ip:port)"] - TcpAdapter["Transparent TCP adapter"] - NameLookup --> PolicyDns - PolicyDns --> DnsAnswer - DnsAnswer --> NativeConnect - NativeConnect --> TcpAdapter - end - - subgraph LocalApis["Sandbox-local services"] - InferenceReq["Request to inference.local"] - PolicyReq["Request to policy.local"] - InferenceAdapter["Inference local adapter"] - PolicyAdapter["Policy local adapter"] - InferenceReq --> InferenceAdapter - PolicyReq --> PolicyAdapter - end - - subgraph Shared["Shared egress pipeline"] - Intent["Egress intent"] - Auth["Authorize and select endpoint"] - Decision["Egress decision"] - Validate["Resolve and validate destination"] - Relay["Relay"] - Deny["Adapter-specific deny response"] - Intent --> Auth - Auth --> Allowed{"Allowed?"} - Allowed -- No --> Deny - Allowed -- Yes --> Decision - Decision --> Validate - Validate --> Relay - end - - User --> ProxyBytes - User --> NameLookup - User --> NativeConnect - User --> InferenceReq - User --> PolicyReq - - Connect --> Intent - Forward --> Intent - TcpAdapter --> Intent - InferenceAdapter --> InferenceResp["Local inference response"] - PolicyAdapter --> PolicyResp["Local policy response"] -``` - -### Relay Flow - -```mermaid -flowchart TD - Start["Authorized egress + destination connector"] - Start --> HasFirst{"First HTTP request already parsed?"} - - HasFirst -- Yes --> ForwardMode{"Selected enforcement"} - ForwardMode -- "L4 only" --> HttpCred["HTTP relay
credential injection only"] - ForwardMode -- "HTTP rules" --> HttpL7["HTTP relay
REST/GraphQL/WebSocket policy"] - ForwardMode -- "TCP app rules" --> BadForward["Deny: HTTP request for TCP app endpoint"] - - HasFirst -- No --> Inspect["Inspect tunnel or native stream bytes"] - Inspect --> SkipTls{"Endpoint says skip TLS handling?"} - SkipTls -- Yes --> TcpBytes["TCP relay
byte copy"] - SkipTls -- No --> Peek["Peek client bytes"] - Peek --> IsTls{"TLS ClientHello?"} - IsTls -- Yes --> Tls["Shared TLS terminator"] - IsTls -- No --> Readable["Readable client stream"] - Tls --> Readable - - Readable --> Mode{"Selected enforcement"} - Mode -- "L4 only" --> SniffHttp{"Looks like HTTP?"} - SniffHttp -- Yes --> HttpCred - SniffHttp -- No --> TcpBytes - - Mode -- "HTTP rules" --> MustHttp{"Looks like HTTP?"} - MustHttp -- Yes --> HttpL7 - MustHttp -- No --> DenyHttp["Deny: expected HTTP"] - - Mode -- "TCP app rules" --> TcpParser["TCP relay
application parser owns loop"] - - HttpCred --> Creds["Resolve and redact credentials"] - HttpL7 --> CredsL7["Resolve and redact credentials"] - CredsL7 --> ParseHttp["Parse and evaluate each HTTP request"] - ParseHttp --> HttpAllowed{"Request allowed?"} - HttpAllowed -- No --> HttpDeny["Local HTTP deny
no upstream connect"] - HttpAllowed -- Yes --> Rewrite["Rewrite configured credential slots"] - Creds --> Rewrite - Rewrite --> HttpDial["Connect or reuse upstream"] - HttpDial --> HttpResponse["Write request and relay response"] - HttpResponse --> Upgrade{"101 WebSocket upgrade?"} - Upgrade -- No --> NextHttp["Continue HTTP request loop"] - Upgrade -- Yes --> WsMode{"WebSocket inspection needed?"} - WsMode -- No --> RawUpgrade["Raw upgraded stream"] - WsMode -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] - NextHttp --> ParseHttp - - TcpParser --> ParserDial["Parser dials upstream when protocol allows"] - TcpBytes --> TcpDial["Connect upstream"] - TcpDial --> ByteCopy["Copy bytes"] -``` - -Relay rules: - -- HTTP credential injection happens in both HTTP modes: L4-only HTTP and - HTTP-inspected. -- Credential injection includes request target, query, headers, opt-in REST - request bodies, and opt-in client-to-server WebSocket text frames. -- HTTP L7 policy is evaluated before upstream dial for each request. -- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101` - upgrade, the WebSocket relay owns frame parsing when text-frame credential - rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe - compression handling is configured. Other upgraded streams remain raw. -- Forward HTTP must stay in the shared HTTP relay loop. It must not evaluate - one request and then switch to raw bidirectional copy. Keeping forward HTTP - single-request with `Connection: close` is also acceptable, but the invariant - is that no follow-on request bytes reach upstream unevaluated. -- `protocol: tcp` means L4 authorization plus byte copy unless HTTP is detected - for credential injection. -- Future TCP application parsers, such as Redis or Postgres, own the full - message loop and can parse multiple commands over one TCP session. - -### CONNECT Adapter - -CONNECT remains the standard explicit proxy tunnel for HTTPS and arbitrary TCP. -It parses the CONNECT line into an egress intent, then waits for the shared -relay to decide if and when an upstream connection should be opened. - -```mermaid -flowchart TD - Client["Client sends CONNECT host:port"] --> Parse["Parse target"] - Parse --> Intent["Build egress intent"] - Intent --> Auth["Shared authorization"] - Auth --> Allowed{"Destination allowed?"} - Allowed -- No --> Deny["CONNECT deny response"] - Allowed -- Yes --> Ready["Return tunnel-ready response"] - Ready --> Relay["Relay inspects tunneled bytes"] - Relay --> Dial["Relay or parser connects upstream when allowed"] -``` - -CONNECT should stay because forward proxy is only a plaintext HTTP request -format. CONNECT is still the generic explicit proxy mode for TLS and non-HTTP -TCP clients. - -### Forward HTTP Adapter - -Forward HTTP is compatibility for clients that send absolute-form HTTP requests. -The adapter parses the first request and hands it to the shared HTTP relay or -an equivalent guarded single-request relay. - -```mermaid -flowchart TD - Req["Absolute-form HTTP request"] --> Parse["Parse URI and first request"] - Parse --> Intent["Build egress intent"] - Intent --> Auth["Shared authorization"] - Auth --> Allowed{"Allowed?"} - Allowed -- No --> Deny["HTTP deny response"] - Allowed -- Yes --> Relay["Shared or guarded HTTP relay"] - Relay --> Mode{"Connection mode"} - Mode -- "Persistent" --> Loop["Evaluate every request on this connection"] - Mode -- "Single request" --> Close["Force Connection: close"] -``` - -### Transparent TCP Adapter - -Transparent TCP supports native clients that do not know they are using a -proxy. The capture mechanism should be network namespace interception into a -userland proxy listener. Since main now uses nftables for sandbox bypass -enforcement, transparent capture should be designed as nftables -REDIRECT/TPROXY state in the inner sandbox network namespace, not as an -iptables path. - -```mermaid -flowchart TD - Policy["Policy load / reload"] --> Register["Register native TCP names"] - Lookup["Userland DNS lookup"] --> Dns["Policy DNS adapter"] - Register --> Dns - Dns --> Answer["Return approved IPs"] - Answer --> Capture["Enable capture for active IP:port"] - Connect["Userland connect(ip:port)"] --> Capture - Capture --> Adapter["Transparent TCP adapter"] - Adapter --> Intent["Build egress intent from original destination"] - Intent --> Shared["Shared authorization and relay"] -``` - -### Policy DNS - -Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is -query-driven: check whether the name is policy-eligible, resolve through trusted -DNS, filter returned IPs, publish the active endpoint mapping, and answer -userland. - -```mermaid -flowchart TD - Query["DNS query from userland"] --> Adapter["Policy DNS adapter"] - Adapter --> Known{"Registered native TCP policy name?"} - Known -- No --> Refuse["NXDOMAIN / REFUSED / SERVFAIL"] - Known -- Yes --> Upstream["Trusted upstream DNS lookup"] - Upstream --> Filter["Filter answers against endpoint policy"] - Filter --> Publish["Publish active mapping and capture rule"] - Publish --> Answer["DNS answer"] -``` - -The later `connect(ip:port)` still creates the egress intent and runs through -normal authorization. - -### Network Enforcement Substrate - -Current main uses nftables for bypass enforcement. It accepts proxy-bound -traffic and loopback, accepts established flows, then rejects and optionally -logs other TCP/UDP traffic for the bypass monitor. That is enforcement, not -native TCP capture. - -```mermaid -flowchart TD - Conn["Userland packet"] --> ProxyDest{"Proxy destination?"} - ProxyDest -- Yes --> AcceptProxy["nftables accept"] - ProxyDest -- No --> Capture{"Future native TCP capture match?"} - Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"] - Capture -- No --> Reject["nftables log + reject bypass"] - Reject --> Monitor["Bypass monitor emits OCSF"] -``` - -The transparent TCP work should extend this nftables model with explicit -capture rules that run before the reject path and are scoped to active policy -DNS mappings. - -### Local Service Adapters - -`inference.local` and `policy.local` are sandbox-local APIs. They should use -the adapter model, but they do not represent normal external egress. - -```mermaid -flowchart TD - A["Request to inference.local"] --> B["Inference local adapter"] - B --> C{"TLS and inference context available?"} - C -- No --> D["Local denial and log"] - C -- Yes --> E["Terminate client TLS"] - E --> F["Parse HTTP request"] - F --> G{"Known inference route?"} - G -- Yes --> H["Route through openshell-router"] - H --> I["Strip caller auth and inject provider auth/model"] - I --> J["Stream response with limits"] - G -- No --> K["403 and close"] -``` - -```mermaid -flowchart TD - A["Request to policy.local"] --> B["Policy local adapter"] - B --> C{"Local route"} - C -- "Current policy" --> D["Policy snapshot response"] - C -- "Recent denials" --> E["Bounded denial summaries"] - C -- "Policy proposal" --> F["Validate and submit proposal"] - D --> G["HTTP response"] - E --> G - F --> G -``` - -### Deployment Modes - -The first implementation can remain embedded in `openshell-sandbox`, but the -proxy should be shaped around explicit runtime contracts. - -| Mode | Shape | Main concern | -|------|-------|--------------| -| Embedded | Current sandbox process owns proxy modules | Lowest migration cost | -| Standalone process | Sandbox supervisor launches a proxy binary | Clear process/API boundary | -| Sidecar | Proxy runs outside the payload container but inside the sandbox boundary | Reliable process identity across namespaces | - -A pluggable proxy must expose the configured userland surfaces, implement the -gateway APIs it needs, and prove equivalent policy enforcement through tests. -The nftables rules that force or reject userland traffic belong to the sandbox -network boundary even if the proxy process later moves into a standalone binary -or sidecar. - -## Implementation plan - -The migration plan lives in [implementation-plan.md](implementation-plan.md). -The intended order is: first add regression coverage, then introduce the shared -authorization result and destination validation, then preserve the current -forward HTTP single-request/guarded-relay invariant, then add shared TLS -handling, TCP parser boundaries, nftables-backed policy DNS capture, local -service adapters, and finally the runtime boundary cleanup. - -## Risks - -- Tightening endpoint metadata failures from fail-open to deny may expose - latent policy or Rego errors. -- Deterministic endpoint selection may reject policies that currently load but - only work by accident. -- Transparent TCP capture adds network namespace interception complexity. -- Transparent TCP capture must coexist with the current nftables bypass - reject/log table without creating gaps where direct egress bypasses the proxy. -- Sidecar mode needs a reliable identity source for binary/path scoped policy. -- `policy.local` expands the sandbox-local control surface and needs strict - route validation, body limits, redaction, and gateway authentication. - -## Alternatives - -- Keep patching each current proxy path separately. This has the lowest short - term cost but keeps the security surface duplicated. -- Replace CONNECT with forward proxy. This does not work for arbitrary TCP and - is not a replacement for HTTPS tunnels. -- Build only transparent TCP. This helps native clients but does not replace - explicit proxy support used by common HTTP tooling. - -## Open questions - -1. Should overlapping endpoint metadata be rejected at policy load time, or - should policy name plus endpoint index define precedence? -2. Should missing TLS state fail closed for credential-capable or inspected - endpoints? -3. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted, - or should DNS query correlation be required for stricter modes? -4. What TTL cap and stale-generation grace period should policy DNS use? -5. Which process identity source should sidecar mode use when it cannot inspect - payload process metadata through local `/proc`? -6. Which proxy capabilities should be negotiated with the gateway at startup? - -## Expected result - -Adding a new HTTP-family protocol parser should require parser code, policy -schema/Rego support, tests, and docs. It should not require new CONNECT and -forward-proxy branches. REST, GraphQL, WebSocket upgrade policy, request-body -credential rewrite, and WebSocket text-frame rewrite should all remain behind -the shared HTTP/WebSocket relay boundary. - -Adding a native TCP application parser should require policy DNS/capture -support, a TCP application parser, policy rules, tests, and docs. Plain -`protocol: tcp` remains L4 authorization plus byte relay. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md deleted file mode 100644 index b428fed145..0000000000 --- a/rfc/0004-sandbox-proxy-egress-adapter/current-shape.md +++ /dev/null @@ -1,167 +0,0 @@ -# Current Shape Appendix - -This appendix records the current proxy shape and the review findings that -motivate the adapter model. The main RFC intentionally keeps these details out -of the direction document. - -## Current Entry Points - -The sandbox proxy currently handles multiple userland-facing paths in the same -large module: - -- CONNECT proxy traffic for HTTPS and generic TCP tunnels. -- Forward HTTP proxy traffic for absolute-form HTTP requests. -- Local service routes such as `inference.local`. -- Network namespace bypass enforcement through nftables reject/log rules. -- Policy and endpoint metadata lookups through OPA/Rego. -- DNS resolution and endpoint validation for CONNECT and forward HTTP egress. -- Credential injection and redaction for provider-backed HTTP egress. -- Opt-in REST request-body credential rewrite. -- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement. - -The issue is not that these features exist. The issue is that entry mechanisms, -policy evaluation, endpoint metadata lookup, credential injection, and byte -relay decisions are interleaved. - -## Current CONNECT Shape - -```mermaid -flowchart TD - Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"] - Parse --> L4["Evaluate network policy"] - L4 --> Allowed{"Allowed?"} - Allowed -- No --> Deny["CONNECT denial"] - Allowed -- Yes --> Meta["Query endpoint metadata"] - Meta --> Config{"L7 or credential config?"} - Config -- No --> Raw["Open upstream and copy bytes"] - Config -- Yes --> Tunnel["Return tunnel-ready response"] - Tunnel --> Inspect["Parse tunneled HTTP when possible"] - Inspect --> L7["Evaluate HTTP policy"] - L7 --> Inject["Inject credentials if configured"] - Inject --> Upstream["Write upstream and relay response"] -``` - -This path has the strongest HTTP relay behavior because it can keep parsing -requests on a long-lived tunnel and enforce L7 rules per request. - -## Current Forward HTTP Shape - -```mermaid -flowchart TD - Client["Absolute-form HTTP request"] --> Parse["Parse first request"] - Parse --> L4["Evaluate network policy"] - L4 --> Allowed{"Allowed?"} - Allowed -- No --> Deny["HTTP denial"] - Allowed -- Yes --> L7{"Matching L7 endpoint?"} - L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"] - Eval --> Rewrite["Rewrite to origin-form + configured credentials"] - L7 -- No --> Rewrite - Rewrite --> Close["Force Connection: close except WebSocket upgrade"] - Close --> Upstream["Open upstream"] - Upstream --> Relay["Guarded HTTP relay / upgrade relay"] -``` - -The latest main branch no longer has the old raw-copy-after-first-request shape -for ordinary forward HTTP. It rewrites ordinary requests with `Connection: -close`, uses guarded HTTP relay helpers for body handling, and sends allowed -WebSocket upgrades through the same upgrade relay. That is a narrower surface -than the historical bidirectional copy, but it is still implemented separately -from the CONNECT relay path. - -## Current Network Namespace Enforcement - -```mermaid -flowchart TD - Start["Process in sandbox network namespace"] --> Dest{"Destination"} - Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"] - Dest -- "Loopback" --> Loopback["Accept loopback"] - Dest -- "Established/related" --> Established["Accept response packet"] - Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"] - Reject --> Monitor["Bypass monitor reads dmesg"] - Monitor --> OCSF["OCSF network + detection events"] -``` - -The sandbox now installs an `inet` nftables filter table for bypass -enforcement. The table accepts proxy-bound traffic, loopback, and established -flows, then rejects and optionally logs other TCP/UDP traffic. It does not -currently redirect native TCP connections into the proxy. - -## Current Local Service Shape - -```mermaid -flowchart TD - Request["Request to local name"] --> Match{"Known local route?"} - Match -- "inference.local" --> Inference["Inference routing logic"] - Match -- "policy.local" --> Policy["Policy local logic"] - Match -- No --> External["Normal egress path"] - Inference --> LocalResponse["Local response"] - Policy --> LocalResponse -``` - -Local routes are userland-facing proxy surfaces. They should stay distinct from -external egress while still fitting the adapter model. - -## Findings To Preserve - -### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes - -The historical forward path evaluated at most the first absolute-form request, -rewrote it, then switched to bidirectional copy. Bytes already buffered after -the first header block, or later pipelined requests on the same client/upstream -connection, could reach upstream without the CONNECT L7 relay's per-request -parser/evaluator. - -Latest main mitigates this by forcing ordinary forward HTTP to one request per -connection and by using guarded relay helpers. The adapter model should -preserve the invariant either by keeping forward HTTP single-request/close or -by passing the first parsed request into a shared HTTP relay loop. - -### Endpoint config is not tied to deterministic matched policy - -The policy name used for L4 authorization and logging can be selected through a -different precedence rule than endpoint metadata. With overlapping host, port, -and binary rules, allowed IPs, TLS behavior, enforcement, and -`allow_encoded_slash` can come from a different endpoint than the policy name -logged and used for L4 allow. - -The adapter model requires authorization to return one decision with one -deterministic matched endpoint. - -### Endpoint metadata query failures fail open to L4 behavior - -If endpoint metadata lookup fails, callers can interpret the result as no L7 -configuration and downgrade to credential-only or raw L4 relay. - -The adapter model treats endpoint metadata as part of the authorization result. -Failure to materialize required metadata should deny rather than erase extended -configuration. - -### Control-plane port block only applies on one resolution path - -Blocked control-plane ports are enforced inside one allowed-IPs validation -path, while the normal host-based path uses a different validation route. - -The adapter model moves resolution, allowed IP checks, SSRF checks, and -control-plane port blocks into shared destination validation. - -## Existing Feature Inventory - -The refactor should preserve: - -- CONNECT explicit proxy support. -- Forward HTTP explicit proxy support. -- nftables bypass reject/log enforcement. -- Provider credential injection and redaction. -- REST request-body credential rewrite. -- WebSocket text-frame credential rewrite. -- REST endpoint method/path policy. -- GraphQL L7 policy. -- WebSocket transport and GraphQL-over-WebSocket policy. -- Inference routing through `inference.local`. -- Agent-facing policy routes through `policy.local`. -- Timeout and resource tracking for client, upstream, and local service work. -- Structured OCSF logging for network and HTTP policy outcomes. -- SSRF and internal address protections. -- Control-plane port protection. -- `allowed_ips` endpoint restrictions. -- TLS termination for inspectable client connections. diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md new file mode 100644 index 0000000000..cbbb973747 --- /dev/null +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -0,0 +1,404 @@ +--- +authors: + - "@johntmyers" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/1107 + - https://github.com/NVIDIA/OpenShell/pull/1083 + - https://github.com/NVIDIA/OpenShell/pull/1151 + - https://github.com/NVIDIA/OpenShell/pull/1286 + - https://github.com/NVIDIA/OpenShell/pull/1511 +--- + +# RFC 0005 - Sandbox Proxy Egress Adapter Model + + + +## Summary + +Refactor sandbox egress around one shared authorization and relay pipeline. +CONNECT, forward HTTP, native TCP capture, policy DNS, `inference.local`, +`policy.local`, and metadata loopback should become narrow adapters that +translate userland entry points into common runtime intents. Policy evaluation, +destination validation, credential injection, request-body rewrite, WebSocket +handling, protocol parsing, and upstream dialing should happen behind shared +boundaries. + +The codebase has already moved in this direction by splitting networking into +`openshell-supervisor-network` and process/netns work into +`openshell-supervisor-process`. This RFC proposes the next internal boundary: +make proxy entry mechanisms pluggable without duplicating authorization, +destination validation, or relay behavior. + +Supporting detail lives in: + +- [Current shape appendix](current-shape.md) +- [Technical design appendix](technical-design.md) +- [Implementation plan](implementation-plan.md) + +## Motivation + +The sandbox proxy supports several connection surfaces: explicit CONNECT, +forward HTTP, local inference and policy APIs, metadata loopback, TLS +termination, REST and GraphQL inspection, WebSocket inspection, credential +injection, and nftables-backed bypass detection. These features are valuable, +but changes to policy and enforcement still tend to touch multiple entry paths. + +The risk is asymmetric enforcement. A security fix can be added to CONNECT and +missed in forward HTTP; endpoint metadata can be selected differently from the +logged policy; a credential path can gain request-body or WebSocket support +without the same behavior existing in another relay mode. + +The target shape separates three concerns: + +- **Adapters** describe how userland reached the networking component. +- **Authorization** decides whether the egress is allowed and what endpoint + behavior applies. +- **Relays** own bytes, credentials, protocol parsing, and upstream dialing. + +This also prepares the proxy for future deployment modes. Today the proxy runs +inside the sandbox supervisor process. The networking leaf can already run in a +network-only mode, and a future standalone binary or sidecar should be possible +if it implements the same userland surfaces, gateway APIs, and policy +enforcement contracts. + +## Non-goals + +- Replace CONNECT with forward proxy as the only explicit proxy mode. +- Add SOCKS support. +- Add HTTP/2 L7 parsing in this refactor. Inspected HTTP paths should continue + to reject unsupported h2c upgrades instead of silently upgrading to raw + traffic. +- Redesign provider credential storage. +- Reintroduce iptables as the sandbox packet filtering backend. +- Use eBPF connect hooks for transparent capture. Native TCP capture needs a + userland proxy in the byte stream for TLS termination and protocol parsing. + +## Proposal + +### Migration Big Rocks + +1. **Transport and local-service adapters.** CONNECT, forward HTTP, + transparent TCP, policy DNS, `inference.local`, `policy.local`, and metadata + loopback become small adapters. They parse their surface and produce either + an egress intent, a local response, or a DNS answer. They do not duplicate + policy evaluation. +2. **Egress intent and decision.** Shared authorization evaluates L4 policy and + endpoint selection once per connection intent and returns one decision + containing the matched policy, matched endpoint, process identity, allowed + IP metadata, TLS behavior, protocol enforcement, and credential injection + plan. +3. **Relays.** Relays receive an authorized destination connector, not an + already-open upstream socket. HTTP relays evaluate every request before + upstream write. TCP relays copy bytes for L4-only endpoints or hand the + stream to a protocol parser when endpoint policy requires native protocol + enforcement. + +### Unified Adapter Flow + +```mermaid +flowchart TD + User["Userland payload / harness"] + + subgraph ExplicitProxy["Explicit proxy listener"] + ProxyBytes["HTTP proxy bytes"] + IsConnect{"CONNECT request?"} + Connect["CONNECT adapter"] + Forward["Forward HTTP adapter"] + ProxyBytes --> IsConnect + IsConnect -- Yes --> Connect + IsConnect -- No --> Forward + end + + subgraph NativeTcp["Policy DNS + native TCP"] + NameLookup["Userland DNS lookup"] + PolicyDns["Policy DNS adapter"] + DnsAnswer["DNS answer + active mapping"] + NativeConnect["Userland connect(ip:port)"] + TcpAdapter["Transparent TCP adapter"] + NameLookup --> PolicyDns + PolicyDns --> DnsAnswer + DnsAnswer --> NativeConnect + NativeConnect --> TcpAdapter + end + + subgraph LocalApis["Sandbox-local services"] + InferenceReq["Request to inference.local"] + PolicyReq["Request to policy.local"] + MetadataReq["Request to metadata loopback"] + InferenceAdapter["Inference local adapter"] + PolicyAdapter["Policy local adapter"] + MetadataAdapter["Metadata loopback adapter"] + InferenceReq --> InferenceAdapter + PolicyReq --> PolicyAdapter + MetadataReq --> MetadataAdapter + end + + subgraph Shared["Shared external egress pipeline"] + Intent["EgressIntent"] + Auth["Authorize and select endpoint"] + Decision["EgressDecision"] + Validate["Resolve and validate destination"] + Relay["Relay"] + Deny["Adapter-specific deny response"] + Intent --> Auth + Auth --> Allowed{"Allowed?"} + Allowed -- No --> Deny + Allowed -- Yes --> Decision + Decision --> Validate + Validate --> Relay + end + + User --> ProxyBytes + User --> NameLookup + User --> NativeConnect + User --> InferenceReq + User --> PolicyReq + User --> MetadataReq + + Connect --> Intent + Forward --> Intent + TcpAdapter --> Intent + InferenceAdapter --> InferenceResp["Local inference response"] + PolicyAdapter --> PolicyResp["Local policy response"] + MetadataAdapter --> MetadataResp["Local metadata credential response"] +``` + +Each adapter still owns its response shape. If authorization denies a CONNECT +intent, the CONNECT adapter returns a tunnel denial. If forward HTTP is denied, +the forward adapter returns an HTTP denial. If policy DNS refuses a name, it +returns the appropriate DNS response. The shared layer decides the outcome; the +adapter renders it for its protocol. + +### Relay Flow + +```mermaid +flowchart TD + Start["Authorized egress + destination connector"] + Start --> FirstReq{"First HTTP request already parsed?"} + + FirstReq -- Yes --> ForwardMode{"decision.endpoint.enforcement"} + ForwardMode -- "None" --> HttpCred["HTTP relay
credential injection only"] + ForwardMode -- "Http" --> HttpL7["HTTP relay
REST/GraphQL/WebSocket policy"] + ForwardMode -- "TcpApplication" --> BadForward["Deny: HTTP request for TCP app endpoint"] + + FirstReq -- No --> TlsPolicy{"TLS handling skipped?"} + TlsPolicy -- Yes --> Readable["Readable client stream"] + TlsPolicy -- No --> Peek["Peek client bytes"] + Peek --> Tls{"TLS ClientHello?"} + Tls -- Yes --> Terminate["Shared TLS terminator"] + Tls -- No --> Readable + Terminate --> Readable + + Readable --> Enforce{"decision.endpoint.enforcement"} + Enforce -- "None" --> Sniff{"Looks like HTTP?"} + Sniff -- Yes --> HttpCred + Sniff -- No --> TcpRelay["TcpRelay
byte copy"] + + Enforce -- "Http" --> MustHttp{"Looks like HTTP?"} + MustHttp -- Yes --> HttpL7 + MustHttp -- No --> DenyHttp["Deny: expected HTTP"] + + Enforce -- "TcpApplication" --> TcpParser["TcpRelay
protocol parser owns loop"] + + HttpCred --> ReqLoop["HTTP request loop"] + HttpL7 --> ReqLoop + ReqLoop --> ReqPolicy{"Request allowed?"} + ReqPolicy -- No --> ReqDeny["Local HTTP deny
no upstream write"] + ReqPolicy -- Yes --> StaticCreds["Resolve static placeholders"] + StaticCreds --> TokenGrant["Apply endpoint token grant if configured"] + TokenGrant --> Rewrite["Rewrite configured credential slots"] + Rewrite --> HttpDial["Connect or reuse upstream"] + HttpDial --> HttpResponse["Write request and relay response"] + HttpResponse --> Upgrade{"101 WebSocket upgrade?"} + Upgrade -- No --> ReqLoop + Upgrade -- Yes --> WsInspect{"WebSocket inspection or rewrite configured?"} + WsInspect -- No --> RawUpgrade["Raw upgraded stream"] + WsInspect -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] + + TcpParser --> ParserDial["Parser calls connector when protocol allows"] + TcpRelay --> TcpDial["Connect upstream"] + TcpDial --> ByteCopy["Copy bytes"] +``` + +Relay rules: + +- HTTP credential injection happens in both HTTP modes: L4-only HTTP and + HTTP-inspected. +- Credential injection includes static placeholder rewrite and endpoint-bound + dynamic token grants. Token grants run after policy allow and before upstream + write; failures deny without forwarding the request. +- Static credential rewrite covers request target, query, headers, opt-in REST + request bodies, and opt-in client-to-server WebSocket text frames. +- HTTP L7 policy is evaluated before upstream write for each request. +- WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101` + upgrade, the WebSocket relay owns frame parsing when text-frame credential + rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe + compression handling is configured. Other upgraded streams remain raw. +- Forward HTTP must stay in the shared HTTP relay loop or in an equivalent + guarded single-request relay. It must not evaluate one request and then + switch to raw bidirectional copy. +- `protocol: tcp` or an omitted protocol means L4 authorization plus byte copy, + except that HTTP-looking streams may still use HTTP credential injection. +- Future native protocol parsers, such as Redis, Postgres, or MySQL, own the + full message loop and can parse multiple commands or queries on one TCP + session. + +### Adapter Responsibilities + +CONNECT remains the generic explicit proxy mode for HTTPS and arbitrary TCP. +The CONNECT adapter parses `CONNECT host:port` into an `EgressIntent`, asks the +shared authorization boundary for an `EgressDecision`, returns the tunnel-ready +response only after the connection is allowed, and then hands the tunnel to the +relay. The upstream connection is opened by the HTTP relay or TCP parser when +payload policy allows it. + +Forward HTTP is compatibility for clients that send absolute-form HTTP +requests. The adapter parses the first request, rewrites proxy framing only at +the relay boundary, rejects `https://` absolute-form requests, rejects +unsupported h2c upgrades on inspected routes, and either stays in a shared HTTP +request loop or forces `Connection: close` for a guarded single request. + +Transparent TCP is for native clients that do not know they are using a proxy. +It depends on policy DNS and nftables capture: DNS answers create active +endpoint mappings, userland later calls `connect(ip:port)`, nftables redirects +matching traffic to a userland listener, and the TCP adapter recovers the +original destination before building an intent. + +Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is +query-driven: check whether the name is policy-eligible, resolve through +trusted DNS, filter returned IPs, publish the active endpoint mapping, and +answer userland. The later `connect(ip:port)` still runs through normal +authorization. + +Local service adapters stay outside the normal external egress relay: +`inference.local` routes chat, completion, model discovery, embeddings, and +provider-specific inference traffic through the router with local limits; +`policy.local` exposes current policy, denial summaries, proposal submission, +and proposal wait routes; metadata loopback serves provider metadata +credentials to SDKs that bypass HTTP proxy variables. + +### Network Enforcement Substrate + +Current main uses nftables for sandbox bypass enforcement. It accepts +proxy-bound traffic, loopback, and established flows, then rejects and +optionally logs other TCP/UDP traffic for the bypass monitor. That is +enforcement, not native TCP capture. + +```mermaid +flowchart TD + Packet["Userland packet"] --> ProxyDest{"Proxy destination?"} + ProxyDest -- Yes --> AcceptProxy["nftables accept"] + ProxyDest -- No --> Capture{"Future native TCP capture match?"} + Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"] + Capture -- No --> Reject["nftables log + reject bypass"] + Reject --> Monitor["Bypass monitor emits OCSF"] +``` + +Transparent TCP work should extend this nftables model with explicit capture +rules that run before the reject path and are scoped to active policy DNS +mappings. It should not add a parallel iptables path. + +### Deployment Modes + +| Mode | Shape | Status | +|------|-------|--------| +| Embedded supervisor | `openshell-sandbox` orchestrates `openshell-supervisor-network` and `openshell-supervisor-process` | Current | +| Network-only supervisor | Networking, policy, proxy, local services, and background tasks run without a payload process leaf | Current runtime mode | +| Standalone proxy binary | Supervisor launches networking as a separate process with explicit APIs | Future packaging/API work | +| Sidecar proxy | Proxy runs outside the payload container but inside the sandbox boundary | Future isolation mode | + +A pluggable proxy must expose the right userland surfaces, implement the +gateway APIs it needs, and prove equivalent policy enforcement through tests. +The nftables rules that force or reject userland traffic belong to the sandbox +network boundary even if the proxy process later moves into a standalone binary +or sidecar. + +## Implementation plan + +The detailed migration plan lives in [implementation-plan.md](implementation-plan.md). +The intended order is: + +1. Add regression coverage around the current split, forward HTTP invariants, + endpoint selection, token grants, WebSocket/body rewrite, metadata loopback, + and nftables bypass enforcement. +2. Introduce `EgressIntent` and `EgressDecision` inside + `openshell-supervisor-network`. +3. Move destination validation and endpoint metadata materialization behind the + shared decision and connector boundary. +4. Consolidate forward HTTP, CONNECT HTTP inspection, credential injection, + request-body rewrite, and WebSocket handling behind shared HTTP/WebSocket + relay code. +5. Move TLS detection and termination ahead of the HTTP/TCP relay split. +6. Add the TCP relay/parser boundary, then policy DNS and native TCP capture. +7. Treat local services and deployment modes as explicit runtime contracts. + +## Risks + +- Tightening endpoint metadata failures from fail-open to deny may expose + latent policy or Rego errors. +- Deterministic endpoint selection may reject policies that currently load but + only work by accident. +- Token grants add a runtime dependency on SPIFFE Workload API and token + endpoints. Failures should remain fail-closed and sanitized. +- Transparent TCP capture adds network namespace interception complexity and + must coexist with the nftables bypass reject/log table. +- Sidecar mode needs a reliable identity source for binary/path scoped policy. +- Metadata loopback and `policy.local` expand sandbox-local control surfaces + and need strict route validation, body limits, redaction, and authentication + boundaries. +- Provider-composed policy rules use a reserved namespace. Decisions and logs + must distinguish provider-derived policy from user-authored policy without + exposing provider rules as editable sandbox proposals. + +## Alternatives + +### Keep patching each entry path + +This has the lowest short-term cost but keeps security behavior duplicated +across CONNECT, forward HTTP, and local services. It also makes future TCP +application protocol support harder because each parser must be wired through +multiple entry mechanisms. + +### Replace CONNECT with forward proxy + +Forward proxy only covers plaintext absolute-form HTTP requests. It is not a +replacement for HTTPS tunnels, WebSocket tunnels, or arbitrary TCP clients. +CONNECT should remain the generic explicit proxy mode. + +### Build only transparent TCP + +Transparent TCP helps native clients but does not replace explicit proxy +support used by common HTTP tooling. It also requires policy DNS and nftables +capture before it can safely preserve endpoint identity. + +## Prior art + +The current `openshell-supervisor-network` split is the immediate prior step: +it already separates proxy, OPA, L7, inference routing, policy-local routes, +TLS, and token grants from process supervision. + +The current `openshell-supervisor-process` netns and bypass monitor are the +packet-enforcement substrate. Transparent TCP should extend that nftables +model rather than creating a second firewall path. + +The existing L7 relay is the behavioral prior art for this RFC. It already +proves per-request HTTP evaluation, GraphQL parsing, WebSocket frame handling, +request-body rewrite, and token-grant injection can live behind relay +boundaries. + +## Open questions + +1. Should overlapping endpoint metadata be rejected at policy load time, or + should policy name plus endpoint index define precedence? +2. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted, + or should DNS query correlation be required for stricter modes? +3. What TTL cap and stale-generation grace period should policy DNS use? +4. Which process identity source should sidecar mode use when it cannot inspect + payload process metadata through local `/proc`? +5. Which proxy capabilities should be negotiated with the gateway at startup? +6. Should metadata loopback be modeled as an adapter inside + `openshell-supervisor-network`, or remain orchestrated by `openshell-sandbox` + with shared credential/provider helpers? diff --git a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md new file mode 100644 index 0000000000..d4090e4f67 --- /dev/null +++ b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md @@ -0,0 +1,223 @@ +# Current Shape Appendix + +This appendix records the current proxy shape and the review findings that +motivate the adapter model. The main RFC intentionally keeps these details out +of the direction document. + +## Current Runtime Split + +The proxy is no longer only a large module inside `openshell-sandbox`. +Current main has three relevant runtime owners: + +```mermaid +flowchart TD + Sandbox["openshell-sandbox
orchestrator"] + Network["openshell-supervisor-network
proxy, OPA, L7, TLS, inference,
policy.local, token grants"] + Process["openshell-supervisor-process
process leaf, SSH, netns,
nftables, bypass monitor"] + Denials["Denial/activity aggregators"] + Gateway["Gateway policy/provider APIs"] + + Sandbox --> Network + Sandbox --> Process + Network --> Denials + Process --> Denials + Sandbox --> Gateway + Network --> Gateway +``` + +`openshell-sandbox` creates the shared network namespace, owns denial/activity +channels, starts the policy poll loop, starts networking, starts the metadata +loopback server when needed, and then optionally starts the process leaf. If +`process_enabled` is false, the supervisor can run in network-only mode and +keep networking/background tasks alive until shutdown. + +`openshell-supervisor-network` owns the explicit proxy listener, OPA engine +integration, L7 enforcement, TLS termination, inference routing, policy-local +routes, identity cache, provider credential injection, and token grants. + +`openshell-supervisor-process` owns process execution, SSH, network namespace +helpers, nftables bypass rules, and the bypass monitor that turns nftables LOG +entries into OCSF events. + +## Current Userland-Facing Surfaces + +The networking surface currently includes: + +- CONNECT proxy traffic for HTTPS and generic TCP tunnels. +- Forward HTTP proxy traffic for absolute-form HTTP requests. +- `inference.local` for local inference routing. +- `policy.local` for current policy, denial summaries, proposal submission, + and proposal wait routes. +- GCE metadata loopback for SDKs that bypass HTTP proxy variables. +- nftables bypass enforcement for direct TCP/UDP egress that does not enter + the proxy. +- OPA/Rego policy and endpoint metadata lookups. +- DNS resolution and endpoint validation for CONNECT and forward HTTP egress. +- Static provider credential injection and redaction. +- Endpoint-bound dynamic token grant injection. +- Opt-in REST request-body credential rewrite. +- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement. + +The issue is not that these features exist. The issue is that entry mechanisms, +policy evaluation, endpoint metadata lookup, credential injection, and byte +relay decisions are still interleaved. + +## Current CONNECT Shape + +```mermaid +flowchart TD + Client["Client CONNECT host:port"] --> Parse["Parse CONNECT target"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["CONNECT denial"] + Allowed -- Yes --> Meta["Query endpoint metadata"] + Meta --> Config{"L7, TLS, or credential config?"} + Config -- No --> Tunnel["Return tunnel-ready response"] + Config -- Yes --> Tunnel + Tunnel --> Inspect["Inspect tunneled bytes when possible"] + Inspect --> Relay["HTTP/WebSocket/TCP relay selection"] + Relay --> Inject["Static credentials and token grants if configured"] + Inject --> Upstream["Open upstream when relay policy allows"] +``` + +CONNECT is still the strongest entry shape because the tunnel relay can keep +parsing HTTP requests on long-lived connections and enforce request policy per +request. + +## Current Forward HTTP Shape + +```mermaid +flowchart TD + Client["Absolute-form HTTP request"] --> Parse["Parse first request"] + Parse --> L4["Evaluate network policy"] + L4 --> Allowed{"Allowed?"} + Allowed -- No --> Deny["HTTP denial"] + Allowed -- Yes --> L7{"Matching L7 endpoint?"} + L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"] + Eval --> Guard["Reject unsupported h2c upgrade when inspected"] + Guard --> Rewrite["Rewrite to origin-form + configured credentials"] + L7 -- No --> Rewrite + Rewrite --> Token["Apply token grant if endpoint-bound"] + Token --> Close["Force Connection: close except WebSocket upgrade"] + Close --> Upstream["Open upstream"] + Upstream --> Relay["Guarded HTTP relay / upgrade relay"] +``` + +Latest main no longer has the old raw-copy-after-first-request shape for +ordinary forward HTTP. It rewrites ordinary requests with `Connection: close`, +uses guarded HTTP relay helpers for body handling, rejects inspected h2c +upgrades, injects token grants, and sends allowed WebSocket upgrades through +the upgrade relay. That is a narrower surface than the historical bidirectional +copy, but it is still orchestrated separately from the CONNECT relay path. + +## Current Local Service Shape + +```mermaid +flowchart TD + Request["Request to local name"] --> Match{"Known local route?"} + Match -- "inference.local" --> Inference["Inference route adapter"] + Match -- "policy.local" --> Policy["Policy local adapter"] + Match -- "metadata loopback" --> Metadata["Metadata credential server"] + Match -- No --> External["Normal egress path"] + Inference --> InferenceResp["Local inference response"] + Policy --> PolicyResp["Local policy response"] + Metadata --> MetadataResp["Metadata response"] +``` + +`inference.local` now covers buffered and streaming inference shapes including +chat/completion routes, model discovery, embeddings, and provider-specific +routes. `policy.local` supports the agentic approval loop: agents can submit +narrow proposals and wait on approval/reload before retrying. Metadata +loopback exists for provider credentials consumed by SDKs that do not honor +HTTP proxy variables. + +These are userland-facing network surfaces. They should stay distinct from +external egress while still fitting the adapter model. + +## Current Network Namespace Enforcement + +```mermaid +flowchart TD + Start["Process in sandbox network namespace"] --> Dest{"Destination"} + Dest -- "Proxy host_ip:port" --> Proxy["Accept to sandbox proxy"] + Dest -- "Loopback" --> Loopback["Accept loopback"] + Dest -- "Established/related" --> Established["Accept response packet"] + Dest -- "Other TCP/UDP" --> Reject["nftables log + reject"] + Reject --> Monitor["Bypass monitor reads dmesg"] + Monitor --> OCSF["OCSF network + detection events"] +``` + +The process leaf installs an `inet` nftables filter table for bypass +enforcement. The table accepts proxy-bound traffic, loopback, and established +flows, then rejects and optionally logs other TCP/UDP traffic. It does not +currently redirect native TCP connections into the proxy. + +## Findings To Preserve + +### Invariant: forward proxy must not relay unevaluated follow-on HTTP bytes + +The historical forward path evaluated at most the first absolute-form request, +rewrote it, then switched to bidirectional copy. Bytes already buffered after +the first header block, or later pipelined requests on the same client/upstream +connection, could reach upstream without the CONNECT L7 relay's per-request +parser/evaluator. + +Latest main mitigates this by forcing ordinary forward HTTP to one request per +connection and by using guarded relay helpers. The adapter model should +preserve the invariant either by keeping forward HTTP single-request/close or +by passing the first parsed request into a shared HTTP relay loop. + +### Endpoint config is not tied to deterministic matched policy + +The policy name used for L4 authorization and logging can be selected through a +different precedence rule than endpoint metadata. With overlapping host, port, +and binary rules, allowed IPs, TLS behavior, enforcement, and +`allow_encoded_slash` can come from a different endpoint than the policy name +logged and used for L4 allow. + +The adapter model requires authorization to return one decision with one +deterministic matched endpoint. + +### Endpoint metadata query failures should not erase enforcement + +If endpoint metadata lookup fails, callers can interpret the result as no L7 +configuration and downgrade to credential-only or raw L4 relay. + +The adapter model treats endpoint metadata as part of the authorization result. +Failure to materialize required metadata should deny rather than erase extended +configuration. + +### Destination validation must be shared + +Private address checks, `allowed_ips`, exact declared private endpoint trust, +trusted gateway aliases, SSRF checks, and control-plane port blocks have grown +over time. They should be centralized so CONNECT, forward HTTP, future +transparent TCP, and local-service egress use the same resolved-destination +rules. + +## Existing Feature Inventory + +The refactor should preserve: + +- CONNECT explicit proxy support. +- Forward HTTP explicit proxy support. +- Network-only supervisor mode. +- nftables bypass reject/log enforcement. +- Provider credential injection and redaction. +- Dynamic token grant injection through SPIFFE-backed provider credentials. +- REST request-body credential rewrite. +- WebSocket text-frame credential rewrite. +- REST endpoint method/path policy. +- GraphQL-over-HTTP policy. +- WebSocket transport and GraphQL-over-WebSocket policy. +- h2c rejection on inspected HTTP routes. +- Inference routing through `inference.local`, including embeddings. +- Agent-facing policy advisor routes through `policy.local`. +- GCE metadata loopback for supported provider credentials. +- Timeout and resource tracking for client, upstream, and local service work. +- Structured OCSF logging for network and HTTP policy outcomes. +- SSRF and internal address protections. +- Exact declared private endpoint handling. +- Control-plane port protection. +- `allowed_ips` endpoint restrictions. +- TLS auto-detection and termination for inspectable client connections. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md similarity index 58% rename from rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md rename to rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md index 94ba53b7f0..3b00a512b0 100644 --- a/rfc/0004-sandbox-proxy-egress-adapter/implementation-plan.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md @@ -7,28 +7,46 @@ direction-focused. - Add tests for forward HTTP pipelining and keep-alive follow-on requests, including the current `Connection: close` mitigation. +- Add tests for forward HTTP h2c rejection on inspected endpoints. - Add tests for overlapping endpoint metadata selection. - Add tests for endpoint metadata query failures. - Add tests for control-plane port blocking through all destination validation paths. +- Add tests for exact declared private endpoint trust and `allowed_ips` + behavior across CONNECT and forward HTTP. +- Add tests proving static credential injection works in L4-only HTTP and + HTTP-inspected paths. +- Add tests proving token grant success injects the configured header and token + grant failure does not forward upstream. +- Add tests for REST request-body credential rewrite, WebSocket text-frame + credential rewrite, WebSocket GraphQL policy, and compression handling. +- Add tests for `policy.local` proposal wait behavior and `inference.local` + buffered/streaming route limits. +- Add tests for metadata loopback startup/failure behavior when provider + credentials require it. - Add nftables bypass enforcement tests that verify proxy-bound traffic is accepted while direct TCP/UDP egress is rejected and logged when available. ## Phase 1 - Authorization Result -- Introduce `EgressIntent` and `EgressDecision`. +- Introduce `EgressIntent` and `EgressDecision` inside + `openshell-supervisor-network`. - Make authorization return matched policy and matched endpoint metadata together. +- Include policy source on the decision: user-authored, provider-derived, or + local-service internal. +- Include protocol enforcement and credential injection plan on the decision. - Fail closed when required endpoint metadata cannot be materialized. - Emit consistent OCSF network denial events from the shared boundary. ## Phase 2 - Shared Destination Validation -- Move DNS resolution, allowed IP filtering, SSRF checks, and control-plane port - checks into one destination validation path. +- Move DNS resolution, allowed IP filtering, SSRF checks, exact declared + endpoint handling, trusted gateway aliases, and control-plane port checks + into one destination validation path. - Return an `UpstreamConnector` rather than an opened upstream socket. -- Add tests proving CONNECT, forward HTTP, and transparent TCP use the same - validation behavior. +- Add tests proving CONNECT, forward HTTP, and future transparent TCP use the + same validation behavior. ## Phase 3 - Forward HTTP Adapter @@ -36,15 +54,20 @@ direction-focused. request and builds an egress intent. - Route the parsed first request into the shared HTTP relay or preserve the current guarded single-request relay behavior. +- Preserve `https://` absolute-form rejection. +- Preserve h2c rejection on inspected routes. - Keep the no-raw-copy invariant after the first request. -## Phase 4 - HTTP And WebSocket Relay Consolidation +## Phase 4 - HTTP, WebSocket, And Credential Relay Consolidation - Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket upgrade policy, credential resolution, redaction, request rewrite, upstream dial, and response relay. - Evaluate every HTTP request before upstream write. - Ensure denied HTTP requests do not create upstream TCP sessions. +- Preserve static placeholder rewrite for target, query, and headers. +- Preserve dynamic token grant injection after request allow and before + upstream write. - Preserve opt-in REST request-body credential rewrite behind the shared HTTP relay, including bounded buffering, supported content-type handling, `Content-Length` recomputation, and fail-closed unresolved placeholders. @@ -58,13 +81,14 @@ direction-focused. - Move client-side TLS detection and termination before the HTTP/TCP relay split. - Keep endpoint TLS behavior on `EgressDecision`. +- Treat `tls: skip` as the explicit opt-out for TLS handling. - Remove duplicate HTTP-specific and TCP-specific TLS termination decisions. ## Phase 6 - TCP Relay And Parser Boundary -- Rename raw TCP relay concepts to `TcpRelay`. +- Use `TcpRelay` for byte relay and TCP application parser dispatch. +- Keep `protocol: tcp` or omitted protocol as L4 authorization plus byte copy. - Add a TCP application parser dispatch point for future protocol enforcement. -- Keep `protocol: tcp` as L4 authorization plus byte copy. - Let TCP application parsers own their message loop and call the connector when protocol state allows. @@ -75,6 +99,7 @@ direction-focused. - Publish active DNS answer state and capture rules. - Implement nftables REDIRECT/TPROXY capture rules ahead of the bypass reject path; do not add a parallel iptables path. +- Coordinate capture rule ownership with `openshell-supervisor-process::netns`. - Implement transparent TCP adapter lookup from captured original destination to active endpoint generation. - Decide TTL and stale-generation behavior. @@ -82,35 +107,51 @@ direction-focused. ## Phase 8 - Local Service Adapters - Model `inference.local` as a local adapter with TLS termination, route - validation, provider auth injection, streaming limits, and OCSF logging. + validation, provider auth injection, streaming/buffered limits, and OCSF + logging. - Model `policy.local` as a local adapter for current policy, bounded denial - summaries, and policy proposals. -- Keep both paths outside normal external egress relay. + summaries, policy proposals, and proposal wait. +- Decide whether metadata loopback remains orchestrated in `openshell-sandbox` + or moves behind a local adapter boundary in `openshell-supervisor-network`. +- Keep these paths outside normal external egress relay while preserving + credential redaction and route validation. ## Phase 9 - Runtime Boundary -- Keep embedded mode for the first migration. +- Keep embedded supervisor mode as the first migration target. +- Treat the existing `openshell-supervisor-network` and + `openshell-supervisor-process` split as the structural baseline. - Define the proxy runtime API needed for a future standalone binary: - configured listeners, policy updates, gateway calls, telemetry, and shutdown. + configured listeners, policy updates, provider credentials, token grants, + gateway calls, telemetry, denial/activity events, and shutdown. - Identify process identity requirements for standalone and sidecar modes. +- Add capability negotiation with the gateway if standalone proxy versions can + differ from gateway versions. ## Phase 10 - Cleanup - Remove duplicated endpoint metadata queries from relay paths. -- Remove duplicated deny rendering where adapters can own response shape. +- Remove duplicated destination validation and deny rendering where adapters + can own response shape. - Remove any remaining forward HTTP raw-copy fallback. +- Remove stale references to iptables or static `/etc/hosts` native TCP + mapping from proxy design docs. - Update architecture docs once implementation lands. ## Testing Plan - Unit-test each adapter's intent construction and deny response shape. - Unit-test authorization precedence for overlapping policy and endpoint rules. +- Unit-test provider-derived rule namespace handling and `policy.local` + filtering. - Integration-test shared destination validation across CONNECT, forward HTTP, and transparent TCP. - Integration-test HTTP keep-alive and pipelined requests with REST, GraphQL, and WebSocket upgrade enforcement. - Integration-test credential injection in L4-only HTTP and HTTP-inspected paths. +- Integration-test token grant success, cache hit, malformed token, resolver + unavailable, and token endpoint failure. - Integration-test REST request-body credential rewrite for JSON, form-url-encoded, `text/*`, unsupported content types, chunked framing, body caps, and unresolved placeholders. @@ -123,5 +164,5 @@ direction-focused. application parsers. - Integration-test policy DNS TTL, stale generation handling, and captured connect correlation. -- Integration-test `inference.local` and `policy.local` body limits, timeout - behavior, redaction, and local denial responses. +- Integration-test `inference.local`, `policy.local`, and metadata loopback + body limits, timeout behavior, redaction, and local denial responses. diff --git a/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md similarity index 57% rename from rfc/0004-sandbox-proxy-egress-adapter/technical-design.md rename to rfc/0005-sandbox-proxy-egress-adapter/technical-design.md index b13e259f45..837a638d9e 100644 --- a/rfc/0004-sandbox-proxy-egress-adapter/technical-design.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md @@ -1,7 +1,19 @@ # Technical Design Appendix -This appendix carries the implementation-level design details behind the main -RFC. +This appendix carries implementation-level design details behind the main RFC. + +## Existing Runtime Boundary + +`openshell-supervisor-network::run::run_networking` is the current networking +startup boundary. It builds policy-local context, waits for policy binary +symlink resolution, creates the identity cache, writes the TLS CA, builds TLS +state, resolves inference routes, wires provider credentials and token grants, +and starts the proxy. + +This is a useful outer boundary, but it is not yet the proxy adapter boundary. +The proxy still needs internal `EgressIntent` and `EgressDecision` boundaries +so CONNECT, forward HTTP, local routes, and future native TCP capture do not +duplicate policy and relay orchestration. ## Shared Data Boundaries @@ -11,14 +23,16 @@ RFC. It should carry: -- entry transport: CONNECT, forward HTTP, transparent TCP, or local HTTP; +- entry transport: CONNECT, forward HTTP, transparent TCP, local HTTP, policy + DNS, or metadata loopback; - requested destination host/port or captured original IP/port; - process identity inputs collected by the adapter/runtime; - optional first HTTP request for forward proxy traffic; -- optional local service route. +- optional local service route; +- policy generation or DNS mapping generation when relevant. -Adapters build intents. They should not query endpoint metadata or select -relays. +Adapters build intents. They should not query endpoint metadata, select TLS +mode, or select relays. ### EgressDecision @@ -28,15 +42,19 @@ It should carry: - allow or deny; - deterministic matched policy identifier; +- whether the policy is user-authored, provider-derived, or local-service + internal; - deterministic matched endpoint identifier and endpoint metadata; - process identity used for evaluation; - destination and allowed IP constraints; - TLS behavior; - protocol enforcement; +- credential injection plan; - logging context and denial reason. Relay code should read this decision. It should not query OPA again for -endpoint metadata, TLS mode, allowed IPs, or parser selection. +endpoint metadata, TLS mode, allowed IPs, credential behavior, or parser +selection. ## Protocol Enforcement @@ -46,15 +64,14 @@ Use a protocol enforcement value derived from endpoint policy: |-----------------|-------------|----------------| | omitted / `tcp` | None | L4 authorization plus byte relay, with optional HTTP sniff for credential injection | | `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite | -| `graphql` | HTTP | HTTP request parser with GraphQL rules | +| `graphql` | HTTP | HTTP request parser with GraphQL-over-HTTP rules | | `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy | | future `redis`, `postgres`, `mysql`, ... | TCP application | Protocol-specific TCP parser owns the message loop | `protocol: tcp` is effectively the default L4 mode. It should not run TCP -application parsers. - -Avoid using the term "provider" for these parser concepts because providers -are already a first-class credential and routing domain in OpenShell. +application parsers. Avoid using the term "provider" for parser concepts +because providers are already a first-class credential and routing domain in +OpenShell. ## Suggested Types @@ -65,7 +82,9 @@ enum EgressTransport { Connect, ForwardHttp, TransparentTcp, + PolicyDns, LocalHttp, + MetadataLoopback, } struct EgressIntent { @@ -74,6 +93,7 @@ struct EgressIntent { process: ProcessIdentity, first_request: Option, local_route: Option, + generation: Option, } struct EgressDecision { @@ -83,11 +103,23 @@ struct EgressDecision { log_context: EgressLogContext, } +struct MatchedPolicy { + id: PolicyId, + source: PolicySource, +} + +enum PolicySource { + User, + ProviderDerived, + LocalService, +} + struct MatchedEndpoint { id: EndpointId, allowed_ips: AllowedIpPolicy, tls: TlsPolicy, enforcement: ProtocolEnforcement, + credentials: CredentialInjectionPlan, } enum ProtocolEnforcement { @@ -104,10 +136,30 @@ enum HttpL7Protocol { struct HttpL7Config { protocol: HttpL7Protocol, + path: EndpointPathScope, allow_encoded_slash: bool, + enforcement_mode: L7EnforcementMode, websocket_credential_rewrite: bool, request_body_credential_rewrite: bool, websocket_graphql_policy: bool, + graphql_max_body_bytes: usize, +} + +struct CredentialInjectionPlan { + static_placeholders: StaticPlaceholderPlan, + token_grant: Option, +} + +struct StaticPlaceholderPlan { + http_target_query_header: bool, + rest_request_body: bool, + websocket_text_frames: bool, +} + +struct TokenGrantPlan { + provider_key: String, + auth_style: TokenGrantAuthStyle, + token_endpoint: String, } struct RelayContext { @@ -122,26 +174,19 @@ struct RelayContext { validated destination and lets relays/parsers open an upstream connection only after protocol policy allows it. -## Module Layout - -A future split could look like: - -| Module | Responsibility | -|--------|----------------| -| `proxy::adapter::connect` | Parse CONNECT and render CONNECT responses | -| `proxy::adapter::forward_http` | Parse absolute-form HTTP and preserve first request | -| `proxy::adapter::transparent_tcp` | Recover captured original destination | -| `proxy::adapter::policy_dns` | Answer eligible DNS queries and publish active mappings | -| `proxy::adapter::local` | Implement `inference.local` and `policy.local` surfaces | -| `proxy::auth` | Build decisions from intents and OPA results | -| `proxy::destination` | Resolve, filter, and validate destinations | -| `proxy::netfilter` | Own nftables bypass and future transparent capture rules | -| `proxy::relay::http` | HTTP request loop, credentials, REST/GraphQL/WebSocket upgrade policy | -| `proxy::relay::websocket` | WebSocket frame validation, text-frame rewrite, and message policy | -| `proxy::relay::tcp` | TCP byte relay and TCP application parser dispatch | -| `proxy::relay::tls` | Shared client-side TLS termination | -| `proxy::parser` | HTTP, WebSocket, and TCP application parser traits/config | -| `proxy::telemetry` | OCSF and tracing helpers | +## Current Owners And Proposed Cleanup + +| Current owner | Current responsibility | Proposed cleanup | +|---------------|------------------------|------------------| +| `openshell-sandbox` | Orchestrator, policy poll loop, denial/activity channels, metadata loopback startup, network-only lifecycle | Keep as orchestration; avoid embedding per-entry proxy policy decisions | +| `openshell-supervisor-network::run` | Networking startup and handles | Become the stable runtime API for embedded and future standalone modes | +| `openshell-supervisor-network::proxy` | CONNECT, forward HTTP, local route dispatch, destination validation, denial rendering | Split into adapters, authorization, destination, relay selection, and adapter response rendering | +| `openshell-supervisor-network::opa` | Policy engine and Rego queries | Return deterministic `EgressDecision` data instead of separate policy and endpoint lookups | +| `openshell-supervisor-network::l7` | REST, GraphQL, WebSocket, inference helpers, TLS, token grants | Keep as protocol/relay implementation behind shared relay boundaries | +| `openshell-supervisor-network::policy_local` | `policy.local` state and routes | Model as a local adapter with explicit limits and proposal/wait behavior | +| `openshell-supervisor-process::netns` | nftables bypass rules and namespace helpers | Remain owner of bypass enforcement; coordinate future capture rules with network proxy mappings | +| `openshell-supervisor-process::bypass_monitor` | nftables LOG parsing and OCSF bypass telemetry | Remain telemetry producer for bypass violations | +| `openshell-core::secrets` and provider credential state | Static placeholder sources and dynamic credential metadata | Feed credential injection plans; do not leak secrets into decision logs | ## Policy DNS And Resolved TCP State @@ -194,15 +239,23 @@ Two acceptable approaches: Endpoint metadata query failures should fail closed when metadata is required for the selected endpoint. They should not silently downgrade to L4 behavior. +Provider-derived policies use a reserved rule-name namespace. The gateway and +sandbox sync should prevent user-authored `_provider_*` rules, and +`policy.local` proposal surfaces should not expose provider-derived rules as +editable user policy. `EgressDecision` should still identify provider-derived +matches for logging and debugging. + ## Credential Injection Boundary -Credential injection belongs in the HTTP relay: +Credential injection belongs in the HTTP/WebSocket relay after policy allow and +before upstream write. -1. Authorization selects the endpoint and confirms credentials may be used. -2. The HTTP relay resolves credentials only when it has an allowed HTTP request. -3. Secrets are redacted from logs and policy-visible metadata. -4. The final upstream request or frame is rewritten with real credentials - immediately before write. +1. Authorization selects the endpoint and computes a credential injection plan. +2. The HTTP relay resolves credentials only when it has an allowed request. +3. Static placeholder values are resolved and redacted from logs. +4. Endpoint-bound token grants obtain or reuse a dynamic access token. +5. The final upstream request or WebSocket frame is rewritten immediately + before write. Both L4-only HTTP and HTTP-inspected paths can inject credentials. The difference is whether REST, GraphQL, or WebSocket policy is evaluated before @@ -215,7 +268,8 @@ Credential rewrite slots should be explicit: - client-to-server WebSocket text frames only when `websocket_credential_rewrite` is enabled; - GraphQL-over-WebSocket connection/control messages when they are carried in - text frames and the endpoint enables the WebSocket rewrite path. + text frames and the endpoint enables the WebSocket rewrite path; +- token grant headers for endpoint-bound provider credentials. Request-body rewrite is REST-only. It should buffer bounded UTF-8 textual bodies, including JSON, form-url-encoded, and `text/*`, recompute @@ -223,6 +277,12 @@ bodies, including JSON, form-url-encoded, and `text/*`, recompute credential markers, and fail closed when a reserved placeholder cannot be resolved safely. Binary WebSocket frames are not rewritten. +Token grants are dynamic credential injection. They use provider metadata to +request a SPIFFE JWT-SVID, exchange it for an OAuth2 access token, cache the +token, and inject either an `Authorization: Bearer` header or a configured +custom header. Token grant failures should return a local relay error and must +not forward the request upstream. + ## Parser Boundary Protocol parsers operate on streams owned by the relay. @@ -241,6 +301,24 @@ Protocol parsers operate on streams owned by the relay. This avoids a separate dial strategy enum. The parser knows which protocol milestone is sufficient to call the validated connector. +## Local Service Adapter Boundary + +Local services are network surfaces but not normal external egress: + +- `inference.local` terminates local client traffic, validates known inference + routes, strips caller auth, injects provider routing/auth, and applies + streaming or buffered limits based on route type. +- `policy.local` serves policy snapshots, denial summaries, proposal + submission, and proposal wait. It should never expose secrets or provider + rules as editable policy. +- Metadata loopback serves provider metadata credentials for SDKs that bypass + HTTP proxy variables. It should use the same provider credential state and + redaction discipline as other credential paths. + +These adapters may call gateway APIs or local credential helpers, but they +should not bypass policy and credential invariants that apply to external +egress. + ## Timeout And Resource Ownership | Owner | Resource | @@ -254,6 +332,7 @@ milestone is sufficient to call the validated connector. | TCP relay | Byte-copy idle timeout and half-close handling | | TCP parser | Protocol message timeouts and parser-specific limits | | Local service adapter | Local route body limits, response caps, gateway call timeout | +| Token grant resolver | SPIFFE Workload API timeout, token endpoint timeout, cache TTL | Timeouts should be recorded in telemetry at the owner boundary that can explain the failure. From 4ad15bf297362f304ed0c94cd92379215eaca5b0 Mon Sep 17 00:00:00 2001 From: John Myers Date: Mon, 6 Jul 2026 08:49:23 -0700 Subject: [PATCH 4/8] docs(rfc): account for supervisor middleware Signed-off-by: John Myers --- .../README.md | 77 +++++++---- .../current-shape.md | 28 ++++ .../implementation-plan.md | 40 ++++-- .../technical-design.md | 124 ++++++++++++++---- 4 files changed, 213 insertions(+), 56 deletions(-) diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md index cbbb973747..34197c1679 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/README.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -8,6 +8,8 @@ links: - https://github.com/NVIDIA/OpenShell/pull/1151 - https://github.com/NVIDIA/OpenShell/pull/1286 - https://github.com/NVIDIA/OpenShell/pull/1511 + - https://github.com/NVIDIA/OpenShell/pull/1738 + - https://github.com/NVIDIA/OpenShell/pull/2027 --- # RFC 0005 - Sandbox Proxy Egress Adapter Model @@ -22,9 +24,9 @@ Refactor sandbox egress around one shared authorization and relay pipeline. CONNECT, forward HTTP, native TCP capture, policy DNS, `inference.local`, `policy.local`, and metadata loopback should become narrow adapters that translate userland entry points into common runtime intents. Policy evaluation, -destination validation, credential injection, request-body rewrite, WebSocket -handling, protocol parsing, and upstream dialing should happen behind shared -boundaries. +destination validation, supervisor middleware, credential injection, +request-body rewrite, WebSocket handling, protocol processing, and upstream +dialing should happen behind shared boundaries. The codebase has already moved in this direction by splitting networking into `openshell-supervisor-network` and process/netns work into @@ -43,8 +45,9 @@ Supporting detail lives in: The sandbox proxy supports several connection surfaces: explicit CONNECT, forward HTTP, local inference and policy APIs, metadata loopback, TLS termination, REST and GraphQL inspection, WebSocket inspection, credential -injection, and nftables-backed bypass detection. These features are valuable, -but changes to policy and enforcement still tend to touch multiple entry paths. +injection, supervisor middleware, and nftables-backed bypass detection. These +features are valuable, but changes to policy and enforcement still tend to +touch multiple entry paths. The risk is asymmetric enforcement. A security fix can be added to CONNECT and missed in forward HTTP; endpoint metadata can be selected differently from the @@ -89,11 +92,11 @@ enforcement contracts. endpoint selection once per connection intent and returns one decision containing the matched policy, matched endpoint, process identity, allowed IP metadata, TLS behavior, protocol enforcement, and credential injection - plan. + and middleware plans. 3. **Relays.** Relays receive an authorized destination connector, not an already-open upstream socket. HTTP relays evaluate every request before upstream write. TCP relays copy bytes for L4-only endpoints or hand the - stream to a protocol parser when endpoint policy requires native protocol + stream to a protocol processor when endpoint policy requires native protocol enforcement. ### Unified Adapter Flow @@ -177,12 +180,12 @@ adapter renders it for its protocol. ```mermaid flowchart TD Start["Authorized egress + destination connector"] - Start --> FirstReq{"First HTTP request already parsed?"} + Start --> FirstReq{"Forward HTTP adapter supplied first request?"} FirstReq -- Yes --> ForwardMode{"decision.endpoint.enforcement"} ForwardMode -- "None" --> HttpCred["HTTP relay
credential injection only"] ForwardMode -- "Http" --> HttpL7["HTTP relay
REST/GraphQL/WebSocket policy"] - ForwardMode -- "TcpApplication" --> BadForward["Deny: HTTP request for TCP app endpoint"] + ForwardMode -- "ProtocolProcessor" --> BadForward["Deny: HTTP request for native protocol endpoint"] FirstReq -- No --> TlsPolicy{"TLS handling skipped?"} TlsPolicy -- Yes --> Readable["Readable client stream"] @@ -201,13 +204,18 @@ flowchart TD MustHttp -- Yes --> HttpL7 MustHttp -- No --> DenyHttp["Deny: expected HTTP"] - Enforce -- "TcpApplication" --> TcpParser["TcpRelay
protocol parser owns loop"] + Enforce -- "ProtocolProcessor" --> Processor["TcpRelay
protocol processor owns loop"] HttpCred --> ReqLoop["HTTP request loop"] HttpL7 --> ReqLoop ReqLoop --> ReqPolicy{"Request allowed?"} ReqPolicy -- No --> ReqDeny["Local HTTP deny
no upstream write"] - ReqPolicy -- Yes --> StaticCreds["Resolve static placeholders"] + ReqPolicy -- Yes --> Middleware{"Supervisor middleware chain configured?"} + Middleware -- Yes --> MwEval["Evaluate HTTP_REQUEST / PRE_CREDENTIALS
allow, deny, or mutate"] + Middleware -- No --> StaticCreds["Resolve static placeholders"] + MwEval --> MwAllowed{"Middleware allowed?"} + MwAllowed -- No --> MwDeny["Local middleware deny
no credential injection"] + MwAllowed -- Yes --> StaticCreds StaticCreds --> TokenGrant["Apply endpoint token grant if configured"] TokenGrant --> Rewrite["Rewrite configured credential slots"] Rewrite --> HttpDial["Connect or reuse upstream"] @@ -218,7 +226,7 @@ flowchart TD WsInspect -- No --> RawUpgrade["Raw upgraded stream"] WsInspect -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] - TcpParser --> ParserDial["Parser calls connector when protocol allows"] + Processor --> ProcessorDial["Processor calls connector when protocol allows"] TcpRelay --> TcpDial["Connect upstream"] TcpDial --> ByteCopy["Copy bytes"] ``` @@ -227,9 +235,19 @@ Relay rules: - HTTP credential injection happens in both HTTP modes: L4-only HTTP and HTTP-inspected. +- Supervisor middleware is a typed relay hook. V1 middleware runs on parsed + HTTP requests at `HTTP_REQUEST / PRE_CREDENTIALS`, after network and request + policy admit the request and before OpenShell injects credentials. +- Middleware can allow, deny, replace the bounded request body, add approved + headers, and emit audit-safe findings/metadata. External middleware must not + receive OpenShell-managed credentials. - Credential injection includes static placeholder rewrite and endpoint-bound dynamic token grants. Token grants run after policy allow and before upstream write; failures deny without forwarding the request. +- Middleware-transformed content must not create a new path for resolving + OpenShell credential placeholders unless the middleware hook is explicitly + trusted as credential-capable. The safe default is to fail closed on newly + introduced reserved placeholders before credential injection. - Static credential rewrite covers request target, query, headers, opt-in REST request bodies, and opt-in client-to-server WebSocket text frames. - HTTP L7 policy is evaluated before upstream write for each request. @@ -242,9 +260,10 @@ Relay rules: switch to raw bidirectional copy. - `protocol: tcp` or an omitted protocol means L4 authorization plus byte copy, except that HTTP-looking streams may still use HTTP credential injection. -- Future native protocol parsers, such as Redis, Postgres, or MySQL, own the +- Future native protocol processors, such as Redis, Postgres, or MySQL, own the full message loop and can parse multiple commands or queries on one TCP - session. + session. A processor may be in-tree, middleware-backed, or a combination + where in-tree framing exposes typed middleware hooks. ### Adapter Responsibilities @@ -252,8 +271,8 @@ CONNECT remains the generic explicit proxy mode for HTTPS and arbitrary TCP. The CONNECT adapter parses `CONNECT host:port` into an `EgressIntent`, asks the shared authorization boundary for an `EgressDecision`, returns the tunnel-ready response only after the connection is allowed, and then hands the tunnel to the -relay. The upstream connection is opened by the HTTP relay or TCP parser when -payload policy allows it. +relay. The upstream connection is opened by the HTTP relay or protocol +processor when payload policy allows it. Forward HTTP is compatibility for clients that send absolute-form HTTP requests. The adapter parses the first request, rewrites proxy framing only at @@ -312,6 +331,10 @@ mappings. It should not add a parallel iptables path. A pluggable proxy must expose the right userland surfaces, implement the gateway APIs it needs, and prove equivalent policy enforcement through tests. +If supervisor middleware is configured, the proxy runtime must also receive the +effective middleware service registry, validate/refresh bindings, enforce +`fail_open` and `fail_closed`, buffer within configured caps, invoke middleware +on the request path, and emit middleware OCSF events. The nftables rules that force or reject userland traffic belong to the sandbox network boundary even if the proxy process later moves into a standalone binary or sidecar. @@ -322,17 +345,18 @@ The detailed migration plan lives in [implementation-plan.md](implementation-pla The intended order is: 1. Add regression coverage around the current split, forward HTTP invariants, - endpoint selection, token grants, WebSocket/body rewrite, metadata loopback, - and nftables bypass enforcement. + endpoint selection, supervisor middleware, token grants, WebSocket/body + rewrite, metadata loopback, and nftables bypass enforcement. 2. Introduce `EgressIntent` and `EgressDecision` inside `openshell-supervisor-network`. 3. Move destination validation and endpoint metadata materialization behind the shared decision and connector boundary. -4. Consolidate forward HTTP, CONNECT HTTP inspection, credential injection, - request-body rewrite, and WebSocket handling behind shared HTTP/WebSocket - relay code. +4. Consolidate forward HTTP, CONNECT HTTP inspection, supervisor middleware, + credential injection, request-body rewrite, and WebSocket handling behind + shared HTTP/WebSocket relay code. 5. Move TLS detection and termination ahead of the HTTP/TCP relay split. -6. Add the TCP relay/parser boundary, then policy DNS and native TCP capture. +6. Add the TCP relay/protocol processor boundary, then policy DNS and native + TCP capture. 7. Treat local services and deployment modes as explicit runtime contracts. ## Risks @@ -352,6 +376,9 @@ The intended order is: - Provider-composed policy rules use a reserved namespace. Decisions and logs must distinguish provider-derived policy from user-authored policy without exposing provider rules as editable sandbox proposals. +- Supervisor middleware adds a synchronous request-path dependency. Body caps, + timeout behavior, registry reloads, and `fail_open` choices must be visible + in telemetry so operators can diagnose whether content inspection ran. ## Alternatives @@ -389,6 +416,12 @@ proves per-request HTTP evaluation, GraphQL parsing, WebSocket frame handling, request-body rewrite, and token-grant injection can live behind relay boundaries. +RFC 0009 supervisor middleware is the extension prior art. It defines +`HTTP_REQUEST / PRE_CREDENTIALS` as a supervisor-owned hook that can inspect, +deny, or transform admitted HTTP requests before credentials are injected. RFC +0005 should place that hook inside the shared relay rather than making each +adapter wire middleware separately. + ## Open questions 1. Should overlapping endpoint metadata be rejected at policy load time, or diff --git a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md index d4090e4f67..5d51929fc0 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md @@ -134,6 +134,33 @@ HTTP proxy variables. These are userland-facing network surfaces. They should stay distinct from external egress while still fitting the adapter model. +## Adjacent In-Flight Supervisor Middleware Shape + +PRs #1738 and #2027 propose supervisor middleware as an HTTP request hook in +the proxy relay. That work is adjacent to this RFC rather than a separate entry +adapter. + +```mermaid +flowchart TD + Req["Parsed admitted HTTP request"] --> Policy["Network and request policy already allowed"] + Policy --> Hook["HTTP_REQUEST / PRE_CREDENTIALS middleware"] + Hook --> Outcome{"Middleware outcome"} + Outcome -- "deny" --> Deny["Local deny, no credential injection"] + Outcome -- "allow / mutate" --> Creds["Credential injection"] + Creds --> Upstream["Upstream write"] +``` + +The proposed middleware chain is selected by admitted destination host, runs in +deterministic order, buffers bounded request bodies, applies `fail_open` or +`fail_closed`, emits audit-safe findings, and runs before OpenShell-managed +credentials are injected. It can inspect WebSocket upgrade requests because +they are HTTP requests, but it does not inspect post-upgrade WebSocket frames +in v1. + +RFC 0005 should account for this by treating middleware as part of the shared +request processing plan. CONNECT and forward HTTP should not each learn how to +select and invoke middleware independently. + ## Current Network Namespace Enforcement ```mermaid @@ -205,6 +232,7 @@ The refactor should preserve: - nftables bypass reject/log enforcement. - Provider credential injection and redaction. - Dynamic token grant injection through SPIFFE-backed provider credentials. +- Supervisor middleware `HTTP_REQUEST / PRE_CREDENTIALS` when it lands. - REST request-body credential rewrite. - WebSocket text-frame credential rewrite. - REST endpoint method/path policy. diff --git a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md index 3b00a512b0..0e91cda9f1 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md @@ -18,6 +18,9 @@ direction-focused. HTTP-inspected paths. - Add tests proving token grant success injects the configured header and token grant failure does not forward upstream. +- Add tests proving supervisor middleware runs after request policy and before + credential injection, including allow, deny, mutation, `fail_open`, + `fail_closed`, and body-cap behavior. - Add tests for REST request-body credential rewrite, WebSocket text-frame credential rewrite, WebSocket GraphQL policy, and compression handling. - Add tests for `policy.local` proposal wait behavior and `inference.local` @@ -35,7 +38,8 @@ direction-focused. together. - Include policy source on the decision: user-authored, provider-derived, or local-service internal. -- Include protocol enforcement and credential injection plan on the decision. +- Include protocol enforcement, supervisor middleware, and credential injection + plans on the decision. - Fail closed when required endpoint metadata cannot be materialized. - Emit consistent OCSF network denial events from the shared boundary. @@ -58,13 +62,20 @@ direction-focused. - Preserve h2c rejection on inspected routes. - Keep the no-raw-copy invariant after the first request. -## Phase 4 - HTTP, WebSocket, And Credential Relay Consolidation +## Phase 4 - HTTP, WebSocket, Middleware, And Credential Relay Consolidation - Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket - upgrade policy, credential resolution, redaction, request rewrite, upstream - dial, and response relay. + upgrade policy, supervisor middleware, credential resolution, redaction, + request rewrite, upstream dial, and response relay. - Evaluate every HTTP request before upstream write. - Ensure denied HTTP requests do not create upstream TCP sessions. +- Run `HTTP_REQUEST / PRE_CREDENTIALS` middleware after request allow and + before static or dynamic credential injection. +- Preserve middleware ordering, body caps, failure policy, safe header + mutation, findings, and metadata emission. +- Reject or strip newly introduced reserved credential placeholders from + middleware-transformed content unless a future hook is explicitly + credential-capable. - Preserve static placeholder rewrite for target, query, and headers. - Preserve dynamic token grant injection after request allow and before upstream write. @@ -84,13 +95,16 @@ direction-focused. - Treat `tls: skip` as the explicit opt-out for TLS handling. - Remove duplicate HTTP-specific and TCP-specific TLS termination decisions. -## Phase 6 - TCP Relay And Parser Boundary +## Phase 6 - TCP Relay And Protocol Processor Boundary -- Use `TcpRelay` for byte relay and TCP application parser dispatch. +- Use `TcpRelay` for byte relay and native protocol processor dispatch. - Keep `protocol: tcp` or omitted protocol as L4 authorization plus byte copy. -- Add a TCP application parser dispatch point for future protocol enforcement. -- Let TCP application parsers own their message loop and call the connector +- Add a native protocol processor dispatch point for future protocol + enforcement. +- Let protocol processors own their message loop and call the connector when protocol state allows. +- Allow processors to expose typed middleware hooks instead of requiring all + payload logic to live in-tree. ## Phase 7 - Policy DNS And Transparent TCP @@ -123,7 +137,8 @@ direction-focused. `openshell-supervisor-process` split as the structural baseline. - Define the proxy runtime API needed for a future standalone binary: configured listeners, policy updates, provider credentials, token grants, - gateway calls, telemetry, denial/activity events, and shutdown. + supervisor middleware registry, gateway calls, telemetry, denial/activity + events, and shutdown. - Identify process identity requirements for standalone and sidecar modes. - Add capability negotiation with the gateway if standalone proxy versions can differ from gateway versions. @@ -152,6 +167,9 @@ direction-focused. paths. - Integration-test token grant success, cache hit, malformed token, resolver unavailable, and token endpoint failure. +- Integration-test supervisor middleware allow/deny/mutate, unavailable + service, unresolved binding, body over-capacity, safe header mutation, + finding emission, and no-credential-visible behavior. - Integration-test REST request-body credential rewrite for JSON, form-url-encoded, `text/*`, unsupported content types, chunked framing, body caps, and unresolved placeholders. @@ -160,8 +178,8 @@ direction-focused. safe compression negotiation. - Integration-test TLS termination before HTTP/TCP relay split. - Integration-test `protocol: tcp` byte-copy behavior. -- Add parser harness tests before adding Redis, Postgres, or similar TCP - application parsers. +- Add protocol processor harness tests before adding Redis, Postgres, or + similar native protocol enforcement. - Integration-test policy DNS TTL, stale generation handling, and captured connect correlation. - Integration-test `inference.local`, `policy.local`, and metadata loopback diff --git a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md index 837a638d9e..44b28d22a2 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md @@ -8,7 +8,8 @@ This appendix carries implementation-level design details behind the main RFC. startup boundary. It builds policy-local context, waits for policy binary symlink resolution, creates the identity cache, writes the TLS CA, builds TLS state, resolves inference routes, wires provider credentials and token grants, -and starts the proxy. +and starts the proxy. The supervisor middleware work extends this boundary with +middleware registry construction and reload behavior. This is a useful outer boundary, but it is not yet the proxy adapter boundary. The proxy still needs internal `EgressIntent` and `EgressDecision` boundaries @@ -50,11 +51,12 @@ It should carry: - TLS behavior; - protocol enforcement; - credential injection plan; +- supervisor middleware plan; - logging context and denial reason. Relay code should read this decision. It should not query OPA again for -endpoint metadata, TLS mode, allowed IPs, credential behavior, or parser -selection. +endpoint metadata, TLS mode, allowed IPs, credential behavior, middleware +selection, or processor selection. ## Protocol Enforcement @@ -66,10 +68,10 @@ Use a protocol enforcement value derived from endpoint policy: | `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite | | `graphql` | HTTP | HTTP request parser with GraphQL-over-HTTP rules | | `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy | -| future `redis`, `postgres`, `mysql`, ... | TCP application | Protocol-specific TCP parser owns the message loop | +| future `redis`, `postgres`, `mysql`, ... | Protocol processor | Protocol-specific processor owns framing, middleware hooks, and the message loop | -`protocol: tcp` is effectively the default L4 mode. It should not run TCP -application parsers. Avoid using the term "provider" for parser concepts +`protocol: tcp` is effectively the default L4 mode. It should not run native +protocol processors. Avoid using the term "provider" for processor concepts because providers are already a first-class credential and routing domain in OpenShell. @@ -100,6 +102,7 @@ struct EgressDecision { outcome: PolicyOutcome, matched_policy: Option, endpoint: Option, + request_processing: RequestProcessingPlan, log_context: EgressLogContext, } @@ -119,13 +122,17 @@ struct MatchedEndpoint { allowed_ips: AllowedIpPolicy, tls: TlsPolicy, enforcement: ProtocolEnforcement, +} + +struct RequestProcessingPlan { + middleware: SupervisorMiddlewarePlan, credentials: CredentialInjectionPlan, } enum ProtocolEnforcement { None, Http(HttpL7Config), - TcpApplication(TcpApplicationConfig), + ProtocolProcessor(ProtocolProcessorConfig), } enum HttpL7Protocol { @@ -162,6 +169,32 @@ struct TokenGrantPlan { token_endpoint: String, } +struct SupervisorMiddlewarePlan { + stages: Vec, + min_body_limit: Option, + registry_generation: PolicyGeneration, +} + +struct SupervisorMiddlewareStage { + policy_name: String, + binding_id: String, + operation: MiddlewareOperation, + phase: MiddlewarePhase, + order: i32, + on_error: MiddlewareOnError, + config: MiddlewareConfig, +} + +enum MiddlewareOperation { + HttpRequest, + Future(String), +} + +enum MiddlewarePhase { + PreCredentials, + Future(String), +} + struct RelayContext { decision: EgressDecision, connector: UpstreamConnector, @@ -171,8 +204,8 @@ struct RelayContext { ``` `UpstreamConnector` is the relay-owned dial boundary. It encapsulates the -validated destination and lets relays/parsers open an upstream connection only -after protocol policy allows it. +validated destination and lets relays/processors open an upstream connection +only after protocol policy allows it. ## Current Owners And Proposed Cleanup @@ -184,6 +217,7 @@ after protocol policy allows it. | `openshell-supervisor-network::opa` | Policy engine and Rego queries | Return deterministic `EgressDecision` data instead of separate policy and endpoint lookups | | `openshell-supervisor-network::l7` | REST, GraphQL, WebSocket, inference helpers, TLS, token grants | Keep as protocol/relay implementation behind shared relay boundaries | | `openshell-supervisor-network::policy_local` | `policy.local` state and routes | Model as a local adapter with explicit limits and proposal/wait behavior | +| `openshell-supervisor-middleware` | Middleware registry, built-ins, service contract, and chain execution | Treat as a relay hook dependency selected by `EgressDecision`, not as adapter-specific policy logic | | `openshell-supervisor-process::netns` | nftables bypass rules and namespace helpers | Remain owner of bypass enforcement; coordinate future capture rules with network proxy mappings | | `openshell-supervisor-process::bypass_monitor` | nftables LOG parsing and OCSF bypass telemetry | Remain telemetry producer for bypass violations | | `openshell-core::secrets` and provider credential state | Static placeholder sources and dynamic credential metadata | Feed credential injection plans; do not leak secrets into decision logs | @@ -248,13 +282,15 @@ matches for logging and debugging. ## Credential Injection Boundary Credential injection belongs in the HTTP/WebSocket relay after policy allow and -before upstream write. +supervisor middleware, and before upstream write. 1. Authorization selects the endpoint and computes a credential injection plan. -2. The HTTP relay resolves credentials only when it has an allowed request. -3. Static placeholder values are resolved and redacted from logs. -4. Endpoint-bound token grants obtain or reuse a dynamic access token. -5. The final upstream request or WebSocket frame is rewritten immediately +2. Supervisor middleware runs on the admitted request before credentials are + visible. +3. The HTTP relay resolves credentials only when it has an allowed request. +4. Static placeholder values are resolved and redacted from logs. +5. Endpoint-bound token grants obtain or reuse a dynamic access token. +6. The final upstream request or WebSocket frame is rewritten immediately before write. Both L4-only HTTP and HTTP-inspected paths can inject credentials. The @@ -283,22 +319,63 @@ token, and inject either an `Authorization: Bearer` header or a configured custom header. Token grant failures should return a local relay error and must not forward the request upstream. -## Parser Boundary +Middleware-transformed content should be treated as untrusted input from a +credential perspective. External middleware must not receive OpenShell-managed +credentials, and it should not be able to synthesize new reserved credential +placeholders that OpenShell later resolves into secrets. Unless a future hook +is explicitly built-in-only and credential-capable, the relay should fail +closed or strip newly introduced reserved placeholders before static +placeholder rewrite and token grant injection. + +## Supervisor Middleware Boundary + +Supervisor middleware is a typed relay hook, not a replacement for protocol +framing. The relay or protocol processor must first parse enough structure to +construct the operation-specific middleware input. + +For v1, the operation is `HTTP_REQUEST / PRE_CREDENTIALS`: + +1. Network policy, destination validation, and request policy admit the + request. +2. The HTTP relay selects the middleware chain from the request processing + plan. +3. The relay buffers the request body within the smallest selected stage limit. +4. The chain evaluates in deterministic order. +5. A deny short-circuits before credential injection or upstream write. +6. An allow can replace the request body, add approved headers, emit findings, + and pass metadata forward. +7. The transformed request then enters credential injection and upstream write. -Protocol parsers operate on streams owned by the relay. +Middleware selection is independent from the matched endpoint policy. It is a +request processing plan selected by admitted destination host, order, and +binding metadata. The decision boundary should materialize it with the same +policy generation used for endpoint selection so a long-lived tunnel cannot mix +old endpoint policy with a new middleware registry. + +V1 middleware can inspect WebSocket upgrade requests because those are HTTP +requests. It does not inspect post-upgrade WebSocket frames. A future frame +hook should be a separate operation such as `WEBSOCKET_MESSAGE / +BEFORE_FORWARD` owned by the WebSocket relay. + +## Protocol Processor Boundary + +Protocol processors operate on streams owned by the relay. - HTTP parsing converts bytes into request metadata, evaluates request policy, + runs the `HTTP_REQUEST / PRE_CREDENTIALS` middleware hook when configured, and loops for keep-alive or pipelined requests. - WebSocket parsing starts only after an allowed HTTP upgrade. It validates the handshake/frame stream and owns client-to-server text-frame inspection when credential rewrite, transport message policy, GraphQL-over-WebSocket policy, or compression handling is configured. -- TCP application parsers read client and upstream streams as needed and own - their message loop. -- A TCP parser can deny before dialing, dial for a server handshake, or keep - evaluating commands/queries throughout the session. - -This avoids a separate dial strategy enum. The parser knows which protocol +- Native TCP protocol processors read client and upstream streams as needed + and own their message loop. +- A protocol processor can deny before dialing, dial for a server handshake, or + keep evaluating commands/queries throughout the session. +- A protocol processor may be in-tree, middleware-backed, or a hybrid where + in-tree framing exposes typed middleware operations for content evaluation. + +This avoids a separate dial strategy enum. The processor knows which protocol milestone is sufficient to call the validated connector. ## Local Service Adapter Boundary @@ -330,9 +407,10 @@ egress. | HTTP relay | Per-request read/write deadlines, body caps, request-body rewrite caps, upstream reuse | | WebSocket relay | Upgrade validation, frame limits, text-frame rewrite, compression limits, message policy | | TCP relay | Byte-copy idle timeout and half-close handling | -| TCP parser | Protocol message timeouts and parser-specific limits | +| Protocol processor | Protocol message timeouts, middleware hook timeouts, and processor-specific limits | | Local service adapter | Local route body limits, response caps, gateway call timeout | | Token grant resolver | SPIFFE Workload API timeout, token endpoint timeout, cache TTL | +| Middleware runner | Service timeout, body cap, failure policy, registry generation | Timeouts should be recorded in telemetry at the owner boundary that can explain the failure. From 25da5619fe62cef884c88e4a92aab544f29add97 Mon Sep 17 00:00:00 2001 From: John Myers Date: Mon, 6 Jul 2026 09:07:27 -0700 Subject: [PATCH 5/8] docs(rfc): include json-rpc and mcp l7 protocols Signed-off-by: John Myers --- .../README.md | 29 ++++++++++++------- .../current-shape.md | 9 ++++-- .../implementation-plan.md | 11 +++++-- .../technical-design.md | 12 +++++++- 4 files changed, 44 insertions(+), 17 deletions(-) diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md index 34197c1679..f9b4c47829 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/README.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -10,6 +10,8 @@ links: - https://github.com/NVIDIA/OpenShell/pull/1511 - https://github.com/NVIDIA/OpenShell/pull/1738 - https://github.com/NVIDIA/OpenShell/pull/2027 + - https://github.com/NVIDIA/OpenShell/pull/1865 + - https://github.com/NVIDIA/OpenShell/pull/1938 --- # RFC 0005 - Sandbox Proxy Egress Adapter Model @@ -44,10 +46,10 @@ Supporting detail lives in: The sandbox proxy supports several connection surfaces: explicit CONNECT, forward HTTP, local inference and policy APIs, metadata loopback, TLS -termination, REST and GraphQL inspection, WebSocket inspection, credential -injection, supervisor middleware, and nftables-backed bypass detection. These -features are valuable, but changes to policy and enforcement still tend to -touch multiple entry paths. +termination, REST, GraphQL, JSON-RPC, MCP, and WebSocket inspection, +credential injection, supervisor middleware, and nftables-backed bypass +detection. These features are valuable, but changes to policy and enforcement +still tend to touch multiple entry paths. The risk is asymmetric enforcement. A security fix can be added to CONNECT and missed in forward HTTP; endpoint metadata can be selected differently from the @@ -184,7 +186,7 @@ flowchart TD FirstReq -- Yes --> ForwardMode{"decision.endpoint.enforcement"} ForwardMode -- "None" --> HttpCred["HTTP relay
credential injection only"] - ForwardMode -- "Http" --> HttpL7["HTTP relay
REST/GraphQL/WebSocket policy"] + ForwardMode -- "Http" --> HttpL7["HTTP relay
REST/GraphQL/JSON-RPC/MCP/WebSocket policy"] ForwardMode -- "ProtocolProcessor" --> BadForward["Deny: HTTP request for native protocol endpoint"] FirstReq -- No --> TlsPolicy{"TLS handling skipped?"} @@ -235,6 +237,9 @@ Relay rules: - HTTP credential injection happens in both HTTP modes: L4-only HTTP and HTTP-inspected. +- HTTP-inspected endpoints include `rest`, `graphql`, `json-rpc`, `mcp`, and + `websocket`. JSON-RPC and MCP are HTTP L7 protocols, not native TCP protocol + processors. - Supervisor middleware is a typed relay hook. V1 middleware runs on parsed HTTP requests at `HTTP_REQUEST / PRE_CREDENTIALS`, after network and request policy admit the request and before OpenShell injects credentials. @@ -250,7 +255,9 @@ Relay rules: introduced reserved placeholders before credential injection. - Static credential rewrite covers request target, query, headers, opt-in REST request bodies, and opt-in client-to-server WebSocket text frames. -- HTTP L7 policy is evaluated before upstream write for each request. +- HTTP L7 policy is evaluated before upstream write for each request. JSON-RPC + and MCP evaluation parse bounded JSON-RPC-over-HTTP bodies; MCP adds + tool-aware selectors for `tools/call`. - WebSocket upgrade policy is evaluated as HTTP first. After an allowed `101` upgrade, the WebSocket relay owns frame parsing when text-frame credential rewrite, WebSocket transport policy, GraphQL-over-WebSocket policy, or safe @@ -352,8 +359,8 @@ The intended order is: 3. Move destination validation and endpoint metadata materialization behind the shared decision and connector boundary. 4. Consolidate forward HTTP, CONNECT HTTP inspection, supervisor middleware, - credential injection, request-body rewrite, and WebSocket handling behind - shared HTTP/WebSocket relay code. + credential injection, request-body rewrite, JSON-RPC/MCP inspection, and + WebSocket handling behind shared HTTP/WebSocket relay code. 5. Move TLS detection and termination ahead of the HTTP/TCP relay split. 6. Add the TCP relay/protocol processor boundary, then policy DNS and native TCP capture. @@ -412,9 +419,9 @@ packet-enforcement substrate. Transparent TCP should extend that nftables model rather than creating a second firewall path. The existing L7 relay is the behavioral prior art for this RFC. It already -proves per-request HTTP evaluation, GraphQL parsing, WebSocket frame handling, -request-body rewrite, and token-grant injection can live behind relay -boundaries. +proves per-request HTTP evaluation, GraphQL parsing, JSON-RPC/MCP body +inspection, WebSocket frame handling, request-body rewrite, and token-grant +injection can live behind relay boundaries. RFC 0009 supervisor middleware is the extension prior art. It defines `HTTP_REQUEST / PRE_CREDENTIALS` as a supervisor-owned hook that can inspect, diff --git a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md index 5d51929fc0..312b3f029d 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md @@ -56,7 +56,8 @@ The networking surface currently includes: - Static provider credential injection and redaction. - Endpoint-bound dynamic token grant injection. - Opt-in REST request-body credential rewrite. -- L7 REST, GraphQL, WebSocket, and GraphQL-over-WebSocket enforcement. +- L7 REST, GraphQL, JSON-RPC, MCP, WebSocket, and + GraphQL-over-WebSocket enforcement. The issue is not that these features exist. The issue is that entry mechanisms, policy evaluation, endpoint metadata lookup, credential injection, and byte @@ -76,7 +77,7 @@ flowchart TD Config -- Yes --> Tunnel Tunnel --> Inspect["Inspect tunneled bytes when possible"] Inspect --> Relay["HTTP/WebSocket/TCP relay selection"] - Relay --> Inject["Static credentials and token grants if configured"] + Relay --> Inject["Middleware, static credentials, and token grants if configured"] Inject --> Upstream["Open upstream when relay policy allows"] ``` @@ -93,7 +94,7 @@ flowchart TD L4 --> Allowed{"Allowed?"} Allowed -- No --> Deny["HTTP denial"] Allowed -- Yes --> L7{"Matching L7 endpoint?"} - L7 -- Yes --> Eval["Evaluate REST/GraphQL/WebSocket policy"] + L7 -- Yes --> Eval["Evaluate REST/GraphQL/JSON-RPC/MCP/WebSocket policy"] Eval --> Guard["Reject unsupported h2c upgrade when inspected"] Guard --> Rewrite["Rewrite to origin-form + configured credentials"] L7 -- No --> Rewrite @@ -237,6 +238,8 @@ The refactor should preserve: - WebSocket text-frame credential rewrite. - REST endpoint method/path policy. - GraphQL-over-HTTP policy. +- JSON-RPC-over-HTTP method policy. +- MCP Streamable HTTP method and tool policy. - WebSocket transport and GraphQL-over-WebSocket policy. - h2c rejection on inspected HTTP routes. - Inference routing through `inference.local`, including embeddings. diff --git a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md index 0e91cda9f1..93cf0f5fac 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md @@ -23,6 +23,8 @@ direction-focused. `fail_closed`, and body-cap behavior. - Add tests for REST request-body credential rewrite, WebSocket text-frame credential rewrite, WebSocket GraphQL policy, and compression handling. +- Add tests for JSON-RPC method policy, batch behavior, response-frame denial, + and MCP method/tool policy. - Add tests for `policy.local` proposal wait behavior and `inference.local` buffered/streaming route limits. - Add tests for metadata loopback startup/failure behavior when provider @@ -65,8 +67,8 @@ direction-focused. ## Phase 4 - HTTP, WebSocket, Middleware, And Credential Relay Consolidation - Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket - upgrade policy, supervisor middleware, credential resolution, redaction, - request rewrite, upstream dial, and response relay. + upgrade policy, JSON-RPC/MCP policy, supervisor middleware, credential + resolution, redaction, request rewrite, upstream dial, and response relay. - Evaluate every HTTP request before upstream write. - Ensure denied HTTP requests do not create upstream TCP sessions. - Run `HTTP_REQUEST / PRE_CREDENTIALS` middleware after request allow and @@ -86,6 +88,9 @@ direction-focused. opt-in client-to-server text-frame credential rewrite, WebSocket transport message policy, GraphQL-over-WebSocket policy, and raw passthrough for other upgraded protocols. +- Preserve JSON-RPC and MCP handling behind the shared HTTP relay, including + bounded body inspection, JSON-RPC batch evaluation, MCP `tools/call` tool + selectors, and audit-safe logging that omits params and tool arguments. ## Phase 5 - Shared TLS Termination @@ -176,6 +181,8 @@ direction-focused. - Integration-test WebSocket text-frame credential rewrite, raw upgraded passthrough, WebSocket message policy, GraphQL-over-WebSocket policy, and safe compression negotiation. +- Integration-test JSON-RPC method allow/deny, batch denial, response-frame + handling, MCP method profile behavior, and MCP tool selector enforcement. - Integration-test TLS termination before HTTP/TCP relay split. - Integration-test `protocol: tcp` byte-copy behavior. - Add protocol processor harness tests before adding Redis, Postgres, or diff --git a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md index 44b28d22a2..cebb87caa1 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md @@ -67,6 +67,8 @@ Use a protocol enforcement value derived from endpoint policy: | omitted / `tcp` | None | L4 authorization plus byte relay, with optional HTTP sniff for credential injection | | `rest` | HTTP | HTTP request parser with REST rules, plus opt-in request-body and WebSocket text-frame credential rewrite | | `graphql` | HTTP | HTTP request parser with GraphQL-over-HTTP rules | +| `json-rpc` | HTTP | HTTP request parser plus bounded JSON-RPC-over-HTTP method inspection | +| `mcp` | HTTP | HTTP request parser plus bounded MCP Streamable HTTP method/tool inspection | | `websocket` | HTTP | HTTP upgrade policy followed by WebSocket frame policy or GraphQL-over-WebSocket policy | | future `redis`, `postgres`, `mysql`, ... | Protocol processor | Protocol-specific processor owns framing, middleware hooks, and the message loop | @@ -138,6 +140,8 @@ enum ProtocolEnforcement { enum HttpL7Protocol { Rest, Graphql, + JsonRpc, + Mcp, Websocket, } @@ -150,6 +154,8 @@ struct HttpL7Config { request_body_credential_rewrite: bool, websocket_graphql_policy: bool, graphql_max_body_bytes: usize, + json_rpc_max_body_bytes: usize, + mcp_strict_tool_names: bool, } struct CredentialInjectionPlan { @@ -215,7 +221,7 @@ only after protocol policy allows it. | `openshell-supervisor-network::run` | Networking startup and handles | Become the stable runtime API for embedded and future standalone modes | | `openshell-supervisor-network::proxy` | CONNECT, forward HTTP, local route dispatch, destination validation, denial rendering | Split into adapters, authorization, destination, relay selection, and adapter response rendering | | `openshell-supervisor-network::opa` | Policy engine and Rego queries | Return deterministic `EgressDecision` data instead of separate policy and endpoint lookups | -| `openshell-supervisor-network::l7` | REST, GraphQL, WebSocket, inference helpers, TLS, token grants | Keep as protocol/relay implementation behind shared relay boundaries | +| `openshell-supervisor-network::l7` | REST, GraphQL, JSON-RPC, MCP, WebSocket, inference helpers, TLS, token grants | Keep as protocol/relay implementation behind shared relay boundaries | | `openshell-supervisor-network::policy_local` | `policy.local` state and routes | Model as a local adapter with explicit limits and proposal/wait behavior | | `openshell-supervisor-middleware` | Middleware registry, built-ins, service contract, and chain execution | Treat as a relay hook dependency selected by `EgressDecision`, not as adapter-specific policy logic | | `openshell-supervisor-process::netns` | nftables bypass rules and namespace helpers | Remain owner of bypass enforcement; coordinate future capture rules with network proxy mappings | @@ -364,6 +370,10 @@ Protocol processors operate on streams owned by the relay. - HTTP parsing converts bytes into request metadata, evaluates request policy, runs the `HTTP_REQUEST / PRE_CREDENTIALS` middleware hook when configured, and loops for keep-alive or pipelined requests. +- JSON-RPC and MCP processing are HTTP L7 processors: they parse bounded + JSON-RPC-over-HTTP request bodies after HTTP parsing and before upstream + forwarding. Generic JSON-RPC policy matches methods; MCP policy can also + match `tools/call` tool names. - WebSocket parsing starts only after an allowed HTTP upgrade. It validates the handshake/frame stream and owns client-to-server text-frame inspection when credential rewrite, transport message policy, GraphQL-over-WebSocket policy, From 26a29a249d0a97e40c055b902cefd988b812f69c Mon Sep 17 00:00:00 2001 From: John Myers Date: Mon, 6 Jul 2026 10:24:39 -0700 Subject: [PATCH 6/8] docs(rfc): make process identity optional Signed-off-by: John Myers --- .../README.md | 26 ++++++++--- .../current-shape.md | 7 +++ .../implementation-plan.md | 12 ++++- .../technical-design.md | 44 +++++++++++++++++-- 4 files changed, 79 insertions(+), 10 deletions(-) diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md index f9b4c47829..da4cdbb049 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/README.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -92,9 +92,9 @@ enforcement contracts. policy evaluation. 2. **Egress intent and decision.** Shared authorization evaluates L4 policy and endpoint selection once per connection intent and returns one decision - containing the matched policy, matched endpoint, process identity, allowed - IP metadata, TLS behavior, protocol enforcement, and credential injection - and middleware plans. + containing the matched policy, matched endpoint, optional process identity + evidence used for evaluation, allowed IP metadata, TLS behavior, protocol + enforcement, and credential injection and middleware plans. 3. **Relays.** Relays receive an authorized destination connector, not an already-open upstream socket. HTTP relays evaluate every request before upstream write. TCP relays copy bytes for L4-only endpoints or hand the @@ -342,6 +342,18 @@ If supervisor middleware is configured, the proxy runtime must also receive the effective middleware service registry, validate/refresh bindings, enforce `fail_open` and `fail_closed`, buffer within configured caps, invoke middleware on the request path, and emit middleware OCSF events. + +Process identity is mode-dependent. Embedded supervisor mode can usually +resolve the workload process, binary, and ancestors. Network-only, standalone, +and sidecar modes may intentionally have no local process identity. In those +modes the adapter should pass an explicit unavailable identity envelope, and +the decision should record identity as unavailable rather than treating it as +an accidental lookup failure. Authorization must not turn a missing identity +into a broader allow. Process-scoped predicates should either be treated as +non-matching for that runtime or rejected during policy/capability validation. +Policies that require binary/path scoping need an explicit capability check or +fallback rule before they are allowed to run in identity-less modes. + The nftables rules that force or reject userland traffic belong to the sandbox network boundary even if the proxy process later moves into a standalone binary or sidecar. @@ -376,7 +388,9 @@ The intended order is: endpoints. Failures should remain fail-closed and sanitized. - Transparent TCP capture adds network namespace interception complexity and must coexist with the nftables bypass reject/log table. -- Sidecar mode needs a reliable identity source for binary/path scoped policy. +- Sidecar mode may intentionally lack process identity. Binary/path scoped + policy needs a reliable identity source or must be rejected/ignored for that + deployment mode. - Metadata loopback and `policy.local` expand sandbox-local control surfaces and need strict route validation, body limits, redaction, and authentication boundaries. @@ -436,8 +450,8 @@ adapter wire middleware separately. 2. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted, or should DNS query correlation be required for stricter modes? 3. What TTL cap and stale-generation grace period should policy DNS use? -4. Which process identity source should sidecar mode use when it cannot inspect - payload process metadata through local `/proc`? +4. Which policy features should be disabled, rejected, or treated as + non-matching when the proxy runtime advertises no process identity support? 5. Which proxy capabilities should be negotiated with the gateway at startup? 6. Should metadata loopback be modeled as an adapter inside `openshell-supervisor-network`, or remain orchestrated by `openshell-sandbox` diff --git a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md index 312b3f029d..67506117b1 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md @@ -39,6 +39,13 @@ routes, identity cache, provider credential injection, and token grants. helpers, nftables bypass rules, and the bypass monitor that turns nftables LOG entries into OCSF events. +In embedded supervisor mode, the network leaf can usually use process metadata +resolved by the process/orchestrator side for binary-scoped policy and OCSF +context. Future network-only, standalone, or sidecar proxy modes may +intentionally lack that metadata. The adapter model should preserve this as an +explicit "identity unavailable" state rather than fabricating an empty binary +identity. + ## Current Userland-Facing Surfaces The networking surface currently includes: diff --git a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md index 93cf0f5fac..e4a15ee578 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md @@ -14,6 +14,9 @@ direction-focused. paths. - Add tests for exact declared private endpoint trust and `allowed_ips` behavior across CONNECT and forward HTTP. +- Add tests for identity-less runtime modes where process identity is + intentionally unavailable and binary/path scoped policy does not + accidentally match. - Add tests proving static credential injection works in L4-only HTTP and HTTP-inspected paths. - Add tests proving token grant success injects the configured header and token @@ -42,6 +45,9 @@ direction-focused. local-service internal. - Include protocol enforcement, supervisor middleware, and credential injection plans on the decision. +- Include process identity availability and fields used on the decision. Treat + missing process identity as an explicit runtime mode, not as an implicit + lookup failure. - Fail closed when required endpoint metadata cannot be materialized. - Emit consistent OCSF network denial events from the shared boundary. @@ -144,7 +150,10 @@ direction-focused. configured listeners, policy updates, provider credentials, token grants, supervisor middleware registry, gateway calls, telemetry, denial/activity events, and shutdown. -- Identify process identity requirements for standalone and sidecar modes. +- Advertise process identity capability for embedded, network-only, + standalone, and sidecar modes. Reject policies that require unavailable + identity dimensions, or define those predicates as non-matching for the + runtime mode. - Add capability negotiation with the gateway if standalone proxy versions can differ from gateway versions. @@ -164,6 +173,7 @@ direction-focused. - Unit-test authorization precedence for overlapping policy and endpoint rules. - Unit-test provider-derived rule namespace handling and `policy.local` filtering. +- Unit-test identity-available and identity-unavailable authorization inputs. - Integration-test shared destination validation across CONNECT, forward HTTP, and transparent TCP. - Integration-test HTTP keep-alive and pipelined requests with REST, GraphQL, diff --git a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md index cebb87caa1..09032e1213 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md @@ -27,7 +27,7 @@ It should carry: - entry transport: CONNECT, forward HTTP, transparent TCP, local HTTP, policy DNS, or metadata loopback; - requested destination host/port or captured original IP/port; -- process identity inputs collected by the adapter/runtime; +- optional process identity inputs collected by the adapter/runtime; - optional first HTTP request for forward proxy traffic; - optional local service route; - policy generation or DNS mapping generation when relevant. @@ -46,7 +46,7 @@ It should carry: - whether the policy is user-authored, provider-derived, or local-service internal; - deterministic matched endpoint identifier and endpoint metadata; -- process identity used for evaluation; +- process identity availability and any identity fields used for evaluation; - destination and allowed IP constraints; - TLS behavior; - protocol enforcement; @@ -94,7 +94,7 @@ enum EgressTransport { struct EgressIntent { transport: EgressTransport, destination: RequestedDestination, - process: ProcessIdentity, + process: ProcessIdentityEvidence, first_request: Option, local_route: Option, generation: Option, @@ -104,10 +104,27 @@ struct EgressDecision { outcome: PolicyOutcome, matched_policy: Option, endpoint: Option, + process: EvaluatedProcessIdentity, request_processing: RequestProcessingPlan, log_context: EgressLogContext, } +enum ProcessIdentityEvidence { + Available(ProcessIdentity), + Unavailable(ProcessIdentityUnavailableReason), +} + +enum ProcessIdentityUnavailableReason { + RuntimeMode, + UnsupportedAdapter, + LookupFailed, +} + +struct EvaluatedProcessIdentity { + evidence: ProcessIdentityEvidence, + fields_used: Vec, +} + struct MatchedPolicy { id: PolicyId, source: PolicySource, @@ -213,6 +230,27 @@ struct RelayContext { validated destination and lets relays/processors open an upstream connection only after protocol policy allows it. +## Process Identity Availability + +Process identity is evidence, not an always-present input. Embedded supervisor +mode can usually populate binary, PID, ancestry, command-line path, and binary +hash data. Network-only, standalone binary, or sidecar proxy modes may +intentionally have no local process metadata. That should be represented as +`ProcessIdentityEvidence::Unavailable(RuntimeMode)`, not as an empty string +that accidentally matches policy. + +Authorization should evaluate only identity dimensions that are available. If +no identity is available, binary/path scoped policy should either be skipped as +non-matching or rejected at policy/capability validation time, depending on the +runtime mode contract. The important invariant is that missing identity must +not produce a synthetic binary, broaden a binary-scoped allow rule, or cause +the relay to query process metadata later. During `EgressDecision` hydration, +absent process evidence should be carried forward as unavailable identity and +process-derived fields should be omitted from the hydrated decision. The +decision should record identity availability and fields used so OCSF logs and +deny responses can distinguish "policy denied this binary" from "this runtime +did not provide process identity." + ## Current Owners And Proposed Cleanup | Current owner | Current responsibility | Proposed cleanup | From 83920476a0fa80923ecacd57a25be26f0f54665e Mon Sep 17 00:00:00 2001 From: John Myers Date: Mon, 6 Jul 2026 10:31:06 -0700 Subject: [PATCH 7/8] docs(rfc): clarify relay flow diagram Signed-off-by: John Myers --- .../README.md | 86 ++++++++++--------- 1 file changed, 47 insertions(+), 39 deletions(-) diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md index da4cdbb049..113657a740 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/README.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -182,57 +182,65 @@ adapter renders it for its protocol. ```mermaid flowchart TD Start["Authorized egress + destination connector"] - Start --> FirstReq{"Forward HTTP adapter supplied first request?"} + Start --> FirstReq{"Forward HTTP adapter
already has first request?"} - FirstReq -- Yes --> ForwardMode{"decision.endpoint.enforcement"} - ForwardMode -- "None" --> HttpCred["HTTP relay
credential injection only"] - ForwardMode -- "Http" --> HttpL7["HTTP relay
REST/GraphQL/JSON-RPC/MCP/WebSocket policy"] - ForwardMode -- "ProtocolProcessor" --> BadForward["Deny: HTTP request for native protocol endpoint"] + FirstReq -- Yes --> ForwardEnforcement{"Endpoint enforcement"} + ForwardEnforcement -- "None or HTTP" --> HttpReq["Parsed HTTP request"] + ForwardEnforcement -- "Protocol processor" --> BadForward["Deny: HTTP request for native protocol endpoint"] - FirstReq -- No --> TlsPolicy{"TLS handling skipped?"} - TlsPolicy -- Yes --> Readable["Readable client stream"] - TlsPolicy -- No --> Peek["Peek client bytes"] + FirstReq -- No --> Prepare["Prepare readable client stream"] + Prepare --> TlsPolicy{"TLS handling enabled?"} + TlsPolicy -- No --> Readable["Client stream"] + TlsPolicy -- Yes --> Peek["Peek client bytes"] Peek --> Tls{"TLS ClientHello?"} Tls -- Yes --> Terminate["Shared TLS terminator"] Tls -- No --> Readable Terminate --> Readable - Readable --> Enforce{"decision.endpoint.enforcement"} - Enforce -- "None" --> Sniff{"Looks like HTTP?"} - Sniff -- Yes --> HttpCred - Sniff -- No --> TcpRelay["TcpRelay
byte copy"] + Readable --> Enforce{"Endpoint enforcement"} + Enforce -- "None" --> Sniff{"HTTP request detected?"} + Sniff -- Yes --> ParseHttp["Parse HTTP request"] + Sniff -- No --> TcpRelay["TcpRelay
connect upstream and copy bytes"] + ParseHttp --> HttpReq - Enforce -- "Http" --> MustHttp{"Looks like HTTP?"} - MustHttp -- Yes --> HttpL7 + Enforce -- "HTTP" --> MustHttp{"HTTP request detected?"} + MustHttp -- Yes --> ParseHttp MustHttp -- No --> DenyHttp["Deny: expected HTTP"] - Enforce -- "ProtocolProcessor" --> Processor["TcpRelay
protocol processor owns loop"] - - HttpCred --> ReqLoop["HTTP request loop"] - HttpL7 --> ReqLoop - ReqLoop --> ReqPolicy{"Request allowed?"} - ReqPolicy -- No --> ReqDeny["Local HTTP deny
no upstream write"] - ReqPolicy -- Yes --> Middleware{"Supervisor middleware chain configured?"} - Middleware -- Yes --> MwEval["Evaluate HTTP_REQUEST / PRE_CREDENTIALS
allow, deny, or mutate"] - Middleware -- No --> StaticCreds["Resolve static placeholders"] - MwEval --> MwAllowed{"Middleware allowed?"} - MwAllowed -- No --> MwDeny["Local middleware deny
no credential injection"] - MwAllowed -- Yes --> StaticCreds - StaticCreds --> TokenGrant["Apply endpoint token grant if configured"] - TokenGrant --> Rewrite["Rewrite configured credential slots"] - Rewrite --> HttpDial["Connect or reuse upstream"] - HttpDial --> HttpResponse["Write request and relay response"] - HttpResponse --> Upgrade{"101 WebSocket upgrade?"} - Upgrade -- No --> ReqLoop - Upgrade -- Yes --> WsInspect{"WebSocket inspection or rewrite configured?"} - WsInspect -- No --> RawUpgrade["Raw upgraded stream"] - WsInspect -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] - - Processor --> ProcessorDial["Processor calls connector when protocol allows"] - TcpRelay --> TcpDial["Connect upstream"] - TcpDial --> ByteCopy["Copy bytes"] + Enforce -- "Protocol processor" --> Processor["TcpRelay hands stream to protocol processor"] + Processor --> ProcessorOwns["Processor owns message loop
and calls connector when allowed"] + + subgraph HttpLoop["HTTP relay request loop"] + HttpReq --> HttpMode{"HTTP endpoint policy?"} + HttpMode -- "L4-only HTTP" --> ReqAllowed["Request admitted by connection decision"] + HttpMode -- "REST / GraphQL / JSON-RPC / MCP / WebSocket" --> ReqPolicy{"Request policy allowed?"} + ReqPolicy -- No --> ReqDeny["Local HTTP deny
no upstream write"] + ReqPolicy -- Yes --> ReqAllowed + ReqAllowed --> Middleware{"Supervisor middleware
configured?"} + Middleware -- Yes --> MwEval["Run HTTP_REQUEST / PRE_CREDENTIALS middleware"] + Middleware -- No --> Creds["Resolve static placeholders
and token grants"] + MwEval --> MwAllowed{"Middleware allowed?"} + MwAllowed -- No --> MwDeny["Local middleware deny
no credential injection"] + MwAllowed -- Yes --> Creds + Creds --> Rewrite["Inject credentials into configured slots"] + Rewrite --> HttpDial["Connect or reuse upstream"] + HttpDial --> HttpResponse["Write request and relay response"] + HttpResponse --> Upgrade{"101 WebSocket upgrade?"} + Upgrade -- No --> NextReq{"Another HTTP request
on this connection?"} + NextReq -- Yes --> HttpReq + NextReq -- No --> Done["HTTP relay done"] + Upgrade -- Yes --> WsInspect{"WebSocket inspection
or rewrite configured?"} + WsInspect -- No --> RawUpgrade["Raw upgraded stream"] + WsInspect -- Yes --> WsRelay["WebSocket relay
text-frame rewrite / message policy"] + end ``` +Read this as two phases. The top half chooses the relay shape from the adapter +surface and endpoint enforcement. The `HTTP relay request loop` only receives a +parsed HTTP request. Supervisor middleware is not another policy funnel; it is +an optional request-path hook after HTTP policy allows the request and before +OpenShell-managed credential injection. + Relay rules: - HTTP credential injection happens in both HTTP modes: L4-only HTTP and From c18e06a01afaeaf6ddcbd609864b62f0c7230493 Mon Sep 17 00:00:00 2001 From: John Myers <9696606+johntmyers@users.noreply.github.com> Date: Wed, 15 Jul 2026 10:23:37 -0700 Subject: [PATCH 8/8] docs(rfc): address proxy adapter review feedback Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --- .../README.md | 228 ++++++--- .../current-shape.md | 48 +- .../implementation-plan.md | 441 ++++++++++-------- .../technical-design.md | 209 ++++++--- 4 files changed, 586 insertions(+), 340 deletions(-) diff --git a/rfc/0005-sandbox-proxy-egress-adapter/README.md b/rfc/0005-sandbox-proxy-egress-adapter/README.md index 113657a740..4059bedd32 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/README.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/README.md @@ -1,9 +1,10 @@ --- authors: - "@johntmyers" -state: draft +state: review links: - https://github.com/NVIDIA/OpenShell/issues/1107 + - https://github.com/NVIDIA/OpenShell/pull/2155 - https://github.com/NVIDIA/OpenShell/pull/1083 - https://github.com/NVIDIA/OpenShell/pull/1151 - https://github.com/NVIDIA/OpenShell/pull/1286 @@ -22,13 +23,20 @@ See rfc/README.md for the full RFC process and state definitions. ## Summary -Refactor sandbox egress around one shared authorization and relay pipeline. -CONNECT, forward HTTP, native TCP capture, policy DNS, `inference.local`, -`policy.local`, and metadata loopback should become narrow adapters that -translate userland entry points into common runtime intents. Policy evaluation, -destination validation, supervisor middleware, credential injection, -request-body rewrite, WebSocket handling, protocol processing, and upstream -dialing should happen behind shared boundaries. +Refactor sandbox egress around shared authorization, destination-validation, +and relay boundaries. CONNECT, forward HTTP, native TCP capture, policy DNS, +`inference.local`, `policy.local`, and metadata loopback become narrow adapters +that translate userland entry points into common runtime intents. Policy +evaluation, destination validation, supervisor middleware, credential +injection, request-body rewrite, WebSocket handling, protocol processing, and +upstream dialing happen behind shared boundaries. + +The RFC describes the complete forward-looking architecture. It is designed +to land incrementally across multiple pull requests. The first milestone only +restructures CONNECT and forward HTTP, extracts shared primitives, and +preserves every current user-facing feature and enforcement behavior. Later +milestones add policy DNS, transparent TCP capture, native protocol processors, +and optional deployment shapes on top of those boundaries. The codebase has already moved in this direction by splitting networking into `openshell-supervisor-network` and process/netns work into @@ -63,11 +71,11 @@ The target shape separates three concerns: behavior applies. - **Relays** own bytes, credentials, protocol parsing, and upstream dialing. -This also prepares the proxy for future deployment modes. Today the proxy runs -inside the sandbox supervisor process. The networking leaf can already run in a -network-only mode, and a future standalone binary or sidecar should be possible -if it implements the same userland surfaces, gateway APIs, and policy -enforcement contracts. +The first milestone targets the current embedded/network-only supervisor +runtime and preserves its existing user-facing behavior while the internal +seams move. The same boundaries then support policy DNS and transparent TCP, +native protocol processing, and future deployment modes without duplicating +authorization or relay logic. ## Non-goals @@ -80,6 +88,13 @@ enforcement contracts. - Reintroduce iptables as the sandbox packet filtering backend. - Use eBPF connect hooks for transparent capture. Native TCP capture needs a userland proxy in the byte stream for TLS termination and protocol parsing. +- Add policy-declared supervisor-proxied host-local endpoints. Issue + [#1633](https://github.com/NVIDIA/OpenShell/issues/1633) can consume these + boundaries in separate feature work. +- Change the existing endpoint-only runtime's process-identity semantics during + the compatibility refactor. Future identity-less deployment modes require an + explicit capability contract and cannot inherit endpoint-only behavior by + accident. ## Proposal @@ -101,6 +116,13 @@ enforcement contracts. stream to a protocol processor when endpoint policy requires native protocol enforcement. +The first implementation milestone populates a compatibility +`EgressDecision` through the existing separate queries so type extraction does +not change behavior. That transitional envelope is not the target single +authorization result. Generation-consistent materialization and deterministic +endpoint selection cut over separately after shadow comparison, before later +transport adapters depend on the result. + ### Unified Adapter Flow ```mermaid @@ -120,7 +142,7 @@ flowchart TD subgraph NativeTcp["Policy DNS + native TCP"] NameLookup["Userland DNS lookup"] PolicyDns["Policy DNS adapter"] - DnsAnswer["DNS answer + active mapping"] + DnsAnswer["DNS answer + correlated active mapping"] NativeConnect["Userland connect(ip:port)"] TcpAdapter["Transparent TCP adapter"] NameLookup --> PolicyDns @@ -174,8 +196,8 @@ flowchart TD Each adapter still owns its response shape. If authorization denies a CONNECT intent, the CONNECT adapter returns a tunnel denial. If forward HTTP is denied, the forward adapter returns an HTTP denial. If policy DNS refuses a name, it -returns the appropriate DNS response. The shared layer decides the outcome; the -adapter renders it for its protocol. +returns the appropriate DNS response. The shared layer decides the outcome; +the adapter renders it for its protocol. ### Relay Flow @@ -287,7 +309,8 @@ The CONNECT adapter parses `CONNECT host:port` into an `EgressIntent`, asks the shared authorization boundary for an `EgressDecision`, returns the tunnel-ready response only after the connection is allowed, and then hands the tunnel to the relay. The upstream connection is opened by the HTTP relay or protocol -processor when payload policy allows it. +processor when payload policy allows it. The compatibility milestone preserves +the current raw-relay dial point until the processor boundary exists. Forward HTTP is compatibility for clients that send absolute-form HTTP requests. The adapter parses the first request, rewrites proxy framing only at @@ -296,16 +319,16 @@ unsupported h2c upgrades on inspected routes, and either stays in a shared HTTP request loop or forces `Connection: close` for a guarded single request. Transparent TCP is for native clients that do not know they are using a proxy. -It depends on policy DNS and nftables capture: DNS answers create active -endpoint mappings, userland later calls `connect(ip:port)`, nftables redirects -matching traffic to a userland listener, and the TCP adapter recovers the -original destination before building an intent. +It depends on policy DNS and nftables capture: DNS answers create correlated +active endpoint mappings, userland later calls `connect(ip:port)`, nftables +redirects matching traffic to a userland listener, and the TCP adapter recovers +the original destination before building an intent. Policy DNS replaces static `/etc/hosts` snapshots for native TCP names. It is query-driven: check whether the name is policy-eligible, resolve through trusted DNS, filter returned IPs, publish the active endpoint mapping, and answer userland. The later `connect(ip:port)` still runs through normal -authorization. +authorization and must correlate with the mapping created by that DNS answer. Local service adapters stay outside the normal external egress relay: `inference.local` routes chat, completion, model discovery, embeddings, and @@ -318,22 +341,23 @@ credentials to SDKs that bypass HTTP proxy variables. Current main uses nftables for sandbox bypass enforcement. It accepts proxy-bound traffic, loopback, and established flows, then rejects and -optionally logs other TCP/UDP traffic for the bypass monitor. That is +optionally logs other TCP/UDP traffic for the bypass monitor. That is current enforcement, not native TCP capture. ```mermaid flowchart TD Packet["Userland packet"] --> ProxyDest{"Proxy destination?"} ProxyDest -- Yes --> AcceptProxy["nftables accept"] - ProxyDest -- No --> Capture{"Future native TCP capture match?"} + ProxyDest -- No --> Capture{"Active policy-DNS capture match?"} Capture -- Yes --> Redirect["nftables redirect/TPROXY to transparent adapter"] Capture -- No --> Reject["nftables log + reject bypass"] Reject --> Monitor["Bypass monitor emits OCSF"] ``` -Transparent TCP work should extend this nftables model with explicit capture -rules that run before the reject path and are scoped to active policy DNS -mappings. It should not add a parallel iptables path. +Transparent TCP extends this nftables model with explicit capture rules that +run before the reject path and are scoped to unexpired, policy-correlated DNS +mappings. It does not add a parallel iptables path. The compatibility milestone +leaves the current table unchanged; capture arrives in a later feature phase. ### Deployment Modes @@ -347,58 +371,92 @@ mappings. It should not add a parallel iptables path. A pluggable proxy must expose the right userland surfaces, implement the gateway APIs it needs, and prove equivalent policy enforcement through tests. If supervisor middleware is configured, the proxy runtime must also receive the -effective middleware service registry, validate/refresh bindings, enforce +effective middleware service registry, validate and refresh bindings, enforce `fail_open` and `fail_closed`, buffer within configured caps, invoke middleware on the request path, and emit middleware OCSF events. -Process identity is mode-dependent. Embedded supervisor mode can usually -resolve the workload process, binary, and ancestors. Network-only, standalone, -and sidecar modes may intentionally have no local process identity. In those -modes the adapter should pass an explicit unavailable identity envelope, and -the decision should record identity as unavailable rather than treating it as -an accidental lookup failure. Authorization must not turn a missing identity -into a broader allow. Process-scoped predicates should either be treated as -non-matching for that runtime or rejected during policy/capability validation. -Policies that require binary/path scoping need an explicit capability check or -fallback rule before they are allowed to run in identity-less modes. - -The nftables rules that force or reject userland traffic belong to the sandbox -network boundary even if the proxy process later moves into a standalone binary -or sidecar. +Process identity is mode-dependent. Embedded supervisor mode normally requires +successful workload process, binary, and ancestor resolution; a lookup failure +continues to deny. A trusted runtime can explicitly select the existing +endpoint-only mode, in which identity is recorded as intentionally unavailable +and policy evaluation keeps its current endpoint-only semantics. The refactor +must not represent either case as a fabricated empty identity or accidentally +convert a lookup failure into endpoint-only evaluation. + +Future standalone and sidecar modes must advertise identity capability. A mode +without local identity needs an explicit unavailable-identity contract and +policy validation for binary/path predicates; it does not automatically inherit +endpoint-only semantics. The nftables rules that force, capture, or reject +userland traffic remain owned by the sandbox network boundary even if the proxy +process later moves into a standalone binary or sidecar. + +### Migration And Operational Contract + +Mechanical adapter, destination, and relay extractions ship as isolated, +revertible changes without a permanent feature flag. The deterministic +authorization result is different: it first runs beside the legacy queries in +shadow mode, reports audit-safe mismatches through internal telemetry, and +retains the legacy evaluator through the cutover observation window. + +Policy DNS, transparent TCP, native processors, and alternate deployment modes +land only after the shared authorization and relay contracts are authoritative. +Each is a separate, feature-bearing pull request or series with its own +capability gating, migration, telemetry, and rollback plan; they are not bundled +into the compatibility-only refactor. + +Adapter response bytes and OCSF event class, action, disposition, severity, +status, destination, actor, firewall rule, message, and status detail are +compatibility surfaces. Moving code does not justify changing them. Performance +is also measured at each phase; fewer OPA evaluations are a target to verify, +not an unmeasured claim. ## Implementation plan The detailed migration plan lives in [implementation-plan.md](implementation-plan.md). The intended order is: -1. Add regression coverage around the current split, forward HTTP invariants, - endpoint selection, supervisor middleware, token grants, WebSocket/body - rewrite, metadata loopback, and nftables bypass enforcement. -2. Introduce `EgressIntent` and `EgressDecision` inside - `openshell-supervisor-network`. -3. Move destination validation and endpoint metadata materialization behind the - shared decision and connector boundary. -4. Consolidate forward HTTP, CONNECT HTTP inspection, supervisor middleware, - credential injection, request-body rewrite, JSON-RPC/MCP inspection, and - WebSocket handling behind shared HTTP/WebSocket relay code. -5. Move TLS detection and termination ahead of the HTTP/TCP relay split. -6. Add the TCP relay/protocol processor boundary, then policy DNS and native - TCP capture. -7. Treat local services and deployment modes as explicit runtime contracts. +1. Lock down current responses, OCSF events, policy outcomes, credential + behavior, upstream-dial timing, and local-service behavior with regression + coverage. +2. Introduce compatibility `EgressIntent` and `EgressDecision` envelopes while + preserving current lookup precedence and failure defaults. +3. Centralize destination validation behind an unopened connector. +4. Materialize one generation-consistent authorization decision, compare it + against the legacy queries, then cut over deterministic endpoint selection + and fail-closed metadata handling as an independently reviewable step. +5. Consolidate HTTP request-loop, credential, WebSocket, and middleware relay + behavior in separately shippable subphases. +6. Consolidate TLS handling and existing raw TCP relay selection. +7. Preserve current local-service boundaries and remove compatibility plumbing. +8. Add the native protocol-processor dispatch contract, then add protocol + implementations as independently reviewed features. +9. Add policy DNS state and transparent TCP capture with mandatory + DNS-answer-to-connect correlation and separate mapping generations. +10. Define capability-checked standalone or sidecar runtime contracts and + complete cleanup after each boundary is in use. + +Steps 1 through 7 are the compatibility foundation and may themselves span +several pull requests. Steps 8 through 10 are later feature milestones; their +presence in this RFC defines the direction without adding them to the initial +refactor branch. ## Risks -- Tightening endpoint metadata failures from fail-open to deny may expose - latent policy or Rego errors. -- Deterministic endpoint selection may reject policies that currently load but - only work by accident. +- Tightening L7 and TLS metadata failures from fail-open to deny may expose + latent policy or Rego errors. `allowed_ips` and SSRF validation already fail + more conservatively; tests must cover each query independently. +- Deterministic endpoint selection may change ambiguous overlapping policies. + The new decision must shadow the legacy queries and report mismatches before + any semantic cutover. - Token grants add a runtime dependency on SPIFFE Workload API and token endpoints. Failures should remain fail-closed and sanitized. -- Transparent TCP capture adds network namespace interception complexity and - must coexist with the nftables bypass reject/log table. -- Sidecar mode may intentionally lack process identity. Binary/path scoped - policy needs a reliable identity source or must be rejected/ignored for that - deployment mode. +- Transparent TCP capture adds network-namespace interception and mutable DNS + mapping state. It must coexist with nftables reject/log enforcement, prevent + unrelated bare-IP connections from inheriting DNS authorization, and fail + closed across policy or mapping generation changes. +- Sidecar or standalone modes may intentionally lack process identity. + Binary/path-scoped policy needs an advertised identity capability and policy + validation; missing identity cannot silently broaden an allow. - Metadata loopback and `policy.local` expand sandbox-local control surfaces and need strict route validation, body limits, redaction, and authentication boundaries. @@ -408,6 +466,16 @@ The intended order is: - Supervisor middleware adds a synchronous request-path dependency. Body caps, timeout behavior, registry reloads, and `fail_open` choices must be visible in telemetry so operators can diagnose whether content inspection ran. +- Moving OCSF emission sites can accidentally change event class, action, + disposition, severity, message, or actor/destination fields. Adapter response + shapes and OCSF schemas are compatibility requirements, not cleanup targets. +- New decision/context objects add per-connection allocations on a hot path. + Performance claims require before/after measurements of OPA evaluation count, + allocation volume, and connection/request latency. +- Structural phases back out by reverting their isolated commits. The + deterministic-decision cutover must retain the legacy evaluator long enough + for shadow comparison and immediate rollback; a permanent feature flag is not + required for the purely mechanical phases. ## Alternatives @@ -427,8 +495,10 @@ CONNECT should remain the generic explicit proxy mode. ### Build only transparent TCP Transparent TCP helps native clients but does not replace explicit proxy -support used by common HTTP tooling. It also requires policy DNS and nftables -capture before it can safely preserve endpoint identity. +support used by common HTTP tooling. It also requires the shared authorization, +destination, and relay boundaries plus policy DNS and nftables capture before +it can safely preserve endpoint identity. For that reason it is a later phase +of this RFC, not the first implementation change. ## Prior art @@ -437,8 +507,8 @@ it already separates proxy, OPA, L7, inference routing, policy-local routes, TLS, and token grants from process supervision. The current `openshell-supervisor-process` netns and bypass monitor are the -packet-enforcement substrate. Transparent TCP should extend that nftables -model rather than creating a second firewall path. +packet-enforcement substrate. Transparent TCP extends that nftables model in a +later phase rather than creating a second firewall path. The existing L7 relay is the behavioral prior art for this RFC. It already proves per-request HTTP evaluation, GraphQL parsing, JSON-RPC/MCP body @@ -454,13 +524,17 @@ adapter wire middleware separately. ## Open questions 1. Should overlapping endpoint metadata be rejected at policy load time, or - should policy name plus endpoint index define precedence? -2. Should direct IP connects to a policy-DNS-resolved TCP endpoint be accepted, - or should DNS query correlation be required for stricter modes? -3. What TTL cap and stale-generation grace period should policy DNS use? -4. Which policy features should be disabled, rejected, or treated as - non-matching when the proxy runtime advertises no process identity support? -5. Which proxy capabilities should be negotiated with the gateway at startup? -6. Should metadata loopback be modeled as an adapter inside + should one documented policy/endpoint precedence key select the complete + decision? The initial compatibility refactor does not choose between them. +2. What mismatch-free observation window is sufficient before the + deterministic decision replaces the legacy endpoint queries? +3. Should metadata loopback be modeled as an adapter inside `openshell-supervisor-network`, or remain orchestrated by `openshell-sandbox` with shared credential/provider helpers? +4. What TTL cap should policy DNS use, and should policy reload immediately + invalidate all active mappings or permit a bounded drain period that cannot + authorize new connections? +5. Which original-destination mechanism and nftables redirect mode should each + supported runtime use while keeping capture rules ahead of bypass rejection? +6. Which identity capabilities must standalone and sidecar runtimes advertise + before the gateway accepts binary/path-scoped policy for them? diff --git a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md index 67506117b1..0a12c6d957 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/current-shape.md @@ -39,12 +39,12 @@ routes, identity cache, provider credential injection, and token grants. helpers, nftables bypass rules, and the bypass monitor that turns nftables LOG entries into OCSF events. -In embedded supervisor mode, the network leaf can usually use process metadata +In embedded supervisor mode, the network leaf normally uses process metadata resolved by the process/orchestrator side for binary-scoped policy and OCSF -context. Future network-only, standalone, or sidecar proxy modes may -intentionally lack that metadata. The adapter model should preserve this as an -explicit "identity unavailable" state rather than fabricating an empty binary -identity. +context. Current runtime configuration can instead select endpoint-only policy +evaluation. The adapter model must preserve that intentional mode separately +from an identity lookup failure; the latter continues to deny when binary +identity is required. ## Current Userland-Facing Surfaces @@ -204,19 +204,37 @@ by passing the first parsed request into a shared HTTP relay loop. ### Endpoint config is not tied to deterministic matched policy -The policy name used for L4 authorization and logging can be selected through a -different precedence rule than endpoint metadata. With overlapping host, port, -and binary rules, allowed IPs, TLS behavior, enforcement, and -`allow_encoded_slash` can come from a different endpoint than the policy name -logged and used for L4 allow. +The policy name used for L4 authorization and logging is the lexicographically +smallest matching policy. L7 candidates are collected independently and later +selected by request-path specificity. TLS and `allowed_ips` use the first +extended endpoint config returned by a separate query. Exact-declared-host is +another independent existential query. With overlapping host, port, and binary +rules, those results can describe different endpoints and policies on the same +connection. The adapter model requires authorization to return one decision with one deterministic matched endpoint. +### Policy materialization can span generations + +The L4 decision, L7 route, TLS mode, `allowed_ips`, and exact-declared-host +signal are materialized through separate engine calls. The L7 route records a +generation for its tunnel evaluator, but the sibling metadata queries are not +all asserted against the L4 decision generation. A reload during setup can +therefore assemble one logical connection decision from different policy +generations. + +The target decision carries one top-level policy generation. Every +policy-derived field must be evaluated from that generation, and relay startup +must reject a stale decision before an upstream request is written. + ### Endpoint metadata query failures should not erase enforcement -If endpoint metadata lookup fails, callers can interpret the result as no L7 -configuration and downgrade to credential-only or raw L4 relay. +Failure behavior is not uniform today. L7 configuration failure becomes no L7 +configuration, and TLS configuration failure becomes automatic TLS handling; +either can erase intended enforcement. `allowed_ips` and the downstream SSRF +validation path are already more conservative. The migration must test each +query independently instead of describing all metadata failures as equivalent. The adapter model treats endpoint metadata as part of the authorization result. Failure to materialize required metadata should deny rather than erase extended @@ -226,9 +244,9 @@ configuration. Private address checks, `allowed_ips`, exact declared private endpoint trust, trusted gateway aliases, SSRF checks, and control-plane port blocks have grown -over time. They should be centralized so CONNECT, forward HTTP, future -transparent TCP, and local-service egress use the same resolved-destination -rules. +over time. They should be centralized so CONNECT and forward HTTP use the same +resolved-destination rules. Existing local services remain outside normal +external destination validation. ## Existing Feature Inventory diff --git a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md index e4a15ee578..20a685c42e 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/implementation-plan.md @@ -1,203 +1,258 @@ # Implementation Plan This plan is intentionally separate from the main RFC so the proposal can stay -direction-focused. - -## Phase 0 - Regression Tests - -- Add tests for forward HTTP pipelining and keep-alive follow-on requests, - including the current `Connection: close` mitigation. -- Add tests for forward HTTP h2c rejection on inspected endpoints. -- Add tests for overlapping endpoint metadata selection. -- Add tests for endpoint metadata query failures. -- Add tests for control-plane port blocking through all destination validation - paths. -- Add tests for exact declared private endpoint trust and `allowed_ips` - behavior across CONNECT and forward HTTP. -- Add tests for identity-less runtime modes where process identity is - intentionally unavailable and binary/path scoped policy does not - accidentally match. -- Add tests proving static credential injection works in L4-only HTTP and - HTTP-inspected paths. -- Add tests proving token grant success injects the configured header and token - grant failure does not forward upstream. -- Add tests proving supervisor middleware runs after request policy and before - credential injection, including allow, deny, mutation, `fail_open`, - `fail_closed`, and body-cap behavior. -- Add tests for REST request-body credential rewrite, WebSocket text-frame - credential rewrite, WebSocket GraphQL policy, and compression handling. -- Add tests for JSON-RPC method policy, batch behavior, response-frame denial, - and MCP method/tool policy. -- Add tests for `policy.local` proposal wait behavior and `inference.local` - buffered/streaming route limits. -- Add tests for metadata loopback startup/failure behavior when provider - credentials require it. -- Add nftables bypass enforcement tests that verify proxy-bound traffic is - accepted while direct TCP/UDP egress is rejected and logged when available. - -## Phase 1 - Authorization Result - -- Introduce `EgressIntent` and `EgressDecision` inside +direction-focused. The RFC is an incremental roadmap, not one pull request. +Phases 0 through 7 form the compatibility foundation: they restructure current +CONNECT, forward HTTP, raw TCP, and local-service behavior without adding a new +user-facing transport. Phases 8 and later add the forward-looking capabilities +after the shared contracts are authoritative. + +## Phase 0 - Compatibility Baseline + +- Cover CONNECT and forward HTTP allow/deny responses, including exact status, + headers, and adapter-specific error bodies. +- Cover forward HTTP pipelining, keep-alive follow-on requests, the current + `Connection: close` mitigation, `https://` absolute-form rejection, and h2c + rejection on inspected endpoints. +- Cover the current overlapping-policy outcomes separately for matched policy, + L7 route selection, TLS, `allowed_ips`, and exact-declared-host. +- Inject failures into L7, TLS, `allowed_ips`, and exact-declared-host queries. + Record the current fail-open or fail-closed result for each query rather than + treating all endpoint metadata errors as equivalent. +- Cover control-plane ports, cloud metadata, always-blocked addresses, exact + declared private endpoints, IP-literal synthesis, trusted gateway aliases, + and explicit `allowed_ips` through CONNECT and forward HTTP. +- Cover identity-required success/failure, unsupported-platform behavior where + possible, and intentional endpoint-only evaluation. Prove an empty + `exec.path` cannot satisfy binary-scoped policy while identity is required. +- Cover static credential injection, token grants, REST body rewrite, + WebSocket text-frame rewrite and policy, GraphQL, JSON-RPC, and MCP behavior. +- Cover `inference.local`, `policy.local`, metadata loopback, and unchanged + nftables bypass reject/log behavior. +- Capture stable OCSF event class, activity/action/disposition, severity, + status, destination, actor, firewall rule, message, and status detail for + representative allow and deny paths. +- Record a performance baseline for OPA evaluations, per-connection + allocations, and CONNECT/forward request latency. + +## Phase 1 - Adapters And Compatibility Decision Envelope + +- Introduce CONNECT and forward HTTP `EgressIntent` construction inside `openshell-supervisor-network`. -- Make authorization return matched policy and matched endpoint metadata - together. -- Include policy source on the decision: user-authored, provider-derived, or - local-service internal. -- Include protocol enforcement, supervisor middleware, and credential injection - plans on the decision. -- Include process identity availability and fields used on the decision. Treat - missing process identity as an explicit runtime mode, not as an implicit - lookup failure. -- Fail closed when required endpoint metadata cannot be materialized. -- Emit consistent OCSF network denial events from the shared boundary. +- Introduce a transitional `EgressDecision` carrying L4 outcome, policy + generation, process evidence, and endpoint fields while preserving the + current query timing, precedence, and failure defaults. +- Keep `LookupFailed`/unsupported identity as a denial when identity is + required. Keep explicitly configured endpoint-only mode behavior unchanged. +- Keep adapter-specific responses and OCSF emission at the protocol boundary. +- Do not claim the transitional decision is one atomic OPA result; document its + compatibility hydration until Phase 3 cuts over. + +This phase is a mechanical extraction. It must be independently shippable and +revertible without changing user-visible policy or relay behavior. ## Phase 2 - Shared Destination Validation -- Move DNS resolution, allowed IP filtering, SSRF checks, exact declared - endpoint handling, trusted gateway aliases, and control-plane port checks - into one destination validation path. -- Return an `UpstreamConnector` rather than an opened upstream socket. -- Add tests proving CONNECT, forward HTTP, and future transparent TCP use the - same validation behavior. - -## Phase 3 - Forward HTTP Adapter - -- Convert forward HTTP into an adapter that parses the first absolute-form - request and builds an egress intent. -- Route the parsed first request into the shared HTTP relay or preserve the - current guarded single-request relay behavior. -- Preserve `https://` absolute-form rejection. -- Preserve h2c rejection on inspected routes. -- Keep the no-raw-copy invariant after the first request. - -## Phase 4 - HTTP, WebSocket, Middleware, And Credential Relay Consolidation - -- Centralize HTTP request parsing, REST policy, GraphQL policy, WebSocket - upgrade policy, JSON-RPC/MCP policy, supervisor middleware, credential - resolution, redaction, request rewrite, upstream dial, and response relay. -- Evaluate every HTTP request before upstream write. -- Ensure denied HTTP requests do not create upstream TCP sessions. -- Run `HTTP_REQUEST / PRE_CREDENTIALS` middleware after request allow and - before static or dynamic credential injection. -- Preserve middleware ordering, body caps, failure policy, safe header - mutation, findings, and metadata emission. -- Reject or strip newly introduced reserved credential placeholders from - middleware-transformed content unless a future hook is explicitly - credential-capable. -- Preserve static placeholder rewrite for target, query, and headers. -- Preserve dynamic token grant injection after request allow and before - upstream write. -- Preserve opt-in REST request-body credential rewrite behind the shared HTTP - relay, including bounded buffering, supported content-type handling, - `Content-Length` recomputation, and fail-closed unresolved placeholders. -- Preserve WebSocket upgrade handling behind the shared relay, including - opt-in client-to-server text-frame credential rewrite, WebSocket transport - message policy, GraphQL-over-WebSocket policy, and raw passthrough for other - upgraded protocols. -- Preserve JSON-RPC and MCP handling behind the shared HTTP relay, including - bounded body inspection, JSON-RPC batch evaluation, MCP `tools/call` tool - selectors, and audit-safe logging that omits params and tool arguments. - -## Phase 5 - Shared TLS Termination - -- Move client-side TLS detection and termination before the HTTP/TCP relay - split. -- Keep endpoint TLS behavior on `EgressDecision`. -- Treat `tls: skip` as the explicit opt-out for TLS handling. -- Remove duplicate HTTP-specific and TCP-specific TLS termination decisions. - -## Phase 6 - TCP Relay And Protocol Processor Boundary - -- Use `TcpRelay` for byte relay and native protocol processor dispatch. -- Keep `protocol: tcp` or omitted protocol as L4 authorization plus byte copy. -- Add a native protocol processor dispatch point for future protocol - enforcement. -- Let protocol processors own their message loop and call the connector - when protocol state allows. -- Allow processors to expose typed middleware hooks instead of requiring all - payload logic to live in-tree. - -## Phase 7 - Policy DNS And Transparent TCP +- Move DNS resolution, explicit `allowed_ips`, exact declared endpoints, + implicit IP-literal handling, trusted gateway aliases, SSRF checks, + cloud-metadata blocks, and control-plane-port blocks into one validator. +- Represent the selected validation mode explicitly instead of passing an + ambiguous collection of booleans. +- Return an unopened `UpstreamConnector` so adapters and relays preserve the + current point at which upstream TCP is created. +- Prove CONNECT and forward HTTP retain their existing denial responses, OCSF + fields, and dial timing while using the shared validator. + +## Phase 3 - Generation-Consistent Authorization Cutover + +- Define either rejection of ambiguous overlapping endpoint metadata or one + documented policy/endpoint precedence key before changing enforcement. +- Add one OPA result that materializes matched policy/source, matched endpoint, + destination constraints, TLS, HTTP enforcement, credential plan, and + middleware selection from one policy generation. +- Attach a generation-pinned `TunnelPolicyEngine` to relay context for + per-request REST, GraphQL, JSON-RPC, MCP, and WebSocket evaluation. Relays do + not rematerialize connection-level endpoint policy. +- Run the new query in shadow mode beside legacy queries. Emit internal, + audit-safe mismatch telemetry without changing existing OCSF network/HTTP + events or enforcement. +- Add reload-race tests proving every materialized field matches the top-level + generation and stale decisions stop before upstream request write. +- Make deterministic selection and fail-closed L7/TLS metadata errors a + dedicated cutover only after mismatch cases are understood. Retain the + legacy evaluator temporarily for immediate rollback. + +This is the only phase that intentionally tightens ambiguous or error behavior; +it must not be hidden inside the structural refactor commits. + +## Phase 4 - Forward HTTP Adapter + +- Keep absolute-form parsing and adapter-specific errors at the forward HTTP + boundary. +- Pass the buffered first request into a shared HTTP relay, or retain the + guarded single-request/`Connection: close` path until Phase 5a is ready. +- Preserve `https://` absolute-form rejection and inspected h2c rejection. +- Preserve the invariant that no unevaluated follow-on request can reach raw + bidirectional copy. + +## Phase 5 - Relay Consolidation + +### Phase 5a - HTTP request loop + +- Centralize HTTP parsing and per-request REST, GraphQL, JSON-RPC, and MCP + evaluation behind the generation-pinned request-policy handle. +- Evaluate every request before upstream write and preserve the current rule + that a denied request does not create an upstream session. +- Preserve bounded JSON-RPC/MCP inspection and audit-safe logging that omits + params and tool arguments. + +### Phase 5b - Credential injection + +- Unify static target/query/header rewrite, endpoint-bound token grants, and + opt-in REST request-body rewrite after request allow and before upstream + write. +- Preserve buffering limits, supported content types, `Content-Length` + recomputation, redaction, token caching, and fail-closed unresolved secrets. + +### Phase 5c - WebSocket + +- Move allowed upgrades behind the shared relay while preserving raw upgraded + passthrough, opt-in text-frame credential rewrite, WebSocket transport policy, + GraphQL-over-WebSocket policy, and safe compression behavior. + +### Phase 5d - Supervisor middleware + +- Land only after the supervisor middleware dependency is available. +- Run `HTTP_REQUEST / PRE_CREDENTIALS` after request allow and before static or + dynamic credential injection. +- Preserve ordering, body caps, `fail_open`/`fail_closed`, safe headers, + findings, metadata, and rejection of middleware-introduced credential + placeholders. + +Each subphase must be independently testable and shippable; Phase 5 is not a +single flag-day cutover. + +## Phase 6 - Shared TLS And TCP Relay Boundary + +- Move client-side TLS detection and termination before the HTTP/raw-TCP relay + split without changing handshake, certificate, or upstream-connect timing. +- Keep endpoint TLS behavior on `EgressDecision` and preserve `tls: skip` as the + explicit raw-tunnel path. +- Use one existing raw `TcpRelay` byte-copy primitive for L4 traffic. +- Add a protocol-processor dispatch contract without enabling a concrete new + protocol in the compatibility milestone. +- Let processors own their message loop and call the validated connector only + when protocol state allows. Permit in-tree, middleware-backed, and hybrid + processors with typed middleware operations. + +## Phase 7 - Existing Local Services And Cleanup + +- Keep `inference.local` as a local adapter with its existing TLS, route, + provider-auth, streaming/buffered limit, and OCSF behavior. +- Keep `policy.local` as a local adapter for current policy, bounded denial + summaries, proposals, and proposal wait. +- Decide whether metadata loopback remains orchestrated by `openshell-sandbox` + or moves behind a local adapter boundary; preserve startup/failure behavior + either way. +- Keep the local-routing and destination contracts extensible for issue + [#1633](https://github.com/NVIDIA/OpenShell/issues/1633), while leaving its + policy surface and host-loopback authorization to separate feature work. +- Remove compatibility endpoint queries only after Phase 3 is authoritative. +- Remove duplicated destination/relay plumbing without centralizing + adapter-specific response rendering. +- Update the living architecture documentation once each implemented boundary + reflects current code. + +Completion of Phase 7 is the compatibility milestone: existing user-facing +features and capabilities are preserved on the new internal structure. The +following phases are feature-bearing work and land in separate pull requests or +series. + +## Phase 8 - Policy DNS And Transparent TCP - Add policy DNS registration for native TCP endpoint names. -- Replace static host-file mapping with query-driven DNS answers. -- Publish active DNS answer state and capture rules. -- Implement nftables REDIRECT/TPROXY capture rules ahead of the bypass reject - path; do not add a parallel iptables path. -- Coordinate capture rule ownership with `openshell-supervisor-process::netns`. -- Implement transparent TCP adapter lookup from captured original destination - to active endpoint generation. -- Decide TTL and stale-generation behavior. - -## Phase 8 - Local Service Adapters - -- Model `inference.local` as a local adapter with TLS termination, route - validation, provider auth injection, streaming/buffered limits, and OCSF - logging. -- Model `policy.local` as a local adapter for current policy, bounded denial - summaries, policy proposals, and proposal wait. -- Decide whether metadata loopback remains orchestrated in `openshell-sandbox` - or moves behind a local adapter boundary in `openshell-supervisor-network`. -- Keep these paths outside normal external egress relay while preserving - credential redaction and route validation. - -## Phase 9 - Runtime Boundary - -- Keep embedded supervisor mode as the first migration target. -- Treat the existing `openshell-supervisor-network` and - `openshell-supervisor-process` split as the structural baseline. -- Define the proxy runtime API needed for a future standalone binary: - configured listeners, policy updates, provider credentials, token grants, - supervisor middleware registry, gateway calls, telemetry, denial/activity +- Replace static host-file mapping with query-driven DNS answers resolved + through trusted DNS and filtered through destination controls. +- Store normalized name, endpoint ID, IP/port, policy generation, distinct DNS + mapping generation, mapping ID, and expiration in active mapping state. +- Require every captured connect to correlate with the unexpired mapping + created by its policy-eligible DNS answer. Do not allow unrelated bare-IP + traffic to inherit a policy-DNS decision. +- Publish mapping and nftables capture updates atomically from the adapter's + perspective. +- Add nftables REDIRECT/TPROXY capture rules ahead of the bypass reject path; + do not add a parallel iptables path. +- Coordinate capture-rule ownership with + `openshell-supervisor-process::netns` and preserve reject/log fallback for + unmatched traffic. +- Recover the original destination, construct a transparent-TCP intent, and run + normal generation-consistent authorization and destination validation. +- Define TTL caps, policy-reload invalidation, stale-mapping behavior, and + rollback before enabling capture by default. + +## Phase 9 - Native Protocol Processors + +- Add concrete Redis, Postgres, MySQL, or other processors one protocol at a + time, each with a separately reviewed policy schema and operational limits. +- Keep omitted/`tcp` endpoints on raw L4 byte copy; never infer a native + processor from traffic alone. +- Test multi-message sessions, pre-dial denial, handshake-required dialing, + per-command/query evaluation, middleware hooks, timeouts, and redaction. +- Capability-gate policy that names a processor unavailable in the running + proxy build. + +## Phase 10 - Runtime Boundary + +- Keep embedded and network-only supervisor modes as the migration baseline. +- Define the proxy runtime API needed for a future standalone binary or + sidecar: configured listeners, policy updates, provider credentials, token + grants, middleware registry, gateway calls, telemetry, denial/activity events, and shutdown. -- Advertise process identity capability for embedded, network-only, - standalone, and sidecar modes. Reject policies that require unavailable - identity dimensions, or define those predicates as non-matching for the - runtime mode. -- Add capability negotiation with the gateway if standalone proxy versions can - differ from gateway versions. - -## Phase 10 - Cleanup - -- Remove duplicated endpoint metadata queries from relay paths. -- Remove duplicated destination validation and deny rendering where adapters - can own response shape. -- Remove any remaining forward HTTP raw-copy fallback. -- Remove stale references to iptables or static `/etc/hosts` native TCP - mapping from proxy design docs. -- Update architecture docs once implementation lands. - -## Testing Plan - -- Unit-test each adapter's intent construction and deny response shape. -- Unit-test authorization precedence for overlapping policy and endpoint rules. -- Unit-test provider-derived rule namespace handling and `policy.local` - filtering. -- Unit-test identity-available and identity-unavailable authorization inputs. -- Integration-test shared destination validation across CONNECT, forward HTTP, - and transparent TCP. -- Integration-test HTTP keep-alive and pipelined requests with REST, GraphQL, - and WebSocket upgrade enforcement. -- Integration-test credential injection in L4-only HTTP and HTTP-inspected - paths. -- Integration-test token grant success, cache hit, malformed token, resolver - unavailable, and token endpoint failure. -- Integration-test supervisor middleware allow/deny/mutate, unavailable - service, unresolved binding, body over-capacity, safe header mutation, - finding emission, and no-credential-visible behavior. -- Integration-test REST request-body credential rewrite for JSON, - form-url-encoded, `text/*`, unsupported content types, chunked framing, body - caps, and unresolved placeholders. -- Integration-test WebSocket text-frame credential rewrite, raw upgraded - passthrough, WebSocket message policy, GraphQL-over-WebSocket policy, and - safe compression negotiation. -- Integration-test JSON-RPC method allow/deny, batch denial, response-frame - handling, MCP method profile behavior, and MCP tool selector enforcement. -- Integration-test TLS termination before HTTP/TCP relay split. -- Integration-test `protocol: tcp` byte-copy behavior. -- Add protocol processor harness tests before adding Redis, Postgres, or - similar native protocol enforcement. -- Integration-test policy DNS TTL, stale generation handling, and captured - connect correlation. -- Integration-test `inference.local`, `policy.local`, and metadata loopback - body limits, timeout behavior, redaction, and local denial responses. +- Advertise process-identity and protocol-processor capabilities. Reject policy + that requires unavailable binary/path identity or processor support. +- Represent intentional runtime identity unavailability separately from the + existing endpoint-only mode and from lookup failure. +- Add gateway capability negotiation if proxy and gateway versions can differ. + +## Phase 11 - Final Cleanup + +- Remove any compatibility query/evaluator retained for deterministic-decision + rollback after its observation window closes. +- Remove stale static `/etc/hosts`, iptables, or single-process assumptions from + proxy design and architecture documentation as the corresponding later phase + lands. +- Keep adapter-specific response rendering and OCSF contracts at their protocol + boundaries. + +## Testing And Operational Validation + +- Unit-test adapter intent construction, response rendering, explicit + destination modes, identity evidence, and authorization precedence. +- Integration-test destination validation across CONNECT and forward HTTP, + then reuse the same suite for transparent TCP when Phase 8 lands. +- Integration-test HTTP keep-alive/pipelining, REST, GraphQL, JSON-RPC, MCP, + WebSocket, credentials, token grants, middleware, and TLS/raw-TCP selection. +- Integration-test `inference.local`, `policy.local`, and metadata loopback body + limits, timeouts, redaction, and local denial responses. +- Compare OCSF fixtures before and after each migration subphase. +- Exercise policy reload between L4 decision, endpoint materialization, relay + startup, and long-lived per-request evaluation. +- Add protocol-processor harness tests before adding Redis, Postgres, MySQL, or + similar enforcement. Each concrete processor adds multi-message, handshake, + timeout, denial, redaction, and middleware coverage. +- Integration-test policy DNS filtering, TTL, distinct mapping generations, + policy-reload invalidation, atomic capture-rule updates, original-destination + recovery, captured-connect correlation, and rejection of unrelated bare-IP + connects. +- Test standalone/sidecar capability negotiation and prove missing identity or + processor support fails during policy validation rather than broadening an + allow at runtime. +- Re-run the performance baseline after the compatibility envelope, after the + single-decision query, and after relay consolidation. Treat reduced OPA calls + as a measured result rather than an assumed benefit. +- Back out structural phases by reverting their isolated commits. Keep shadow + comparison and the legacy evaluator available through the deterministic + cutover observation window. +- Gate later transport/runtime phases independently so disabling policy DNS or + transparent capture restores the existing explicit-proxy and bypass-reject + behavior without reverting the compatibility foundation. diff --git a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md index 09032e1213..b03d07b36a 100644 --- a/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md +++ b/rfc/0005-sandbox-proxy-egress-adapter/technical-design.md @@ -14,7 +14,9 @@ middleware registry construction and reload behavior. This is a useful outer boundary, but it is not yet the proxy adapter boundary. The proxy still needs internal `EgressIntent` and `EgressDecision` boundaries so CONNECT, forward HTTP, local routes, and future native TCP capture do not -duplicate policy and relay orchestration. +duplicate policy and relay orchestration. The first implementation milestone +wires only current surfaces; later milestones add new adapters to the same +contract. ## Shared Data Boundaries @@ -30,7 +32,8 @@ It should carry: - optional process identity inputs collected by the adapter/runtime; - optional first HTTP request for forward proxy traffic; - optional local service route; -- policy generation or DNS mapping generation when relevant. +- policy generation and, for policy DNS/transparent TCP, a distinct DNS + mapping generation and correlation handle. Adapters build intents. They should not query endpoint metadata, select TLS mode, or select relays. @@ -42,6 +45,7 @@ mode, or select relays. It should carry: - allow or deny; +- one top-level policy generation used for every policy-derived field; - deterministic matched policy identifier; - whether the policy is user-authored, provider-derived, or local-service internal; @@ -52,11 +56,17 @@ It should carry: - protocol enforcement; - credential injection plan; - supervisor middleware plan; +- the request-policy selection needed to create a pinned per-request L7 + evaluator when HTTP inspection is configured; - logging context and denial reason. Relay code should read this decision. It should not query OPA again for endpoint metadata, TLS mode, allowed IPs, credential behavior, middleware -selection, or processor selection. +selection, or relay selection. Long-lived HTTP relays still evaluate each +request through the generation-pinned L7 evaluator carried in `RelayContext`; +that is request authorization, not endpoint rematerialization. Later native +protocol processors use the same pattern with a generation-pinned protocol +evaluator for per-command or per-query decisions. ## Protocol Enforcement @@ -75,7 +85,7 @@ Use a protocol enforcement value derived from endpoint policy: `protocol: tcp` is effectively the default L4 mode. It should not run native protocol processors. Avoid using the term "provider" for processor concepts because providers are already a first-class credential and routing domain in -OpenShell. +OpenShell. Concrete native processors land after the shared dispatch contract. ## Suggested Types @@ -97,10 +107,11 @@ struct EgressIntent { process: ProcessIdentityEvidence, first_request: Option, local_route: Option, - generation: Option, + correlation: Option, } struct EgressDecision { + policy_generation: PolicyGeneration, outcome: PolicyOutcome, matched_policy: Option, endpoint: Option, @@ -115,8 +126,9 @@ enum ProcessIdentityEvidence { } enum ProcessIdentityUnavailableReason { - RuntimeMode, - UnsupportedAdapter, + EndpointOnlyMode, + DeclaredRuntimeMode(RuntimeMode), + UnsupportedPlatform, LookupFailed, } @@ -138,11 +150,23 @@ enum PolicySource { struct MatchedEndpoint { id: EndpointId, - allowed_ips: AllowedIpPolicy, + destination: DestinationValidationPlan, tls: TlsPolicy, enforcement: ProtocolEnforcement, } +struct DestinationValidationPlan { + address_authorization: AddressAuthorization, +} + +enum AddressAuthorization { + DefaultPublicOnly, + ExplicitAllowedIps(Vec), + ExactDeclaredHost, + ImplicitIpLiteral(IpAddr), + TrustedGatewayAlias { expected_ip: IpAddr }, +} + struct RequestProcessingPlan { middleware: SupervisorMiddlewarePlan, credentials: CredentialInjectionPlan, @@ -220,36 +244,70 @@ enum MiddlewarePhase { struct RelayContext { decision: EgressDecision, + request_policy: Option, + protocol_policy: Option, connector: UpstreamConnector, deadlines: RelayDeadlines, telemetry: RelayTelemetry, } + +struct ResolvedEndpointCorrelation { + policy_generation: PolicyGeneration, + mapping_generation: DnsMappingGeneration, + mapping_id: DnsMappingId, +} + +struct PinnedRequestPolicy { + generation: PolicyGeneration, + evaluator: TunnelPolicyEngine, +} + +struct PinnedProtocolPolicy { + generation: PolicyGeneration, + evaluator: ProtocolPolicyEngine, +} ``` `UpstreamConnector` is the relay-owned dial boundary. It encapsulates the -validated destination and lets relays/processors open an upstream connection -only after protocol policy allows it. +validated destination and lets relays or processors open an upstream connection +only after current request or protocol policy allows it. + +`DestinationValidationPlan` selects one current validation mode. All modes +retain control-plane-port and cloud-metadata blocks. Default and explicit IP +paths retain the always-blocked loopback, link-local, and unspecified-address +checks. `ImplicitIpLiteral` is synthesized only for an explicitly declared IP +host. `TrustedGatewayAlias` may accept the one runtime-discovered gateway IP but +does not become a general private-address exemption. + +`policy_generation`, the optional pinned request/protocol evaluators, endpoint +metadata, and middleware selection must describe one policy snapshot. +Authorization asserts that every sub-materialization used that generation. If +the generation changes before relay startup, the adapter receives a +stale-policy denial rather than a mixed decision. ## Process Identity Availability -Process identity is evidence, not an always-present input. Embedded supervisor -mode can usually populate binary, PID, ancestry, command-line path, and binary -hash data. Network-only, standalone binary, or sidecar proxy modes may -intentionally have no local process metadata. That should be represented as -`ProcessIdentityEvidence::Unavailable(RuntimeMode)`, not as an empty string -that accidentally matches policy. - -Authorization should evaluate only identity dimensions that are available. If -no identity is available, binary/path scoped policy should either be skipped as -non-matching or rejected at policy/capability validation time, depending on the -runtime mode contract. The important invariant is that missing identity must -not produce a synthetic binary, broaden a binary-scoped allow rule, or cause -the relay to query process metadata later. During `EgressDecision` hydration, -absent process evidence should be carried forward as unavailable identity and -process-derived fields should be omitted from the hydrated decision. The -decision should record identity availability and fields used so OCSF logs and -deny responses can distinguish "policy denied this binary" from "this runtime -did not provide process identity." +Process identity is evidence, not a string to fabricate when lookup fails. +Embedded mode normally populates binary, PID, ancestry, command-line path, and +binary hash data. When binary identity is required, `LookupFailed` and +`UnsupportedPlatform` remain denials. An explicitly configured endpoint-only +runtime records `Unavailable(EndpointOnlyMode)` and keeps the current +endpoint-only policy behavior. This RFC does not silently turn one state into +the other or change the endpoint-only trust contract. + +A future standalone or sidecar runtime that intentionally lacks local identity +uses `DeclaredRuntimeMode`, not `EndpointOnlyMode`, and advertises that +capability to policy validation. The runtime contract must define binary/path +predicates as unavailable and reject incompatible policy before traffic starts, +unless a later accepted policy design specifies a different fail-closed rule. + +The decision records identity availability and fields used so OCSF logs and +deny responses distinguish a binary policy denial, an identity lookup failure, +and intentional endpoint-only evaluation. Tests must prove an empty synthetic +`exec.path` cannot satisfy a binary-scoped rule while identity is required. +Adding new identity-less deployment modes, or changing how binary predicates +behave in endpoint-only mode, requires the capability work in the later runtime +phase and cannot be smuggled into the compatibility refactor. ## Current Owners And Proposed Cleanup @@ -268,22 +326,32 @@ did not provide process identity." ## Policy DNS And Resolved TCP State -Policy DNS should be query-driven rather than a static `/etc/hosts` snapshot. +Policy DNS is query-driven rather than a static `/etc/hosts` snapshot. 1. Policy load registers eligible native TCP endpoint names. -2. Userland performs DNS lookup. -3. Policy DNS checks whether the name is registered for native TCP. -4. Policy DNS resolves through trusted upstream DNS. -5. Answers are filtered against endpoint metadata and SSRF controls. -6. The adapter publishes the DNS answer, endpoint generation, and capture rule. +2. Userland performs a DNS lookup. +3. Policy DNS checks whether the name is eligible in the current policy + generation. +4. Policy DNS resolves through trusted upstream DNS and filters answers through + endpoint metadata and SSRF controls. +5. The adapter publishes the DNS answer plus an active mapping containing the + normalized name, endpoint identifier, IP/port, policy generation, distinct + DNS mapping generation, opaque mapping ID, and expiration. +6. Capture rules are updated atomically for the active mapping. 7. Userland later calls `connect(ip:port)`. -8. Transparent TCP recovers the original destination and maps it to the active - endpoint generation. -9. Normal egress authorization and relay selection run. - -The resolved endpoint store is therefore not a preemptive global DNS snapshot. -It is active state produced by policy-eligible lookups and consumed by -transparent TCP connects. +8. Transparent TCP recovers the original destination and requires an unexpired + mapping that correlates that connect with the policy-eligible DNS answer. +9. Normal egress authorization and relay selection run against a policy + generation consistent with the mapping contract. + +The resolved endpoint store is active state produced by policy-eligible lookups +and consumed by transparent TCP connects. Policy generation and DNS mapping +generation are separate values: a DNS refresh can replace mappings without a +policy reload, while a policy reload can invalidate mappings whose endpoint +contract is no longer current. A captured connection with no mapping, a stale +mapping, or a mismatched endpoint/port fails closed. An unrelated bare-IP +connection cannot inherit a policy-DNS authorization merely because it targets +an IP currently present in the mapping store. ## nftables Boundary @@ -293,20 +361,27 @@ loopback, and established/related flows, then rejects and optionally logs other TCP/UDP traffic. The bypass monitor reads those log lines and emits OCSF network and detection events. -Transparent TCP capture should build on this same nftables substrate: +Transparent TCP capture builds on this same nftables substrate in a later +feature phase: -- capture rules must run before the generic bypass reject rules; -- capture rules should be scoped to active policy DNS IP/port mappings; -- capture state should be updated atomically with endpoint generation changes; +- capture rules run before the generic bypass reject rules; +- capture rules are scoped to active policy-DNS IP/port mappings; +- mapping and capture-rule updates are atomic from the adapter's perspective; - reject/log rules remain the fallback for unmatched TCP/UDP egress; -- VM or Podman driver nftables rules are infrastructure NAT/isolation and - should not be treated as the proxy policy enforcement point. +- VM or Podman driver nftables rules are infrastructure NAT/isolation and are + not the proxy policy enforcement point. + +The initial CONNECT/forward refactor does not change the installed table. This +section defines the consumer contract that the shared adapter and decision +boundaries must support when transparent capture lands. ## Endpoint Selection And OPA -OPA/Rego should return policy and endpoint metadata through one deterministic -authorization result. It should not let policy name and endpoint config be -selected by different precedence rules. +Today the matched policy name, L7 candidates, first TLS/`allowed_ips` endpoint, +and exact-declared-host signal are selected through independent rules. OPA/Rego +should return policy and endpoint metadata through one deterministic +authorization result. It should not let those fields describe different +matches. Two acceptable approaches: @@ -316,6 +391,16 @@ Two acceptable approaches: Endpoint metadata query failures should fail closed when metadata is required for the selected endpoint. They should not silently downgrade to L4 behavior. +The top-level decision generation must also match every policy-derived field; +reload during materialization yields a stale decision instead of mixing +generations. + +This semantic cutover is separate from introducing the Rust types. The new +query first runs in shadow mode beside the legacy queries, records audit-safe +mismatches through internal telemetry, and preserves legacy enforcement. After +the precedence rule is accepted and mismatch cases are understood, a dedicated +change switches the authoritative result and retains the legacy evaluator long +enough for immediate rollback. Provider-derived policies use a reserved rule-name namespace. The gateway and sandbox sync should prevent user-authored `_provider_*` rules, and @@ -416,15 +501,18 @@ Protocol processors operate on streams owned by the relay. handshake/frame stream and owns client-to-server text-frame inspection when credential rewrite, transport message policy, GraphQL-over-WebSocket policy, or compression handling is configured. -- Native TCP protocol processors read client and upstream streams as needed - and own their message loop. +- Native TCP protocol processors read client and upstream streams as needed and + own their message loop. - A protocol processor can deny before dialing, dial for a server handshake, or - keep evaluating commands/queries throughout the session. + keep evaluating commands or queries throughout the session. - A protocol processor may be in-tree, middleware-backed, or a hybrid where in-tree framing exposes typed middleware operations for content evaluation. -This avoids a separate dial strategy enum. The processor knows which protocol -milestone is sufficient to call the validated connector. +HTTP and WebSocket relays receive the generation-pinned request evaluator +because request policy must continue throughout long-lived sessions. No +processor rematerializes endpoint, TLS, allowed-IP, credential, or middleware +selection. This avoids a separate dial-strategy enum: each processor knows +which protocol milestone is sufficient to call the validated connector. ## Local Service Adapter Boundary @@ -444,6 +532,17 @@ These adapters may call gateway APIs or local credential helpers, but they should not bypass policy and credential invariants that apply to external egress. +Issue [#1633](https://github.com/NVIDIA/OpenShell/issues/1633) is a prospective +consumer of these boundaries, not a feature defined by this RFC. A +policy-declared host-local endpoint should use an explicit local-routing adapter +or destination mode; it must not become a general loopback exemption in the +external destination validator. Its feature design still needs to choose the +policy surface (reserved hostname versus endpoint flag), define authorization +before the supervisor connects to host loopback, and specify driver/runtime +capabilities. That work can reuse `EgressIntent`, adapter-specific responses, +and the unopened connector boundary without changing this RFC's compatibility +milestone. + ## Timeout And Resource Ownership | Owner | Resource |