Skip to content

rfc-draft-diego-c2c-mtls.md

Latest commit

 

History

History
271 lines (187 loc) · 11.4 KB

rfc-draft-diego-c2c-mtls.md

File metadata and controls

271 lines (187 loc) · 11.4 KB

Meta

  • Name: Full mTLS for Diego Container-to-Container Traffic
  • Start Date: 2026-02-13
  • Author(s): @rkoster
  • Status: Draft
  • RFC Pull Request: community#1437

Summary

This RFC proposes implementing full mutual TLS (mTLS) for container-to-container (C2C) traffic in Diego, enabling applications to both authenticate themselves and verify the identity of connecting applications.

The approach introduces a new HTTP-based listener on port 62443 that runs alongside the existing TCP-based C2C port (61443), providing a dual opt-in model for both operators and application authors. This enables zero-trust networking where all app-to-app traffic can be authenticated and authorization decisions can be made based on app/space/org identity.

Problem

Cloud Foundry's Diego runtime currently runs Envoy as a per-container sidecar for handling inbound traffic only. While this provides TLS termination and route integrity, several limitations prevent true zero-trust app-to-app communication:

No Client-Side (Egress) Proxy

Envoy is not configured as an egress proxy. Applications cannot automatically inject their instance identity certificate when making outbound connections. The current workaround requires applications to manually use CF_INSTANCE_CERT and CF_INSTANCE_KEY environment variables to present their identity.

No Identity Extraction on Server Side

Even when mTLS is enabled, the receiving application has no way to know who connected. Envoy does not forward client certificate information via headers (no XFCC header configuration).

C2C Port (61443) is TCP-Based

The C2C listener (port 61443) is a TCP proxy, not an HTTP proxy. This means:

  • No HTTP connection manager
  • No XFCC header support
  • No mTLS client certificate validation
  • No HTTP/2 support

Existing mTLS Only Affects Route Integrity

The existing BOSH property containers.proxy.require_and_verify_client_certificates only affects route integrity ports (Gorouter → App), not C2C traffic.

Current Architecture

flowchart LR
    subgraph Container
        Envoy["Envoy<br/>(port 61001)"]
        App["App<br/>(port 8080)"]
        Envoy -->|plain TCP| App
    end
    Gorouter -->|TLS| Envoy
    
    Note["Envoy only handles INBOUND traffic<br/>(DownstreamTlsContext)"]
Loading

Current Certificate Setup

Diego generates two separate certificates per container in the credmanager:

Certificate SAN Contents Purpose mTLS Support HTTP/2
Instance Identity Container IP address Route integrity (Gorouter → App) Yes (optional) Yes
C2C Certificate Internal route hostnames (e.g., app.apps.internal) Container-to-container TLS No No

Both certificates include app identity in the certificate's Subject.OrganizationalUnit field (separate from the SAN). These claims are set by Cloud Controller and passed through the BBS CertificateProperties model:

  • app:<app-guid>
  • space:<space-guid>
  • organization:<org-guid>

Proposal

Design Principles

  1. Backwards compatibility: Keep port 61443 unchanged for existing applications
  2. Dual opt-in: Both operator and application author must opt-in
  3. New port for new behavior: Port 62443 for HTTP-based mTLS with XFCC header support

Port Allocation

Port Type Purpose mTLS XFCC Status
61001+ HTTP Route integrity Optional No Existing
61443 TCP C2C TLS (TCP-based) No No Existing (unchanged)
62443 HTTP C2C mTLS with identity Required Yes New
61445 HTTP Egress proxy (client-side) N/A N/A New

Implementation Phases

The implementation is divided into three phases to allow gradual adoption and testing.

Phase 1: Server-Side - New HTTP-based C2C mTLS Port (62443)

Add a new Envoy listener on port 62443 that provides HTTP-based C2C with mTLS and XFCC header forwarding.

flowchart TB
    subgraph ContainerB["Container B (Server)"]
        subgraph Listeners
            L61443["Port 61443<br/>(TCP)"]
            L62443["Port 62443<br/>(HTTP, new)"]
        end
        AppB["App B<br/>(port 8080)"]
        
        L61443 -->|"TLS termination only<br/>No client cert validation<br/>No XFCC header"| AppB
        L62443 -->|"mTLS required<br/>XFCC header forwarded"| AppB
    end
    
    ClientA["Container A<br/>(Client)"] -->|"Option 1: TLS (TCP proxy)"| L61443
    ClientA -->|"Option 2: mTLS with identity"| L62443
Loading

Changes Required

New Port Constant: Add C2CMTLSPort = 62443 alongside the existing C2CTLSPort = 61443.

New HTTP-based Listener: Create a new Envoy listener with:

  • HTTP connection manager (not TCP proxy)
  • DownstreamTlsContext with RequireClientCertificate: true
  • Validation context trusting the instance identity CA
  • XFCC header set from the verified client certificate, with any existing headers removed to prevent spoofing
  • Subject, URI, and DNS fields included in the XFCC header

BOSH Properties:

containers.proxy.enable_c2c_mtls_listener:
  description: "Enable the HTTP-based C2C mTLS listener on port 62443. This listener 
    requires client certificates and forwards caller identity via the X-Forwarded-Client-Cert header."
  default: false

Port Reservation: Add validation to prevent applications from using port 62443, similar to the existing reservation for port 61443.

Result

When an operator enables containers.proxy.enable_c2c_mtls_listener:

  • Port 62443 becomes available on all containers
  • Clients connecting to 62443 MUST present a valid instance identity certificate
  • The receiving app gets an X-Forwarded-Client-Cert header with caller identity:
X-Forwarded-Client-Cert: Hash=abc123;Subject="CN=instance-guid,OU=app:client-app-guid,OU=space:space-guid"

Phase 2: Client-Side Egress Proxy (Port 61445)

Enable Envoy to act as an HTTP proxy for outbound connections, automatically injecting the instance identity certificate.

flowchart LR
    subgraph ContainerA["Container A"]
        AppA["App A"]
        EgressProxy["Envoy Egress<br/>(port 61445)"]
        AppA -->|HTTP| EgressProxy
    end
    
    subgraph ContainerB["Container B"]
        IngressProxy["Envoy Ingress<br/>(port 62443)"]
    end
    
    EgressProxy -->|"mTLS<br/>(auto cert injection)"| IngressProxy
    
    Note["Certificate contains:<br/>OU=app:client-app-guid<br/>OU=space:space-guid"]
Loading

Changes Required

Egress Proxy Port: Reserve port 61445 for the egress proxy.

Egress Listener: Create a new Envoy listener that acts as an HTTP CONNECT proxy, bound to 127.0.0.1:61445 (localhost only). The proxy accepts all outbound traffic but only injects the instance identity certificate for connections to C2C mTLS backends.

UpstreamTlsContext: Configure the egress proxy to attach the instance identity certificate only when connecting to C2C mTLS backends, identified by destination port (62443) and hostname pattern matching the configured internal_domains.

Environment Variable: Inject a CF-specific environment variable for applications to discover the egress proxy:

CF_INSTANCE_MTLS_PROXY=http://127.0.0.1:61445

Enforcing Proxy Usage: Operators can make the egress proxy mandatory for all applications by setting HTTP_PROXY and HTTPS_PROXY to http://127.0.0.1:61445 using running environment variable groups. This ensures all HTTP/HTTPS traffic from applications is routed through the egress proxy, with instance identity certificates injected only for C2C mTLS backends.

BOSH Properties:

containers.proxy.enable_egress_proxy:
  description: "Enable Envoy egress proxy for outbound mTLS on port 61445"
  default: false

containers.proxy.internal_domains:
  description: "Domains for C2C mTLS backends where the instance identity certificate should be injected."
  example: ["apps.internal.", "my.apps.internal."]
  default: []

This follows the same naming and format as the internal_domains property in bosh-dns-adapter.

Considerations

  • SNI Handling: Envoy needs to extract the target hostname for proper TLS handshake
  • NO_PROXY: Applications should configure NO_PROXY for traffic that should not go through the proxy
  • Non-HTTP Traffic: The HTTP CONNECT-based egress proxy only supports HTTP/HTTPS traffic. Support for TCP-based protocols could be addressed in a follow-up RFC.
  • Performance: Additional hop adds latency
  • Memory: Additional memory allocation may be needed for the egress proxy

Phase 3: Full Integration

With both phases implemented, the complete mTLS flow enables zero-trust app-to-app communication:

sequenceDiagram
    participant AppA as App A
    participant EgressA as Envoy Egress<br/>(Container A:61445)
    participant IngressB as Envoy Ingress<br/>(Container B:62443)
    participant AppB as App B

    AppA->>EgressA: HTTP request
    Note over EgressA: Attaches instance cert<br/>OU=app:aaa-guid<br/>OU=space:xxx
    EgressA->>IngressB: mTLS connection
    Note over IngressB: Validates client cert<br/>against instance identity CA
    IngressB->>AppB: HTTP request +<br/>X-Forwarded-Client-Cert header
    Note over AppB: Reads XFCC header<br/>to authorize request
    AppB-->>IngressB: HTTP response
    IngressB-->>EgressA: Response
    EgressA-->>AppA: Response
Loading

Opt-in Model

Actor How to Opt-in
Operator Set containers.proxy.enable_c2c_mtls_listener: true and containers.proxy.enable_egress_proxy: true
Server App Listen for connections and read X-Forwarded-Client-Cert header
Client App Connect to port 62443 (directly or via egress proxy at 61445)

App-Level Authorization

Applications can parse the XFCC header to make authorization decisions based on the caller's app GUID, space GUID, or organization GUID embedded in the certificate's OrganizationalUnit field.

Benefits

  • Backwards compatible: port 61443 stays as-is, existing apps keep working
  • Opt-in for both operators (BOSH properties) and app authors (connect to 62443)
  • Apps can migrate gradually, no big-bang required
  • Server apps can verify who's calling and authorize based on app/space/org
  • No more manual certificate handling for C2C calls

Key Repositories

The following repositories require modifications to implement this proposal:

Repository Purpose
cloudfoundry/executor Envoy config generation, environment variable injection
cloudfoundry/diego-release BOSH property definitions

References