bug: proxy-init OUTBOUND_PORTS_EXCLUDE missing port 443 — blocks external HTTPS from agent pods

[Ladas]

Problem

Agent pods with AuthBridge sidecar injection cannot make outbound HTTPS calls to external endpoints (e.g., LLM APIs, MaaS proxies). The weather-service agent gets openai.APIConnectionError: Connection error. when calling litellm-prod.apps.maas.redhatworkshops.io:443.

Root Cause

proxy-init (init-iptables.sh line 225) has a catch-all rule that redirects ALL outbound TCP to Envoy port 15123:

iptables -t nat -A PROXY_OUTPUT -p tcp -j REDIRECT --to-port 15123

OUTBOUND_PORTS_EXCLUDE is hardcoded to "8080" (Keycloak port) in container_builder.go:439:

{
    Name:  "OUTBOUND_PORTS_EXCLUDE",
    Value: "8080",
},

Port 443 is not excluded, so outbound HTTPS to any external endpoint gets redirected to the Envoy outbound listener. The Envoy ext_proc handles HTTP token exchange for in-cluster services but cannot passthrough external HTTPS connections.

Evidence

• Plain pods in the same namespace (with ambient.istio.io/redirection: enabled) can reach external HTTPS endpoints — confirmed by kubectl exec test on strm24
• Pods without AuthBridge sidecar (strm24 weather-service, no webhook injection) reach the MaaS endpoint fine (HTTP 401 = connected)
• Pods with AuthBridge sidecar (proxy-init:v0.6.0-alpha.3) get Connection error — confirmed on every HyperShift CI run since May 16
• Previous version worked: authbridge-envoy:v0.5.0-rc.2 (before the proxy rename/architecture change) did not have this issue — CI passed on May 15

Proposed Fix

Add port 443 to OUTBOUND_PORTS_EXCLUDE in container_builder.go:

{
    Name:  "OUTBOUND_PORTS_EXCLUDE",
    Value: "8080,443",
},

Or better — make it configurable via annotation or operator config so workloads can specify additional excluded ports without rebuilding the webhook.

Impact

• All HyperShift E2E CI runs fail (agent conversation tests)
• Any agent workload that needs to call external HTTPS APIs (LLM, webhooks, external tools) is broken
• Kind CI is unaffected (no AuthBridge sidecar injection on Kind)

Environment

• proxy-init:v0.6.0-alpha.3
• authbridge:v0.6.0-alpha.3
• Istio ambient mode
• OCP 4.20.21 HyperShift

[huang195]

@Ladas AuthBridge should allow outbound ports to be configured to be excluded. This is externalized in the Kagenti UI:

Is this knob not working?

[huang195]

After deeper digging into the actual May 16+ HyperShift CI runs, the diagnosis here is wrong. Sharing the corrected root cause so we don't fix the wrong thing.

What's actually in the failing pod

I pulled the May 17 CI run (authbridge:v0.6.0-alpha.3 — the same image you cite):

Containers:
  agent (...)
  authbridge-proxy (ghcr.io/kagenti/kagenti-extensions/authbridge:v0.6.0-alpha.3)
Init containers:        ← empty

There is no proxy-init container in the pod. It's authbridge-proxy in proxy-sidecar mode (HTTP_PROXY env-based, no iptables anywhere). The proxy-init:v0.6.0-alpha.3 reference came from the chart's pinned image versions, not the running pod.

The init-iptables.sh:225 line you cite is actually a DNAT rule for ztunnel; the catch-all REDIRECT is at line 253. But it doesn't matter — that script never runs in proxy-sidecar mode.

What's actually breaking

Two latent design choices collided on May 14:

1. kagenti-extensions authbridge/authlib/listener/forwardproxy/server.go:170-173 (since Apr 15): the forward proxy explicitly rejects HTTPS CONNECT with 405:

   if r.Method == http.MethodConnect {
       http.Error(w, `{"error":"HTTPS CONNECT not supported — only HTTP proxy"}`, http.StatusMethodNotAllowed)
   }

2. kagenti-operator pod_mutator.go:935 injects HTTPS_PROXY=http://127.0.0.1:<forward-proxy-port> on every agent container in proxy-sidecar mode.

3. kagenti-operator commit 21a2258 (May 14) flipped the cluster-default mode from envoy-sidecar to proxy-sidecar. Pre-flip, envoy-sidecar's TLS passthrough filter chain handled external HTTPS (per kagenti/CLAUDE.md gotcha feature: Remove initContainer and Replace it with Kagenti Sidecar Containers #4). Post-flip, agents land in proxy-sidecar mode by default.

End-to-end: agent's openai-python honors HTTPS_PROXY and sends CONNECT litellm-prod...:443 to localhost:8081 → forward proxy returns 405 → openai raises APIConnectionError: Connection error.

Why the proposed fix doesn't apply

Adding 443 to OUTBOUND_PORTS_EXCLUDE only affects proxy-init iptables (envoy-sidecar mode). The actual failing pods are in proxy-sidecar mode with no proxy-init. The annotation knob exists end-to-end — it's just irrelevant to this failure.

Fix path

Opening kagenti-extensions PR shortly: implement CONNECT in the forward proxy as raw TCP passthrough for HTTPS targets, matching envoy-sidecar's TLS-passthrough behavior. The token-exchange plugin can't operate on encrypted bytes anyway, so passthrough is the only sensible behavior for CONNECT — same as Envoy in envoy-sidecar mode.

Will close this issue once that lands. Suggesting we also retitle to reflect the actual scope ("forward proxy rejects HTTPS CONNECT in proxy-sidecar mode") so the conversation matches what's broken.

[huang195]

@Ladas we might not be able to pick up the fix in the hypershift e2e test any time soon. See #maintainers channel on Kagenti Slack. I've tested the weather agent using LiteLLM using https endpoint, and it works fine with this fix. Once our stable release picks this up, this part of the E2E test will pass again.

[Ladas]

@huang195 ok, I think I have a temporary fix for it in kagenti/kagenti, but we're dealing with other hcp issues and we're trying never versions to fix it.