proxy-sidecar: NO_PROXY too restrictive — breaks K8s API and external endpoints

[akram]

Problem

In proxy-sidecar mode, the webhook injects HTTP_PROXY / HTTPS_PROXY / NO_PROXY env vars into all app containers via injectHTTPProxyEnv() in pod_mutator.go. The current NO_PROXY value is hardcoded to:

127.0.0.1,localhost

This is too restrictive and causes two classes of failures.

Root cause

The authbridge forward proxy is not a general-purpose HTTP proxy — it only handles inter-agent traffic for token exchange. When a request arrives for a destination it doesn't know (no matching route, not an agent), it rejects it.

Because NO_PROXY only excludes 127.0.0.1,localhost, all outbound HTTPS traffic from agent containers gets routed through the authbridge forward proxy — including traffic that has nothing to do with inter-agent communication:

agent container → "I want to reach kubernetes.default.svc"
               → HTTPS_PROXY intercepts → authbridge forward proxy :8082
               → "unknown destination, not an agent route → 405"

The forward proxy never even attempts authentication or token exchange for these requests. It simply doesn't support them and returns an error immediately.

Failure 1: Kubernetes API calls → 405 Method Not Allowed

Any agent that calls the K8s API (e.g. the orchestrator discovering AgentCard CRDs) gets its request intercepted by the forward proxy:

GET https://kubernetes.default.svc/apis/agent.kagenti.dev/v1alpha1/namespaces/redbank-demo-9/agentcards
→ routed to authbridge forward proxy → 405 Method Not Allowed

WARNING:k8s_discovery:K8s API request failed: 405 Method Not Allowed
WARNING:discovery:No peer agents discovered — orchestrator has no tools.

The orchestrator cannot discover peers, and user requests fail with "Internal server error".

Failure 2: External LLM endpoints → Connection error

Agents calling external LLM endpoints (e.g. https://litellm-prod.apps.example.com/v1) are also intercepted. The forward proxy cannot establish HTTPS connections to arbitrary external hosts:

openai.APIConnectionError: Connection error.

Current workaround

Users must manually set NO_PROXY on each agent deployment to include cluster services and external endpoints. The webhook's injectHTTPProxyEnv() does check if !envExists before injecting, so pre-set values are preserved — but this is not documented and error-prone.

Workaround branch: https://github.com/akram/redbank-demo-2/tree/feat/proxy-sidecar-mode

Proposed solution

Minimum fix — always include cluster-internal destinations

The injected NO_PROXY should at minimum include:

127.0.0.1,localhost,kubernetes.default.svc,.svc,.cluster.local,10.0.0.0/8

These destinations never need token exchange and should never be routed through the authbridge forward proxy. This should not be overridable by users, as removing these entries always breaks things.

OpenShift-aware defaults

On OpenShift clusters, the operator could read the cluster network config to build a more accurate NO_PROXY:

oc get network.config cluster -o jsonpath='{.spec.clusterNetwork[*].cidr},{.spec.serviceNetwork[*]}'

This would automatically cover pod and service CIDRs without hardcoding 10.0.0.0/8.

The operator could also read the cluster proxy config:

oc get proxy cluster -o jsonpath='{.spec.noProxy}'

to inherit cluster-wide proxy exclusions (.cluster.local, node network CIDRs, etc.).

Operator-level configurability

Add a noProxy field to PlatformConfig so cluster admins can extend the base NO_PROXY with site-specific exclusions (e.g. external LLM endpoints, internal registries):

# kagenti-platform-config
proxy:
  noProxy:
    # Always included (not overridable):
    # 127.0.0.1,localhost,kubernetes.default.svc,.svc,.cluster.local
    # Admin-configurable additions:
    additional:
      - ".example.com"
      - ".internal.corp"
      - "10.0.0.0/8"

This way:

• Base entries (localhost, K8s API, cluster services) are always present and not overridable
• Cluster admins can add site-specific entries via kagenti-platform-config
• Per-workload overrides remain possible via pre-setting NO_PROXY in the deployment spec (existing !envExists check), but only for adding entries beyond the base + admin list

Environment

• Cluster: ROSA (OpenShift on AWS)
• Operator: built from main (commit 2073d03)
• Mode: proxy-sidecar
• Relevant code: internal/webhook/injector/pod_mutator.go → injectHTTPProxyEnv()

References

• Upstream authbridge-proxy port-stealing: ✨ Add listeners and single binary — four modes, one codebase (Phase 3) cortex#312
• Workaround branch: https://github.com/akram/redbank-demo-2/tree/feat/proxy-sidecar-mode

[huang195]

@akram which Kagenti branch or release are you running?

[akram]

@huang195 I am using v0.6.0-alpha.5 for the kagenti backend. And for the operator that is the current main branch; (I rebuilt it on purpose); the commit id is 2073d03)

[akram]

Update: PR rossoctl/cortex#430 partially resolves this

After rebuilding authbridge with the CONNECT passthrough fix from rossoctl/cortex#430 (merged May 22), we confirmed that:

• K8s API calls (kubernetes.default.svc) now work without NO_PROXY — the forward proxy tunnels the HTTPS CONNECT request instead of rejecting it with 405
• External LLM endpoints also work — same CONNECT tunnel mechanism
• Agent discovery succeeds: orchestrator finds 2 peers via AgentCard CRDs

What's fixed

The 405 Method Not Allowed and Connection error failures no longer occur. The forward proxy handles arbitrary HTTPS destinations as raw TCP tunnels.

What remains

NO_PROXY is no longer required for correctness, but is still recommended as a best practice:

• Avoids an unnecessary hop through the forward proxy for traffic that doesn't need token exchange (K8s API, external LLM, cluster services)
• Reduces latency and load on the authbridge sidecar
• Follows the principle of least surprise — operators expect NO_PROXY to exclude cluster-internal traffic

The proposal to include sensible defaults in the injected NO_PROXY (cluster services, K8s API) and make it configurable via PlatformConfig remains valid as an optimization, but is no longer a P0 blocker.

Severity change

Before PR #430: Critical — proxy-sidecar mode broken out of the box
After PR #430: Enhancement — unnecessary proxy hop for non-agent traffic