diff --git a/.agents/skills/debug-openshell-cluster/SKILL.md b/.agents/skills/debug-openshell-cluster/SKILL.md index 82ce9650cd..f1a9f297ac 100644 --- a/.agents/skills/debug-openshell-cluster/SKILL.md +++ b/.agents/skills/debug-openshell-cluster/SKILL.md @@ -163,6 +163,13 @@ kubectl -n openshell get secret \ openshell-jwt-keys ``` +When `server.tls.clientCaSecretName=""`, the chart intentionally omits +`client_ca_path` and the `tls-client-ca` mount, even with built-in PKI or +cert-manager. That is expected; do not treat a missing `tls-client-ca` pod +mount as a defect (`openshell-server-client-ca` may still exist from PKI). +User auth is OIDC or trusted proxy (`server.auth.allowUnauthenticatedUsers=true`); +supervisor transport still uses `openshell-client-tls`. + In cert-manager installs, `certManager.enabled=true` makes cert-manager own TLS generation. The Helm chart should still render the `openshell-certgen` pre-install/pre-upgrade hook in JWT-only mode to create `openshell-jwt-keys`, @@ -350,6 +357,7 @@ openshell logs | `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled | | HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://` | | HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https:// --oidc-issuer `, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster | +| Browser `ERR_BAD_SSL_CLIENT_AUTH_CERT` or gateway logs show client cert verification when OIDC or direct HTTPS is expected | Listener client-CA verification still enabled (`clientCaSecretName` unset or `client_ca_path` in ConfigMap) | Set `server.tls.clientCaSecretName=""`, upgrade chart, confirm ConfigMap omits `client_ca_path` | ## Reporting diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 8723535d1a..5e9fec45f1 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -226,7 +226,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files. | server.sandboxJwt.ttlSecs | int | `3600` | Token TTL in seconds. Defaults to 3600 (1h). | | server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. | | server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. | -| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). | +| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). Do not set to null; omit the key to use the default secret name above. | | server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. | | server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). | | service.healthPort | int | `8081` | Gateway health service port. | diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index 5931047e5f..630d2e3073 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -84,7 +84,7 @@ spec: - name: tls-cert mountPath: /etc/openshell-tls/server readOnly: true - {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + {{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }} - name: tls-client-ca mountPath: /etc/openshell-tls/client-ca readOnly: true @@ -144,7 +144,7 @@ spec: - name: tls-cert secret: secretName: {{ .Values.server.tls.certSecretName }} - {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + {{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }} - name: tls-client-ca secret: {{- if or (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 1b4598088f..940c63ad2c 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -83,6 +83,20 @@ so a released chart automatically pulls the matching image without extra overrid ghcr.io/nvidia/openshell/supervisor {{- end }} +{{/* +Whether the gateway listener should verify client certificates (mTLS). +An explicit empty server.tls.clientCaSecretName disables client-CA wiring in +both gateway.toml and the workload, overriding built-in PKI and cert-manager +defaults. +*/}} +{{- define "openshell.gatewayClientCaEnabled" -}} +{{- if .Values.server.disableTls -}} +{{- else if eq .Values.server.tls.clientCaSecretName "" -}} +{{- else if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) -}} +true +{{- end -}} +{{- end -}} + {{/* Whether Helm must propagate a supervisor image override into gateway.toml. The chart's documented repository and empty tag are the gateway-owned default. @@ -213,4 +227,7 @@ Validate chart values that Helm would otherwise accept silently. {{- if and (eq $workloadKind "statefulset") (gt $replicaCount 1) (not (get $workload "allowMultiReplicaStatefulSet" | default false)) -}} {{- fail "replicaCount > 1 with workload.kind=statefulset requires workload.allowMultiReplicaStatefulSet=true; use workload.kind=deployment for external database-backed multi-replica gateways." -}} {{- end -}} +{{- if kindIs "invalid" .Values.server.tls.clientCaSecretName -}} +{{- fail "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode" -}} +{{- end -}} {{- end }} diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index 36a579250b..d879c20cae 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -75,8 +75,10 @@ data: [openshell.gateway.tls] cert_path = "/etc/openshell-tls/server/tls.crt" key_path = "/etc/openshell-tls/server/tls.key" + {{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }} client_ca_path = "/etc/openshell-tls/client-ca/ca.crt" {{- end }} + {{- end }} {{- if .Values.server.auth.allowUnauthenticatedUsers }} diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index aee396c38f..3c9ebcb901 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -308,6 +308,72 @@ tests: path: data["gateway.toml"] pattern: '\[openshell\.gateway\.tls\]' + - it: renders client_ca_path for built-in PKI by default + template: templates/gateway-config.yaml + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=\s*"/etc/openshell-tls/client-ca/ca\.crt"' + + - it: omits client_ca_path when clientCaSecretName is empty for HTTPS-only mode + template: templates/gateway-config.yaml + set: + server.tls.clientCaSecretName: "" + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + + - it: omits client_ca_path for cert-manager shared CA when clientCaSecretName is empty + template: templates/gateway-config.yaml + set: + certManager.enabled: true + certManager.clientCaFromServerTlsSecret: true + server.tls.clientCaSecretName: "" + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + + - it: omits client_ca_path when cert-manager owns TLS and no client CA secret is set + template: templates/gateway-config.yaml + set: + certManager.enabled: true + certManager.clientCaFromServerTlsSecret: false + pkiInitJob.enabled: true + server.tls.clientCaSecretName: "" + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + - it: renders server_sans from certManager.serverDnsNames set: certManager.enabled: true @@ -328,6 +394,14 @@ tests: path: spec.template.spec.containers[0].args content: "sqlite:/var/openshell/openshell.db" + - it: fails when clientCaSecretName is null + template: templates/statefulset.yaml + set: + server.tls.clientCaSecretName: null + asserts: + - failedTemplate: + errorMessage: "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode" + - it: fails when legacy postgres.enabled is set template: templates/statefulset.yaml set: diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index a7b02310cf..92f5d20f62 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -83,3 +83,39 @@ tests: name: tls-client-ca mountPath: /etc/openshell-tls/client-ca readOnly: true + + # Explicit HTTPS-only opt-out must suppress built-in PKI client-CA wiring even + # when pkiInitJob remains enabled (the default install path from #2095). + - it: omits the client CA volume for built-in PKI when clientCaSecretName is empty + template: templates/statefulset.yaml + set: + pkiInitJob.enabled: true + certManager.enabled: false + server.tls.clientCaSecretName: "" + asserts: + - lengthEqual: + path: spec.template.spec.volumes + count: 3 + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: tls-client-ca + mountPath: /etc/openshell-tls/client-ca + readOnly: true + + - it: omits the client CA volume when cert-manager shares server CA and clientCaSecretName is empty + template: templates/statefulset.yaml + set: + certManager.enabled: true + certManager.clientCaFromServerTlsSecret: true + server.tls.clientCaSecretName: "" + asserts: + - lengthEqual: + path: spec.template.spec.volumes + count: 3 + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: tls-client-ca + mountPath: /etc/openshell-tls/client-ca + readOnly: true diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index e89a234912..ba695651e3 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -245,6 +245,7 @@ server: certSecretName: openshell-server-tls # -- K8s secret with ca.crt for client certificate verification (mTLS). # Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). + # Do not set to null; omit the key to use the default secret name above. clientCaSecretName: openshell-server-client-ca # -- K8s secret mounted into sandbox pods for mTLS to the server. clientTlsSecretName: openshell-client-tls diff --git a/docs/kubernetes/access-control.mdx b/docs/kubernetes/access-control.mdx index 5409a4b11d..c93317a1ba 100644 --- a/docs/kubernetes/access-control.mdx +++ b/docs/kubernetes/access-control.mdx @@ -37,9 +37,12 @@ helm upgrade openshell \ --version \ --namespace openshell \ --set server.oidc.issuer=https://your-idp.example.com/realms/openshell \ - --set server.oidc.audience=openshell-cli + --set server.oidc.audience=openshell-cli \ + --set server.tls.clientCaSecretName="" ``` +Set `server.tls.clientCaSecretName=""` when the gateway terminates TLS directly and browsers or CLI clients connect without client certificates. The chart omits `client_ca_path` from `gateway.toml` and does not mount the client-CA volume, leaving HTTPS-only transport with OIDC for user authentication. Do not set the value to `null`; omit the key to use the chart default, or set it to `""` to disable client certificate verification. + The `audience` value must match the client ID configured in your identity provider for the OpenShell resource server. ### OIDC values reference @@ -67,6 +70,7 @@ helm upgrade openshell \ --namespace openshell \ --set server.oidc.issuer=https://your-idp.example.com/realms/openshell \ --set server.oidc.audience=openshell-cli \ + --set server.tls.clientCaSecretName="" \ --set server.oidc.rolesClaim=realm_access.roles \ --set server.oidc.adminRole=openshell-admin \ --set server.oidc.userRole=openshell-user @@ -96,6 +100,8 @@ helm upgrade openshell \ The gateway still serves TLS and sandbox supervisors still authenticate with gateway-minted sandbox JWTs. User-facing CLI/API calls without OIDC or mTLS credentials are accepted as an unauthenticated local developer principal. The proxy is responsible for authenticating callers and forwarding only authorized traffic. +When the gateway terminates TLS directly and callers connect without client certificates, also set `server.tls.clientCaSecretName=""` as described in the OIDC section above. + To also disable TLS entirely (when the proxy terminates TLS before the request reaches the gateway): ```shell diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 247774a6c0..8e09633df9 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -105,6 +105,8 @@ grpc_rate_limit_requests = 120 grpc_rate_limit_window_seconds = 60 # Gateway listener TLS (distinct from the per-driver guest_tls_*). +# client_ca_path is optional; omit it for HTTPS-only listeners that do not +# verify client certificates. [openshell.gateway.tls] cert_path = "/etc/openshell/certs/gateway.pem" key_path = "/etc/openshell/certs/gateway-key.pem"