From 7228c1a973467f784a2e472618cd96ef00e436c6 Mon Sep 17 00:00:00 2001 From: Robby Cochran Date: Tue, 26 May 2026 16:51:12 -0700 Subject: [PATCH 1/3] feat: update harness for latest OpenShell with Helm chart and provider system Modernize the deployment to use the official OpenShell Helm chart and provider-based credential management instead of manual TLS generation, DaemonSet supervisor installation, and raw K8s secret extraction. Changes: - deploy.sh: use Helm pkiInitJob for TLS, remove DaemonSet supervisor (Helm handles init-container sideloading), remove manual credential secrets, make images configurable with upstream defaults - setup-providers.sh: new script to register credentials via `openshell provider create` instead of storing them as K8s secrets - ocp-sandbox.sh: use --provider flags for credential injection, auto-detect registered providers, keep file uploads for ADC/GWS - Remove obsolete docs (openshift-deploy.md, agent-observability-spike.md, quay-pull-secret.yaml.template) - Update README, sandbox-CLAUDE.md, sandbox-harness.md for new workflow --- .gitignore | 1 + README.md | 141 ++++++++------ agent-observability-spike.md | 211 -------------------- deploy.sh | 207 +++++++++----------- ocp-sandbox.sh | 226 ++++++++++++---------- openshift-deploy.md | 342 --------------------------------- quay-pull-secret.yaml.template | 15 -- sandbox-CLAUDE.md | 16 +- sandbox-harness.md | 160 ++++++--------- setup-providers.sh | 105 ++++++++++ 10 files changed, 478 insertions(+), 946 deletions(-) delete mode 100644 agent-observability-spike.md delete mode 100644 openshift-deploy.md delete mode 100644 quay-pull-secret.yaml.template create mode 100755 setup-providers.sh diff --git a/.gitignore b/.gitignore index 0f786e7..b67be0e 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ *.json.key acs-ai-sa.json kubeconfig +quay-pull-secret.yaml # Local state *.log diff --git a/README.md b/README.md index a314a70..ea7ccb8 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ Deploy OpenShell sandboxes on OpenShift with Claude Code (Vertex AI), Atlassian ## What This Is -A harness for running AI agent sandboxes on an OpenShift cluster using [OpenShell](https://github.com/NVIDIA/OpenShell). Each sandbox gets: +A deployment harness for running AI agent sandboxes on OpenShift using [OpenShell](https://github.com/NVIDIA/OpenShell). Each sandbox gets: -- **Claude Code** via Google Vertex AI (no Anthropic API key needed) +- **Claude Code** via Google Vertex AI (or direct Anthropic API) - **Jira/Confluence** via mcp-atlassian MCP server - **Gmail, Calendar, Drive** via gws CLI - **GitHub** via gh CLI (pre-authenticated) @@ -17,63 +17,71 @@ A harness for running AI agent sandboxes on an OpenShift cluster using [OpenShel - OpenShift cluster with `KUBECONFIG` set - `kubectl`, `helm` on PATH -- Docker or OrbStack (for building images — BuildKit required) -- Rust toolchain 1.85+ (for building the OpenShell CLI) -- `gcloud auth application-default login` completed (for Vertex AI) -- quay.io registry credentials +- OpenShell CLI (`openshell`) installed or built from source +- NVIDIA/OpenShell repo cloned alongside this repo (for the Helm chart) +- `gcloud auth application-default login` completed (if using Vertex AI) ## Quick Start ```shell -# 1. Deploy OpenShell to the cluster +# 1. Deploy OpenShell to the cluster (Helm chart + CRD + SCCs) ./deploy.sh -# 2. Launch an interactive Claude sandbox -./ocp-sandbox.sh --name my-agent +# 2. Start port-forward to the gateway +kubectl port-forward svc/openshell -n openshell 18443:8080 + +# 3. Register your credentials as providers +export GITHUB_TOKEN="ghp_..." +export JIRA_URL="https://mysite.atlassian.net" +export JIRA_USERNAME="user@example.com" +export JIRA_API_TOKEN="ATATT..." +./setup-providers.sh -# 3. Once connected, type 'claude' to start +# 4. Launch an interactive Claude sandbox +./ocp-sandbox.sh --name my-agent ``` ## Files | File | Purpose | |------|---------| -| `deploy.sh` | Full OpenShell deployment (namespace, CRD, SCCs, TLS, supervisor, Helm) | +| `deploy.sh` | Deploy OpenShell (namespace, CRD, SCCs, Helm chart) | +| `setup-providers.sh` | Register credentials with the OpenShell provider system | | `ocp-sandbox.sh` | Launch/rejoin Claude sandboxes with all integrations | | `vertex-policy.yaml` | Network policy for sandbox egress | | `sandbox-CLAUDE.md` | Agent instructions injected into sandboxes | -| `sandbox-harness.md` | Operations doc (credential rotation, architecture) | | `verify-integrations.py` | Integration test script for all tools | -| `openshift-deploy.md` | Step-by-step deployment guide | | `future-ideas.md` | Roadmap (observability, CronJobs, web UI, memory) | -## Credentials (K8s Secrets) +## How Credentials Work + +This harness uses the **OpenShell provider system** instead of raw K8s secrets. Credentials are: + +1. Registered once via `setup-providers.sh` (stored in the gateway database) +2. Automatically injected as environment variables into sandbox pods +3. Attached to sandboxes with `--provider` flags -All credentials are stored as K8s secrets in the `openshell` namespace: +| Provider | Type | Env Vars Injected | +|----------|------|-------------------| +| `github` | github | `GITHUB_TOKEN` | +| `atlassian` | generic | `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` | +| `anthropic` | anthropic | `ANTHROPIC_API_KEY` | -| Secret | Purpose | -|--------|---------| -| `openshell-server-tls` | Gateway TLS cert | -| `openshell-server-client-ca` | CA for mTLS | -| `openshell-client-tls` | Client mTLS cert | -| `openshell-ssh-handshake` | SSH HMAC key | -| `quay-pull-secret` | Registry auth | -| `github-token` | GitHub PAT | -| `atlassian-creds` | Jira URL, email, API token | -| `gws-credentials` | Google Workspace OAuth | -| `gcp-adc` | Vertex AI Application Default Credentials | +File-based credentials (GCP ADC, GWS OAuth) are uploaded at sandbox creation time via `--upload`. -See `sandbox-harness.md` for rotation and revocation instructions. +### Updating Credentials -## Container Images +```shell +# Update a single credential +openshell provider update github --credential GITHUB_TOKEN="ghp_new_token" -All images pushed to `quay.io/rcochran/scratch` with tags: +# Re-discover from environment +export GITHUB_TOKEN="ghp_new_token" +openshell provider update github --from-existing -| Tag | Source | Purpose | -|-----|--------|---------| -| `openshell-gateway-dev` | `Dockerfile.images --target gateway` | Gateway server | -| `openshell-supervisor-dev` | `Dockerfile.images --target supervisor` | Sandbox supervisor | -| `openshell-sandbox-base` | `ghcr.io/nvidia/openshell-community/sandboxes/base` | Sandbox base image | +# Or re-run the setup script +./setup-providers.sh +``` ## Sandbox Usage @@ -81,34 +89,59 @@ All images pushed to `quay.io/rcochran/scratch` with tags: # Interactive Claude session ./ocp-sandbox.sh --name dev -# Keep sandbox alive after disconnect -./ocp-sandbox.sh --name dev --keep - # Reconnect to a running sandbox ./ocp-sandbox.sh --rejoin dev -# Shell without Claude +# Shell mode (type 'claude' to start) ./ocp-sandbox.sh --name debug --shell + +# Delete sandbox after exit +./ocp-sandbox.sh --name ephemeral --no-keep ``` +## Customizing Images + +Override image sources with environment variables: + +```shell +export GATEWAY_IMAGE_REPO=quay.io/myrepo/openshell-gateway +export GATEWAY_IMAGE_TAG=v1.0.0 +export SUPERVISOR_IMAGE_REPO=quay.io/myrepo/openshell-supervisor +export SANDBOX_IMAGE=quay.io/myrepo/sandbox-base:latest +export PULL_SECRET=my-registry-secret +./deploy.sh +``` + +By default, upstream images from `ghcr.io/nvidia/openshell/` are used. + ## Architecture ``` Your Mac OpenShift Cluster -┌──────────┐ port-forward ┌─────────────────────────┐ -│ openshell├───────────────────▶│ Gateway (StatefulSet) │ -│ CLI │ mTLS :18443 │ ├─ gRPC API │ -└──────────┘ │ ├─ SSH tunnel │ - │ └─ sandbox lifecycle │ - │ │ - │ Sandbox Pods │ - │ ├─ Claude Code │ - │ ├─ mcp-atlassian │ - │ ├─ gws CLI │ - │ ├─ gh CLI │ - │ └─ Network proxy │ - │ │ - │ Supervisor DaemonSet │ - │ └─ /opt/openshell/bin │ - └─────────────────────────┘ +┌──────────┐ port-forward ┌──────────────────────────────┐ +│ openshell├───────────────────▶│ Gateway (StatefulSet) │ +│ CLI │ mTLS :18443 │ ├─ gRPC API │ +│ │ │ ├─ SSH tunnel │ +│ │ │ ├─ Provider credential store │ +│ │ │ └─ Sandbox lifecycle mgmt │ +└──────────┘ │ │ + │ Sandbox Pods │ + │ ├─ Claude Code │ + │ ├─ mcp-atlassian │ + │ ├─ gws CLI │ + │ ├─ gh CLI │ + │ └─ Network proxy │ + │ │ + │ Supervisor (init-container) │ + │ └─ sideloaded per sandbox │ + └──────────────────────────────┘ ``` + +### Key Differences from Manual Deployment + +This harness relies on the official OpenShell Helm chart, which handles: + +- **TLS/PKI** — auto-generated via a pre-install certgen job (no manual OpenSSL) +- **Supervisor** — sideloaded into each sandbox pod as an init-container (no DaemonSet) +- **Credentials** — managed by the provider system with refresh support (no raw K8s secrets) +- **Gateway config** — TOML-based ConfigMap rendered by Helm diff --git a/agent-observability-spike.md b/agent-observability-spike.md deleted file mode 100644 index 50fef80..0000000 --- a/agent-observability-spike.md +++ /dev/null @@ -1,211 +0,0 @@ -# Agent Observability Spike - -## Problem - -When running multiple agent sessions on the OCP cluster, there's no way to: -- See what agents are doing in real time -- Search past agent activity across sessions -- Debug why a task failed or what tools were called -- Compare agent behavior across runs - -## Goals - -1. Structured logs from every agent session, queryable by session/agent/time/tool -2. Web UI accessible from browser — search, filter, live tail -3. Works with what's already on the cluster or easily installable -4. Minimal per-sandbox overhead (no custom sidecars if avoidable) - -## Architecture - -### Data Sources - -Three layers of structured data: - -| Source | Format | What it captures | -|--------|--------|-----------------| -| Claude Code `--output-format stream-json` | JSONL | Every assistant message, tool call, tool result, thinking block, token usage | -| OpenShell supervisor OCSF logs | JSONL | Network decisions, process lifecycle, policy enforcement, SSH events | -| Harness metadata | Labels/annotations | Session name, profile, who started it, which tools enabled | - -### Option A: Loki + Grafana (OCP-native, recommended) - -``` -┌─────────────────┐ ┌──────────────┐ ┌─────────────┐ -│ Sandbox Pod │────▶│ Loki │────▶│ Grafana │ -│ stdout/stderr │ │ (log store) │ │ (web UI) │ -│ JSONL output │ │ │ │ search, │ -│ │ │ labels: │ │ filter, │ -│ supervisor OCSF │ │ - sandbox │ │ live tail │ -│ claude stream │ │ - agent │ │ │ -└─────────────────┘ │ - session │ └─────────────┘ - └──────────────┘ -``` - -**How it works:** -- OCP cluster logging (Vector/Fluentd → Loki) already collects pod stdout -- Claude Code JSONL goes to stdout → captured automatically -- Supervisor OCSF logs go to stderr → captured automatically -- Pod labels (`openshell.ai/sandbox-id`, session name) become Loki labels -- Grafana provides search UI, dashboards, live tail - -**What's needed:** -- Enable OpenShift Logging operator (if not already installed) -- Configure LokiStack CR in the cluster -- Create Grafana dashboards for agent activity -- Configure Claude Code to output structured JSON in the harness - -**Pros:** Zero per-sandbox config. Uses OCP's built-in log pipeline. Grafana is already familiar. -**Cons:** Loki is text-based search, not field-level structured queries. Log retention is cluster-scoped. - -### Option B: OpenSearch + Dashboards - -``` -┌─────────────────┐ ┌──────────────┐ ┌──────────────┐ -│ Sandbox Pod │────▶│ OpenSearch │────▶│ Dashboards │ -│ stdout/stderr │ │ (index) │ │ (web UI) │ -│ JSONL output │ │ │ │ full-text + │ -│ │ │ structured │ │ field search│ -│ supervisor OCSF │ │ field index │ │ │ -└─────────────────┘ └──────────────┘ └──────────────┘ -``` - -**How it works:** -- OCP cluster logging can route to OpenSearch instead of Loki -- JSONL output is parsed into structured fields (tool name, model, tokens, etc.) -- OpenSearch supports field-level queries: "show me all Bash tool calls that failed" -- Dashboards (Kibana fork) provides rich filtering and visualization - -**Pros:** Full structured field search. Better for high cardinality (many tools, many sessions). -**Cons:** Heavier to run. More storage. More setup. - -### Option C: Lightweight custom viewer (PVC + web app) - -``` -┌─────────────────┐ ┌──────────────┐ ┌──────────────┐ -│ Sandbox Pod │────▶│ Shared PVC │────▶│ Viewer Pod │ -│ writes JSONL │ │ /logs/ │ │ (web app) │ -│ to /logs/ │ │ session-1/ │ │ search, │ -│ │ │ session-2/ │ │ browse, │ -│ │ │ ... │ │ live tail │ -└─────────────────┘ └──────────────┘ └──────────────┘ -``` - -**How it works:** -- Shared PVC (ReadWriteMany) mounted in every sandbox at `/logs/` -- Claude Code output tee'd to `/logs//claude.jsonl` -- Supervisor OCSF logs tee'd to `/logs//ocsf.jsonl` -- Small viewer pod serves a web UI that reads the PVC -- Search via `grep`/`jq` or a small sqlite index - -**Pros:** Simplest. No operators. Total control over format. Works on any cluster. -**Cons:** No real-time streaming. Manual index. PVC storage limits. RWX storage class needed. - -## Recommendation - -**Start with Option A (Loki + Grafana)** if OpenShift Logging is available on the cluster. It's the lowest effort and uses the standard OCP observability stack. - -**Fall back to Option C** if cluster logging isn't available or too heavy. It's the simplest to build and gives you the core use case (search past sessions, browse activity). - -**Option B** is overkill unless you have 10+ concurrent agents and need field-level analytics. - -## Implementation Steps (Option A) - -### Phase 1: Enable structured output (harness changes only) - -1. Add `--output-format stream-json` to the claude launch command -2. Tee claude output to both terminal and stdout (so cluster logging captures it) -3. Add session labels to sandbox pods via the harness -4. Verify logs appear in Loki via `oc logs` or Grafana - -### Phase 2: Install/configure logging stack - -1. Check if OpenShift Logging operator is already installed -2. If not, install via OperatorHub: - - Loki Operator → create LokiStack - - Cluster Logging Operator → create ClusterLogging CR -3. Configure log forwarding to include `openshell` namespace -4. Verify pod logs flow to Loki - -### Phase 3: Grafana dashboards - -1. Create a "Agent Activity" dashboard: - - Session timeline (start/end, duration) - - Tool call frequency and latency - - Token usage per session - - Errors and failures -2. Create a "Live Agent" panel: - - Live tail of current session stdout - - Filter by sandbox name -3. Create saved searches: - - "All Jira tool calls today" - - "Failed tool invocations" - - "Sessions by agent profile" - -### Phase 4: Enrichment (optional) - -1. Parse Claude Code JSONL in a log pipeline (Vector transform) -2. Extract structured fields: tool_name, tool_result_status, model, tokens_in, tokens_out -3. Create Loki structured metadata labels for fast filtering -4. Build cost tracking (tokens × price per model) - -## Implementation Steps (Option C fallback) - -### Phase 1: Shared log PVC - -```yaml -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: agent-logs - namespace: openshell -spec: - accessModes: [ReadWriteMany] - resources: - requests: - storage: 10Gi -``` - -Mount in sandbox pods (same pattern as the policy ConfigMap). - -### Phase 2: Log capture in harness - -```shell -# In the startup script -LOGDIR="/logs/$(date +%Y%m%d)-${SANDBOX_NAME}" -mkdir -p "$LOGDIR" -claude --output-format stream-json --dangerously-skip-permissions \ - 2>&1 | tee "$LOGDIR/claude.jsonl" -``` - -### Phase 3: Viewer - -A simple Go/Python web app: -- Serves at `agent-logs.openshell.svc:8080` -- Lists sessions by date -- Renders JSONL as a conversation view -- Full-text search via grep/ripgrep -- Expose via OCP Route for browser access - -## Claude Code Structured Output Fields - -When `--output-format stream-json` is used, each line is a JSON object: - -```json -{"type": "assistant", "message": {"role": "assistant", "content": [...]}, "usage": {"input_tokens": 1234, "output_tokens": 567}} -{"type": "tool_use", "name": "Bash", "input": {"command": "ls"}} -{"type": "tool_result", "name": "Bash", "content": "file1.txt\nfile2.txt", "is_error": false} -{"type": "system", "message": "Session started", "session_id": "abc123"} -``` - -Key fields for search/filtering: -- `type` — assistant, tool_use, tool_result, system, error -- `name` — tool name (Bash, Read, Write, mcp__atlassian__*, etc.) -- `is_error` — whether the tool call failed -- `usage.input_tokens`, `usage.output_tokens` — cost tracking - -## Open Questions - -1. Does the OCP cluster have OpenShift Logging installed? (`oc get csv -n openshift-logging`) -2. Is there a default RWX storage class? (needed for Option C shared PVC) -3. What retention do you want? 7 days? 30 days? Indefinite? -4. Should the viewer be accessible outside the cluster (OCP Route) or only via port-forward? diff --git a/deploy.sh b/deploy.sh index 55466da..760d519 100755 --- a/deploy.sh +++ b/deploy.sh @@ -1,18 +1,23 @@ #!/usr/bin/env bash -# Deploy OpenShell to an OpenShift cluster. -# Assumes: KUBECONFIG set, kubectl/helm on PATH, quay.io images already pushed. +# Deploy OpenShell to an OpenShift cluster using the official Helm chart. # # Usage: # ./deploy.sh # full deploy # ./deploy.sh --kubeconfig ./kubeconfig # explicit kubeconfig -# ./deploy.sh --skip-images # skip image build (use existing) +# +# Environment variables (all optional, sensible defaults provided): +# OPENSHELL_REPO — path to NVIDIA/OpenShell checkout (default: ../OpenShell) +# GATEWAY_IMAGE_REPO — gateway image repo (default: ghcr.io/nvidia/openshell/gateway) +# GATEWAY_IMAGE_TAG — gateway image tag (default: chart appVersion) +# SUPERVISOR_IMAGE_REPO — supervisor image repo (default: ghcr.io/nvidia/openshell/supervisor) +# SANDBOX_IMAGE — sandbox base image (default: ghcr.io/nvidia/openshell-community/sandboxes/base:latest) +# PULL_SECRET — imagePullSecrets name (default: none) +# GATEWAY_NAME — CLI gateway name (default: ocp) set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -# Parse args -SKIP_IMAGES=true # images are pre-built on quay.io while [[ $# -gt 0 ]]; do case $1 in --kubeconfig) export KUBECONFIG="$2"; shift 2 ;; @@ -20,144 +25,110 @@ while [[ $# -gt 0 ]]; do esac done -# Require OpenShell repo to be cloned alongside this harness OPENSHELL_REPO="${OPENSHELL_REPO:-$(cd "$SCRIPT_DIR/.." && pwd)/OpenShell}" if [[ ! -d "$OPENSHELL_REPO/deploy/helm/openshell" ]]; then - echo "OpenShell repo not found at $OPENSHELL_REPO" + echo "ERROR: OpenShell repo not found at $OPENSHELL_REPO" echo "Set OPENSHELL_REPO or clone NVIDIA/OpenShell alongside this repo" exit 1 fi +GATEWAY_NAME="${GATEWAY_NAME:-ocp}" + echo "Using OpenShell repo: $OPENSHELL_REPO" echo "Using KUBECONFIG: ${KUBECONFIG:-default}" echo "" -# ── Step 1: Namespace ── -echo "=== Creating namespace ===" +# ── Step 1: Namespace ────────────────────────────────────────────────── +echo "=== Step 1: Creating namespace ===" kubectl create ns openshell 2>/dev/null || true -kubectl label ns openshell pod-security.kubernetes.io/enforce=privileged --overwrite -kubectl label ns openshell pod-security.kubernetes.io/warn=privileged --overwrite +kubectl label ns openshell \ + pod-security.kubernetes.io/enforce=privileged \ + pod-security.kubernetes.io/warn=privileged \ + --overwrite -# ── Step 2: Sandbox CRD + controller ── -echo "=== Installing Sandbox CRD ===" +# ── Step 2: Sandbox CRD + controller ────────────────────────────────── +echo "=== Step 2: Installing Sandbox CRD ===" kubectl apply -f "$OPENSHELL_REPO/deploy/kube/manifests/agent-sandbox.yaml" -# ── Step 3: SCCs ── -echo "=== Granting SCCs ===" -kubectl create clusterrolebinding openshell-sa-anyuid --clusterrole=system:openshift:scc:anyuid --serviceaccount=openshell:openshell 2>/dev/null || true -kubectl create clusterrolebinding openshell-sa-privileged --clusterrole=system:openshift:scc:privileged --serviceaccount=openshell:openshell 2>/dev/null || true -kubectl create clusterrolebinding openshell-default-privileged --clusterrole=system:openshift:scc:privileged --serviceaccount=openshell:default 2>/dev/null || true -kubectl create clusterrolebinding agent-sandbox-admin --clusterrole=cluster-admin --serviceaccount=agent-sandbox-system:agent-sandbox-controller 2>/dev/null || true - -# ── Step 4: TLS certificates ── -echo "=== Generating TLS certificates ===" -TLSDIR="$HOME/.openshell-ocp-tls" -mkdir -p "$TLSDIR" - -if [[ ! -f "$TLSDIR/ca.crt" ]]; then - openssl req -x509 -newkey rsa:2048 -nodes -keyout "$TLSDIR/ca.key" -out "$TLSDIR/ca.crt" -days 365 -subj "/CN=openshell-ca" 2>/dev/null - openssl req -newkey rsa:2048 -nodes -keyout "$TLSDIR/server.key" -out "$TLSDIR/server.csr" \ - -subj "/CN=openshell.openshell.svc.cluster.local" \ - -addext "subjectAltName=DNS:openshell.openshell.svc.cluster.local,DNS:openshell,DNS:localhost,IP:127.0.0.1" 2>/dev/null - openssl x509 -req -in "$TLSDIR/server.csr" -CA "$TLSDIR/ca.crt" -CAkey "$TLSDIR/ca.key" -CAcreateserial \ - -out "$TLSDIR/server.crt" -days 365 \ - -extfile <(echo "subjectAltName=DNS:openshell.openshell.svc.cluster.local,DNS:openshell,DNS:localhost,IP:127.0.0.1") 2>/dev/null - openssl req -newkey rsa:2048 -nodes -keyout "$TLSDIR/client.key" -out "$TLSDIR/client.csr" -subj "/CN=openshell-client" 2>/dev/null - openssl x509 -req -in "$TLSDIR/client.csr" -CA "$TLSDIR/ca.crt" -CAkey "$TLSDIR/ca.key" -CAcreateserial \ - -out "$TLSDIR/client.crt" -days 365 2>/dev/null - echo " Generated new certs at $TLSDIR" -else - echo " Using existing certs at $TLSDIR" +# ── Step 3: OpenShift SCCs ──────────────────────────────────────────── +echo "=== Step 3: Granting OpenShift SCCs ===" +kubectl create clusterrolebinding openshell-sa-anyuid \ + --clusterrole=system:openshift:scc:anyuid \ + --serviceaccount=openshell:openshell 2>/dev/null || true +kubectl create clusterrolebinding openshell-sa-privileged \ + --clusterrole=system:openshift:scc:privileged \ + --serviceaccount=openshell:openshell 2>/dev/null || true +kubectl create clusterrolebinding openshell-default-privileged \ + --clusterrole=system:openshift:scc:privileged \ + --serviceaccount=openshell:default 2>/dev/null || true +kubectl create clusterrolebinding agent-sandbox-admin \ + --clusterrole=cluster-admin \ + --serviceaccount=agent-sandbox-system:agent-sandbox-controller 2>/dev/null || true + +# Also grant privileged to the sandbox service account created by Helm +kubectl create clusterrolebinding openshell-sandbox-privileged \ + --clusterrole=system:openshift:scc:privileged \ + --serviceaccount=openshell:openshell-sandbox 2>/dev/null || true + +# ── Step 4: Pull secret (optional) ─────────────────────────────────── +if [[ -f "$SCRIPT_DIR/quay-pull-secret.yaml" ]]; then + echo "=== Step 4: Applying pull secret ===" + kubectl apply -n openshell -f "$SCRIPT_DIR/quay-pull-secret.yaml" fi -# ── Step 5: K8s secrets ── -echo "=== Creating K8s secrets ===" -kubectl create secret tls openshell-server-tls -n openshell --cert="$TLSDIR/server.crt" --key="$TLSDIR/server.key" 2>/dev/null || true -kubectl create secret generic openshell-server-client-ca -n openshell --from-file=ca.crt="$TLSDIR/ca.crt" 2>/dev/null || true -kubectl create secret generic openshell-client-tls -n openshell --from-file=ca.crt="$TLSDIR/ca.crt" --from-file=tls.crt="$TLSDIR/client.crt" --from-file=tls.key="$TLSDIR/client.key" 2>/dev/null || true -kubectl create secret generic openshell-ssh-handshake -n openshell --from-literal=secret="$(openssl rand -hex 32)" 2>/dev/null || true -kubectl apply -n openshell -f "$SCRIPT_DIR/quay-pull-secret.yaml" 2>/dev/null || true - -# Credential secrets — create only if they don't exist -kubectl get secret github-token -n openshell >/dev/null 2>&1 || echo "WARNING: github-token secret missing — create manually" -kubectl get secret atlassian-creds -n openshell >/dev/null 2>&1 || echo "WARNING: atlassian-creds secret missing — create manually" -kubectl get secret gws-credentials -n openshell >/dev/null 2>&1 || echo "WARNING: gws-credentials secret missing — create manually" -kubectl get secret gcp-adc -n openshell >/dev/null 2>&1 || echo "WARNING: gcp-adc secret missing — create manually" - -# ── Step 6: Supervisor DaemonSet ── -echo "=== Deploying supervisor DaemonSet ===" -cat <<'EOF' | kubectl apply -f - -apiVersion: apps/v1 -kind: DaemonSet -metadata: - name: openshell-supervisor-installer - namespace: openshell -spec: - selector: - matchLabels: - app: openshell-supervisor-installer - template: - metadata: - labels: - app: openshell-supervisor-installer - spec: - serviceAccountName: default - imagePullSecrets: - - name: quay-pull-secret - initContainers: - - name: install - image: quay.io/rcochran/scratch:openshell-supervisor-dev - command: ["sh", "-c", "mkdir -p /host/opt/openshell/bin && cp /usr/local/bin/openshell-sandbox /host/opt/openshell/bin/openshell-sandbox && chmod 755 /host/opt/openshell/bin/openshell-sandbox && chcon -t container_file_t /host/opt/openshell/bin && chcon -t container_file_t /host/opt/openshell/bin/openshell-sandbox && echo installed"] - securityContext: - privileged: true - volumeMounts: - - name: host-root - mountPath: /host - containers: - - name: pause - image: registry.k8s.io/pause:3.10 - volumes: - - name: host-root - hostPath: - path: / - tolerations: - - operator: Exists -EOF +# ── Step 5: Helm install gateway ────────────────────────────────────── +echo "=== Step 5: Deploying gateway via Helm ===" -# ── Step 7: Helm install gateway ── -echo "=== Deploying gateway via Helm ===" -helm upgrade --install openshell "$OPENSHELL_REPO/deploy/helm/openshell" -n openshell \ - --set image.repository=quay.io/rcochran/scratch \ - --set image.tag=openshell-gateway-dev \ - --set image.pullPolicy=Always \ - --set imagePullSecrets[0].name=quay-pull-secret \ - --set server.sandboxImage="quay.io/rcochran/scratch:openshell-sandbox-base" \ - --set server.sandboxImagePullPolicy=Always \ - --set server.grpcEndpoint="https://openshell.openshell.svc.cluster.local:8080" \ - --set server.dbUrl="sqlite:/var/openshell/openshell.db" \ +HELM_ARGS=( + --set server.sandboxImagePullPolicy=Always + --set server.dbUrl="sqlite:/var/openshell/openshell.db" + --set pkiInitJob.enabled=true + --set pkiInitJob.serverDnsNames[0]=openshell.openshell.svc.cluster.local --set service.type=ClusterIP +) + +[[ -n "${GATEWAY_IMAGE_REPO:-}" ]] && HELM_ARGS+=(--set image.repository="$GATEWAY_IMAGE_REPO") +[[ -n "${GATEWAY_IMAGE_TAG:-}" ]] && HELM_ARGS+=(--set image.tag="$GATEWAY_IMAGE_TAG" --set image.pullPolicy=Always) +[[ -n "${SUPERVISOR_IMAGE_REPO:-}" ]] && HELM_ARGS+=(--set supervisor.image.repository="$SUPERVISOR_IMAGE_REPO") +[[ -n "${SANDBOX_IMAGE:-}" ]] && HELM_ARGS+=(--set server.sandboxImage="$SANDBOX_IMAGE") +[[ -n "${PULL_SECRET:-}" ]] && HELM_ARGS+=(--set imagePullSecrets[0].name="$PULL_SECRET") + +helm upgrade --install openshell "$OPENSHELL_REPO/deploy/helm/openshell" -n openshell \ + "${HELM_ARGS[@]}" echo "=== Waiting for gateway ===" -kubectl rollout status statefulset/openshell -n openshell --timeout=120s - -# ── Step 8: Gateway config ── -echo "=== Configuring CLI gateway ===" -mkdir -p "$HOME/.config/openshell/gateways/ocp/mtls" -cp "$TLSDIR/ca.crt" "$HOME/.config/openshell/gateways/ocp/mtls/" -cp "$TLSDIR/client.crt" "$HOME/.config/openshell/gateways/ocp/mtls/tls.crt" -cp "$TLSDIR/client.key" "$HOME/.config/openshell/gateways/ocp/mtls/tls.key" -cat > "$HOME/.config/openshell/gateways/ocp/metadata.json" <<'GWEOF' -{"name":"ocp","gateway_endpoint":"https://127.0.0.1:18443","is_remote":false,"gateway_port":18443,"auth_mode":"mtls"} -GWEOF +kubectl rollout status statefulset/openshell -n openshell --timeout=180s + +# ── Step 6: Configure local CLI gateway ─────────────────────────────── +echo "=== Step 6: Configuring local CLI gateway ===" +GW_DIR="$HOME/.config/openshell/gateways/$GATEWAY_NAME" +MTLS_DIR="$GW_DIR/mtls" +mkdir -p "$MTLS_DIR" + +kubectl get secret openshell-client-tls -n openshell \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > "$MTLS_DIR/ca.crt" +kubectl get secret openshell-client-tls -n openshell \ + -o jsonpath='{.data.tls\.crt}' | base64 -d > "$MTLS_DIR/tls.crt" +kubectl get secret openshell-client-tls -n openshell \ + -o jsonpath='{.data.tls\.key}' | base64 -d > "$MTLS_DIR/tls.key" + +cat > "$GW_DIR/metadata.json" </dev/null; then + REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" + CLI="$REPO_ROOT/OpenShell/target/debug/openshell" +fi +[[ -x "$CLI" ]] || { echo "ERROR: openshell CLI not found. Install it or set OPENSHELL_CLI."; exit 1; } -# ── Parse args ────────────────────────────────────────────────────────── +# ── Parse args ───────────────────────────────────────────────────────── SHELL_MODE=false -KEEP_FLAG="--no-keep" +NO_KEEP=false REJOIN="" NAME_ARGS=() +EXTRA_PROVIDERS=() while [[ $# -gt 0 ]]; do case $1 in - --shell) SHELL_MODE=true; shift ;; - --keep) KEEP_FLAG=""; shift ;; - --rejoin) REJOIN="$2"; shift 2 ;; - --name) NAME_ARGS=(--name "$2"); shift 2 ;; - *) shift ;; + --shell) SHELL_MODE=true; shift ;; + --no-keep) NO_KEEP=true; shift ;; + --rejoin) REJOIN="$2"; shift 2 ;; + --name) NAME_ARGS=(--name "$2"); shift 2 ;; + --provider) EXTRA_PROVIDERS+=(--provider "$2"); shift 2 ;; + *) shift ;; esac done -# ── Ensure port-forward ──────────────────────────────────────────────── +# ── Ensure port-forward ─────────────────────────────────────────────── if ! lsof -i :18443 -sTCP:LISTEN >/dev/null 2>&1; then echo "Starting port-forward..." nohup kubectl port-forward svc/openshell -n openshell 18443:8080 >/tmp/openshell-pf.log 2>&1 & sleep 2 fi -# ── Ensure gateway config ────────────────────────────────────────────── -if [[ ! -f "$HOME/.config/openshell/gateways/ocp/metadata.json" ]]; then - TLSDIR="$HOME/.openshell-ocp-tls" - mkdir -p "$HOME/.config/openshell/gateways/ocp/mtls" - cp "$TLSDIR/ca.crt" "$HOME/.config/openshell/gateways/ocp/mtls/" - cp "$TLSDIR/client.crt" "$HOME/.config/openshell/gateways/ocp/mtls/tls.crt" - cp "$TLSDIR/client.key" "$HOME/.config/openshell/gateways/ocp/mtls/tls.key" - cat > "$HOME/.config/openshell/gateways/ocp/metadata.json" <<'EOF' -{"name":"ocp","gateway_endpoint":"https://127.0.0.1:18443","is_remote":false,"gateway_port":18443,"auth_mode":"mtls"} -EOF -fi - -# ── Rejoin mode ───────────────────────────────────────────────────────── +# ── Rejoin mode ──────────────────────────────────────────────────────── if [[ -n "$REJOIN" ]]; then - echo "Reconnecting to $REJOIN — type 'claude' to start." + echo "Reconnecting to $REJOIN..." exec "$CLI" sandbox connect "$REJOIN" fi -# ── Fetch credentials ─────────────────────────────────────────────────── -echo "Fetching credentials..." -GH_TOKEN=$(kubectl get secret github-token -n openshell -o jsonpath='{.data.GITHUB_TOKEN}' | base64 -d) -JIRA_URL=$(kubectl get secret atlassian-creds -n openshell -o jsonpath='{.data.JIRA_URL}' | base64 -d) -JIRA_USER=$(kubectl get secret atlassian-creds -n openshell -o jsonpath='{.data.JIRA_USERNAME}' | base64 -d) -JIRA_TOKEN=$(kubectl get secret atlassian-creds -n openshell -o jsonpath='{.data.JIRA_API_TOKEN}' | base64 -d) -GWS_CS_B64=$(kubectl get secret gws-credentials -n openshell -o jsonpath='{.data.client_secret\.json}') -GWS_CR_B64=$(kubectl get secret gws-credentials -n openshell -o jsonpath='{.data.credentials\.enc}') -GWS_EK_B64=$(kubectl get secret gws-credentials -n openshell -o jsonpath='{.data.encryption_key}') -GWS_TC_B64=$(kubectl get secret gws-credentials -n openshell -o jsonpath='{.data.token_cache\.json}') - -[[ -f "$HOME/.config/gcloud/application_default_credentials.json" ]] || { - echo "No ADC. Run: gcloud auth application-default login"; exit 1 -} +# ── Build provider flags from registered providers ───────────────────── +PROVIDER_FLAGS=() +if "$CLI" provider get github &>/dev/null 2>&1; then + PROVIDER_FLAGS+=(--provider github) +fi +if "$CLI" provider get atlassian &>/dev/null 2>&1; then + PROVIDER_FLAGS+=(--provider atlassian) +fi +if "$CLI" provider get anthropic &>/dev/null 2>&1; then + PROVIDER_FLAGS+=(--provider anthropic) +fi +PROVIDER_FLAGS+=("${EXTRA_PROVIDERS[@]}") -VERTEX_PROJECT="itpc-gcp-hcm-pe-eng-claude" -VERTEX_REGION="us-east5" +# ── Build upload flags ───────────────────────────────────────────────── +UPLOAD_ARGS=() -# ── Create sandbox ────────────────────────────────────────────────────── -# Phase 1: create sandbox + upload ADC + run setup (setup consumes stdin, that's ok) -# Phase 2: connect with clean stdin for Claude -# -# The sandbox is always created with --keep so we can connect in phase 2. -# If the user didn't pass --keep, we note it and delete after disconnect. -USER_WANTS_KEEP=false -[[ "$KEEP_FLAG" == "" ]] && USER_WANTS_KEEP=true +# Vertex AI ADC +ADC_FILE="${GOOGLE_APPLICATION_CREDENTIALS:-$HOME/.config/gcloud/application_default_credentials.json}" +if [[ -f "$ADC_FILE" ]]; then + UPLOAD_ARGS+=(--upload "$ADC_FILE:/tmp/adc.json") +fi + +VERTEX_PROJECT="${VERTEX_PROJECT:-}" +VERTEX_REGION="${VERTEX_REGION:-}" + +# GWS credentials directory +GWS_CONFIG_DIR="${GWS_CONFIG_DIR:-$HOME/.config/gws}" +GWS_UPLOAD=false +if [[ -d "$GWS_CONFIG_DIR" && -f "$GWS_CONFIG_DIR/client_secret.json" ]]; then + GWS_UPLOAD=true +fi + +# ── Sandbox env vars (non-provider, Vertex AI config) ────────────────── +ENV_BLOCK="" +if [[ -f "$ADC_FILE" ]]; then + ENV_BLOCK+="export GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json +" + ENV_BLOCK+="export CLAUDE_CODE_USE_VERTEX=1 +" + ENV_BLOCK+="export CLOUD_ML_REGION=global +" + [[ -n "$VERTEX_PROJECT" ]] && ENV_BLOCK+="export ANTHROPIC_VERTEX_PROJECT_ID=$VERTEX_PROJECT +export GOOGLE_CLOUD_PROJECT=$VERTEX_PROJECT +" + [[ -n "$VERTEX_REGION" ]] && ENV_BLOCK+="export GOOGLE_CLOUD_LOCATION=$VERTEX_REGION +" +fi +ENV_BLOCK+='export GOOGLE_WORKSPACE_CLI_CONFIG_DIR=/tmp/gws-config +' +ENV_BLOCK+='export PATH="/sandbox/.local/bin:$PATH" +' +# ── Build Jira env for MCP config ────────────────────────────────────── +# The provider injects JIRA_URL, JIRA_USERNAME, JIRA_API_TOKEN as env vars. +# We reference them in the claude.json MCP config at startup. + +# ── Create sandbox ───────────────────────────────────────────────────── echo "Creating sandbox..." + +KEEP_ARGS=() +$NO_KEEP && KEEP_ARGS=(--no-keep) + +GWS_SETUP="" +if $GWS_UPLOAD; then + GWS_SETUP=' +mkdir -p /tmp/gws-config +cp /sandbox/gws-upload/* /tmp/gws-config/ 2>/dev/null || true +' + UPLOAD_ARGS+=(--upload "$GWS_CONFIG_DIR:/sandbox/gws-upload") +fi + "$CLI" sandbox create \ --policy "$SCRIPT_DIR/vertex-policy.yaml" \ - --upload "$HOME/.config/gcloud/application_default_credentials.json:/tmp/adc.json" \ - --no-bootstrap \ - --keep \ + "${UPLOAD_ARGS[@]}" \ + "${PROVIDER_FLAGS[@]}" \ + "${KEEP_ARGS[@]}" \ "${NAME_ARGS[@]}" \ -- bash -c ' -# ── Setup (runs once, stdin is consumed, thats fine) ── +# ── Sandbox setup (runs once) ────────────────────────────────────── rm -rf /sandbox/.claude/plugins /sandbox/.claude.json 2>/dev/null -# Env file for reconnects -cat > /sandbox/.openshell-env < /sandbox/.openshell-env </dev/null || { echo ". ~/.openshell-env 2>/dev/null" >> /sandbox/.bashrc @@ -119,17 +146,29 @@ grep -q openshell-env /sandbox/.bashrc 2>/dev/null || { } . /sandbox/.openshell-env +'"$GWS_SETUP"' -# GWS config -mkdir -p /tmp/gws-config -echo "'"$GWS_CS_B64"'" | base64 -d > /tmp/gws-config/client_secret.json -echo "'"$GWS_CR_B64"'" | base64 -d > /tmp/gws-config/credentials.enc -echo "'"$GWS_EK_B64"'" | base64 -d > /tmp/gws-config/.encryption_key -echo "'"$GWS_TC_B64"'" | base64 -d > /tmp/gws-config/token_cache.json - -# Claude config +# Claude MCP config — uses env vars injected by the atlassian provider cat > /sandbox/.claude.json </dev/null 2>&1 || true mkdir -p /sandbox/.local/bin -curl -fsSL -L https://github.com/googleworkspace/cli/releases/download/v0.22.5/google-workspace-cli-x86_64-unknown-linux-gnu.tar.gz /dev/null | tar xz -C /sandbox/.local/bin 2>/dev/null || true +curl -fsSL -L https://github.com/googleworkspace/cli/releases/download/v0.22.5/google-workspace-cli-x86_64-unknown-linux-gnu.tar.gz /dev/null \ + | tar xz -C /sandbox/.local/bin 2>/dev/null || true -# Auth gh -echo "$GITHUB_TOKEN" | gh auth login --with-token /dev/null 2>&1 || true +# Auth gh CLI using the token injected by the github provider +if [[ -n "${GITHUB_TOKEN:-}" ]]; then + echo "$GITHUB_TOKEN" | gh auth login --with-token /dev/null 2>&1 || true +fi echo "Setup complete." ' 2>&1 -# Get sandbox name +# ── Get sandbox name ─────────────────────────────────────────────────── if [[ ${#NAME_ARGS[@]} -gt 0 ]]; then SANDBOX_NAME="${NAME_ARGS[1]}" else - SANDBOX_NAME=$("$CLI" sandbox list 2>&1 | grep Ready | tail -1 | awk '{print $1}') + SANDBOX_NAME=$("$CLI" sandbox list --names 2>&1 | tail -1) fi echo "Connecting to $SANDBOX_NAME..." - -# Phase 2: connect with clean stdin — Claude gets pristine I/O -if $SHELL_MODE; then - "$CLI" sandbox connect "$SANDBOX_NAME" -else - "$CLI" sandbox connect "$SANDBOX_NAME" -fi - -# Cleanup if user didn't want --keep -if ! $USER_WANTS_KEEP; then - echo "Cleaning up sandbox $SANDBOX_NAME..." - "$CLI" sandbox delete "$SANDBOX_NAME" 2>/dev/null || true -fi +"$CLI" sandbox connect "$SANDBOX_NAME" diff --git a/openshift-deploy.md b/openshift-deploy.md deleted file mode 100644 index 85c1c33..0000000 --- a/openshift-deploy.md +++ /dev/null @@ -1,342 +0,0 @@ -# Deploy OpenShell to OpenShift - -Living document. Updated as we progress through each step. - -## Overview - -Deploy OpenShell gateway + sandbox infrastructure to an existing OpenShift cluster. This does NOT enable user namespace isolation (PR #983) — it uses the standard privileged sandbox model. - -## Prerequisites - -- [x] OpenShift cluster with `KUBECONFIG` set: `export KUBECONFIG=$PWD/kubeconfig` - - OCP 4.21 / K8s 1.34.6 / CRI-O 1.34.6 / RHEL CoreOS 9.6 / kernel 5.14 / x86_64 - - 3 masters + 3 workers (`api.rc-test-fact.ocp.infra.rox.systems:6443`) - - Default StorageClass: `ssd-csi` (PVCs work) -- [ ] `kubectl`, `helm`, `podman` on PATH -- [ ] OpenShell repo checked out (branch: `rc/openshift-deploy`) -- [ ] Rust toolchain installed (for building binaries) -- [ ] quay.io credentials (see `registry-credentials.md`) - -## High-Level Plan - -| Step | What | Status | -|------|------|--------| -| 1 | Build x86_64 images (gateway + supervisor) + native CLI | **done** | -| 2 | Create `openshell` namespace + label for privileged pods | **done** | -| 3 | Install Sandbox CRD + controller | **done** | -| 4 | Grant OpenShift SCCs (anyuid for gateway, privileged for sandboxes) | **done** | -| 5 | Generate mTLS certificates + create K8s secrets | **done** (certs at `~/.openshell-ocp-tls/`) | -| 6 | Push images to quay.io + create imagePullSecret | **done** | -| 7 | Install supervisor binary on nodes via DaemonSet (uses image from step 6) | **done** (6/6 nodes) | -| 8 | Deploy gateway with Helm | **done** (openshell-0 Running) | -| 9 | Configure CLI with mTLS + port-forward | **done** | -| 10 | Verify: create a sandbox and run a command | **done** | - -## Step 1: Build images (cross-compile for x86_64) - -Building on Apple Silicon for x86_64 nodes. The Dockerfile requires BuildKit (cache mounts, `$BUILDPLATFORM`), so use Docker (OrbStack) not podman. The Dockerfile handles Rust cross-compilation internally via `cross-build.sh`. - -```shell -# Login to quay.io (podman for push, Docker for build) -podman login -u "rcochran+rcochran_quay_scratch" \ - -p "1Q3TUPVODGOU0UE1KM7XQJ8SNR9JV042JVDNIQ88XWUGRRHZ5183CON2N87VS5ED" quay.io - -# Gateway image (Docker BuildKit) -DOCKER_BUILDKIT=1 docker build --platform linux/amd64 \ - -f deploy/docker/Dockerfile.images --target gateway \ - -t quay.io/rcochran/scratch:openshell-gateway-dev . - -# Supervisor image (reuses cached Rust deps from gateway build) -DOCKER_BUILDKIT=1 docker build --platform linux/amd64 \ - -f deploy/docker/Dockerfile.images --target supervisor \ - -t quay.io/rcochran/scratch:openshell-supervisor-dev . -``` - -Build the CLI locally (needs Rust 1.85+ for edition 2024 — install via `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`): - -```shell -cargo build -p openshell-cli -``` - -## Step 2: Create namespace - -```shell -kubectl create ns openshell -kubectl label ns openshell pod-security.kubernetes.io/enforce=privileged --overwrite -kubectl label ns openshell pod-security.kubernetes.io/warn=privileged --overwrite -``` - -## Step 3: Install Sandbox CRD + controller - -```shell -kubectl apply -f deploy/kube/manifests/agent-sandbox.yaml -``` - -This creates the `Sandbox` CRD (`agents.x-k8s.io/v1alpha1`) and deploys the controller in `agent-sandbox-system` namespace. - -## Step 4: Grant SCCs - -Gateway needs `anyuid` (runs as UID 1000). Sandbox pods need `privileged` (SYS_ADMIN, NET_ADMIN, SYS_PTRACE, SYSLOG + hostPath). - -```shell -# Gateway SA -kubectl create clusterrolebinding openshell-sa-anyuid \ - --clusterrole=system:openshift:scc:anyuid \ - --serviceaccount=openshell:openshell - -# Sandbox pods via openshell SA -kubectl create clusterrolebinding openshell-sa-privileged \ - --clusterrole=system:openshift:scc:privileged \ - --serviceaccount=openshell:openshell - -# Sandbox pods via default SA -kubectl create clusterrolebinding openshell-default-privileged \ - --clusterrole=system:openshift:scc:privileged \ - --serviceaccount=openshell:default - -# Sandbox CRD controller needs cluster-admin for ownerReferences -kubectl create clusterrolebinding agent-sandbox-admin \ - --clusterrole=cluster-admin \ - --serviceaccount=agent-sandbox-system:agent-sandbox-controller -``` - -## Step 5: Generate mTLS certificates + secrets - -```shell -TLSDIR=$(mktemp -d) - -# CA -openssl req -x509 -newkey rsa:2048 -nodes \ - -keyout $TLSDIR/ca.key -out $TLSDIR/ca.crt \ - -days 365 -subj "/CN=openshell-ca" 2>/dev/null - -# Server cert (SANs cover in-cluster DNS + localhost for port-forward) -openssl req -newkey rsa:2048 -nodes \ - -keyout $TLSDIR/server.key -out $TLSDIR/server.csr \ - -subj "/CN=openshell.openshell.svc.cluster.local" \ - -addext "subjectAltName=DNS:openshell.openshell.svc.cluster.local,DNS:openshell,DNS:localhost,IP:127.0.0.1" 2>/dev/null - -openssl x509 -req -in $TLSDIR/server.csr \ - -CA $TLSDIR/ca.crt -CAkey $TLSDIR/ca.key -CAcreateserial \ - -out $TLSDIR/server.crt -days 365 \ - -extfile <(echo "subjectAltName=DNS:openshell.openshell.svc.cluster.local,DNS:openshell,DNS:localhost,IP:127.0.0.1") 2>/dev/null - -# Client cert -openssl req -newkey rsa:2048 -nodes \ - -keyout $TLSDIR/client.key -out $TLSDIR/client.csr \ - -subj "/CN=openshell-client" 2>/dev/null - -openssl x509 -req -in $TLSDIR/client.csr \ - -CA $TLSDIR/ca.crt -CAkey $TLSDIR/ca.key -CAcreateserial \ - -out $TLSDIR/client.crt -days 365 2>/dev/null - -# Create K8s secrets -kubectl create secret tls openshell-server-tls -n openshell \ - --cert=$TLSDIR/server.crt --key=$TLSDIR/server.key - -kubectl create secret generic openshell-server-client-ca -n openshell \ - --from-file=ca.crt=$TLSDIR/ca.crt - -kubectl create secret generic openshell-client-tls -n openshell \ - --from-file=ca.crt=$TLSDIR/ca.crt \ - --from-file=tls.crt=$TLSDIR/client.crt \ - --from-file=tls.key=$TLSDIR/client.key - -kubectl create secret generic openshell-ssh-handshake -n openshell \ - --from-literal=secret=$(openssl rand -hex 32) -``` - -Note: `openshell-client-tls` must be generic (not `kubernetes.io/tls`) because it needs `ca.crt` in addition to `tls.crt` and `tls.key`. - -## Step 6: Push images to quay.io + create imagePullSecret - -Push the images built in step 1, plus pull and re-tag the sandbox base image: - -```shell -# Push gateway and supervisor (built in step 1) -podman push quay.io/rcochran/scratch:openshell-gateway-dev -podman push quay.io/rcochran/scratch:openshell-supervisor-dev - -# Pull the sandbox base image (multi-arch, will get amd64) and re-tag -podman pull --platform linux/amd64 ghcr.io/nvidia/openshell-community/sandboxes/base:latest -podman tag ghcr.io/nvidia/openshell-community/sandboxes/base:latest \ - quay.io/rcochran/scratch:openshell-sandbox-base -podman push quay.io/rcochran/scratch:openshell-sandbox-base -``` - -Create imagePullSecret so the cluster can pull from quay.io: - -```shell -export KUBECONFIG=$PWD/kubeconfig -kubectl apply -n openshell -f architecture/plans/quay-pull-secret.yaml -``` - -Note: The quay.io repo must be **public**, or you also need the imagePullSecret on the `default` SA for sandbox pods and the `agent-sandbox-system` namespace for the CRD controller. - -## Step 7: Install supervisor binary on nodes (DaemonSet) - -The supervisor image was built and pushed in steps 1 and 6. - -Deploy the installer DaemonSet: - -```shell -cat </tmp/pf.log 2>&1 & -``` - -Set up CLI gateway config with mTLS: - -```shell -mkdir -p ~/.config/openshell/gateways/ocp/mtls - -cp $TLSDIR/ca.crt ~/.config/openshell/gateways/ocp/mtls/ -cp $TLSDIR/client.crt ~/.config/openshell/gateways/ocp/mtls/tls.crt -cp $TLSDIR/client.key ~/.config/openshell/gateways/ocp/mtls/tls.key - -cat > ~/.config/openshell/gateways/ocp/metadata.json <<'EOF' -{ - "name": "ocp", - "gateway_endpoint": "https://127.0.0.1:18443", - "is_remote": false, - "gateway_port": 18443, - "auth_mode": "mtls" -} -EOF -``` - -Verify connectivity: - -```shell -OPENSHELL_GATEWAY=ocp target/debug/openshell status -``` - -Expected: - -``` -Server Status - Gateway: ocp - Server: https://127.0.0.1:18443 - Status: Connected -``` - -## Step 10: Verify — create a sandbox - -```shell -export OPENSHELL_GATEWAY=ocp - -target/debug/openshell sandbox create --no-bootstrap -- sh -lc \ - "echo '=== id ==='; id; \ - echo '=== hostname ==='; hostname; \ - echo '=== sandbox-ok ==='" -``` - -## Troubleshooting - -| Symptom | Cause | Fix | -|---------|-------|-----| -| `ErrImageNeverPull` on gateway pod | Image not in internal registry | Push with `podman push --tls-verify=false` | -| `unable to validate against any security context constraint` | Missing SCC grants | Run clusterrolebinding commands from step 4 | -| `cannot set blockOwnerDeletion` | CRD controller lacks RBAC | Grant cluster-admin to controller SA (step 4) | -| `hostPath type check failed: /opt/openshell/bin is not a directory` | Supervisor not installed | Deploy DaemonSet from step 7 | -| `Permission denied` accessing supervisor binary | SELinux blocking hostPath | Ensure `chcon -t container_file_t` was applied (step 7) | -| Gateway `CrashLoopBackOff` with `unable to open database file` | PVC permissions | Use `--set server.dbUrl="sqlite:/tmp/openshell.db"` | -| `dns error: failed to lookup address` from supervisor | DNS not resolving | Use ClusterIP directly in `server.grpcEndpoint` | - -## Cleanup - -```shell -kubectl delete sandbox --all -n openshell -helm uninstall openshell -n openshell -kubectl delete daemonset openshell-supervisor-installer -n openshell -kubectl delete clusterrolebinding openshell-sa-anyuid openshell-sa-privileged \ - openshell-default-privileged agent-sandbox-admin 2>/dev/null -kubectl delete -f deploy/kube/manifests/agent-sandbox.yaml -kubectl delete ns openshell -pkill -f "port-forward.*18443" -rm -rf ~/.config/openshell/gateways/ocp -``` diff --git a/quay-pull-secret.yaml.template b/quay-pull-secret.yaml.template deleted file mode 100644 index 4cf8c8e..0000000 --- a/quay-pull-secret.yaml.template +++ /dev/null @@ -1,15 +0,0 @@ -# Create this secret with your own registry credentials: -# -# kubectl create secret docker-registry quay-pull-secret -n openshell \ -# --docker-server=quay.io \ -# --docker-username="YOUR_ROBOT_ACCOUNT" \ -# --docker-password="YOUR_TOKEN" -# -# Or apply a YAML file: -# apiVersion: v1 -# kind: Secret -# metadata: -# name: quay-pull-secret -# data: -# .dockerconfigjson: -# type: kubernetes.io/dockerconfigjson diff --git a/sandbox-CLAUDE.md b/sandbox-CLAUDE.md index b6335be..cda7a76 100644 --- a/sandbox-CLAUDE.md +++ b/sandbox-CLAUDE.md @@ -1,18 +1,16 @@ # Sandbox Agent Instructions -You are running inside an OpenShell sandbox on an OpenShift cluster. You have access to the following tools and services. +You are running inside an OpenShell sandbox on an OpenShift cluster. Credentials are injected via the OpenShell provider system — they appear as environment variables automatically. ## Tools Available ### GitHub — `gh` CLI - Pre-authenticated. Use `gh` for all GitHub operations. - Examples: `gh repo clone`, `gh pr create`, `gh issue list`, `gh api` -- Do NOT use raw git credential helpers or GITHUB_TOKEN directly in commands. ### Jira & Confluence — mcp-atlassian MCP server -- Connected to `redhat.atlassian.net` via the `atlassian` MCP server. +- Connected via the `atlassian` MCP server (credentials injected by provider). - Use MCP tools for Jira searches, issue creation, comments, and Confluence page reads. -- Project keys: Use JQL for searching (e.g., `project = ROX AND ...`). ### Google Workspace — `gws` CLI - Pre-authenticated for Gmail, Calendar, Drive, Docs, Sheets. @@ -21,7 +19,6 @@ You are running inside an OpenShell sandbox on an OpenShift cluster. You have ac - `gws gmail users messages list --params '{"userId": "me", "maxResults": 5}'` - `gws calendar events list --params '{"calendarId": "primary", "maxResults": 5}'` - `gws drive files list --params '{"pageSize": 10}'` - - `gws docs documents get --params '{"documentId": "DOC_ID"}'` - Use `gws schema ` to discover API parameters. ### Kubernetes — `kubectl` @@ -31,16 +28,15 @@ You are running inside an OpenShell sandbox on an OpenShift cluster. You have ac ### General Tools - `python3`, `pip`, `uv` — Python 3.13 with a virtualenv at `/sandbox/.venv` - `node`, `npm` — Node.js 22 -- `git` — pre-installed -- `curl` — pre-installed +- `git`, `curl` — pre-installed - `cargo` — NOT available (no Rust toolchain in sandbox) -## Claude Code Configuration -- Running via **Vertex AI** (Google Cloud), not direct Anthropic API. +## Configuration +- Running via **Vertex AI** (Google Cloud) unless configured otherwise. - Model selection: Use `--model` flag if the default model isn't available. ## Conventions - Working directory: `/sandbox` - Writable paths: `/sandbox`, `/tmp` - Network: Outbound allowed to Google APIs, GitHub, Atlassian, npm/pypi. -- All credentials are injected at startup and cleaned up on sandbox exit. +- Credentials are managed by the OpenShell provider system and cleaned up on sandbox exit. diff --git a/sandbox-harness.md b/sandbox-harness.md index c7036fe..033ccfa 100644 --- a/sandbox-harness.md +++ b/sandbox-harness.md @@ -1,6 +1,6 @@ -# Sandbox Harness Setup +# Sandbox Harness Operations -Tracks everything needed to configure a Claude Code sandbox session on top of any base image. These steps are image-agnostic — they handle credentials, MCP servers, CLI tools, and network policy. +Tracks credentials, MCP servers, network policy, and the sandbox startup flow. ## Base Image Assumptions @@ -11,145 +11,109 @@ The harness expects the base image to provide: - `gh` CLI - `git`, `curl` -The current base image (`ghcr.io/nvidia/openshell-community/sandboxes/base:latest`) satisfies all of these. +The default base image (`ghcr.io/nvidia/openshell-community/sandboxes/base:latest`) satisfies all of these. -## Credentials (K8s Secrets) +## Credentials -All credentials are stored as K8s secrets in the `openshell` namespace and uploaded into sandboxes at creation time. They are never baked into images. +Credentials are managed by the **OpenShell provider system**. They are stored in the gateway database and injected as environment variables into sandbox pods automatically. -| Secret | Contents | Mount Path | Purpose | -|--------|----------|------------|---------| -| `gcp-adc` | `adc.json` — Google ADC (authorized_user refresh token) | `/tmp/adc.json` | Vertex AI auth for Claude Code | -| `atlassian-creds` | `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` | env vars | mcp-atlassian MCP server | -| `github-token` | `GITHUB_TOKEN` | env var | `gh` CLI auth | -| `gws-credentials` | `client_secret.json`, `credentials.enc`, `.encryption_key`, `token_cache.json` | `/tmp/gws-config/` | Google Workspace CLI (`gws`) | +### Provider-managed credentials -### How to revoke +| Provider | Type | Env Vars | Purpose | +|----------|------|----------|---------| +| `github` | github | `GITHUB_TOKEN` | `gh` CLI auth | +| `atlassian` | generic | `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` | mcp-atlassian MCP server | +| `anthropic` | anthropic | `ANTHROPIC_API_KEY` | Direct Anthropic API (optional) | + +Register via `setup-providers.sh` or manually: +```shell +openshell provider create --name github --type github --credential GITHUB_TOKEN="$GITHUB_TOKEN" +``` + +### File-based credentials (uploaded at sandbox creation) -- **GCP ADC**: `gcloud auth application-default revoke` (invalidates the refresh token everywhere) -- **Atlassian API token**: Revoke at https://id.atlassian.com/manage-profile/security/api-tokens -- **GitHub PAT**: Revoke at https://github.com/settings/tokens -- **GWS OAuth**: Revoke at https://myaccount.google.com/permissions (find the OAuth app and remove access) +| File | Upload Path | Purpose | +|------|-------------|---------| +| GCP ADC (`application_default_credentials.json`) | `/tmp/adc.json` | Vertex AI auth | +| GWS config directory | `/tmp/gws-config/` | Google Workspace CLI | + +These are uploaded via `--upload` flags in `ocp-sandbox.sh`. ### How to rotate ```shell -export KUBECONFIG=$PWD/kubeconfig +# Provider credentials — update in-place +openshell provider update github --credential GITHUB_TOKEN="ghp_new_token" +openshell provider update atlassian \ + --credential JIRA_API_TOKEN="new_token" -# GCP ADC +# GCP ADC — re-authenticate and launch a new sandbox gcloud auth application-default login -kubectl delete secret gcp-adc -n openshell -kubectl create secret generic gcp-adc -n openshell \ - --from-file=adc.json=$HOME/.config/gcloud/application_default_credentials.json - -# Atlassian -kubectl delete secret atlassian-creds -n openshell -kubectl create secret generic atlassian-creds -n openshell \ - --from-literal=JIRA_URL=https://redhat.atlassian.net \ - --from-literal=JIRA_USERNAME= \ - --from-literal=JIRA_API_TOKEN= - -# GitHub -kubectl delete secret github-token -n openshell -kubectl create secret generic github-token -n openshell \ - --from-literal=GITHUB_TOKEN= - -# GWS (re-auth locally first: gws auth login) -kubectl delete secret gws-credentials -n openshell -kubectl create secret generic gws-credentials -n openshell \ - --from-file=client_secret.json=$HOME/.config/gws/client_secret.json \ - --from-file=credentials.enc=$HOME/.config/gws/credentials.enc \ - --from-file=encryption_key=$HOME/.config/gws/.encryption_key \ - --from-file=token_cache.json=$HOME/.config/gws/token_cache.json + +# GWS OAuth — re-authenticate locally +gws auth login +# New sandboxes will pick up the updated files from $GWS_CONFIG_DIR ``` -## Environment Variables +### How to revoke -Set directly in the sandbox command (not via the provider system, which wraps values in proxy placeholders). +- **GCP ADC**: `gcloud auth application-default revoke` +- **Atlassian API token**: https://id.atlassian.com/manage-profile/security/api-tokens +- **GitHub PAT**: https://github.com/settings/tokens +- **GWS OAuth**: https://myaccount.google.com/permissions + +## Environment Variables ### Vertex AI (Claude Code) ```shell GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json CLAUDE_CODE_USE_VERTEX=1 CLOUD_ML_REGION=global -ANTHROPIC_VERTEX_PROJECT_ID=itpc-gcp-hcm-pe-eng-claude -GOOGLE_CLOUD_LOCATION=us-east5 -GOOGLE_CLOUD_PROJECT=itpc-gcp-hcm-pe-eng-claude +ANTHROPIC_VERTEX_PROJECT_ID= +GOOGLE_CLOUD_LOCATION= +GOOGLE_CLOUD_PROJECT= ``` -### Atlassian (mcp-atlassian) -```shell -JIRA_URL=https://redhat.atlassian.net -JIRA_USERNAME= -JIRA_API_TOKEN= -``` - -### GitHub (gh CLI) -```shell -GITHUB_TOKEN= -``` +Set via `VERTEX_PROJECT` and `VERTEX_REGION` environment variables before running `ocp-sandbox.sh`. ## MCP Servers -Installed at sandbox startup via `uv`. Configured via Claude Code's settings file. +Installed at sandbox startup via `pip`. Configured via Claude Code's config file. ### mcp-atlassian - **Source**: https://github.com/sooperset/mcp-atlassian -- **Install**: `uv pip install mcp-atlassian` -- **Config**: Written to `~/.claude/settings.json` at startup -- **Env vars**: `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` +- **Install**: `pip install mcp-atlassian` +- **Config**: Written to `~/.claude.json` at startup +- **Env vars**: Injected by the `atlassian` provider ### gws (Google Workspace CLI) - **Source**: https://github.com/googleworkspace/cli -- **Install**: `npm install -g @googleworkspace/cli` -- **Config**: `GOOGLE_WORKSPACE_CLI_CONFIG_DIR=/tmp/gws-config` (files uploaded from K8s secret) -- **Auth**: Uses pre-authenticated OAuth credentials (encrypted at rest in the secret) -- **Capabilities**: Gmail, Calendar, Drive, Docs, Sheets, Chat, Admin — 100+ API methods - -### Future MCP servers -Add entries here as we add more. The pattern is: -1. Add the package to the startup install command -2. Add the MCP server config to the settings.json template -3. Add required env vars to the credentials section -4. Add required network endpoints to the policy +- **Install**: Downloaded from GitHub releases +- **Config**: `GOOGLE_WORKSPACE_CLI_CONFIG_DIR=/tmp/gws-config` +- **Auth**: Uses pre-authenticated OAuth credentials uploaded from local machine ## Network Policy -File: `architecture/plans/vertex-policy.yaml` - -Current allowed endpoints: +File: `vertex-policy.yaml` | Pattern | Purpose | |---------|---------| | `*.googleapis.com:443` | Vertex AI, GCP auth | | `*.google.com:443` | Google OAuth | -| `*.anthropic.com:443` | Claude telemetry (statsig) | -| `github.com:443`, `*.github.com:443`, `*.githubusercontent.com:443` | GitHub API, repos | +| `*.anthropic.com:443` | Claude telemetry | +| `github.com:443`, `*.github.com:443` | GitHub API, repos | +| `*.atlassian.net:443`, `*.atl-paas.net:443` | Jira/Confluence | | `registry.npmjs.org:443` | npm packages | | `pypi.org:443`, `*.pythonhosted.org:443` | Python packages | -| `*.atlassian.net:443` | Jira/Confluence | -| `*.atl-paas.net:443` | Atlassian CDN/auth | - -## Startup Script - -The sandbox startup script (`ocp-sandbox.sh`) does: -1. Ensures port-forward is running -2. Ensures gateway config exists -3. Uploads ADC file -4. Sets all env vars -5. Installs MCP servers via `uv` -6. Writes Claude Code MCP config -7. Launches Claude Code (or shell) -## Custom Image Strategy (Future) +## Custom Image Strategy -When building platform-specific images, the harness separates cleanly: +The harness separates cleanly into image and configuration layers: -- **Image layer**: Base OS, runtimes (node, python, rust), CLI tools (gh, git, claude) -- **Harness layer** (this doc): Credentials, MCP servers, env vars, network policy +- **Image layer**: Base OS, runtimes, CLI tools +- **Harness layer**: Providers, MCP servers, env vars, network policy -The harness works on any image that meets the base assumptions above. To support a different platform: +To use a different base image: 1. Build an image with the required tools -2. Push to quay.io with a new tag (e.g., `scratch:openshell-sandbox-rust`) -3. Use `--from ` or update `server.sandboxImage` in the Helm values -4. The harness script works unchanged +2. Set `SANDBOX_IMAGE` before running `deploy.sh`, or use `--from ` on sandbox create +3. The harness scripts work unchanged diff --git a/setup-providers.sh b/setup-providers.sh new file mode 100755 index 0000000..52df86d --- /dev/null +++ b/setup-providers.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# Register credential providers with the OpenShell gateway. +# +# Run this after deploy.sh and after starting the port-forward. +# Credentials are read from environment variables on your machine and +# stored in the gateway's database — they are injected into sandbox pods +# automatically when you pass --provider flags to sandbox create. +# +# Usage: +# export GITHUB_TOKEN="ghp_..." +# export JIRA_URL="https://mysite.atlassian.net" +# export JIRA_USERNAME="user@example.com" +# export JIRA_API_TOKEN="ATATT..." +# export ANTHROPIC_API_KEY="sk-ant-..." # if using Anthropic directly +# ./setup-providers.sh +# +# For Vertex AI: no env var needed here — ADC file is uploaded at sandbox creation time. +# +# To update credentials later, re-run this script or use: +# openshell provider update --credential KEY=NEW_VALUE + +set -euo pipefail + +GATEWAY_NAME="${GATEWAY_NAME:-ocp}" +export OPENSHELL_GATEWAY="$GATEWAY_NAME" + +CLI="${OPENSHELL_CLI:-openshell}" +if ! command -v "$CLI" &>/dev/null; then + REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" + CLI="$REPO_ROOT/OpenShell/target/debug/openshell" +fi + +if ! "$CLI" gateway select "$GATEWAY_NAME" &>/dev/null; then + echo "ERROR: Gateway '$GATEWAY_NAME' not registered. Run deploy.sh first." + exit 1 +fi + +created=0 +skipped=0 +failed=0 + +register_provider() { + local name="$1" type="$2" + shift 2 + local cred_args=("$@") + + if "$CLI" provider get "$name" &>/dev/null 2>&1; then + echo " ↻ $name — already exists, updating credentials" + if "$CLI" provider update "$name" "${cred_args[@]}" 2>/dev/null; then + ((created++)) + else + echo " ✗ $name — update failed" + ((failed++)) + fi + else + if "$CLI" provider create --name "$name" --type "$type" "${cred_args[@]}" 2>/dev/null; then + echo " ✓ $name — registered" + ((created++)) + else + echo " ✗ $name — creation failed" + ((failed++)) + fi + fi +} + +echo "Registering providers with gateway '$GATEWAY_NAME'..." +echo "" + +# ── GitHub ───────────────────────────────────────────────────────────── +if [[ -n "${GITHUB_TOKEN:-}" ]]; then + register_provider github github \ + --credential "GITHUB_TOKEN=$GITHUB_TOKEN" +else + echo " – github — skipped (GITHUB_TOKEN not set)" + ((skipped++)) +fi + +# ── Atlassian (Jira + Confluence) ───────────────────────────────────── +if [[ -n "${JIRA_URL:-}" && -n "${JIRA_USERNAME:-}" && -n "${JIRA_API_TOKEN:-}" ]]; then + register_provider atlassian generic \ + --credential "JIRA_URL=$JIRA_URL" \ + --credential "JIRA_USERNAME=$JIRA_USERNAME" \ + --credential "JIRA_API_TOKEN=$JIRA_API_TOKEN" +else + echo " – atlassian — skipped (JIRA_URL, JIRA_USERNAME, or JIRA_API_TOKEN not set)" + ((skipped++)) +fi + +# ── Anthropic (direct API) ──────────────────────────────────────────── +if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then + register_provider anthropic anthropic \ + --credential "ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY" +else + echo " – anthropic — skipped (ANTHROPIC_API_KEY not set)" + ((skipped++)) +fi + +echo "" +echo "Done: $created registered, $skipped skipped, $failed failed." +echo "" +echo "Verify with:" +echo " openshell provider list" +echo "" +echo "Providers are attached to sandboxes with --provider flags:" +echo " openshell sandbox create --provider github --provider atlassian ..." From d465af465355eee0e267f79fa07fff7e6c9e888f Mon Sep 17 00:00:00 2001 From: Robby Cochran Date: Wed, 27 May 2026 08:10:37 -0700 Subject: [PATCH 2/3] fix(deploy): resolve image tags and auth for OpenShift deployment The local Helm chart uses appVersion 0.0.0 as a dev placeholder, which causes ImagePullBackOff for certgen and supervisor images. Auto-resolve the latest release tag from GitHub when no GATEWAY_IMAGE_TAG is set. Also set supervisor.image.tag to match the gateway tag and enable allowUnauthenticatedUsers for single-user dev clusters where the Helm chart does not render the mTLS auth config section. --- deploy.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/deploy.sh b/deploy.sh index 760d519..43c0dff 100755 --- a/deploy.sh +++ b/deploy.sh @@ -79,16 +79,26 @@ fi # ── Step 5: Helm install gateway ────────────────────────────────────── echo "=== Step 5: Deploying gateway via Helm ===" +# Resolve image tag — the local chart's appVersion is 0.0.0 (dev placeholder), +# so we default to the latest release tag when no override is provided. +if [[ -z "${GATEWAY_IMAGE_TAG:-}" ]]; then + GATEWAY_IMAGE_TAG=$(gh api repos/NVIDIA/OpenShell/releases/latest --jq '.tag_name' 2>/dev/null | sed 's/^v//' || echo "latest") + echo " Resolved image tag: $GATEWAY_IMAGE_TAG" +fi + HELM_ARGS=( --set server.sandboxImagePullPolicy=Always --set server.dbUrl="sqlite:/var/openshell/openshell.db" --set pkiInitJob.enabled=true --set pkiInitJob.serverDnsNames[0]=openshell.openshell.svc.cluster.local --set service.type=ClusterIP + --set image.tag="$GATEWAY_IMAGE_TAG" + --set image.pullPolicy=Always + --set supervisor.image.tag="$GATEWAY_IMAGE_TAG" + --set server.auth.allowUnauthenticatedUsers=true ) [[ -n "${GATEWAY_IMAGE_REPO:-}" ]] && HELM_ARGS+=(--set image.repository="$GATEWAY_IMAGE_REPO") -[[ -n "${GATEWAY_IMAGE_TAG:-}" ]] && HELM_ARGS+=(--set image.tag="$GATEWAY_IMAGE_TAG" --set image.pullPolicy=Always) [[ -n "${SUPERVISOR_IMAGE_REPO:-}" ]] && HELM_ARGS+=(--set supervisor.image.repository="$SUPERVISOR_IMAGE_REPO") [[ -n "${SANDBOX_IMAGE:-}" ]] && HELM_ARGS+=(--set server.sandboxImage="$SANDBOX_IMAGE") [[ -n "${PULL_SECRET:-}" ]] && HELM_ARGS+=(--set imagePullSecrets[0].name="$PULL_SECRET") From 5a33ff322c99c40f70a74bc241061f3d96d65e2f Mon Sep 17 00:00:00 2001 From: Robby Cochran Date: Thu, 28 May 2026 14:54:31 -0700 Subject: [PATCH 3/3] feat: decompose ADC into provider credentials with L7 body rewrite ADC secrets (client_secret, refresh_token) are stored as provider credentials and resolved by the proxy during OAuth token exchange. Non-secret fields are injected as literal env vars since they're read locally by Google's auth library. Atlassian credentials use base64-encoded literal env vars (Basic auth hides placeholders from the proxy). MCP config written via python3 to prevent JSON injection. Network policy splits Google endpoints: L7 with body rewrite on oauth2.googleapis.com only, plain L4 passthrough for Vertex AI streaming. Adds downloads.claude.ai for skill/plugin installs. --- README.md | 47 ++++------- credentials.md | 205 +++++++++++++++++++++++++++++++++++++++++++++ ocp-sandbox.sh | 205 ++++++++++++++++++++++++++++++--------------- sandbox-harness.md | 52 ++++++++++-- setup-providers.sh | 85 +++++++++++++------ vertex-policy.yaml | 26 +++++- 6 files changed, 493 insertions(+), 127 deletions(-) create mode 100644 credentials.md diff --git a/README.md b/README.md index ea7ccb8..3e6c482 100644 --- a/README.md +++ b/README.md @@ -30,14 +30,16 @@ A deployment harness for running AI agent sandboxes on OpenShift using [OpenShel # 2. Start port-forward to the gateway kubectl port-forward svc/openshell -n openshell 18443:8080 -# 3. Register your credentials as providers +# 3. Register provider credentials (GitHub, Anthropic, GCP ADC) export GITHUB_TOKEN="ghp_..." -export JIRA_URL="https://mysite.atlassian.net" -export JIRA_USERNAME="user@example.com" -export JIRA_API_TOKEN="ATATT..." +# ADC secrets are auto-extracted from your local ADC file by setup-providers.sh ./setup-providers.sh # 4. Launch an interactive Claude sandbox +# Atlassian creds are passed directly (not via provider) +export JIRA_URL="https://mysite.atlassian.net" +export JIRA_USERNAME="user@example.com" +export JIRA_API_TOKEN="ATATT..." ./ocp-sandbox.sh --name my-agent ``` @@ -49,39 +51,24 @@ export JIRA_API_TOKEN="ATATT..." | `setup-providers.sh` | Register credentials with the OpenShell provider system | | `ocp-sandbox.sh` | Launch/rejoin Claude sandboxes with all integrations | | `vertex-policy.yaml` | Network policy for sandbox egress | +| `credentials.md` | Credential flows, mechanisms, and rotation guide | | `sandbox-CLAUDE.md` | Agent instructions injected into sandboxes | | `verify-integrations.py` | Integration test script for all tools | | `future-ideas.md` | Roadmap (observability, CronJobs, web UI, memory) | -## How Credentials Work - -This harness uses the **OpenShell provider system** instead of raw K8s secrets. Credentials are: +## Credentials -1. Registered once via `setup-providers.sh` (stored in the gateway database) -2. Automatically injected as environment variables into sandbox pods -3. Attached to sandboxes with `--provider` flags +See [credentials.md](credentials.md) for the full credential reference — how each credential is stored, transported, and consumed in sandboxes. -| Provider | Type | Env Vars Injected | -|----------|------|-------------------| -| `github` | github | `GITHUB_TOKEN` | -| `atlassian` | generic | `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` | -| `anthropic` | anthropic | `ANTHROPIC_API_KEY` | +**Quick summary:** -File-based credentials (GCP ADC, GWS OAuth) are uploaded at sandbox creation time via `--upload`. - -### Updating Credentials - -```shell -# Update a single credential -openshell provider update github --credential GITHUB_TOKEN="ghp_new_token" - -# Re-discover from environment -export GITHUB_TOKEN="ghp_new_token" -openshell provider update github --from-existing - -# Or re-run the setup script -./setup-providers.sh -``` +| Credential | Mechanism | Setup | +|------------|-----------|-------| +| GitHub | Provider (Bearer auth) | `setup-providers.sh` | +| GCP ADC | Provider (decomposed, L7 body rewrite) | `setup-providers.sh` | +| Anthropic | Provider (Bearer auth) | `setup-providers.sh` (optional, not needed for Vertex) | +| Atlassian | Literal env vars (Basic auth) | Set `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` before `ocp-sandbox.sh` | +| Google Workspace | File upload | Pre-authenticate with `gws auth login` | ## Sandbox Usage diff --git a/credentials.md b/credentials.md new file mode 100644 index 0000000..6e3d6ad --- /dev/null +++ b/credentials.md @@ -0,0 +1,205 @@ +# Credential Flows + +How credentials are configured, stored, and consumed in sandboxes. + +## Mechanisms + +OpenShell supports two credential delivery mechanisms. Which one applies +depends on whether the consuming tool sends the credential over HTTP +(where the proxy can intercept it) or reads it locally. + +### Provider credentials (proxy-resolved) + +Credentials registered with `openshell provider create` are stored in the +gateway database and injected into sandbox pods as **placeholder** environment +variables (e.g., `openshell:resolve:env:v2_GITHUB_TOKEN`). The real value +never enters the sandbox. + +When the sandbox makes an HTTP request, the network proxy inspects headers +and (with L7 enabled) request bodies, replacing placeholders with real values +on the wire. The sandbox process only ever sees the placeholder string. + +This works for: +- **Bearer auth** — placeholder appears directly in `Authorization: Bearer ` +- **POST body params** — placeholder appears in form-encoded fields (requires `request_body_credential_rewrite: true` on the endpoint) + +### Literal env vars (direct injection) + +Some tools use auth schemes that transform credentials before sending them, +making placeholders unrecognizable to the proxy. These credentials must be +passed as real values. + +`ocp-sandbox.sh` base64-encodes values on the host and decodes them inside +the sandbox to survive multi-layer shell expansion without injection risk. + +### File upload + +Credentials stored as encrypted files that are read locally (not sent over +HTTP) can't use placeholders at all. These are uploaded at sandbox creation +via `--upload` and copied into place during setup. + +## Credentials + +### GitHub (`github` provider) + +| | | +|---|---| +| **Source** | `GITHUB_TOKEN` env var on host | +| **Registration** | `setup-providers.sh` → `openshell provider create --name github --type github` | +| **Sandbox delivery** | Provider placeholder in env | +| **Consumption** | `gh` CLI sends `Authorization: Bearer ` → proxy resolves | + +### GCP ADC (`gcp-adc` provider) + +Google's OAuth client library reads an Application Default Credentials file +and POSTs `client_secret` and `refresh_token` to `oauth2.googleapis.com` +as form-encoded body parameters during token exchange. + +| | | +|---|---| +| **Source** | `~/.config/gcloud/application_default_credentials.json` (or `$GOOGLE_APPLICATION_CREDENTIALS`) | +| **Registration** | `setup-providers.sh` extracts all 7 fields via `jq` → `openshell provider create --name gcp-adc --type generic` | +| **Sandbox delivery** | All fields injected as provider placeholder env vars | +| **Reconstruction** | Sandbox startup writes `/tmp/adc.json` from env vars — secret fields contain placeholders | +| **Consumption** | Google OAuth library POSTs secrets to `oauth2.googleapis.com` → proxy resolves via L7 `request_body_credential_rewrite` | + +**Provider credentials:** + +| Key | Secret? | Purpose | +|-----|---------|---------| +| `ADC_CLIENT_ID` | No | OAuth client identifier | +| `ADC_CLIENT_SECRET` | Yes | OAuth client secret | +| `ADC_REFRESH_TOKEN` | Yes | OAuth refresh token | +| `ADC_ACCOUNT` | No | GCP account email (may be empty) | +| `ADC_QUOTA_PROJECT_ID` | No | Billing/quota project | +| `ADC_TYPE` | No | Credential type (usually `authorized_user`) | +| `ADC_UNIVERSE_DOMAIN` | No | API universe (usually `googleapis.com`) | + +**Required env vars at launch:** + +| Var | Purpose | +|-----|---------| +| `VERTEX_PROJECT` | GCP project ID for Vertex AI | +| `VERTEX_REGION` | GCP region (optional, defaults to `global` via `CLOUD_ML_REGION`) | + +**L7 policy requirement:** The `google_apis` network policy must have +`request_body_credential_rewrite: true` on `*.googleapis.com` and +`*.google.com` endpoints. Without this, the proxy can't resolve placeholders +in POST bodies — only in headers. + +### Anthropic (`anthropic` provider) + +| | | +|---|---| +| **Source** | `ANTHROPIC_API_KEY` env var on host | +| **Registration** | `setup-providers.sh` → `openshell provider create --name anthropic --type anthropic` | +| **Sandbox delivery** | Provider placeholder in env | +| **Consumption** | Claude SDK sends `x-api-key: ` → proxy resolves | + +Not needed when using the Vertex AI path (`CLAUDE_CODE_USE_VERTEX=1`). + +### Atlassian (literal env vars) + +Atlassian credentials use **Basic auth**: `Authorization: Basic base64("username:token")`. +This is incompatible with provider placeholders because: + +1. The placeholder string (e.g., `openshell:resolve:env:v2_JIRA_API_TOKEN`) + would be concatenated with the username and base64-encoded +2. The resulting base64 string is opaque — the proxy can't pattern-match + the placeholder inside it +3. The proxy only recognizes raw placeholder strings, not encoded ones + +So these credentials are passed as **literal values** via `ocp-sandbox.sh`. + +| | | +|---|---| +| **Source** | `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` env vars on host | +| **Registration** | None — not provider-managed | +| **Sandbox delivery** | Base64-encoded in sandbox env file, decoded at shell init | +| **Consumption** | mcp-atlassian reads env vars, constructs Basic auth header, sends to `*.atlassian.net` | + +**Why base64 for transport?** The values pass through multiple shell expansion +stages: host-side double-quote interpolation into `ocp-sandbox.sh`, then +sandbox-side `source` of `.openshell-env`. Special characters (`'`, `"`, +`` ` ``, `$`, `\n`) in credential values could break quoting or enable +injection. Base64 encoding produces a safe ASCII string that survives all +expansion stages, then is decoded to the original value inside the sandbox. + +**Future: OAuth 2.0 3LO.** mcp-atlassian supports OAuth 2.0 with +`ATLASSIAN_OAUTH_CLIENT_SECRET`, `ATLASSIAN_OAUTH_CLOUD_ID`, etc. This uses +Bearer auth for API calls and POSTs `client_secret` during token refresh — +both compatible with provider placeholders (with L7 body rewrite on +`*.atlassian.com`). The `offline_access` scope enables automatic token +refresh inside mcp-atlassian. This would eliminate literal env vars entirely. + +Requires: Atlassian site admin approval of an OAuth 2.0 (3LO) app via +https://developer.atlassian.com/console/myapps/. Setup wizard: +`uvx mcp-atlassian --oauth-setup -v`. + +**If all three vars are unset**, mcp-atlassian is not configured and the +Atlassian MCP server is skipped. + +### Google Workspace (file upload) + +GWS credentials are encrypted by the `gws` CLI using its own encryption +scheme. The files are consumed locally by the `gws` binary — no HTTP +request carries the credential, so provider placeholders can't help. + +| | | +|---|---| +| **Source** | `$GWS_CONFIG_DIR` (default: `~/.config/gws/`) | +| **Registration** | None — file upload only | +| **Sandbox delivery** | Uploaded via `--upload` at sandbox creation, copied to `/tmp/gws-config/` | +| **Consumption** | `gws` CLI reads files directly from `$GOOGLE_WORKSPACE_CLI_CONFIG_DIR` | + +**Files:** + +| File | Purpose | +|------|---------| +| `client_secret.json` | OAuth client configuration | +| `credentials.enc` | Encrypted OAuth credentials | +| `token_cache.json` | Cached access/refresh tokens | +| `.encryption_key` | Encryption key for credentials.enc | + +When OpenShell adds file-based credential projection (issues #1268, #1423), +GWS files can move to the provider system. + +## Network Policy + +The sandbox network proxy enforces egress policy per `vertex-policy.yaml`. + +| Endpoint Pattern | L7 Body Rewrite | Purpose | +|-----------------|-----------------|---------| +| `*.googleapis.com:443` | Yes | Vertex AI, GCP OAuth token exchange | +| `*.google.com:443` | Yes | Google OAuth | +| `*.anthropic.com:443` | No | Claude telemetry | +| `github.com:443`, `*.github.com:443` | No | GitHub API | +| `*.atlassian.net:443`, `*.atl-paas.net:443` | No | Jira/Confluence | +| `registry.npmjs.org:443` | No | npm packages | +| `pypi.org:443`, `*.pythonhosted.org:443` | No | Python packages | + +## Rotation + +```shell +# Provider credentials — update in-place, new sandboxes pick up changes +openshell provider update github --credential GITHUB_TOKEN="ghp_new_token" + +# GCP ADC — re-authenticate, then re-run setup +gcloud auth application-default login +./setup-providers.sh + +# Atlassian — set env vars, launch new sandbox +export JIRA_API_TOKEN="new_token" + +# GWS — re-authenticate locally, new sandboxes upload fresh files +gws auth login +``` + +## Revocation + +| Credential | Revoke at | +|------------|-----------| +| GCP ADC | `gcloud auth application-default revoke` | +| Atlassian API token | https://id.atlassian.com/manage-profile/security/api-tokens | +| GitHub PAT | https://github.com/settings/tokens | +| GWS OAuth | https://myaccount.google.com/permissions | diff --git a/ocp-sandbox.sh b/ocp-sandbox.sh index 9e76086..669ebc6 100755 --- a/ocp-sandbox.sh +++ b/ocp-sandbox.sh @@ -1,8 +1,10 @@ #!/usr/bin/env bash # Create and connect to an OpenShell sandbox on the OCP cluster. # -# Credentials come from the provider system (see setup-providers.sh). -# File-based credentials (ADC, GWS) are uploaded at sandbox creation time. +# Credentials come from the provider system (GitHub, Anthropic, GCP ADC) +# and literal environment variables (Atlassian). GWS credentials are +# uploaded as files. ADC secrets are decomposed into provider credentials +# and reconstructed in-sandbox with placeholder tokens. # # Usage: # ./ocp-sandbox.sh # interactive Claude session @@ -10,6 +12,7 @@ # ./ocp-sandbox.sh --name my-sandbox # named sandbox # ./ocp-sandbox.sh --no-keep # delete sandbox after exit # ./ocp-sandbox.sh --rejoin my-sandbox # reconnect to existing sandbox +# ./ocp-sandbox.sh --editor vscode # open in VS Code set -euo pipefail @@ -31,6 +34,7 @@ NO_KEEP=false REJOIN="" NAME_ARGS=() EXTRA_PROVIDERS=() +EDITOR_ARGS=() while [[ $# -gt 0 ]]; do case $1 in --shell) SHELL_MODE=true; shift ;; @@ -38,6 +42,7 @@ while [[ $# -gt 0 ]]; do --rejoin) REJOIN="$2"; shift 2 ;; --name) NAME_ARGS=(--name "$2"); shift 2 ;; --provider) EXTRA_PROVIDERS+=(--provider "$2"); shift 2 ;; + --editor) EDITOR_ARGS=(--editor "$2"); shift 2 ;; *) shift ;; esac done @@ -52,7 +57,7 @@ fi # ── Rejoin mode ──────────────────────────────────────────────────────── if [[ -n "$REJOIN" ]]; then echo "Reconnecting to $REJOIN..." - exec "$CLI" sandbox connect "$REJOIN" + exec "$CLI" sandbox connect "$REJOIN" "${EDITOR_ARGS[@]}" fi # ── Build provider flags from registered providers ───────────────────── @@ -60,57 +65,108 @@ PROVIDER_FLAGS=() if "$CLI" provider get github &>/dev/null 2>&1; then PROVIDER_FLAGS+=(--provider github) fi -if "$CLI" provider get atlassian &>/dev/null 2>&1; then - PROVIDER_FLAGS+=(--provider atlassian) -fi if "$CLI" provider get anthropic &>/dev/null 2>&1; then PROVIDER_FLAGS+=(--provider anthropic) fi -PROVIDER_FLAGS+=("${EXTRA_PROVIDERS[@]}") - -# ── Build upload flags ───────────────────────────────────────────────── -UPLOAD_ARGS=() -# Vertex AI ADC -ADC_FILE="${GOOGLE_APPLICATION_CREDENTIALS:-$HOME/.config/gcloud/application_default_credentials.json}" -if [[ -f "$ADC_FILE" ]]; then - UPLOAD_ARGS+=(--upload "$ADC_FILE:/tmp/adc.json") +if "$CLI" provider get gcp-adc &>/dev/null 2>&1; then + PROVIDER_FLAGS+=(--provider gcp-adc) fi +PROVIDER_FLAGS+=("${EXTRA_PROVIDERS[@]}") + +# ── Stage upload directory ───────────────────────────────────────────── +# The CLI accepts a single --upload flag, so we stage everything into +# one temp directory and upload it to /sandbox/.harness. +UPLOAD_DIR=$(mktemp -d) +trap 'rm -rf "$UPLOAD_DIR"' EXIT + VERTEX_PROJECT="${VERTEX_PROJECT:-}" VERTEX_REGION="${VERTEX_REGION:-}" +# ── Read non-secret ADC fields from local file ─────────────────────── +# These are injected as literal env vars because Google's auth library +# reads them locally from the JSON file — placeholders won't work. +# Only client_secret and refresh_token are provider-managed (sent via HTTP). +ADC_FILE="${GOOGLE_APPLICATION_CREDENTIALS:-$HOME/.config/gcloud/application_default_credentials.json}" +ADC_CLIENT_ID="" +ADC_ACCOUNT="" +ADC_QUOTA_PROJECT_ID="" +ADC_TYPE="" +ADC_UNIVERSE_DOMAIN="" +if [[ -f "$ADC_FILE" ]] && command -v jq &>/dev/null; then + ADC_CLIENT_ID=$(jq -r '.client_id // empty' "$ADC_FILE") + ADC_ACCOUNT=$(jq -r '.account // empty' "$ADC_FILE") + ADC_QUOTA_PROJECT_ID=$(jq -r '.quota_project_id // empty' "$ADC_FILE") + ADC_TYPE=$(jq -r '.type // empty' "$ADC_FILE") + ADC_UNIVERSE_DOMAIN=$(jq -r '.universe_domain // empty' "$ADC_FILE") +fi + # GWS credentials directory GWS_CONFIG_DIR="${GWS_CONFIG_DIR:-$HOME/.config/gws}" -GWS_UPLOAD=false +HAS_GWS=false if [[ -d "$GWS_CONFIG_DIR" && -f "$GWS_CONFIG_DIR/client_secret.json" ]]; then - GWS_UPLOAD=true + mkdir -p "$UPLOAD_DIR/gws-config" + cp "$GWS_CONFIG_DIR"/* "$UPLOAD_DIR/gws-config/" 2>/dev/null || true + HAS_GWS=true +fi + +# sandbox-CLAUDE.md +if [[ -f "$SCRIPT_DIR/sandbox-CLAUDE.md" ]]; then + cp "$SCRIPT_DIR/sandbox-CLAUDE.md" "$UPLOAD_DIR/CLAUDE.md" fi -# ── Sandbox env vars (non-provider, Vertex AI config) ────────────────── +UPLOAD_ARGS=(--upload "$UPLOAD_DIR:/sandbox/.harness") + +# ── Sandbox env vars ────────────────────────────────────────────────── +# Atlassian credentials are passed as literal env vars (not via provider) +# because mcp-atlassian uses Basic auth — base64-encoding hides the +# placeholder tokens from the proxy's credential resolver. +JIRA_URL="${JIRA_URL:-}" +JIRA_USERNAME="${JIRA_USERNAME:-}" +JIRA_API_TOKEN="${JIRA_API_TOKEN:-}" + ENV_BLOCK="" -if [[ -f "$ADC_FILE" ]]; then - ENV_BLOCK+="export GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json +ENV_BLOCK+="export GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json " - ENV_BLOCK+="export CLAUDE_CODE_USE_VERTEX=1 +ENV_BLOCK+="export CLAUDE_CODE_USE_VERTEX=1 " - ENV_BLOCK+="export CLOUD_ML_REGION=global +ENV_BLOCK+="export CLOUD_ML_REGION=global " - [[ -n "$VERTEX_PROJECT" ]] && ENV_BLOCK+="export ANTHROPIC_VERTEX_PROJECT_ID=$VERTEX_PROJECT +[[ -n "$VERTEX_PROJECT" ]] && ENV_BLOCK+="export ANTHROPIC_VERTEX_PROJECT_ID=$VERTEX_PROJECT export GOOGLE_CLOUD_PROJECT=$VERTEX_PROJECT " - [[ -n "$VERTEX_REGION" ]] && ENV_BLOCK+="export GOOGLE_CLOUD_LOCATION=$VERTEX_REGION +[[ -n "$VERTEX_REGION" ]] && ENV_BLOCK+="export GOOGLE_CLOUD_LOCATION=$VERTEX_REGION " -fi + +# Non-secret ADC fields (literal values, read locally by Google auth library) +[[ -n "$ADC_CLIENT_ID" ]] && ENV_BLOCK+="export ADC_CLIENT_ID=$ADC_CLIENT_ID +" +[[ -n "$ADC_ACCOUNT" ]] && ENV_BLOCK+="export ADC_ACCOUNT=$ADC_ACCOUNT +" +[[ -n "$ADC_QUOTA_PROJECT_ID" ]] && ENV_BLOCK+="export ADC_QUOTA_PROJECT_ID=$ADC_QUOTA_PROJECT_ID +" +[[ -n "$ADC_TYPE" ]] && ENV_BLOCK+="export ADC_TYPE=$ADC_TYPE +" +[[ -n "$ADC_UNIVERSE_DOMAIN" ]] && ENV_BLOCK+="export ADC_UNIVERSE_DOMAIN=$ADC_UNIVERSE_DOMAIN +" + +# Atlassian env vars (literal values, not provider placeholders). +# Base64-encode to avoid shell injection through multi-layer expansion. +[[ -n "$JIRA_URL" ]] && ENV_BLOCK+="export JIRA_URL=\$(echo '$(echo -n "$JIRA_URL" | base64)' | base64 -d) +" +[[ -n "$JIRA_USERNAME" ]] && ENV_BLOCK+="export JIRA_USERNAME=\$(echo '$(echo -n "$JIRA_USERNAME" | base64)' | base64 -d) +" +[[ -n "$JIRA_API_TOKEN" ]] && ENV_BLOCK+="export JIRA_API_TOKEN=\$(echo '$(echo -n "$JIRA_API_TOKEN" | base64)' | base64 -d) +" + ENV_BLOCK+='export GOOGLE_WORKSPACE_CLI_CONFIG_DIR=/tmp/gws-config ' +ENV_BLOCK+='export CLAUDE_DANGEROUSLY_SKIP_PERMISSIONS=1 +' ENV_BLOCK+='export PATH="/sandbox/.local/bin:$PATH" ' -# ── Build Jira env for MCP config ────────────────────────────────────── -# The provider injects JIRA_URL, JIRA_USERNAME, JIRA_API_TOKEN as env vars. -# We reference them in the claude.json MCP config at startup. - # ── Create sandbox ───────────────────────────────────────────────────── echo "Creating sandbox..." @@ -118,12 +174,11 @@ KEEP_ARGS=() $NO_KEEP && KEEP_ARGS=(--no-keep) GWS_SETUP="" -if $GWS_UPLOAD; then +if $HAS_GWS; then GWS_SETUP=' mkdir -p /tmp/gws-config -cp /sandbox/gws-upload/* /tmp/gws-config/ 2>/dev/null || true +cp /sandbox/.harness/gws-config/* /tmp/gws-config/ 2>/dev/null || true ' - UPLOAD_ARGS+=(--upload "$GWS_CONFIG_DIR:/sandbox/gws-upload") fi "$CLI" sandbox create \ @@ -132,44 +187,74 @@ fi "${PROVIDER_FLAGS[@]}" \ "${KEEP_ARGS[@]}" \ "${NAME_ARGS[@]}" \ + "${EDITOR_ARGS[@]}" \ -- bash -c ' # ── Sandbox setup (runs once) ────────────────────────────────────── rm -rf /sandbox/.claude/plugins /sandbox/.claude.json 2>/dev/null +# Move staged files from upload directory +[[ -f /sandbox/.harness/CLAUDE.md ]] && cp /sandbox/.harness/CLAUDE.md /sandbox/CLAUDE.md + # Environment file for reconnects cat > /sandbox/.openshell-env </dev/null || { echo ". ~/.openshell-env 2>/dev/null" >> /sandbox/.bashrc - echo "alias claude=\"claude --dangerously-skip-permissions\"" >> /sandbox/.bashrc } . /sandbox/.openshell-env '"$GWS_SETUP"' -# Claude MCP config — uses env vars injected by the atlassian provider -cat > /sandbox/.claude.json < /tmp/adc.json < /sandbox/.claude.json < /sandbox/.claude/settings.json < /sandbox/.claude/settings.json </dev/null 2>&1 || true +pip install -q mcp-atlassian==0.21.1 &1 || echo "WARNING: mcp-atlassian install failed" mkdir -p /sandbox/.local/bin curl -fsSL -L https://github.com/googleworkspace/cli/releases/download/v0.22.5/google-workspace-cli-x86_64-unknown-linux-gnu.tar.gz /dev/null \ | tar xz -C /sandbox/.local/bin 2>/dev/null || true -# Auth gh CLI using the token injected by the github provider -if [[ -n "${GITHUB_TOKEN:-}" ]]; then - echo "$GITHUB_TOKEN" | gh auth login --with-token /dev/null 2>&1 || true -fi - echo "Setup complete." -' 2>&1 -# ── Get sandbox name ─────────────────────────────────────────────────── -if [[ ${#NAME_ARGS[@]} -gt 0 ]]; then - SANDBOX_NAME="${NAME_ARGS[1]}" -else - SANDBOX_NAME=$("$CLI" sandbox list --names 2>&1 | tail -1) -fi - -echo "Connecting to $SANDBOX_NAME..." -"$CLI" sandbox connect "$SANDBOX_NAME" +# Drop into interactive shell or claude +exec bash -l +' diff --git a/sandbox-harness.md b/sandbox-harness.md index 033ccfa..e5279c0 100644 --- a/sandbox-harness.md +++ b/sandbox-harness.md @@ -17,12 +17,14 @@ The default base image (`ghcr.io/nvidia/openshell-community/sandboxes/base:lates Credentials are managed by the **OpenShell provider system**. They are stored in the gateway database and injected as environment variables into sandbox pods automatically. -### Provider-managed credentials +### Provider-managed credentials (Bearer auth) + +Provider credentials are injected as placeholder env vars. The sandbox proxy +resolves them transparently in HTTP `Authorization: Bearer` headers. | Provider | Type | Env Vars | Purpose | |----------|------|----------|---------| | `github` | github | `GITHUB_TOKEN` | `gh` CLI auth | -| `atlassian` | generic | `JIRA_URL`, `JIRA_USERNAME`, `JIRA_API_TOKEN` | mcp-atlassian MCP server | | `anthropic` | anthropic | `ANTHROPIC_API_KEY` | Direct Anthropic API (optional) | Register via `setup-providers.sh` or manually: @@ -30,22 +32,52 @@ Register via `setup-providers.sh` or manually: openshell provider create --name github --type github --credential GITHUB_TOKEN="$GITHUB_TOKEN" ``` +### Direct env vars (Basic auth) + +Atlassian credentials use Basic auth (`base64(username:token)`), which hides +placeholders from the proxy resolver. These are passed as literal env vars +by `ocp-sandbox.sh` from the host environment. + +| Env Var | Purpose | +|---------|---------| +| `JIRA_URL` | Atlassian site URL | +| `JIRA_USERNAME` | Atlassian email | +| `JIRA_API_TOKEN` | Atlassian API token | + +### Decomposed file credentials (provider-managed) + +GCP ADC credentials are decomposed into individual provider fields so secrets +never exist as plaintext files in the sandbox. All 7 ADC fields are stored as +provider credentials by `setup-providers.sh` (requires `jq`). + +| Provider | Credentials | Source | +|----------|------------|--------| +| `gcp-adc` | `ADC_CLIENT_ID`, `ADC_CLIENT_SECRET`, `ADC_REFRESH_TOKEN`, `ADC_ACCOUNT`, `ADC_QUOTA_PROJECT_ID`, `ADC_TYPE`, `ADC_UNIVERSE_DOMAIN` | `application_default_credentials.json` | + +The ADC file (`/tmp/adc.json`) is reconstructed inside the sandbox from these +env vars. Secret fields contain placeholder tokens; when Google's OAuth library +sends a token exchange POST to `oauth2.googleapis.com`, the proxy resolves +them via L7 `request_body_credential_rewrite`. + ### File-based credentials (uploaded at sandbox creation) | File | Upload Path | Purpose | |------|-------------|---------| -| GCP ADC (`application_default_credentials.json`) | `/tmp/adc.json` | Vertex AI auth | | GWS config directory | `/tmp/gws-config/` | Google Workspace CLI | -These are uploaded via `--upload` flags in `ocp-sandbox.sh`. +GWS credentials are encrypted by the `gws` CLI and cannot be decomposed into +individual fields. They are uploaded via `--upload` in `ocp-sandbox.sh`. +When OpenShell adds file-based credential projection (issues #1268, #1423), +GWS files can move to the provider system. ### How to rotate ```shell # Provider credentials — update in-place openshell provider update github --credential GITHUB_TOKEN="ghp_new_token" -openshell provider update atlassian \ - --credential JIRA_API_TOKEN="new_token" + +# Atlassian — update env vars and launch a new sandbox +export JIRA_API_TOKEN="new_token" # GCP ADC — re-authenticate and launch a new sandbox gcloud auth application-default login @@ -106,6 +138,14 @@ File: `vertex-policy.yaml` | `registry.npmjs.org:443` | npm packages | | `pypi.org:443`, `*.pythonhosted.org:443` | Python packages | +### L7 Configuration + +The `google_apis` policy has L7 `request_body_credential_rewrite` enabled. +This is required because GCP OAuth token exchange sends `client_secret` and +`refresh_token` as form-encoded POST body parameters to `oauth2.googleapis.com`. +Without L7 body rewrite, the proxy can't resolve placeholder tokens in POST +bodies — only in headers and URL paths. + ## Custom Image Strategy The harness separates cleanly into image and configuration layers: diff --git a/setup-providers.sh b/setup-providers.sh index 52df86d..185b367 100755 --- a/setup-providers.sh +++ b/setup-providers.sh @@ -8,13 +8,18 @@ # # Usage: # export GITHUB_TOKEN="ghp_..." -# export JIRA_URL="https://mysite.atlassian.net" -# export JIRA_USERNAME="user@example.com" -# export JIRA_API_TOKEN="ATATT..." # export ANTHROPIC_API_KEY="sk-ant-..." # if using Anthropic directly # ./setup-providers.sh # -# For Vertex AI: no env var needed here — ADC file is uploaded at sandbox creation time. +# Note: Atlassian credentials (JIRA_URL, JIRA_USERNAME, JIRA_API_TOKEN) +# are NOT registered as providers. They are passed as literal env vars +# by ocp-sandbox.sh because mcp-atlassian uses Basic auth, which +# base64-encodes the credentials — hiding the placeholder tokens from +# the proxy's credential resolver. +# +# For Vertex AI: All ADC fields are extracted from the local ADC file +# and registered as the gcp-adc provider. The sandbox reconstructs +# adc.json from provider env vars. Requires jq. # # To update credentials later, re-run this script or use: # openshell provider update --credential KEY=NEW_VALUE @@ -47,18 +52,18 @@ register_provider() { if "$CLI" provider get "$name" &>/dev/null 2>&1; then echo " ↻ $name — already exists, updating credentials" if "$CLI" provider update "$name" "${cred_args[@]}" 2>/dev/null; then - ((created++)) + ((created++)) || true else echo " ✗ $name — update failed" - ((failed++)) + ((failed++)) || true fi else if "$CLI" provider create --name "$name" --type "$type" "${cred_args[@]}" 2>/dev/null; then echo " ✓ $name — registered" - ((created++)) + ((created++)) || true else echo " ✗ $name — creation failed" - ((failed++)) + ((failed++)) || true fi fi } @@ -67,39 +72,71 @@ echo "Registering providers with gateway '$GATEWAY_NAME'..." echo "" # ── GitHub ───────────────────────────────────────────────────────────── +# Token is injected as a placeholder env var. The sandbox proxy resolves +# it transparently in Authorization headers (Bearer auth). if [[ -n "${GITHUB_TOKEN:-}" ]]; then register_provider github github \ --credential "GITHUB_TOKEN=$GITHUB_TOKEN" else echo " – github — skipped (GITHUB_TOKEN not set)" - ((skipped++)) -fi - -# ── Atlassian (Jira + Confluence) ───────────────────────────────────── -if [[ -n "${JIRA_URL:-}" && -n "${JIRA_USERNAME:-}" && -n "${JIRA_API_TOKEN:-}" ]]; then - register_provider atlassian generic \ - --credential "JIRA_URL=$JIRA_URL" \ - --credential "JIRA_USERNAME=$JIRA_USERNAME" \ - --credential "JIRA_API_TOKEN=$JIRA_API_TOKEN" -else - echo " – atlassian — skipped (JIRA_URL, JIRA_USERNAME, or JIRA_API_TOKEN not set)" - ((skipped++)) + ((skipped++)) || true fi # ── Anthropic (direct API) ──────────────────────────────────────────── +# Same Bearer auth pattern — proxy resolves the placeholder in headers. if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then register_provider anthropic anthropic \ --credential "ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY" else echo " – anthropic — skipped (ANTHROPIC_API_KEY not set)" - ((skipped++)) + ((skipped++)) || true +fi + +# ── GCP ADC (Vertex AI) ────────────────────────────────────────────── +# Only the two secret fields (client_secret, refresh_token) are stored +# as provider credentials. These flow through HTTP during OAuth token +# exchange, where the proxy resolves placeholders via L7 body rewrite. +# +# Non-secret fields (client_id, account, type, etc.) are read locally +# from the ADC file by Google's auth library — placeholders wouldn't +# work there. ocp-sandbox.sh reads those from the ADC file at launch +# time and injects them as literal env vars. +ADC_FILE="${GOOGLE_APPLICATION_CREDENTIALS:-$HOME/.config/gcloud/application_default_credentials.json}" +if [[ -f "$ADC_FILE" ]]; then + if ! command -v jq &>/dev/null; then + echo " ✗ gcp-adc — jq required to extract ADC secrets" + ((failed++)) || true + else + ADC_CLIENT_SECRET=$(jq -r '.client_secret // empty' "$ADC_FILE") + ADC_REFRESH_TOKEN=$(jq -r '.refresh_token // empty' "$ADC_FILE") + if [[ -n "$ADC_CLIENT_SECRET" && -n "$ADC_REFRESH_TOKEN" ]]; then + register_provider gcp-adc generic \ + --credential "ADC_CLIENT_SECRET=$ADC_CLIENT_SECRET" \ + --credential "ADC_REFRESH_TOKEN=$ADC_REFRESH_TOKEN" + else + echo " – gcp-adc — skipped (ADC file missing client_secret or refresh_token)" + ((skipped++)) || true + fi + fi +else + echo " – gcp-adc — skipped (no ADC file at $ADC_FILE)" + ((skipped++)) || true fi echo "" echo "Done: $created registered, $skipped skipped, $failed failed." echo "" -echo "Verify with:" +echo "Note: Atlassian credentials are passed directly by ocp-sandbox.sh" +echo "(Basic auth — placeholders can't be used). Set JIRA_URL," +echo "JIRA_USERNAME, JIRA_API_TOKEN in your environment before running" +echo "ocp-sandbox.sh." +echo "" +echo "Note: mcp-atlassian also supports OAuth Bearer auth via" +echo "ATLASSIAN_OAUTH_ACCESS_TOKEN + ATLASSIAN_OAUTH_CLOUD_ID, which" +echo "would work with provider placeholders. However, OAuth access tokens" +echo "expire (~1 hour) and OpenShell doesn't yet support token refresh." +echo "" +echo "Verify providers with:" echo " openshell provider list" echo "" -echo "Providers are attached to sandboxes with --provider flags:" -echo " openshell sandbox create --provider github --provider atlassian ..." +echo "Providers are attached to sandboxes automatically by ocp-sandbox.sh." diff --git a/vertex-policy.yaml b/vertex-policy.yaml index c2cb4b7..7cf8bd4 100644 --- a/vertex-policy.yaml +++ b/vertex-policy.yaml @@ -15,12 +15,36 @@ process: # Testing policy — wildcard binaries on all endpoints. # Tighten binary paths before production. network_policies: + google_oauth: + name: google-oauth + endpoints: + - host: oauth2.googleapis.com + port: 443 + protocol: rest + enforcement: enforce + access: full + request_body_credential_rewrite: true + - host: "*.google.com" + port: 443 + protocol: rest + enforcement: enforce + access: full + request_body_credential_rewrite: true + binaries: + - { path: "/**" } + google_apis: name: google-apis endpoints: - host: "*.googleapis.com" port: 443 - - host: "*.google.com" + binaries: + - { path: "/**" } + + claude_downloads: + name: claude-downloads + endpoints: + - host: downloads.claude.ai port: 443 binaries: - { path: "/**" }