From c8f3f9becf191c24c5e09534253b6ea7aca36645 Mon Sep 17 00:00:00 2001 From: Robby Cochran Date: Thu, 4 Jun 2026 06:38:15 -0700 Subject: [PATCH 1/3] X-Smart-Branch-Parent: main From 3ae4673fe2b5376af865d35328161a1498279f4e Mon Sep 17 00:00:00 2001 From: Robby Cochran Date: Thu, 4 Jun 2026 06:55:31 -0700 Subject: [PATCH 2/3] docs: rewrite README, add SPEC.md, delete dead code, extract TOML parsing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README: rewrite for bin/harness CLI, profiles/, correct paths, add prerequisites, "Why use a sandbox?" section - SPEC.md: implementation-independent specification of the harness (CLI, config, sandbox lifecycle, credential flow, network policy, OCP launcher, testing) - REVIEW.md: 10-agent code review with prioritized recommendations - Delete dead code: sandbox-local.sh, sandbox/configure-mcp.py - Extract TOML parsing to bin/scripts/lib/parse-profile.py (standalone script, shared by profile.sh and launcher) - Rename parse_agent() to parse_profile() - Fix providers.sh broken path ($SCRIPT_DIR → $HARNESS_DIR) - Update PROVIDERS-SPEC.md agents/ → profiles/ Tested: bats 29/29, podman 13/13 (30s) Co-Authored-By: Claude Opus 4.6 (1M context) --- PROVIDERS-SPEC.md | 4 +- README.md | 115 ++++++++------ REVIEW.md | 233 ++++++++++++++++++++++++++++ SPEC.md | 252 +++++++++++++++++++++++++++++++ bin/scripts/lib/parse-profile.py | 40 +++++ bin/scripts/lib/profile.sh | 30 +--- bin/scripts/new.sh | 2 +- bin/scripts/providers.sh | 2 +- sandbox-local.sh | 105 ------------- sandbox/configure-mcp.py | 70 --------- 10 files changed, 609 insertions(+), 244 deletions(-) create mode 100644 REVIEW.md create mode 100644 SPEC.md create mode 100644 bin/scripts/lib/parse-profile.py delete mode 100755 sandbox-local.sh delete mode 100644 sandbox/configure-mcp.py diff --git a/PROVIDERS-SPEC.md b/PROVIDERS-SPEC.md index 368151d..40423ab 100644 --- a/PROVIDERS-SPEC.md +++ b/PROVIDERS-SPEC.md @@ -4,7 +4,7 @@ Three config files drive the harness: - **`providers.toml`** — provider definitions (inputs, types, checks) - **`openshell.toml`** — which providers to enable, inference model -- **`agents/*.toml`** — per-agent sandbox config (image, command, env vars) +- **`profiles/*.toml`** — per-agent sandbox config (image, command, env vars) ## providers.toml @@ -71,7 +71,7 @@ model = "claude-sonnet-4-6" If absent, all providers are enabled. -## agents/*.toml +## profiles/*.toml Per-agent sandbox configuration. `sandbox-podman.sh` and `sandbox-ocp.sh` read these. diff --git a/README.md b/README.md index 5f5b9a6..3197465 100644 --- a/README.md +++ b/README.md @@ -8,46 +8,79 @@ Deploy AI agent sandboxes on Podman (local) or OpenShift using [OpenShell](https - **GitHub** via gh CLI - Network policy enforcement per sandbox +## Prerequisites + +- [OpenShell CLI](https://github.com/NVIDIA/OpenShell) (`openshell`) +- Python 3.11+ (for TOML parsing) +- Podman (local) or kubectl + helm (OCP) +- `gcloud auth application-default login` (for Vertex AI) + +Optional: `gws` CLI (Google Workspace), `bats` (for unit tests) + +## Setup + +```bash +# Add harness CLI to PATH +export PATH="$PWD/bin:$PATH" + +# See available commands +harness +``` + ## Quick Start (Local) ```bash -# 1. Install OpenShell +# Install OpenShell if you haven't curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh -# 2. Verify gateway -./deploy-podman.sh - -# 3. Register providers (one-time) +# Set credentials export GITHUB_TOKEN="ghp_..." export JIRA_API_TOKEN="..." export ANTHROPIC_VERTEX_PROJECT_ID="my-project" export CLOUD_ML_REGION="us-east5" -./setup-providers.sh -# 4. Launch a sandbox -./sandbox-podman.sh +# Create a sandbox (deploys gateway + registers providers if needed) +harness new --local ``` ## Quick Start (OpenShift) ```bash -# 1. Deploy gateway to cluster -./deploy-ocp.sh +# Create a sandbox on the cluster +harness new --remote +``` -# 2. Store credentials + register providers -./setup-creds.sh -./setup-providers.sh +## CLI Reference -# 3. Launch a sandbox -./sandbox-ocp.sh ``` +harness new [--local|--remote] [--profile NAME] [SANDBOX_NAME] + Create a new sandbox. Auto-deploys gateway and registers providers if needed. -## Agent Configs +harness connect [SANDBOX_NAME] + Reconnect to a running sandbox. -Sandboxes are configured via `agents/*.toml`: +harness deploy [--local|--remote] + Deploy or verify the gateway without creating a sandbox. + +harness teardown [--sandboxes] [--providers] [--k8s] + Tear down sandboxes, providers, or k8s resources. + +harness preflight + Check environment prerequisites. + +harness providers + Register providers with the gateway. + +harness test [podman|ocp|all] [--full] + End-to-end validation. +``` + +## Profiles + +Sandboxes are configured via `profiles/*.toml`: ```toml -# agents/default.toml +# profiles/default.toml name = "agent" image = "quay.io/rcochran/openshell:sandbox" command = "claude --bare" @@ -58,49 +91,47 @@ ANTHROPIC_BASE_URL = "https://inference.local" JIRA_URL = "https://mysite.atlassian.net" ``` -Launch with a specific config: `./sandbox-podman.sh research` (uses `agents/research.toml`). +Use a specific profile: `harness new --profile coder` ## Testing ```bash -bats test/preflight.bats # unit tests (29 tests) -./test-flow.sh podman --full # full local validation -./test-flow.sh ocp --full # full OCP validation -make test # build images + test both +harness test podman --full # full local validation +harness test ocp --full # full OCP validation +bats test/preflight.bats # unit tests (29 tests) +make test # build images + test both ``` ## Files -| File | Purpose | +| Path | Purpose | |------|---------| -| `agents/default.toml` | Agent config (image, command, providers, env vars) | +| `bin/harness` | CLI entry point | +| `bin/scripts/` | Subcommand scripts (new, deploy, teardown, etc.) | +| `bin/scripts/lib/` | Shared libraries (profile parsing, providers, common) | +| `profiles/default.toml` | Default sandbox profile | | `providers.toml` | Provider definitions (env/file/check inputs) | -| `openshell.toml` | Which providers to enable, inference model | -| `deploy-podman.sh` | Verify local gateway is running | -| `deploy-ocp.sh` | Deploy OpenShell to OpenShift (Helm + route) | -| `setup-providers.sh` | Register providers with the gateway | -| `setup-creds.sh` | Store GWS + Atlassian config in cluster (OCP only) | -| `sandbox-podman.sh` | Launch sandbox locally | -| `sandbox-ocp.sh` | Launch sandbox on OpenShift | -| `teardown.sh` | Tear down sandboxes, providers, k8s resources | -| `test-flow.sh` | End-to-end validation | -| `openshell-harness-preflight.sh` | Pre-flight environment check | +| `openshell.toml` | Which providers to enable, upstream version pin | +| `sandbox/` | Sandbox image (Dockerfile, startup.sh, policy.yaml, CLAUDE.md) | +| `sandbox/launcher/` | In-cluster launcher image (for OCP sandboxes) | +| `test/` | Tests (preflight.bats, test-flow.sh) | +| `values-ocp.yaml` | Helm values for OpenShift deployment | | `AGENTS.md` | Project principles and workaround tracking | -## Sandbox Usage +## Why Use a Sandbox? -```bash -openshell sandbox connect # reconnect -openshell sandbox list # list running -openshell sandbox delete # delete -``` +Compared to running Claude Code locally: +- **Credential isolation** — sandbox never sees real API tokens (proxy-resolved placeholders) +- **Network policy** — per-binary egress rules (policy.yaml controls which processes reach which hosts) +- **Reproducible environment** — pinned tool versions in Dockerfile +- **Team sharing** — OCP deployment with mTLS, shared gateway, per-user sandboxes ## Architecture ``` Your Mac OpenShift Cluster ┌──────────┐ ┌──────────────────────────────┐ -│ openshell│ OpenShift Route │ Gateway (StatefulSet) │ +│ harness │ OpenShift Route │ Gateway (StatefulSet) │ │ CLI ├──────────────────►│ ├─ gRPC API │ │ │ TLS passthrough │ ├─ inference.local proxy │ │ │ mTLS :443 │ ├─ Provider credential store │ diff --git a/REVIEW.md b/REVIEW.md new file mode 100644 index 0000000..2a75ce7 --- /dev/null +++ b/REVIEW.md @@ -0,0 +1,233 @@ +# Harness-OpenShell Code Review: Prioritized Recommendations + +10 independent reviewers scored the repo across 8 dimensions. Weighted average: **5.3/10**. + +The code architecture is sound (7-8/10) but the documentation is broken (3/10), multi-user support is missing (3/10), and test coverage has critical gaps (4/10). The codebase is ~2100 lines of CLI glue designed to shrink as upstream OpenShell absorbs workarounds — a compiled rewrite would be counterproductive. + +| Dimension | Score | +|---|---| +| New Engineer Onboarding | 3 | +| Bash Code Quality | 6 | +| Security | 5 | +| Architecture | 7 | +| Testing | 4 | +| Upstream Alignment | 7 | +| Developer Workflow | 6 | +| Team Scaling (20-person) | 3 | +| Rewrite Analysis | 8 | +| Strategic Direction | 6 | + +--- + +## 1. README Is Completely Broken + +**Impact: Blocks every new user on step one.** + +Every Quick Start command references scripts deleted in commit `c12db4d`. The Files table lists 9 nonexistent files. `agents/` references should be `profiles/`. Test paths are wrong. No prerequisites listed. No mention of `bin/harness` CLI or how to add it to PATH. + +**What to fix:** +- Rewrite Quick Start to use `harness deploy --local`, `harness providers`, `harness new --local` +- Replace Files table with actual structure: `bin/harness`, `bin/scripts/*.sh`, `profiles/default.toml`, `sandbox/`, `test/` +- Replace all `agents/` references with `profiles/` +- Add Prerequisites section (openshell, python3 3.11+, podman/docker, gcloud, bats) +- Add "How to use the CLI" section with PATH setup and `harness --help` output +- Add "Why use a sandbox instead of local Claude?" section (credential isolation, network policy, reproducibility) +- Fix test command paths to `test/test-flow.sh` or `harness test` + +**Effort:** 2-3 hours +**Flagged by:** Onboarding (critical x3), Developer Workflow (high), Strategic Direction (medium) + +--- + +## 2. Multi-User Is Fundamentally Broken + +**Impact: Cannot deploy to a 20-person team without credential collision and sandbox name conflicts.** + +Five independent single-user assumptions block team scaling: + +| Problem | Where | Consequence | +|---|---|---| +| Sandbox name hardcoded to "agent" | `profiles/default.toml:7` | Two users overwrite each other's ConfigMap | +| Provider names are gateway-global | `providers.sh:76,118` | All 20 users share one GITHUB_TOKEN | +| GWS creds are a single user's OAuth | `creds.sh:54` | All sandboxes mount one person's Gmail | +| Teardown kills all sandboxes | `teardown.sh:63-69` | One user nukes the team | +| Shared Vertex AI quota | `providers.sh:96-99` | 20 sessions exhaust rate limits | + +**What to fix:** +Introduce a `$HARNESS_USER` variable (default: `$USER`) that prefixes: +- Sandbox names: `agent` becomes `rcochran-agent` +- Provider names: `github` becomes `github-rcochran` +- K8s secrets: `openshell-gws` becomes `openshell-gws-rcochran` +- Teardown filter: only delete sandboxes matching `$HARNESS_USER-*` + +This is a ~50-line change across `new.sh`, `providers.sh`, `creds.sh`, and `teardown.sh`. + +**Effort:** 1 day +**Flagged by:** Team Scaling (critical x2, high x4), Security (high) + +--- + +## 3. Dead Code Cleanup + +**Impact: Reduces confusion, aligns codebase with stated "shrink, not grow" principle.** + +| File | Lines | Why dead | +|---|---|---| +| `sandbox-local.sh` | 106 | Superseded by `bin/scripts/new.sh --local` | +| `sandbox/configure-mcp.py` | 71 | `mcp.json` baked into image; startup.sh never calls this | +| `sandbox/__pycache__/` | -- | Artifact of dead configure-mcp.py | + +**Effort:** 30 minutes +**Flagged by:** Architecture (medium), Upstream Alignment (medium x2), Developer Workflow (low), Strategic Direction (medium) + +--- + +## 4. Security: Gateway Auth and Cluster Permissions + +**Impact: Defense-in-depth failures that become critical at team scale.** + +**4a. Gateway allows unauthenticated users** +`values-ocp.yaml:29` sets `allowUnauthenticatedUsers: true`. mTLS is the only gate. If mTLS misconfigures, the gateway is open. Fix: enable OIDC auth in addition to mTLS. + +**4b. cluster-admin ClusterRoleBinding** +`deploy.sh:111-113` grants `cluster-admin` to the sandbox controller. Unrestricted access to every secret in every namespace. Fix: scope the ClusterRole to Sandbox CRDs, pods, and services in the openshell namespace. + +**4c. Privileged SCC on three service accounts** +`deploy.sh:107-110` grants privileged SCC to openshell, openshell-sandbox, and `default`. The `default` SA grant is especially dangerous. Fix: use `restricted` or `nonroot-v2` SCC where possible. + +**4d. mTLS key written without file permissions** +`deploy.sh:206-210` writes `tls.key` without `chmod 600`. Fix: add `chmod 600` immediately after writing. + +**Effort:** 4-6 hours +**Flagged by:** Security (high x4), Team Scaling (high) + +--- + +## 5. Test Coverage Gaps + +**Impact: The most critical code paths (profile parsing, provider flag building, error handling) have zero tests.** + +**What has tests:** `providers.py` preflight checks — 29 bats tests, thorough and well-structured. + +**What has no tests:** + +| Code | Risk | Why it matters | +|---|---|---| +| `lib/profile.sh` (parse_agent, build_provider_flags, stage_harness_dir) | Uses `eval` on Python output | Shell injection if quoting breaks | +| `lib/common.sh` (require_cli, export_gws_creds) | Credential handling | Silent failures leak or lose creds | +| Error paths in deploy.sh, new.sh, teardown.sh | Missing CLI, missing gateway, running sandboxes | Users see cryptic failures | +| `bin/harness` CLI dispatch | Unknown commands, help text | No verification that help matches reality | + +All testable offline using the same stub pattern as `preflight.bats`. + +**Effort:** 1-2 days for profile.sh + common.sh bats tests; 1 day for error path tests +**Flagged by:** Testing (high x3), Developer Workflow (medium) + +--- + +## 6. DRY Violations: Three Duplicated Patterns + +**Impact: Bugs fixed in one copy will not be fixed in the other.** + +| Pattern | Copies | Files | +|---|---|---| +| TOML parsing via inline Python | 2 | `profile.sh:16-29`, `entrypoint.sh:49-63` | +| Provider flag building loop | 3 | `profile.sh:34-44`, `entrypoint.sh:79-87`, `sandbox-local.sh:46-59` | +| GWS credential export | 3 | `common.sh:42-64`, `profile.sh:60-66`, `sandbox-local.sh:68-83` | + +**What to fix:** +- Extract TOML parsing into `lib/parse_profile.py` +- Delete `sandbox-local.sh` (removes 2 of 3 copies) +- Consolidate GWS export into `common.sh:export_gws_creds()` and call from profile.sh + +**Effort:** 3-4 hours (mostly after deleting sandbox-local.sh) +**Flagged by:** Code Quality (medium x3), Architecture (medium) + +--- + +## 7. Broken Path in providers.sh + +**Impact: Profile import likely fails silently.** + +`providers.sh:64` references `$SCRIPT_DIR/sandbox/profiles/` but `SCRIPT_DIR` is `bin/scripts`, so it resolves to `bin/scripts/sandbox/profiles/` which does not exist. Should be `$HARNESS_DIR/sandbox/profiles/`. + +**Effort:** 5 minutes +**Flagged by:** Code Quality (high) + +--- + +## 8. Developer Inner Loop Is Too Slow + +**Impact: Every sandbox config change requires a multi-arch Docker build + push (minutes).** + +Any change to CLAUDE.md, mcp.json, settings.json, startup.sh, or policy.yaml requires `docker buildx build --push` before testing. The `openshell sandbox upload` primitive already exists. + +**What to fix:** Add a `harness sync` subcommand that uploads sandbox config files to a running sandbox without rebuilding the image. + +**Effort:** 2-3 hours +**Flagged by:** Developer Workflow (high), Strategic Direction (medium) + +--- + +## 9. Retry Loop Masks Root Cause + +**Impact: When sandbox creation fails, developers get "Attempt N failed (supervisor race)" with zero diagnostic info.** + +`new.sh:98-116` and `entrypoint.sh:113-126` retry 5 times with backoff but capture no stderr. `test-flow.sh` swallows all output with `&>/dev/null`. + +**What to fix:** Capture and display stderr from failed `openshell sandbox create` attempts. Add `--verbose` flag. Log actual exit codes. + +**Effort:** 1-2 hours +**Flagged by:** Developer Workflow (high), Testing (medium) + +--- + +## 10. Naming Inconsistency: agent vs profile + +**Impact: Low but pervasive confusion.** + +`profile.sh` defines `parse_agent()`. CLI uses `--profile`. Directory is `profiles/`. PROVIDERS-SPEC.md still says `agents/*.toml`. + +**What to fix:** Rename `parse_agent()` to `parse_profile()`, update PROVIDERS-SPEC.md references. + +**Effort:** 30 minutes +**Flagged by:** Code Quality (low), Onboarding (high) + +--- + +## Should We Rewrite? + +**No. With one exception.** + +- The codebase is 2100 lines of CLI glue. 90% is "run this command, check exit code." A Go version would be `os/exec` calls doing the same thing less readably. +- The harness is designed to shrink. AGENTS.md tracks 5 upstream workarounds. Each eliminated removes 50-150 lines. +- Distribution is not an issue. Users already have openshell, kubectl, helm, python3, and podman installed. +- The complexity threshold for rewrite (~5000+ lines, multiple contributors) is not met. + +**The one exception: the in-cluster launcher.** `sandbox/launcher/entrypoint.sh` (138 lines) runs inside a K8s Job, shells out to Python for TOML parsing, and patches gateway metadata.json. A Go binary would produce a scratch-based image (~50MB vs current), eliminate the python3/tomli runtime dependency, and speed up Job startup. A Go rewrite spec already exists at `sandbox/launcher/SPEC.md`. + +**Recommendation:** Rewrite the launcher only. Leave everything else as bash+python. Revisit if the harness exceeds 15 scripts or 3000 lines. + +--- + +## Strategic Direction + +The harness's destiny is to shrink into upstream OpenShell contributions: +1. Submit `sandbox/Dockerfile` as an OpenShell-Community sandbox +2. Upstream the `atlassian.yaml` provider profile +3. Reduce this repo to team-specific profile TOML files and a thin README + +The 10x multiplier is multi-tenant team profiles (`profiles/collector.toml`, `profiles/scanner.toml`) that let the whole RHACS org share one gateway with per-team sandboxes. + +--- + +## Next 3 Things To Do + +### 1. Fix the README (2-3 hours) +Single highest-leverage change. Every new user currently fails on step one. Rewrite Quick Start with actual CLI commands, replace Files table, add prerequisites, add PATH setup. + +### 2. Delete dead code + fix providers.sh path (1 hour) +Remove `sandbox-local.sh`, `sandbox/configure-mcp.py`, `sandbox/__pycache__/`. Fix `$SCRIPT_DIR` to `$HARNESS_DIR` on `providers.sh:64`. Eliminates 177 lines and fixes broken provider import. + +### 3. Add $HARNESS_USER namespacing (1 day) +Prefix sandbox names, provider names, K8s secrets, and teardown filters with `$HARNESS_USER`. Gate to multi-user operation. ~50 lines across 4 files. diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..c5a8183 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,252 @@ +# OpenShell Harness Specification + +This document specifies the behavior of the OpenShell Harness independently of its implementation. Any conforming implementation (bash, Go, Rust, Python) should produce the same observable behavior. + +## Overview + +The harness deploys and manages AI agent sandboxes on two platforms: +- **Local** — Podman containers via a local OpenShell gateway +- **Remote** — Kubernetes pods via an OpenShift-hosted OpenShell gateway + +Each sandbox is an isolated container with a Claude Code agent, credential providers, MCP servers, network policies, and uploaded configuration. + +--- + +## CLI + +The harness exposes a single entry point (`harness`) with subcommands. + +### `harness new [--local|--remote] [--profile NAME] [--name SANDBOX_NAME] [--no-tty]` + +Create a new sandbox. This is the primary command. It performs these steps in order: + +1. **Ensure gateway** — if `--local`, verify the local podman gateway is running. If `--remote`, deploy to OpenShift (Helm chart, CRDs, SCCs, route, mTLS certs). If neither, check for an active gateway. +2. **Ensure providers** — if no providers are registered on the gateway, run provider registration. +3. **Ensure credentials** (remote only) — if K8s secrets for GWS/Atlassian don't exist, create them. +4. **Parse profile** — read `profiles/.toml` (default: `default`). +5. **Stage files** — write `sandbox.env` from profile `[env]`, export GWS credentials. +6. **Create sandbox** — call `openshell sandbox create` with `--from` (image), `--provider` (each provider), `--upload` (staged files), and the startup command. Retry up to 5 times for supervisor race conditions. + +If `--no-tty` is passed, the sandbox runs `startup.sh` and exits (for testing). Otherwise, it runs `startup.sh` then execs into the configured command (e.g., `claude --bare`). + +If `--name` is not provided, the sandbox name comes from the profile's `name` field. + +### `harness connect [SANDBOX_NAME]` + +Reconnect to a running sandbox via `openshell sandbox connect`. + +### `harness deploy [--local|--remote]` + +Deploy or verify the gateway without creating a sandbox. + +**Local:** Find a gateway with endpoint `127.0.0.1`, select it, verify it responds. + +**Remote:** Create namespace, install CRDs, grant SCCs, deploy Helm chart from OCI registry (`oci://ghcr.io/nvidia/openshell/helm-chart`), create TLS passthrough route, register CLI gateway with mTLS certs from the cluster. + +### `harness teardown [--sandboxes] [--providers] [--k8s]` + +Tear down resources. Default (no flags) tears down everything applicable. + +- `--sandboxes` — delete all sandboxes on the active gateway +- `--providers` — delete all providers and inference config (requires no running sandboxes) +- `--k8s` — Helm uninstall, delete CRDs, SCCs, secrets, namespace, and gateway config + +### `harness preflight` + +Read-only environment check. Validates all inputs defined in `providers.toml` for each enabled provider in `openshell.toml`. Reports per-input status with `✓`/`✗` prefixes. + +### `harness providers` + +Register credential providers with the gateway. Reads provider types and credentials from environment variables. Supports `--force` to delete and recreate all providers. + +### `harness test [podman|ocp|all] [--full]` + +End-to-end validation. Quick mode: deploy → providers → gateway check → teardown. Full mode adds: sandbox create → verify env vars, GWS creds, MCP config, Claude responds → sandbox delete → teardown. + +--- + +## Configuration + +### `providers.toml` + +Catalog of provider definitions. Each `[[providers]]` entry has: + +| Field | Required | Description | +|-------|----------|-------------| +| `name` | yes | Unique identifier | +| `type` | yes | `"openshell"` or `"custom"` | +| `description` | yes | Shown in preflight | +| `required` | no | If true, `--strict` preflight fails when inputs missing | +| `method` | no | Registration method (e.g., `"from-gcloud-adc"`) | +| `upstream` | no | Link to upstream issue (custom providers) | + +Each provider has an `inputs` array of inline tables: + +| Field | Required | Description | +|-------|----------|-------------| +| `key` | yes | Env var name, file path, or shell command | +| `kind` | yes | `"env"`, `"file"`, or `"check"` | +| `secret` | no | Mask value in preflight output | + +### `openshell.toml` + +Deployment configuration: + +```toml +providers = ["github", "vertex-local", "atlassian"] +providers-custom = ["gws"] + +[inference] +model = "claude-sonnet-4-6" + +[upstream] +chart-version = "0.0.55" +``` + +### `profiles/.toml` + +Per-sandbox configuration: + +```toml +name = "agent" +image = "quay.io/rcochran/openshell:sandbox" +command = "claude --bare" +keep = true +providers = ["github", "vertex-local", "atlassian"] + +[env] +ANTHROPIC_BASE_URL = "https://inference.local" +ANTHROPIC_API_KEY = "sk-ant-openshell-proxy-managed" +JIRA_URL = "https://mysite.atlassian.net" +JIRA_USERNAME = "user@example.com" +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `name` | `"agent"` | Sandbox name (overridden by `--name`) | +| `image` | none | Container image for the sandbox | +| `command` | `"claude --bare"` | Command to exec after startup | +| `keep` | `true` | Keep sandbox alive after command exits | +| `providers` | `[]` | Provider names to attach | +| `[env]` | `{}` | Environment variables injected into the sandbox | + +--- + +## Sandbox Lifecycle + +### Creation + +1. Profile parsed → `SANDBOX_IMAGE`, `SANDBOX_COMMAND`, `SANDBOX_PROVIDERS`, `SANDBOX_ENV` +2. Files staged to `/tmp/openshell/`: + - `sandbox.env` — export statements from `[env]` section + - `credentials.json` — GWS OAuth credentials (if available) + - `client_secret.json` — GWS OAuth client config (if available) +3. `openshell sandbox create` called with: + - `--from ` — sandbox container image + - `--provider ` — for each provider + - `--upload /tmp/openshell:/sandbox/.config` — files land at `/sandbox/.config/openshell/` + - `-- bash -c '. /sandbox/startup.sh && exec '` (tty mode) + - `-- bash /sandbox/startup.sh` (no-tty mode) +4. On failure (supervisor race), delete sandbox and retry (up to 5 times, 5s between) + +### Startup (inside sandbox) + +`startup.sh` runs once at creation: +1. Source `/sandbox/.config/openshell/sandbox.env` → append to `.bashrc` +2. Run `gh auth setup-git` + +### Connection + +`openshell sandbox connect ` opens an interactive session. Environment variables from `.bashrc` are inherited by Claude Code and its MCP servers. + +### Deletion + +`openshell sandbox delete ` or `harness teardown --sandboxes`. + +--- + +## Credential Flow + +| Credential | Provider Type | How It Works | +|------------|--------------|--------------| +| GitHub | `github` | PAT stored in gateway, proxy-managed. Sandbox sees `GITHUB_TOKEN` placeholder. | +| Vertex AI | `google-vertex-ai` | ADC-based OAuth via `--from-gcloud-adc`. Gateway refreshes tokens automatically. Inference routed through `inference.local`. | +| Atlassian | `atlassian` | API token stored in gateway. Proxy resolves base64 Basic auth header. `JIRA_URL`/`JIRA_USERNAME` injected via `sandbox.env`. | +| GWS | Custom (file upload) | Decrypted OAuth credentials uploaded to `/sandbox/.config/openshell/`. Not proxy-managed. | + +--- + +## Sandbox Image + +The sandbox image extends `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` with: +- `mcp-atlassian` — Jira/Confluence MCP server +- `gws` CLI — Google Workspace +- `policy.yaml` — network egress rules +- `CLAUDE.md` — agent instructions +- `settings.json` — Claude Code permissions (`defaultMode: bypassPermissions`) +- `.mcp.json` — MCP server configuration (auto-loaded by Claude Code) +- `startup.sh` — runtime env setup + +The image is multi-arch (`linux/amd64` + `linux/arm64`), built with `docker buildx`. + +--- + +## Network Policy + +`sandbox/policy.yaml` controls which processes can reach which hosts: + +| Policy | Hosts | Binaries | +|--------|-------|----------| +| `claude_telemetry` | `*.anthropic.com`, `downloads.claude.ai` | claude, node | +| `github_git` | `github.com` (GET info/refs, POST git-upload-pack only) | git | +| `github_downloads` | `*.githubusercontent.com`, `codeload.github.com` | curl, gh, git, uv | +| `google_workspace` | `*.googleapis.com`, `oauth2.googleapis.com` | gws | +| `pypi` | `pypi.org`, `files.pythonhosted.org` | python, pip, uv | +| `npm` | `registry.npmjs.org` | npm, node | + +Git push (`git-receive-pack`) is blocked by default. + +--- + +## OCP-Specific + +### Gateway Deployment + +- Helm chart from `oci://ghcr.io/nvidia/openshell/helm-chart` (version pinned in `openshell.toml`) +- TLS passthrough route at `gateway-openshell.` +- mTLS: client certs copied from `openshell-client-tls` K8s secret to `~/.config/openshell/gateways//mtls/` +- `allowUnauthenticatedUsers: true` (mTLS is the auth layer) + +### In-Cluster Launcher + +For OCP sandboxes, a Kubernetes Job runs the launcher image (`sandbox/launcher/`): +1. Register gateway via `http://` trick (avoids cert generation probe), patch to `https://` + mTLS +2. Parse profile from mounted ConfigMap +3. Build provider flags, create sandbox with retry +4. Upload files (GWS creds, sandbox.env) +5. Run startup.sh via `sandbox exec` + +The launcher connects to the gateway at `https://openshell.openshell.svc.cluster.local:8080` using mounted mTLS certs from `openshell-client-tls`. + +--- + +## Testing + +### Unit Tests (`test/preflight.bats`) + +29 bats tests covering the preflight check engine (`lib/providers.py`). Uses stubbed CLI, isolated temp dirs, and no gateway dependency. Tests: +- env inputs (set/missing/secret/masked) +- file inputs (exists/missing/metadata extraction) +- check inputs (pass/fail/env expansion) +- provider status (all pass/any fail/required/optional) +- config filtering (enabled/disabled/no config) +- CLI detection (present/missing) +- Gateway detection (podman/k8s) + +### Integration Tests (`test/test-flow.sh`) + +End-to-end validation requiring a live gateway: +- Quick mode: deploy → providers → gateway check → teardown +- Full mode: + sandbox create → verify env/GWS/MCP/Claude → delete → teardown +- Targets: `podman`, `ocp`, `all` +- Strips ANSI codes from CLI output for reliable parsing diff --git a/bin/scripts/lib/parse-profile.py b/bin/scripts/lib/parse-profile.py new file mode 100644 index 0000000..3900f0f --- /dev/null +++ b/bin/scripts/lib/parse-profile.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python3 +"""Parse a profile TOML file and output shell variable assignments. + +Usage: + python3 parse-profile.py + +Output (eval-safe shell assignments): + SANDBOX_NAME='agent' + SANDBOX_IMAGE='quay.io/...' + SANDBOX_COMMAND='claude --bare' + SANDBOX_KEEP='true' + SANDBOX_PROVIDERS='github vertex-local atlassian' + SANDBOX_ENV='export KEY=value\n...' +""" +import shlex +import sys + +try: + import tomllib +except ImportError: + import tomli as tomllib + +if len(sys.argv) < 2: + print("Usage: parse-profile.py ", file=sys.stderr) + sys.exit(1) + +with open(sys.argv[1], "rb") as f: + c = tomllib.load(f) + +print(f"SANDBOX_NAME={shlex.quote(c.get('name', 'agent'))}") +print(f"SANDBOX_IMAGE={shlex.quote(c.get('image', ''))}") +print(f"SANDBOX_COMMAND={shlex.quote(c.get('command', 'claude --bare'))}") +print(f"SANDBOX_KEEP={shlex.quote(str(c.get('keep', True)).lower())}") + +providers = c.get("providers", []) +print(f"SANDBOX_PROVIDERS={shlex.quote(' '.join(providers))}") + +env = c.get("env", {}) +lines = [f"export {k}={v}" for k, v in env.items()] +print(f"SANDBOX_ENV={shlex.quote(chr(10).join(lines) + chr(10))}") diff --git a/bin/scripts/lib/profile.sh b/bin/scripts/lib/profile.sh index fc99dd9..f261f5f 100644 --- a/bin/scripts/lib/profile.sh +++ b/bin/scripts/lib/profile.sh @@ -1,32 +1,20 @@ #!/usr/bin/env bash -# Agent config parsing helpers. +# Profile parsing helpers. # # Source from any script: # source "$(dirname "$0")/lib/profile.sh" # # Usage: -# parse_agent agents/default.toml +# parse_profile profiles/default.toml # # Sets: SANDBOX_IMAGE, SANDBOX_COMMAND, SANDBOX_NAME, # # SANDBOX_PROVIDERS, SANDBOX_ENV, SANDBOX_KEEP -parse_agent() { - local agent_file="$1" - [[ -f "$agent_file" ]] || { echo "ERROR: $agent_file not found."; exit 1; } +LIB_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" - eval "$(python3 -c " -import tomllib, sys, shlex -with open(sys.argv[1], 'rb') as f: - c = tomllib.load(f) -print(f'SANDBOX_NAME={shlex.quote(c.get(\"name\", \"agent\"))}') -print(f'SANDBOX_IMAGE={shlex.quote(c.get(\"image\", \"\"))}') -print(f'SANDBOX_COMMAND={shlex.quote(c.get(\"command\", \"claude --bare\"))}') -print(f'SANDBOX_KEEP={shlex.quote(str(c.get(\"keep\", True)).lower())}') -providers = c.get('providers', []) -print(f'SANDBOX_PROVIDERS={shlex.quote(\" \".join(providers))}') -env = c.get('env', {}) -lines = [f'export {k}={v}' for k, v in env.items()] -print(f'SANDBOX_ENV={shlex.quote(chr(10).join(lines) + chr(10))}') -" "$agent_file")" +parse_profile() { + local profile_file="$1" + [[ -f "$profile_file" ]] || { echo "ERROR: $profile_file not found."; exit 1; } + eval "$(python3 "$LIB_DIR/parse-profile.py" "$profile_file")" } # Build provider flags array from SANDBOX_PROVIDERS. @@ -45,10 +33,6 @@ build_provider_flags() { # Stage sandbox.env + GWS credentials to a directory for upload. # The directory name must be "openshell" so upload lands at /sandbox/.config/openshell/. -# -# Usage: -# stage_harness_dir /tmp/openshell -# # Creates: $dir/sandbox.env, $dir/credentials.json, $dir/client_secret.json stage_harness_dir() { local dir="$1" mkdir -p "$dir" diff --git a/bin/scripts/new.sh b/bin/scripts/new.sh index 54035cb..3a6f439 100755 --- a/bin/scripts/new.sh +++ b/bin/scripts/new.sh @@ -62,7 +62,7 @@ fi # ── 4. Parse profile ───────────────────────────────────────────────── PROFILE_FILE="$HARNESS_DIR/profiles/${PROFILE}.toml" NAME_OVERRIDE="$SANDBOX_NAME" -parse_agent "$PROFILE_FILE" +parse_profile "$PROFILE_FILE" [[ -n "$NAME_OVERRIDE" ]] && SANDBOX_NAME="$NAME_OVERRIDE" echo "" diff --git a/bin/scripts/providers.sh b/bin/scripts/providers.sh index d5e168d..4c9f10d 100755 --- a/bin/scripts/providers.sh +++ b/bin/scripts/providers.sh @@ -61,7 +61,7 @@ if $FORCE; then "$CLI" provider profile delete "$id" 2>/dev/null || true done fi -"$CLI" provider profile import --from "$SCRIPT_DIR/sandbox/profiles/" 2>/dev/null || echo " (already imported)" +"$CLI" provider profile import --from "$HARNESS_DIR/sandbox/profiles/" 2>/dev/null || echo " (already imported)" echo "" echo "=== Registering providers ===" diff --git a/sandbox-local.sh b/sandbox-local.sh deleted file mode 100755 index 0cd998b..0000000 --- a/sandbox-local.sh +++ /dev/null @@ -1,105 +0,0 @@ -#!/usr/bin/env bash -# Launch a sandbox on the local Podman/Docker gateway. -# -# Uses the same providers as the OCP workflow (setup-providers.sh). -# GWS credentials are exported from the local gws CLI. -# Atlassian URL/username come from env vars (not K8s secrets). -# -# Prerequisites: -# ./deploy-local.sh # verify gateway running -# ./setup-providers.sh # register providers -# -# Usage: -# ./sandbox-local.sh # interactive Claude session -# ./sandbox-local.sh --name my-sandbox # custom name -# ./sandbox-local.sh --rejoin my-sandbox # reconnect -# ./sandbox-local.sh --no-keep # delete after exit -set -euo pipefail - -CLI="${OPENSHELL_CLI:-openshell}" -command -v "$CLI" &>/dev/null || { echo "ERROR: openshell CLI not found."; exit 1; } - -# Validate we're targeting a local gateway, not the OCP one -GW="${OPENSHELL_GATEWAY:-}" -if [[ "$GW" == "ocp" ]]; then - echo "ERROR: OPENSHELL_GATEWAY=ocp — this script is for local gateways." - echo " Use ./sandbox.sh for OpenShift, or: export OPENSHELL_GATEWAY=" - exit 1 -fi - -# ── Parse args ───────────────────────────────────────────────────────── -EXTRA=() -while [[ $# -gt 0 ]]; do - case $1 in - --rejoin) - echo "Reconnecting to sandbox: $2" - exec "$CLI" sandbox connect "$2" - ;; - --name|--editor) EXTRA+=("$1" "$2"); shift 2 ;; - --no-keep) EXTRA+=("$1"); shift ;; - *) echo "Unknown argument: $1"; exit 1 ;; - esac -done - -# ── Detect registered providers ──────────────────────────────────────── -PROVIDERS=() -for name in github vertex-local atlassian; do - if "$CLI" provider get "$name" &>/dev/null; then - PROVIDERS+=(--provider "$name") - fi -done - -echo "=== Providers ===" -for name in github vertex-local atlassian; do - if "$CLI" provider get "$name" &>/dev/null; then - echo " $name: attached" - else - echo " $name: not registered (skipping)" - fi -done - -# ── Stage files for upload ───────────────────────────────────────────── -UPLOAD_ARGS=() -STAGE="" - -# GWS: export decrypted credentials -echo "" -echo "=== Credentials ===" -if command -v gws &>/dev/null && gws auth status &>/dev/null; then - STAGE=$(mktemp -d) - mkdir -p "$STAGE/creds/gws-config" - if gws auth export --unmasked > "$STAGE/creds/gws-config/credentials.json" 2>/dev/null; then - GWS_DIR="${GWS_CONFIG_DIR:-$HOME/.config/gws}" - [[ -f "$GWS_DIR/client_secret.json" ]] && cp "$GWS_DIR/client_secret.json" "$STAGE/creds/gws-config/" - chmod 600 "$STAGE/creds/gws-config"/* - UPLOAD_ARGS=(--upload "$STAGE/creds:/sandbox/.harness") - echo " GWS: exported" - else - echo " GWS: export failed (skipping)" - rm -rf "$STAGE/creds/gws-config" - fi -else - echo " GWS: not authenticated (skipping)" -fi - -# Atlassian non-secret config -if [[ -n "${JIRA_URL:-}" ]]; then - STAGE="${STAGE:-$(mktemp -d)}" - mkdir -p "$STAGE/creds" - python3 -c "import json,sys; json.dump({'jira_url':sys.argv[1],'jira_username':sys.argv[2]},open(sys.argv[3],'w'))" \ - "$JIRA_URL" "${JIRA_USERNAME:-}" "$STAGE/creds/atlassian.json" - UPLOAD_ARGS=(--upload "$STAGE/creds:/sandbox/.harness") - echo " Atlassian: $JIRA_URL" -else - echo " Atlassian: JIRA_URL not set (skipping)" -fi - -# ── Create sandbox ───────────────────────────────────────────────────── -echo "" -echo "=== Creating sandbox ===" -exec "$CLI" sandbox create \ - --tty \ - ${PROVIDERS[@]+"${PROVIDERS[@]}"} \ - ${UPLOAD_ARGS[@]+"${UPLOAD_ARGS[@]}"} \ - ${EXTRA[@]+"${EXTRA[@]}"} \ - -- bash -c '. /sandbox/startup.sh && exec claude --bare' diff --git a/sandbox/configure-mcp.py b/sandbox/configure-mcp.py deleted file mode 100644 index dea76a0..0000000 --- a/sandbox/configure-mcp.py +++ /dev/null @@ -1,70 +0,0 @@ -#!/usr/bin/env python3 -"""Generate .claude.json MCP server config from environment variables. - -Writes /sandbox/.claude.json with configured MCP servers based on -which credentials are available. Skips servers whose credentials are -not set. - -JIRA_URL and JIRA_USERNAME are read from an uploaded config file -(not secrets — no need for provider placeholders). JIRA_API_TOKEN -comes from the atlassian provider as a placeholder resolved by the proxy. -""" - -import json -import os -import stat - -config = { - "autoUpdates": False, - "hasCompletedOnboarding": True, - "projects": { - "/sandbox": { - "hasTrustDialogAccepted": True, - "allowedTools": [], - }, - }, - "mcpServers": {}, -} - -# ── Atlassian (sooperset/mcp-atlassian) ──────────────────────────────── -# JIRA_URL and JIRA_USERNAME come from either: -# - An uploaded atlassian.json file (sandbox.sh / local workflow) -# - Environment variables injected via secretKeyRef (in-cluster launcher) -atlassian_config = "/sandbox/.harness/creds/atlassian.json" -jira_url = "" -jira_username = "" -if os.path.isfile(atlassian_config): - try: - with open(atlassian_config) as f: - atlassian = json.load(f) - jira_url = atlassian.get("jira_url", "") - jira_username = atlassian.get("jira_username", "") - except (json.JSONDecodeError, OSError) as e: - print(f"WARNING: could not read atlassian config: {e}", flush=True) -# Fall back to env vars (set by in-cluster launcher via secretKeyRef) -if not jira_url: - jira_url = os.environ.get("JIRA_URL", "") -if not jira_username: - jira_username = os.environ.get("JIRA_USERNAME", "") - -if jira_url: - jira_api_token = os.environ.get("JIRA_API_TOKEN", "") - config["mcpServers"]["atlassian"] = { - "type": "stdio", - "command": "/sandbox/.venv/bin/mcp-atlassian", - "args": [], - "env": { - "JIRA_URL": jira_url, - "JIRA_USERNAME": jira_username, - "JIRA_API_TOKEN": jira_api_token, - "CONFLUENCE_URL": jira_url.rstrip("/") + "/wiki", - "CONFLUENCE_USERNAME": jira_username, - "CONFLUENCE_API_TOKEN": jira_api_token, - "READ_ONLY_MODE": os.environ.get("READ_ONLY_MODE", "true"), - }, - } - -path = "/sandbox/.claude.json" -with open(path, "w") as f: - json.dump(config, f, indent=2) -os.chmod(path, stat.S_IRUSR | stat.S_IWUSR) From e099948a170409465cbb30d47d59df0a317e9df0 Mon Sep 17 00:00:00 2001 From: Robby Cochran Date: Thu, 4 Jun 2026 07:03:47 -0700 Subject: [PATCH 3/3] chore: remove REVIEW.md from repo Co-Authored-By: Claude Opus 4.6 (1M context) --- REVIEW.md | 233 ------------------------------------------------------ 1 file changed, 233 deletions(-) delete mode 100644 REVIEW.md diff --git a/REVIEW.md b/REVIEW.md deleted file mode 100644 index 2a75ce7..0000000 --- a/REVIEW.md +++ /dev/null @@ -1,233 +0,0 @@ -# Harness-OpenShell Code Review: Prioritized Recommendations - -10 independent reviewers scored the repo across 8 dimensions. Weighted average: **5.3/10**. - -The code architecture is sound (7-8/10) but the documentation is broken (3/10), multi-user support is missing (3/10), and test coverage has critical gaps (4/10). The codebase is ~2100 lines of CLI glue designed to shrink as upstream OpenShell absorbs workarounds — a compiled rewrite would be counterproductive. - -| Dimension | Score | -|---|---| -| New Engineer Onboarding | 3 | -| Bash Code Quality | 6 | -| Security | 5 | -| Architecture | 7 | -| Testing | 4 | -| Upstream Alignment | 7 | -| Developer Workflow | 6 | -| Team Scaling (20-person) | 3 | -| Rewrite Analysis | 8 | -| Strategic Direction | 6 | - ---- - -## 1. README Is Completely Broken - -**Impact: Blocks every new user on step one.** - -Every Quick Start command references scripts deleted in commit `c12db4d`. The Files table lists 9 nonexistent files. `agents/` references should be `profiles/`. Test paths are wrong. No prerequisites listed. No mention of `bin/harness` CLI or how to add it to PATH. - -**What to fix:** -- Rewrite Quick Start to use `harness deploy --local`, `harness providers`, `harness new --local` -- Replace Files table with actual structure: `bin/harness`, `bin/scripts/*.sh`, `profiles/default.toml`, `sandbox/`, `test/` -- Replace all `agents/` references with `profiles/` -- Add Prerequisites section (openshell, python3 3.11+, podman/docker, gcloud, bats) -- Add "How to use the CLI" section with PATH setup and `harness --help` output -- Add "Why use a sandbox instead of local Claude?" section (credential isolation, network policy, reproducibility) -- Fix test command paths to `test/test-flow.sh` or `harness test` - -**Effort:** 2-3 hours -**Flagged by:** Onboarding (critical x3), Developer Workflow (high), Strategic Direction (medium) - ---- - -## 2. Multi-User Is Fundamentally Broken - -**Impact: Cannot deploy to a 20-person team without credential collision and sandbox name conflicts.** - -Five independent single-user assumptions block team scaling: - -| Problem | Where | Consequence | -|---|---|---| -| Sandbox name hardcoded to "agent" | `profiles/default.toml:7` | Two users overwrite each other's ConfigMap | -| Provider names are gateway-global | `providers.sh:76,118` | All 20 users share one GITHUB_TOKEN | -| GWS creds are a single user's OAuth | `creds.sh:54` | All sandboxes mount one person's Gmail | -| Teardown kills all sandboxes | `teardown.sh:63-69` | One user nukes the team | -| Shared Vertex AI quota | `providers.sh:96-99` | 20 sessions exhaust rate limits | - -**What to fix:** -Introduce a `$HARNESS_USER` variable (default: `$USER`) that prefixes: -- Sandbox names: `agent` becomes `rcochran-agent` -- Provider names: `github` becomes `github-rcochran` -- K8s secrets: `openshell-gws` becomes `openshell-gws-rcochran` -- Teardown filter: only delete sandboxes matching `$HARNESS_USER-*` - -This is a ~50-line change across `new.sh`, `providers.sh`, `creds.sh`, and `teardown.sh`. - -**Effort:** 1 day -**Flagged by:** Team Scaling (critical x2, high x4), Security (high) - ---- - -## 3. Dead Code Cleanup - -**Impact: Reduces confusion, aligns codebase with stated "shrink, not grow" principle.** - -| File | Lines | Why dead | -|---|---|---| -| `sandbox-local.sh` | 106 | Superseded by `bin/scripts/new.sh --local` | -| `sandbox/configure-mcp.py` | 71 | `mcp.json` baked into image; startup.sh never calls this | -| `sandbox/__pycache__/` | -- | Artifact of dead configure-mcp.py | - -**Effort:** 30 minutes -**Flagged by:** Architecture (medium), Upstream Alignment (medium x2), Developer Workflow (low), Strategic Direction (medium) - ---- - -## 4. Security: Gateway Auth and Cluster Permissions - -**Impact: Defense-in-depth failures that become critical at team scale.** - -**4a. Gateway allows unauthenticated users** -`values-ocp.yaml:29` sets `allowUnauthenticatedUsers: true`. mTLS is the only gate. If mTLS misconfigures, the gateway is open. Fix: enable OIDC auth in addition to mTLS. - -**4b. cluster-admin ClusterRoleBinding** -`deploy.sh:111-113` grants `cluster-admin` to the sandbox controller. Unrestricted access to every secret in every namespace. Fix: scope the ClusterRole to Sandbox CRDs, pods, and services in the openshell namespace. - -**4c. Privileged SCC on three service accounts** -`deploy.sh:107-110` grants privileged SCC to openshell, openshell-sandbox, and `default`. The `default` SA grant is especially dangerous. Fix: use `restricted` or `nonroot-v2` SCC where possible. - -**4d. mTLS key written without file permissions** -`deploy.sh:206-210` writes `tls.key` without `chmod 600`. Fix: add `chmod 600` immediately after writing. - -**Effort:** 4-6 hours -**Flagged by:** Security (high x4), Team Scaling (high) - ---- - -## 5. Test Coverage Gaps - -**Impact: The most critical code paths (profile parsing, provider flag building, error handling) have zero tests.** - -**What has tests:** `providers.py` preflight checks — 29 bats tests, thorough and well-structured. - -**What has no tests:** - -| Code | Risk | Why it matters | -|---|---|---| -| `lib/profile.sh` (parse_agent, build_provider_flags, stage_harness_dir) | Uses `eval` on Python output | Shell injection if quoting breaks | -| `lib/common.sh` (require_cli, export_gws_creds) | Credential handling | Silent failures leak or lose creds | -| Error paths in deploy.sh, new.sh, teardown.sh | Missing CLI, missing gateway, running sandboxes | Users see cryptic failures | -| `bin/harness` CLI dispatch | Unknown commands, help text | No verification that help matches reality | - -All testable offline using the same stub pattern as `preflight.bats`. - -**Effort:** 1-2 days for profile.sh + common.sh bats tests; 1 day for error path tests -**Flagged by:** Testing (high x3), Developer Workflow (medium) - ---- - -## 6. DRY Violations: Three Duplicated Patterns - -**Impact: Bugs fixed in one copy will not be fixed in the other.** - -| Pattern | Copies | Files | -|---|---|---| -| TOML parsing via inline Python | 2 | `profile.sh:16-29`, `entrypoint.sh:49-63` | -| Provider flag building loop | 3 | `profile.sh:34-44`, `entrypoint.sh:79-87`, `sandbox-local.sh:46-59` | -| GWS credential export | 3 | `common.sh:42-64`, `profile.sh:60-66`, `sandbox-local.sh:68-83` | - -**What to fix:** -- Extract TOML parsing into `lib/parse_profile.py` -- Delete `sandbox-local.sh` (removes 2 of 3 copies) -- Consolidate GWS export into `common.sh:export_gws_creds()` and call from profile.sh - -**Effort:** 3-4 hours (mostly after deleting sandbox-local.sh) -**Flagged by:** Code Quality (medium x3), Architecture (medium) - ---- - -## 7. Broken Path in providers.sh - -**Impact: Profile import likely fails silently.** - -`providers.sh:64` references `$SCRIPT_DIR/sandbox/profiles/` but `SCRIPT_DIR` is `bin/scripts`, so it resolves to `bin/scripts/sandbox/profiles/` which does not exist. Should be `$HARNESS_DIR/sandbox/profiles/`. - -**Effort:** 5 minutes -**Flagged by:** Code Quality (high) - ---- - -## 8. Developer Inner Loop Is Too Slow - -**Impact: Every sandbox config change requires a multi-arch Docker build + push (minutes).** - -Any change to CLAUDE.md, mcp.json, settings.json, startup.sh, or policy.yaml requires `docker buildx build --push` before testing. The `openshell sandbox upload` primitive already exists. - -**What to fix:** Add a `harness sync` subcommand that uploads sandbox config files to a running sandbox without rebuilding the image. - -**Effort:** 2-3 hours -**Flagged by:** Developer Workflow (high), Strategic Direction (medium) - ---- - -## 9. Retry Loop Masks Root Cause - -**Impact: When sandbox creation fails, developers get "Attempt N failed (supervisor race)" with zero diagnostic info.** - -`new.sh:98-116` and `entrypoint.sh:113-126` retry 5 times with backoff but capture no stderr. `test-flow.sh` swallows all output with `&>/dev/null`. - -**What to fix:** Capture and display stderr from failed `openshell sandbox create` attempts. Add `--verbose` flag. Log actual exit codes. - -**Effort:** 1-2 hours -**Flagged by:** Developer Workflow (high), Testing (medium) - ---- - -## 10. Naming Inconsistency: agent vs profile - -**Impact: Low but pervasive confusion.** - -`profile.sh` defines `parse_agent()`. CLI uses `--profile`. Directory is `profiles/`. PROVIDERS-SPEC.md still says `agents/*.toml`. - -**What to fix:** Rename `parse_agent()` to `parse_profile()`, update PROVIDERS-SPEC.md references. - -**Effort:** 30 minutes -**Flagged by:** Code Quality (low), Onboarding (high) - ---- - -## Should We Rewrite? - -**No. With one exception.** - -- The codebase is 2100 lines of CLI glue. 90% is "run this command, check exit code." A Go version would be `os/exec` calls doing the same thing less readably. -- The harness is designed to shrink. AGENTS.md tracks 5 upstream workarounds. Each eliminated removes 50-150 lines. -- Distribution is not an issue. Users already have openshell, kubectl, helm, python3, and podman installed. -- The complexity threshold for rewrite (~5000+ lines, multiple contributors) is not met. - -**The one exception: the in-cluster launcher.** `sandbox/launcher/entrypoint.sh` (138 lines) runs inside a K8s Job, shells out to Python for TOML parsing, and patches gateway metadata.json. A Go binary would produce a scratch-based image (~50MB vs current), eliminate the python3/tomli runtime dependency, and speed up Job startup. A Go rewrite spec already exists at `sandbox/launcher/SPEC.md`. - -**Recommendation:** Rewrite the launcher only. Leave everything else as bash+python. Revisit if the harness exceeds 15 scripts or 3000 lines. - ---- - -## Strategic Direction - -The harness's destiny is to shrink into upstream OpenShell contributions: -1. Submit `sandbox/Dockerfile` as an OpenShell-Community sandbox -2. Upstream the `atlassian.yaml` provider profile -3. Reduce this repo to team-specific profile TOML files and a thin README - -The 10x multiplier is multi-tenant team profiles (`profiles/collector.toml`, `profiles/scanner.toml`) that let the whole RHACS org share one gateway with per-team sandboxes. - ---- - -## Next 3 Things To Do - -### 1. Fix the README (2-3 hours) -Single highest-leverage change. Every new user currently fails on step one. Rewrite Quick Start with actual CLI commands, replace Files table, add prerequisites, add PATH setup. - -### 2. Delete dead code + fix providers.sh path (1 hour) -Remove `sandbox-local.sh`, `sandbox/configure-mcp.py`, `sandbox/__pycache__/`. Fix `$SCRIPT_DIR` to `$HARNESS_DIR` on `providers.sh:64`. Eliminates 177 lines and fixes broken provider import. - -### 3. Add $HARNESS_USER namespacing (1 day) -Prefix sandbox names, provider names, K8s secrets, and teardown filters with `$HARNESS_USER`. Gate to multi-user operation. ~50 lines across 4 files.