feat(sdk): add Go SDK foundation and core clients (1/4)

brainstorm/00-overview.md

@@ -0,0 +1,36 @@
+# Brainstorm Overview
+
+Last updated: 2026-07-10
+
+## Sessions
+
+| # | Date | Topic | Status | Spec | Issue |
+|---|------|-------|--------|------|-------|
+| 01 | 2026-07-09 | warm-pool-feasibility | active | - | - |
+| 02 | 2026-07-09 | cluster-setup | active | - | - |
+| 03 | 2026-07-09 | warm-pool-measurements | active | - | - |
+| 04 | 2026-07-09 | results-and-recommendations | active | - | - |
+| 05 | 2026-07-10 | k8s-watch-crash-fix | active | - | [#2211](https://github.com/NVIDIA/OpenShell/issues/2211) |
+
+## Structure
+
+01 is the parent document defining the overall feasibility study. 02-04 are execution phases:
+
+- **02** depends on: nothing (first step)
+- **03** depends on: 02 (cluster must be running)
+- **04** depends on: 03 (measurements must be complete)
+
+05 is a standalone bug fix discovered during the warm pool feasibility study.
+
+## Open Threads
+
+- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02)
+- Does env var injection at claim time trigger cold start? (from #01, #03)
+- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02)
+- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03)
+- Should findings be posted to upstream agent-sandbox repo? (from #04)
+- Should the defensive skip use `debug!` or `warn!` level? (from #05)
+
+## Parked Ideas
+
+None.

brainstorm/01-warm-pool-feasibility.md

@@ -0,0 +1,67 @@
+# Brainstorm: Warm Pool Feasibility Study for OpenShell
+
+**Date:** 2026-07-09
+**Status:** active
+
+## Problem Framing
+
+OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable.
+
+The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`.
+
+This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture.
+
+## Approaches Considered
+
+### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell)
+
+Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead.
+
+- Pros: Fast to set up, answers the fundamental feasibility question, no code changes required
+- Cons: Does not account for OpenShell supervisor, identity binding, or policy injection
+
+### B: Measure OpenShell End-to-End with Code Changes
+
+Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency.
+
+- Pros: Realistic numbers that include all OpenShell overhead
+- Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow
+
+### C: Layered Approach (A then B)
+
+Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally.
+
+- Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data
+- Cons: More phases to execute
+
+## Decision
+
+**Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency.
+
+## Key Requirements
+
+1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements
+2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs
+3. **OpenShell deployed via Krzysztof's chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup
+4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control
+5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns)
+6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates
+7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration
+8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope.
+
+## Experiment Phases
+
+This study decomposes into three execution phases, each with its own brainstorm document:
+
+| Phase | Brainstorm | What it covers |
+|-------|------------|----------------|
+| 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment |
+| 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization |
+| 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations |
+
+## Open Questions
+
+- Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD?
+- Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection?
+- Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)?
+- What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers?

brainstorm/02-cluster-setup.md

@@ -0,0 +1,78 @@
+# Brainstorm: Cluster Setup & OpenShell Deployment
+
+**Date:** 2026-07-09
+**Status:** active
+**Parent:** 01-warm-pool-feasibility
+
+## Problem Framing
+
+Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation.
+
+Three components must work together:
+1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+)
+2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim)
+3. OpenShell deployed and functional (can create cold-start sandboxes)
+
+## Approaches Considered
+
+### A: ROSA HCP with Upstream Agent Sandbox Manifests
+
+Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with Krzysztof's chart.
+
+- Pros: Uses upstream directly, most current version, full control over version
+- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration
+
+### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview)
+
+Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with Krzysztof's chart.
+
+- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path
+- Cons: Tech preview may not include extension CRDs yet, version may lag upstream
+
+### C: Both Paths Available
+
+Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them.
+
+- Pros: Best of both worlds, flexibility
+- Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts
+
+## Decision
+
+**Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available.
+
+## Key Requirements
+
+1. **ROSA HCP cluster** via `rosa:create` with the AAET profile
+   - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA)
+   - Region: us-east-2 (matches AAET profile subnets)
+   - Worker nodes: at least 3 for realistic scheduling behavior
+
+2. **Agent Sandbox installation**
+   - Red Hat Agent Sandbox operator from OperatorHub (if available)
+   - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim
+   - Verify all four CRDs are served: `kubectl api-resources | grep agents`
+
+3. **OpenShell deployment**
+   - Clone github.com/2000krysztof/Openshell-Openshift-Deploy
+   - Run `deploy.sh` with default configuration
+   - Verify: `openshell status` shows Connected
+   - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works)
+
+4. **Image pre-pulling**
+   - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes
+   - This removes image pull latency from warm pool measurements
+   - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull
+
+5. **Validation checklist**
+   - Gateway pod is Running
+   - Agent Sandbox controller is Running
+   - Extension CRDs are registered
+   - Cold-start sandbox creation works end-to-end
+   - Images are pre-pulled on all nodes
+
+## Open Questions
+
+- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`.
+- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource?
+- Does Krzysztof's deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes?
+- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously.

brainstorm/03-warm-pool-measurements.md

@@ -0,0 +1,175 @@
+# Brainstorm: Baseline & Warm Pool Measurements
+
+**Date:** 2026-07-09
+**Status:** active
+**Parent:** 01-warm-pool-feasibility
+
+## Problem Framing
+
+With the cluster running, we need a structured measurement plan that answers: how fast can we get a sandbox with warm pooling, and what are the bottlenecks? The measurements must cover cold-start baseline (control), raw Agent Sandbox warm pooling (no OpenShell), and progressively more realistic configurations that approximate what OpenShell would need.
+
+The key unknowns are:
+- What is the actual cold-start latency breakdown on OpenShift?
+- Does warm pooling deliver sub-2s claim-to-ready?
+- Is the readiness probe interval the dominant bottleneck, and can Pod Readiness Gates (KEP-580) or Knative-style readiness wrapping eliminate it?
+- Can env vars be injected at claim time without triggering cold start?
+
+## Approaches Considered
+
+### A: Simple Before/After
+
+Measure cold start, then warm pool, compare. Two data points.
+
+- Pros: Fastest to execute
+- Cons: No insight into what drives the latency, can't identify optimization levers
+
+### B: Phase Breakdown with Configuration Matrix
+
+Measure cold start with per-phase timestamps. Then measure warm pooling across a matrix of configurations (probe intervals, readiness gates, sidecar patterns, env var injection).
+
+- Pros: Identifies specific bottlenecks, tests optimization hypotheses, actionable data
+- Cons: More experiments to run, needs a measurement script
+
+### C: Full Benchmark Suite with Statistical Rigor
+
+N=50+ runs per configuration, p50/p90/p99, automated benchmark harness, CSV output for analysis.
+
+- Pros: Publication-quality data, statistically meaningful
+- Cons: Overkill for a feasibility study, takes days
+
+## Decision
+
+**Approach B: Phase breakdown with configuration matrix.** N=10-20 runs per configuration is enough to see the pattern. We need per-phase timestamps to identify bottlenecks, and we need the configuration matrix to test our optimization hypotheses (readiness gates, sidecar patterns). A simple shell script with `kubectl` timestamps is sufficient.
+
+## Experiment Design
+
+### Experiment 1: Cold-Start Baseline (Control)
+
+Measure OpenShell sandbox creation latency without pooling. Captures the current state.
+
+**What to measure:**
+- Total time: `openshell sandbox create` to sandbox Ready
+- Phase breakdown using `kubectl get events` and pod timestamps:
+  - API call to pod Scheduled
+  - Scheduled to image pulled (should be 0 with pre-pulled images)
+  - Image pulled to init containers complete
+  - Init containers complete to supervisor Ready
+  - Supervisor Ready to SSH available
+
+**Runs:** N=10 with pre-pulled images, N=5 without (to quantify image pull impact)
+
+### Experiment 2: Vanilla Agent Sandbox Warm Pool (No OpenShell)
+
+Measure raw Agent Sandbox warm pooling without OpenShell. This isolates Kubernetes overhead from OpenShell overhead.
+
+**Setup:**
+```yaml
+apiVersion: extensions.agents.x-k8s.io/v1beta1
+kind: SandboxTemplate
+metadata:
+  name: base-template
+spec:
+  sandbox:
+    spec:
+      containers:
+      - name: agent
+        image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest
+---
+apiVersion: extensions.agents.x-k8s.io/v1beta1
+kind: SandboxWarmPool
+metadata:
+  name: base-pool
+spec:
+  templateRef:
+    name: base-template
+  replicas: 5
+```
+
+**What to measure:**
+- Pool fill time: SandboxWarmPool created to all 5 replicas Ready
+- Claim-to-ready time: SandboxClaim created to adopted Sandbox transitioning to Ready
+- Claim-to-ready with default readiness probe (10s periodSeconds)
+- Claim-to-ready with aggressive readiness probe (1s periodSeconds)
+
+**Runs:** N=10 per configuration
+
+### Experiment 3: Pod Readiness Gates (KEP-580)
+
+Test whether Pod Readiness Gates can replace polling-based readiness for faster claim-to-ready.
+
+**How it works:**
+- Add a custom ReadinessGate condition (e.g., `sandbox.openshell.io/claimed`) to the pod template
+- Warm-pooled pods start with the condition missing (defaults to False, pod is NotReady)
+- On claim, an external controller sets the condition to True via a pod status patch
+- Pod transitions to Ready immediately (no probe interval wait)
+
+**What to measure:**
+- Time from condition patch to pod Ready status
+- Compare with probe-based readiness at 1s and 10s intervals
+
+**Runs:** N=10
+
+**KEP-580 status:** GA since Kubernetes 1.14. Available on all OpenShift versions. No feature gate needed.
+
+### Experiment 4: Sidecar Readiness Pattern
+
+Test the Knative-style pattern where a sidecar controls pod readiness.
+
+**How it works:**
+- Define a supervisor-like sidecar (init container with `restartPolicy: Always`)
+- Sidecar starts, runs a readiness HTTP endpoint that returns 503 (not ready)
+- On claim, inject a signal (touch a file in a shared emptyDir, or set an env var)
+- Sidecar detects the signal, flips readiness to 200
+- Pod transitions to Ready
+
+**What to measure:**
+- Time from signal injection to sidecar readiness flip
+- Time from sidecar readiness flip to pod Ready
+- End-to-end claim-to-ready with sidecar pattern
+
+**Runs:** N=10
+
+**KEP-753 status (native sidecars):** GA since Kubernetes 1.33 (April 2025). Requires OpenShift 4.20+. Sidecar readiness probes contribute to pod readiness.
+
+### Experiment 5: Env Var Injection at Claim Time
+
+Test whether SandboxClaim env var injection works without forcing cold start.
+
+**Setup:**
+- SandboxTemplate with `envVarsInjectionPolicy: Allowed`
+- SandboxClaim with env vars simulating OpenShell identity (OPENSHELL_SANDBOX_ID, OPENSHELL_ENDPOINT)
+
+**What to measure:**
+- Does the claim adopt a warm sandbox, or does it trigger cold start?
+- If warm adoption works: claim-to-ready latency
+- If cold start is triggered: document this as a constraint
+
+**Runs:** N=5
+
+### Experiment 6: Combined (Sidecar + Readiness Gates + Env Var Injection)
+
+Combine the best-performing readiness pattern with env var injection. This approximates what a real OpenShell integration would look like.
+
+**What to measure:**
+- End-to-end claim-to-ready with the full stack
+- Compare with cold-start baseline
+
+**Runs:** N=10
+
+## Measurement Script
+
+A shell script that:
+1. Creates a SandboxClaim with `kubectl apply`
+2. Records the creation timestamp
+3. Watches the pod status until Ready (or timeout)
+4. Records the Ready timestamp
+5. Calculates the delta
+6. Collects pod events for phase breakdown
+7. Outputs CSV: run_number, config, create_ts, ready_ts, delta_ms, phases
+
+## Open Questions
+
+- Can we use `kubectl wait --for=condition=Ready` with millisecond precision, or do we need a custom watcher?
+- For the sidecar experiment, what's the simplest possible sidecar binary? A Go binary that listens on :8080 and watches for a file in /tmp/signal/?
+- How does pool replenishment affect back-to-back claim latency? Should we measure burst patterns (claim 5 sandboxes simultaneously)?
+- What happens when the pool is exhausted? Does SandboxClaim fall back to cold start or stay Pending?