Warm Pool Feasibility Study [Spec] #3
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| <!-- | ||
| Sync Impact Report | ||
| ================== | ||
| Version change: (new) → 1.0.0 | ||
| Added principles: | ||
| - I. Safe Arithmetic in Bash Loops | ||
| - II. Non-Zero Exit on CLI Error | ||
| Added sections: | ||
| - Scope (defines applicability to experiment/tooling scripts) | ||
| - Governance | ||
| Removed sections: none (initial constitution) | ||
| Templates requiring updates: | ||
| - plan-template.md: ✅ no update needed (Constitution Check is generic) | ||
| - spec-template.md: ✅ no update needed (no principle-specific refs) | ||
| - tasks-template.md: ✅ no update needed (no principle-specific refs) | ||
| Follow-up TODOs: none | ||
| --> | ||
|
|
||
| # OpenShell Constitution | ||
|
|
||
| ## Core Principles | ||
|
|
||
| ### I. Safe Arithmetic in Bash Loops | ||
|
|
||
| All bash wait/polling loops MUST use `elapsed=$((elapsed + 1))` or | ||
| `(( ++elapsed ))` instead of `(( elapsed++ ))` when running under | ||
| `set -e` (or `set -euo pipefail`). | ||
|
|
||
| **Rationale**: Bash `(( expr ))` returns exit code 1 when the expression | ||
| evaluates to 0. Post-increment (`elapsed++`) returns the value *before* | ||
| incrementing, so when `elapsed=0`, `(( elapsed++ ))` evaluates to 0, | ||
| producing exit code 1. Under `set -e`, this terminates the script on the | ||
| first loop iteration. The substitution form `$((elapsed + 1))` avoids | ||
| this because it is a value expansion, not a command whose exit code is | ||
| checked. Pre-increment (`++elapsed`) also works because it returns the | ||
| value *after* incrementing (1, not 0). | ||
|
|
||
| ### II. Non-Zero Exit on CLI Error | ||
|
|
||
| Usage/help functions called from error paths (unknown options, invalid | ||
| arguments) MUST exit with non-zero status. The recommended pattern is | ||
| `exit "${1:-0}"` in the usage function body, with `usage 0` for help | ||
| requests and `usage 1` for error call sites. | ||
|
|
||
| **Rationale**: When `usage` unconditionally calls `exit 0`, scripts that | ||
| receive invalid arguments appear to succeed. CI pipelines, wrapper | ||
| scripts, and experiment harnesses that check exit codes will miss the | ||
| failure, leading to silent misconfiguration. | ||
|
|
||
| ## Scope | ||
|
|
||
| These principles apply to all bash scripts under `experiments/` and any | ||
| operator-facing shell tooling in the repository. They do not govern Rust, | ||
| Python, or other non-shell code. | ||
|
|
||
| ## Governance | ||
|
|
||
| - This constitution supersedes conflicting practices within its scope. | ||
| - Amendments require documentation and a version bump. | ||
| - Version follows semantic versioning: MAJOR for removed/redefined | ||
| principles, MINOR for new principles, PATCH for clarifications. | ||
| - Bot review triage may surface recurring patterns that become candidate | ||
| principles (see triage workflow Step 14). | ||
|
|
||
| **Version**: 1.0.0 | **Ratified**: 2026-07-10 | **Last Amended**: 2026-07-10 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # Brainstorm Overview | ||
|
|
||
| <<<<<<< HEAD | ||
| Last updated: 2026-07-11 | ||
| ======= | ||
| Last updated: 2026-07-10 | ||
| >>>>>>> origin/main | ||
|
Comment on lines
+3
to
+7
There was a problem hiding this comment. 🟡 Unresolved merge conflict markers in brainstorm overview document Unresolved git merge conflict markers are left throughout the brainstorm overview ( Impact: The brainstorm overview is unreadable, showing duplicate conflicting sections with git markers. Multiple conflict blocks in the fileThe file contains conflict markers at lines 3-7, 17-23, 33-42, 51-62. Each block has both HEAD and origin/main versions of the content left unresolved. For example, the sessions table has two different sets of entries (sessions 05-07 from HEAD vs session 05 from origin/main). 
Was this helpful? React with 👍 or 👎 to provide feedback. |
||
|
|
||
| ## Sessions | ||
|
|
||
| | # | Date | Topic | Status | Spec | Issue | | ||
| |---|------|-------|--------|------|-------| | ||
| | 01 | 2026-07-09 | warm-pool-feasibility | active | - | - | | ||
| | 02 | 2026-07-09 | cluster-setup | active | - | - | | ||
| | 03 | 2026-07-09 | warm-pool-measurements | active | - | - | | ||
| | 04 | 2026-07-09 | results-and-recommendations | active | - | - | | ||
| <<<<<<< HEAD | ||
| | 05 | 2026-07-10 | warm-pool-grpc-push | active | - | - | | ||
| | 06 | 2026-07-11 | warm-pool-grpc-poc | active | - | - | | ||
| | 07 | 2026-07-11 | warm-pool-sandbox-profile | active | - | - | | ||
| ======= | ||
| | 05 | 2026-07-10 | k8s-watch-crash-fix | active | - | [#2211](https://github.com/NVIDIA/OpenShell/issues/2211) | | ||
| >>>>>>> origin/main | ||
|
|
||
| ## Structure | ||
|
|
||
| 01 is the parent document defining the overall feasibility study. 02-04 are execution phases: | ||
|
|
||
| - **02** depends on: nothing (first step) | ||
| - **03** depends on: 02 (cluster must be running) | ||
| - **04** depends on: 03 (measurements must be complete) | ||
|
|
||
| <<<<<<< HEAD | ||
| 05 is the RFC brainstorm for always-on supervisor with gRPC-push identity binding. | ||
|
|
||
| 06-07 are prototype milestones incorporating NVIDIA feedback (2026-07-10): | ||
|
|
||
| - **06** (Milestone 1): ActivateSandbox gRPC PoC with unidentified supervisor. Proves the claim-time flow end-to-end with manual pool setup. | ||
| - **07** (Milestone 2): SandboxProfile entity + workspace-scoped pool lifecycle. Builds on 06, makes pools mergeable upstream. Depends on Derek's workspace PR. | ||
| ======= | ||
| 05 is a standalone bug fix discovered during the warm pool feasibility study. | ||
| >>>>>>> origin/main | ||
|
|
||
| ## Open Threads | ||
|
|
||
| - Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02) | ||
| - Does env var injection at claim time trigger cold start? (from #01, #03) | ||
| - Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02) | ||
| - How does pool exhaustion behave (cold fallback vs. Pending)? (from #03) | ||
| - Should findings be posted to upstream agent-sandbox repo? (from #04) | ||
| <<<<<<< HEAD | ||
| - What is the exact proto definition for ActivateSandbox? (from #05, #06) | ||
| - How does the supervisor discover the gateway endpoint at activation time? (from #06) | ||
| - Should the supervisor support an activation timeout to prevent resource waste? (from #06) | ||
| - Coordinate with Craig's work on issue #1955 (legacy RPC cleanup) (from #06) | ||
| - What is the relationship between SandboxProfile and SandboxTemplate? (from #07) | ||
| - Does Derek's workspace PR define a workspace proto that references SandboxProfiles? (from #07) | ||
| - How to handle pool config for workspaces spanning multiple namespaces? (from #07) | ||
| - Pool utilization metrics: SandboxProfile status field vs. metric endpoint? (from #07) | ||
| ======= | ||
| - Should the defensive skip use `debug!` or `warn!` level? (from #05) | ||
| >>>>>>> origin/main | ||
|
|
||
| ## Parked Ideas | ||
|
|
||
| None. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,71 @@ | ||
| # Brainstorm: Warm Pool Feasibility Study for OpenShell | ||
|
|
||
| **Date:** 2026-07-09 | ||
| **Status:** active | ||
|
|
||
| ## Problem Framing | ||
|
|
||
| OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable. | ||
|
|
||
| The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`. | ||
|
|
||
| This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture. | ||
|
|
||
| ## Approaches Considered | ||
|
|
||
| ### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell) | ||
|
|
||
| Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead. | ||
|
|
||
| - Pros: Fast to set up, answers the fundamental feasibility question, no code changes required | ||
| - Cons: Does not account for OpenShell supervisor, identity binding, or policy injection | ||
|
|
||
| ### B: Measure OpenShell End-to-End with Code Changes | ||
|
|
||
| Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency. | ||
|
|
||
| - Pros: Realistic numbers that include all OpenShell overhead | ||
| - Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow | ||
|
|
||
| ### C: Layered Approach (A then B) | ||
|
|
||
| Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally. | ||
|
|
||
| - Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data | ||
| - Cons: More phases to execute | ||
|
|
||
| ## Decision | ||
|
|
||
| **Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency. | ||
|
|
||
| ## Key Requirements | ||
|
|
||
| 1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements | ||
| 2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs | ||
| <<<<<<< HEAD | ||
| 3. **OpenShell deployed via the OpenShift deploy chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup | ||
| ======= | ||
| 3. **OpenShell deployed via Krzysztof's chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup | ||
| >>>>>>> origin/main | ||
| 4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control | ||
| 5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns) | ||
| 6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates | ||
| 7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration | ||
| 8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope. | ||
|
|
||
| ## Experiment Phases | ||
|
|
||
| This study decomposes into three execution phases, each with its own brainstorm document: | ||
|
|
||
| | Phase | Brainstorm | What it covers | | ||
| |-------|------------|----------------| | ||
| | 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment | | ||
| | 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization | | ||
| | 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations | | ||
|
|
||
| ## Open Questions | ||
|
|
||
| - Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD? | ||
| - Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection? | ||
| - Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)? | ||
| - What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Brainstorm: Cluster Setup & OpenShell Deployment | ||
|
|
||
| **Date:** 2026-07-09 | ||
| **Status:** active | ||
| **Parent:** 01-warm-pool-feasibility | ||
|
|
||
| ## Problem Framing | ||
|
|
||
| Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation. | ||
|
|
||
| Three components must work together: | ||
| 1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+) | ||
| 2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) | ||
| 3. OpenShell deployed and functional (can create cold-start sandboxes) | ||
|
|
||
| ## Approaches Considered | ||
|
|
||
| ### A: ROSA HCP with Upstream Agent Sandbox Manifests | ||
|
|
||
| <<<<<<< HEAD | ||
| Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with the OpenShell OpenShift deploy chart. | ||
| ======= | ||
| Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with Krzysztof's chart. | ||
| >>>>>>> origin/main | ||
|
|
||
| - Pros: Uses upstream directly, most current version, full control over version | ||
| - Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration | ||
|
|
||
| ### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview) | ||
|
|
||
| <<<<<<< HEAD | ||
| Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with the OpenShell OpenShift deploy chart. | ||
| ======= | ||
| Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with Krzysztof's chart. | ||
| >>>>>>> origin/main | ||
|
|
||
| - Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path | ||
| - Cons: Tech preview may not include extension CRDs yet, version may lag upstream | ||
|
|
||
| ### C: Both Paths Available | ||
|
|
||
| Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them. | ||
|
|
||
| - Pros: Best of both worlds, flexibility | ||
| - Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts | ||
|
|
||
| ## Decision | ||
|
|
||
| **Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available. | ||
|
|
||
| ## Key Requirements | ||
|
|
||
| 1. **ROSA HCP cluster** via `rosa:create` with the AAET profile | ||
| - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA) | ||
| - Region: us-east-2 (matches AAET profile subnets) | ||
| - Worker nodes: at least 3 for realistic scheduling behavior | ||
|
|
||
| 2. **Agent Sandbox installation** | ||
| - Red Hat Agent Sandbox operator from OperatorHub (if available) | ||
| - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim | ||
| - Verify all four CRDs are served: `kubectl api-resources | grep agents` | ||
|
|
||
| 3. **OpenShell deployment** | ||
| <<<<<<< HEAD | ||
| - Clone github.com/2000krysztof/Openshell-Openshift-Deploy at a pinned commit/tag for reproducibility | ||
| ======= | ||
| - Clone github.com/2000krysztof/Openshell-Openshift-Deploy | ||
| >>>>>>> origin/main | ||
| - Run `deploy.sh` with default configuration | ||
| - Verify: `openshell status` shows Connected | ||
| - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works) | ||
|
|
||
| 4. **Image pre-pulling** | ||
| - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes | ||
| - This removes image pull latency from warm pool measurements | ||
| - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull | ||
|
|
||
| 5. **Validation checklist** | ||
| - Gateway pod is Running | ||
| - Agent Sandbox controller is Running | ||
| - Extension CRDs are registered | ||
| - Cold-start sandbox creation works end-to-end | ||
| - Images are pre-pulled on all nodes | ||
|
|
||
| ## Open Questions | ||
|
|
||
| - What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`. | ||
| - Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource? | ||
| <<<<<<< HEAD | ||
| - Does the deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes? | ||
| ======= | ||
| - Does Krzysztof's deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes? | ||
| >>>>>>> origin/main | ||
| - How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔴 Unresolved merge conflict markers break the CI workflow file
Unresolved git merge conflict markers are left in the workflow file (
<<<<<<< HEADat.github/workflows/sync-docs.yml:96), so GitHub Actions will fail to parse the YAML and the sync-docs workflow will not run.Impact: The documentation sync workflow is broken and will error on every trigger.
Merge conflict markers in YAML
Lines 96-100 contain literal
<<<<<<< HEAD,=======, and>>>>>>> origin/mainmarkers between two differentsetup-uvaction SHA references. YAML parsers treat these as invalid syntax, causing the entire workflow to fail before any step executes.Was this helpful? React with 👍 or 👎 to provide feedback.