From 77a6571e8457c274e59d7811968007ff2b882722 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Thu, 9 Jul 2026 08:42:11 +0200 Subject: [PATCH 1/4] Add brainstorm: warm pool feasibility study (4 documents) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Parent brainstorm (01) defines the layered measurement approach for evaluating Agent Sandbox warm pooling on OpenShift. Child documents cover cluster setup (02), measurements (03), and results synthesis (04). Assisted-By: 🤖 Claude Code --- brainstorm/00-overview.md | 32 ++++ brainstorm/01-warm-pool-feasibility.md | 67 +++++++ brainstorm/02-cluster-setup.md | 78 +++++++++ brainstorm/03-warm-pool-measurements.md | 175 +++++++++++++++++++ brainstorm/04-results-and-recommendations.md | 114 ++++++++++++ 5 files changed, 466 insertions(+) create mode 100644 brainstorm/00-overview.md create mode 100644 brainstorm/01-warm-pool-feasibility.md create mode 100644 brainstorm/02-cluster-setup.md create mode 100644 brainstorm/03-warm-pool-measurements.md create mode 100644 brainstorm/04-results-and-recommendations.md diff --git a/brainstorm/00-overview.md b/brainstorm/00-overview.md new file mode 100644 index 0000000000..1c5497fd5e --- /dev/null +++ b/brainstorm/00-overview.md @@ -0,0 +1,32 @@ +# Brainstorm Overview + +Last updated: 2026-07-09 + +## Sessions + +| # | Date | Topic | Status | Spec | Issue | +|---|------|-------|--------|------|-------| +| 01 | 2026-07-09 | warm-pool-feasibility | active | - | - | +| 02 | 2026-07-09 | cluster-setup | active | - | - | +| 03 | 2026-07-09 | warm-pool-measurements | active | - | - | +| 04 | 2026-07-09 | results-and-recommendations | active | - | - | + +## Structure + +01 is the parent document defining the overall feasibility study. 02-04 are execution phases: + +- **02** depends on: nothing (first step) +- **03** depends on: 02 (cluster must be running) +- **04** depends on: 03 (measurements must be complete) + +## Open Threads + +- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02) +- Does env var injection at claim time trigger cold start? (from #01, #03) +- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02) +- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03) +- Should findings be posted to upstream agent-sandbox repo? (from #04) + +## Parked Ideas + +None. diff --git a/brainstorm/01-warm-pool-feasibility.md b/brainstorm/01-warm-pool-feasibility.md new file mode 100644 index 0000000000..057b9135fd --- /dev/null +++ b/brainstorm/01-warm-pool-feasibility.md @@ -0,0 +1,67 @@ +# Brainstorm: Warm Pool Feasibility Study for OpenShell + +**Date:** 2026-07-09 +**Status:** active + +## Problem Framing + +OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable. + +The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`. + +This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture. + +## Approaches Considered + +### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell) + +Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead. + +- Pros: Fast to set up, answers the fundamental feasibility question, no code changes required +- Cons: Does not account for OpenShell supervisor, identity binding, or policy injection + +### B: Measure OpenShell End-to-End with Code Changes + +Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency. + +- Pros: Realistic numbers that include all OpenShell overhead +- Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow + +### C: Layered Approach (A then B) + +Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally. + +- Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data +- Cons: More phases to execute + +## Decision + +**Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency. + +## Key Requirements + +1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements +2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs +3. **OpenShell deployed via Krzysztof's chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup +4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control +5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns) +6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates +7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration +8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope. + +## Experiment Phases + +This study decomposes into three execution phases, each with its own brainstorm document: + +| Phase | Brainstorm | What it covers | +|-------|------------|----------------| +| 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment | +| 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization | +| 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations | + +## Open Questions + +- Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD? +- Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection? +- Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)? +- What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers? diff --git a/brainstorm/02-cluster-setup.md b/brainstorm/02-cluster-setup.md new file mode 100644 index 0000000000..7521da60f6 --- /dev/null +++ b/brainstorm/02-cluster-setup.md @@ -0,0 +1,78 @@ +# Brainstorm: Cluster Setup & OpenShell Deployment + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation. + +Three components must work together: +1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+) +2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) +3. OpenShell deployed and functional (can create cold-start sandboxes) + +## Approaches Considered + +### A: ROSA HCP with Upstream Agent Sandbox Manifests + +Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with Krzysztof's chart. + +- Pros: Uses upstream directly, most current version, full control over version +- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration + +### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview) + +Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with Krzysztof's chart. + +- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path +- Cons: Tech preview may not include extension CRDs yet, version may lag upstream + +### C: Both Paths Available + +Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them. + +- Pros: Best of both worlds, flexibility +- Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts + +## Decision + +**Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available. + +## Key Requirements + +1. **ROSA HCP cluster** via `rosa:create` with the AAET profile + - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA) + - Region: us-east-2 (matches AAET profile subnets) + - Worker nodes: at least 3 for realistic scheduling behavior + +2. **Agent Sandbox installation** + - Red Hat Agent Sandbox operator from OperatorHub (if available) + - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim + - Verify all four CRDs are served: `kubectl api-resources | grep agents` + +3. **OpenShell deployment** + - Clone github.com/2000krysztof/Openshell-Openshift-Deploy + - Run `deploy.sh` with default configuration + - Verify: `openshell status` shows Connected + - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works) + +4. **Image pre-pulling** + - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes + - This removes image pull latency from warm pool measurements + - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull + +5. **Validation checklist** + - Gateway pod is Running + - Agent Sandbox controller is Running + - Extension CRDs are registered + - Cold-start sandbox creation works end-to-end + - Images are pre-pulled on all nodes + +## Open Questions + +- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`. +- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource? +- Does Krzysztof's deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes? +- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously. diff --git a/brainstorm/03-warm-pool-measurements.md b/brainstorm/03-warm-pool-measurements.md new file mode 100644 index 0000000000..42b24cfef3 --- /dev/null +++ b/brainstorm/03-warm-pool-measurements.md @@ -0,0 +1,175 @@ +# Brainstorm: Baseline & Warm Pool Measurements + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +With the cluster running, we need a structured measurement plan that answers: how fast can we get a sandbox with warm pooling, and what are the bottlenecks? The measurements must cover cold-start baseline (control), raw Agent Sandbox warm pooling (no OpenShell), and progressively more realistic configurations that approximate what OpenShell would need. + +The key unknowns are: +- What is the actual cold-start latency breakdown on OpenShift? +- Does warm pooling deliver sub-2s claim-to-ready? +- Is the readiness probe interval the dominant bottleneck, and can Pod Readiness Gates (KEP-580) or Knative-style readiness wrapping eliminate it? +- Can env vars be injected at claim time without triggering cold start? + +## Approaches Considered + +### A: Simple Before/After + +Measure cold start, then warm pool, compare. Two data points. + +- Pros: Fastest to execute +- Cons: No insight into what drives the latency, can't identify optimization levers + +### B: Phase Breakdown with Configuration Matrix + +Measure cold start with per-phase timestamps. Then measure warm pooling across a matrix of configurations (probe intervals, readiness gates, sidecar patterns, env var injection). + +- Pros: Identifies specific bottlenecks, tests optimization hypotheses, actionable data +- Cons: More experiments to run, needs a measurement script + +### C: Full Benchmark Suite with Statistical Rigor + +N=50+ runs per configuration, p50/p90/p99, automated benchmark harness, CSV output for analysis. + +- Pros: Publication-quality data, statistically meaningful +- Cons: Overkill for a feasibility study, takes days + +## Decision + +**Approach B: Phase breakdown with configuration matrix.** N=10-20 runs per configuration is enough to see the pattern. We need per-phase timestamps to identify bottlenecks, and we need the configuration matrix to test our optimization hypotheses (readiness gates, sidecar patterns). A simple shell script with `kubectl` timestamps is sufficient. + +## Experiment Design + +### Experiment 1: Cold-Start Baseline (Control) + +Measure OpenShell sandbox creation latency without pooling. Captures the current state. + +**What to measure:** +- Total time: `openshell sandbox create` to sandbox Ready +- Phase breakdown using `kubectl get events` and pod timestamps: + - API call to pod Scheduled + - Scheduled to image pulled (should be 0 with pre-pulled images) + - Image pulled to init containers complete + - Init containers complete to supervisor Ready + - Supervisor Ready to SSH available + +**Runs:** N=10 with pre-pulled images, N=5 without (to quantify image pull impact) + +### Experiment 2: Vanilla Agent Sandbox Warm Pool (No OpenShell) + +Measure raw Agent Sandbox warm pooling without OpenShell. This isolates Kubernetes overhead from OpenShell overhead. + +**Setup:** +```yaml +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: base-template +spec: + sandbox: + spec: + containers: + - name: agent + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest +--- +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: base-pool +spec: + templateRef: + name: base-template + replicas: 5 +``` + +**What to measure:** +- Pool fill time: SandboxWarmPool created to all 5 replicas Ready +- Claim-to-ready time: SandboxClaim created to adopted Sandbox transitioning to Ready +- Claim-to-ready with default readiness probe (10s periodSeconds) +- Claim-to-ready with aggressive readiness probe (1s periodSeconds) + +**Runs:** N=10 per configuration + +### Experiment 3: Pod Readiness Gates (KEP-580) + +Test whether Pod Readiness Gates can replace polling-based readiness for faster claim-to-ready. + +**How it works:** +- Add a custom ReadinessGate condition (e.g., `sandbox.openshell.io/claimed`) to the pod template +- Warm-pooled pods start with the condition missing (defaults to False, pod is NotReady) +- On claim, an external controller sets the condition to True via a pod status patch +- Pod transitions to Ready immediately (no probe interval wait) + +**What to measure:** +- Time from condition patch to pod Ready status +- Compare with probe-based readiness at 1s and 10s intervals + +**Runs:** N=10 + +**KEP-580 status:** GA since Kubernetes 1.14. Available on all OpenShift versions. No feature gate needed. + +### Experiment 4: Sidecar Readiness Pattern + +Test the Knative-style pattern where a sidecar controls pod readiness. + +**How it works:** +- Define a supervisor-like sidecar (init container with `restartPolicy: Always`) +- Sidecar starts, runs a readiness HTTP endpoint that returns 503 (not ready) +- On claim, inject a signal (touch a file in a shared emptyDir, or set an env var) +- Sidecar detects the signal, flips readiness to 200 +- Pod transitions to Ready + +**What to measure:** +- Time from signal injection to sidecar readiness flip +- Time from sidecar readiness flip to pod Ready +- End-to-end claim-to-ready with sidecar pattern + +**Runs:** N=10 + +**KEP-753 status (native sidecars):** GA since Kubernetes 1.33 (April 2025). Requires OpenShift 4.20+. Sidecar readiness probes contribute to pod readiness. + +### Experiment 5: Env Var Injection at Claim Time + +Test whether SandboxClaim env var injection works without forcing cold start. + +**Setup:** +- SandboxTemplate with `envVarsInjectionPolicy: Allowed` +- SandboxClaim with env vars simulating OpenShell identity (OPENSHELL_SANDBOX_ID, OPENSHELL_ENDPOINT) + +**What to measure:** +- Does the claim adopt a warm sandbox, or does it trigger cold start? +- If warm adoption works: claim-to-ready latency +- If cold start is triggered: document this as a constraint + +**Runs:** N=5 + +### Experiment 6: Combined (Sidecar + Readiness Gates + Env Var Injection) + +Combine the best-performing readiness pattern with env var injection. This approximates what a real OpenShell integration would look like. + +**What to measure:** +- End-to-end claim-to-ready with the full stack +- Compare with cold-start baseline + +**Runs:** N=10 + +## Measurement Script + +A shell script that: +1. Creates a SandboxClaim with `kubectl apply` +2. Records the creation timestamp +3. Watches the pod status until Ready (or timeout) +4. Records the Ready timestamp +5. Calculates the delta +6. Collects pod events for phase breakdown +7. Outputs CSV: run_number, config, create_ts, ready_ts, delta_ms, phases + +## Open Questions + +- Can we use `kubectl wait --for=condition=Ready` with millisecond precision, or do we need a custom watcher? +- For the sidecar experiment, what's the simplest possible sidecar binary? A Go binary that listens on :8080 and watches for a file in /tmp/signal/? +- How does pool replenishment affect back-to-back claim latency? Should we measure burst patterns (claim 5 sandboxes simultaneously)? +- What happens when the pool is exhausted? Does SandboxClaim fall back to cold start or stay Pending? diff --git a/brainstorm/04-results-and-recommendations.md b/brainstorm/04-results-and-recommendations.md new file mode 100644 index 0000000000..67f74bf160 --- /dev/null +++ b/brainstorm/04-results-and-recommendations.md @@ -0,0 +1,114 @@ +# Brainstorm: Results Document & OpenShell Recommendations + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +After the measurements are complete, we need to synthesize findings into a document that serves two audiences: + +1. **The OpenShell core team** (Derek, Murnal, Seth): What should OpenShell change to support warm pooling? Which approach from Issue #2157 is backed by data? What are the architectural constraints? +2. **Our Red Hat team**: Is the upstream Agent Sandbox warm pooling viable for enterprise OpenShift deployments? What gaps exist in the Red Hat tech preview? What do we recommend for the beta timeline? + +The document should be concrete enough to inform Issue #2157's design decisions and the Peter Steinberger demo conversation (July 21st). + +## Approaches Considered + +### A: Internal Technical Report + +A detailed technical document with raw data, analysis, and recommendations. Shared internally and referenced in GitHub issue comments. + +- Pros: Complete record, referenceable, can share selectively +- Cons: May be too dense for quick consumption + +### B: GitHub Issue Comment + Summary Doc + +Post key findings as a comment on Issue #2157 with a link to a detailed report. The comment is the "executive summary," the report is the appendix. + +- Pros: Directly visible to the upstream community, invites discussion, keeps the issue alive +- Cons: GitHub comments are hard to update as understanding evolves + +### C: Both (Report + Issue Comment) + +Write the full report as a document, then distill key findings into a GitHub comment on #2157. + +- Pros: Best of both, full technical depth plus community visibility +- Cons: Two artifacts to maintain + +## Decision + +**Approach C: Full report plus GitHub comment.** The report lives in our Obsidian vault as a reference. A distilled comment on #2157 shares our findings with the upstream community and positions our work as a concrete contribution to the warm pooling discussion. + +## Report Structure + +### 1. Executive Summary +- One paragraph: can warm pooling hit sub-2s on OpenShift? +- Key finding: what is the dominant bottleneck? +- Recommendation: which integration path for OpenShell? + +### 2. Experiment Setup +- Cluster configuration (OpenShift version, K8s version, node count, instance type) +- Agent Sandbox version and CRDs installed +- OpenShell version and deployment method +- Image pre-pull status + +### 3. Results + +#### Cold-Start Baseline +- Table: phase breakdown with p50/p90 latencies +- Chart: latency distribution + +#### Warm Pool Results +- Table: configuration matrix with claim-to-ready latencies +- Comparison: default probes vs. aggressive probes vs. readiness gates vs. sidecar pattern +- Finding: which configuration achieves the target? + +#### Health Check Analysis +- Readiness probe interval impact (10s vs. 1s) +- Pod Readiness Gates (KEP-580) performance +- Sidecar readiness pattern performance +- Knative-style comparison + +#### Env Var Injection +- Does claim-time injection work without cold start? +- Which injection policy is needed? +- What can be injected (env vars) vs. what needs another mechanism (TLS certs, files)? + +### 4. OpenShell Architecture Recommendations + +Map findings to OpenShell's architecture: + +- **Kubernetes driver changes:** What needs to change in `driver.rs` to support SandboxClaim creation? +- **Supervisor changes:** Should the supervisor become a native sidecar with late-bind identity? +- **Gateway store changes:** How should the gateway handle sandbox records for warm-pooled sandboxes? +- **Identity binding:** Which mechanism works (env var injection, volume projection, API call)? +- **Configuration surface:** Where should warm pool configuration live (driver config, workspace admin, operator)? +- **Issue #2157 recommendation:** Which of the four alternatives from the issue is best supported by data? +- **Issue #1447 comparison:** Native pool vs. upstream CRDs, with data backing + +### 5. Gaps and Risks +- Missing features in the Agent Sandbox extension API (e.g., volumeClaimTemplates, Issue #453) +- Red Hat tech preview coverage gaps +- KEP-753 availability on the target OpenShift version +- Pool replenishment under burst load +- Identity isolation between warm slot reuse + +### 6. Next Steps +- Concrete list of upstream contributions (issues, PRs, RFCs) +- Internal work items for the 60-day beta sprint +- Demo plan for the Peter Steinberger meeting + +## Key Requirements + +1. **Report saved to Obsidian vault** at `~/Obsidian/ro14nd/09-Meeting-Notes/` with date prefix +2. **GitHub comment on #2157** with distilled findings (use prose:check before posting) +3. **No Google Drive links** in the GitHub comment (public repo rule from CLAUDE.md) +4. **Data tables with raw numbers**, not just qualitative assessments +5. **Clear recommendation** for the OpenShell core team, not just "it depends" + +## Open Questions + +- Should we also post findings to the upstream Agent Sandbox repo (e.g., as a discussion or issue comment on Issue #390)? +- Should the report include a proposed RFC outline for OpenShell warm pooling, or is that a separate step? +- How much of this should feed into Derek's demo prep for Peter Steinberger? From 6e11634ff7bb8de249793c7a1187ab5654228bca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Fri, 10 Jul 2026 17:12:31 +0200 Subject: [PATCH 2/4] docs: add brainstorm for K8s watch stream crash fix (#2211) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assisted-By: 🤖 Claude Code Signed-off-by: Roland Huß --- brainstorm/05-k8s-watch-crash-fix.md | 49 ++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 brainstorm/05-k8s-watch-crash-fix.md diff --git a/brainstorm/05-k8s-watch-crash-fix.md b/brainstorm/05-k8s-watch-crash-fix.md new file mode 100644 index 0000000000..e5f6561114 --- /dev/null +++ b/brainstorm/05-k8s-watch-crash-fix.md @@ -0,0 +1,49 @@ +# Brainstorm: K8s Watch Stream Crash Fix + +**Date:** 2026-07-10 +**Status:** active +**Issue:** [#2211](https://github.com/NVIDIA/OpenShell/issues/2211) + +## Problem Framing + +The Kubernetes driver's `watch_sandboxes()` crashes with a file descriptor leak when non-gateway `Sandbox` resources exist in the namespace. This happens when the Agent Sandbox operator's `SandboxWarmPool` controller creates `Sandbox` objects that the gateway did not create and does not expect. + +The gateway watch stream has no label selector, so it receives events for all `Sandbox` objects. Warm pool sandboxes lack the `openshell.ai/sandbox-id` label and the `sandbox-` name prefix, causing `sandbox_id_from_object()` to return an error. This error propagates as a stream failure, triggering a 2-second reconnect loop that leaks HTTP/2 connections until FD exhaustion crashes the process (13+ restarts observed). + +The same issue affects `list_sandboxes()`, which also uses unfiltered `ListParams::default()`. + +## Approaches Considered + +### A: Minimal targeted fix (chosen) +Add a label selector (`openshell.ai/managed-by=openshell`) to both `watch_sandboxes()` and `list_sandboxes()` to filter at the API server level. Additionally, change the watch loop to skip (with `debug!` log) objects that fail `sandbox_from_object()` / `sandbox_id_from_object()` instead of sending errors to the channel. + +- Pros: Minimal blast radius (one file, two functions). Reduces API server traffic. Defense-in-depth against future unknown object types. +- Cons: None significant. + +### B: Extract shared helper +Same as A, but extract the label selector construction into a helper function to avoid duplication between `watch_sandboxes()` and `list_sandboxes()`. + +- Pros: DRY for selector construction. +- Cons: Over-engineering for a one-liner format string used in two places. + +### C: Change sandbox_from_object signature +Make `sandbox_from_object` return `Option` instead of `Result`, changing all callers to skip `None`. + +- Pros: Treats unknown objects as normal at the type level. +- Cons: Larger change surface. Doesn't reduce API traffic. Loses error information for genuinely malformed objects. + +## Decision + +Approach A: minimal targeted fix with both label selector and defensive skip. + +## Key Requirements + +- Add `openshell.ai/managed-by=openshell` label selector to the `watcher::Config` in `watch_sandboxes()` (`driver.rs:717`) +- Add the same label selector to `ListParams` in `list_sandboxes()` (`driver.rs:488`) +- In the watch loop, all three event branches (Applied at line 741, Deleted at line 761, Restarted at line 782) must `debug!` log and continue instead of sending `Err` to the channel +- PR targets `origin` (fork) first, then submitted upstream to `NVIDIA/OpenShell` +- Unit tests for the skip behavior + +## Open Questions + +- Should the defensive skip use `debug!` or `warn!` level? (`debug!` seems right since the label filter should prevent this from happening in normal operation; it's only a fallback) From 99801e3f58696a9be5e86a9c39fddbde24190dab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Fri, 10 Jul 2026 17:13:05 +0200 Subject: [PATCH 3/4] docs: update brainstorm overview MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Assisted-By: 🤖 Claude Code Signed-off-by: Roland Huß --- brainstorm/00-overview.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/brainstorm/00-overview.md b/brainstorm/00-overview.md index 1c5497fd5e..f2aeaa2785 100644 --- a/brainstorm/00-overview.md +++ b/brainstorm/00-overview.md @@ -1,6 +1,6 @@ # Brainstorm Overview -Last updated: 2026-07-09 +Last updated: 2026-07-10 ## Sessions @@ -10,6 +10,7 @@ Last updated: 2026-07-09 | 02 | 2026-07-09 | cluster-setup | active | - | - | | 03 | 2026-07-09 | warm-pool-measurements | active | - | - | | 04 | 2026-07-09 | results-and-recommendations | active | - | - | +| 05 | 2026-07-10 | k8s-watch-crash-fix | active | - | [#2211](https://github.com/NVIDIA/OpenShell/issues/2211) | ## Structure @@ -19,6 +20,8 @@ Last updated: 2026-07-09 - **03** depends on: 02 (cluster must be running) - **04** depends on: 03 (measurements must be complete) +05 is a standalone bug fix discovered during the warm pool feasibility study. + ## Open Threads - Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02) @@ -26,6 +29,7 @@ Last updated: 2026-07-09 - Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02) - How does pool exhaustion behave (cold fallback vs. Pending)? (from #03) - Should findings be posted to upstream agent-sandbox repo? (from #04) +- Should the defensive skip use `debug!` or `warn!` level? (from #05) ## Parked Ideas From 16354422919aaa780f4534fa85cef027eac97f51 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Roland=20Hu=C3=9F?= Date: Sat, 11 Jul 2026 11:21:38 +0200 Subject: [PATCH 4/4] docs: warm pool feasibility study with RFCs and prototype brainstorm MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Feasibility study for warm pool integration with the Agent Sandbox operator on OpenShift (ROSA HCP 4.22). Key findings: - Claiming a pre-provisioned pod takes ~1.4s vs 16.7s cold start (12x) - Claim latency is image-independent (tested pause + full 3.2GB sandbox) - Identity binding via spec.env bypasses warm pool (creates fresh pod) Two RFCs for identity binding alternatives: - Annotation-based (~3s latency, ~18 points) - gRPC-push with always-on supervisor (sub-2s target, ~24 points) Prototype brainstorm (M1: ActivateSandbox gRPC PoC, M2: SandboxProfile entity with workspace-scoped pool lifecycle) incorporating NVIDIA feedback on unidentified supervisor state and SandboxProfile entity. Includes reproducible experiment scripts, measurement data, and analysis tooling. Assisted-By: 🤖 Claude Code Signed-off-by: Roland Huß --- .github/workflows/sync-docs.yml | 2 +- .gitignore | 10 + .specify/memory/constitution.md | 65 ++ brainstorm/00-overview.md | 50 + brainstorm/01-warm-pool-feasibility.md | 67 ++ brainstorm/02-cluster-setup.md | 78 ++ brainstorm/03-warm-pool-measurements.md | 175 ++++ brainstorm/04-results-and-recommendations.md | 113 +++ brainstorm/05-warm-pool-grpc-push.md | 95 ++ brainstorm/06-warm-pool-grpc-poc.md | 150 +++ brainstorm/07-warm-pool-sandbox-profile.md | 192 ++++ experiments/RESULTS.md | 952 ++++++++++++++++++ experiments/lib/common.sh | 258 +++++ experiments/lib/wait-ready.sh | 80 ++ .../manifests/image-prepull-daemonset.yaml | 34 + experiments/manifests/readiness-gate-pod.yaml | 15 + experiments/manifests/sandbox-claim.yaml | 17 + experiments/manifests/sandbox-template.yaml | 98 ++ experiments/manifests/sidecar-readiness.yaml | 39 + experiments/manifests/warm-pool.yaml | 8 + experiments/measure-cold-start.sh | 253 +++++ experiments/measure-combined.sh | 286 ++++++ experiments/measure-env-injection.sh | 299 ++++++ experiments/measure-readiness-gates.sh | 102 ++ experiments/measure-sidecar-readiness.sh | 105 ++ experiments/measure-warm-pool.sh | 400 ++++++++ experiments/results/.gitignore | 1 + .../cold-start-noprepull.csv | 6 + .../cold-start-prepulled.csv | 11 + .../cold-start-vanilla.csv | 17 + .../env-injection-allowed.csv | 6 + .../env-injection-disallowed.csv | 4 + .../events-wp-default-run-1.json | 556 ++++++++++ .../events-wp-default-run-10.json | 556 ++++++++++ .../events-wp-default-run-2.json | 556 ++++++++++ .../events-wp-default-run-3.json | 588 +++++++++++ .../events-wp-default-run-4.json | 556 ++++++++++ .../events-wp-default-run-5.json | 556 ++++++++++ .../events-wp-default-run-6.json | 556 ++++++++++ .../events-wp-default-run-7.json | 556 ++++++++++ .../events-wp-default-run-8.json | 556 ++++++++++ .../events-wp-default-run-9.json | 588 +++++++++++ .../readiness-gates.csv | 11 + .../warm-pool-burst.csv | 26 + .../warm-pool-default.csv | 11 + .../cold-start-prepulled.csv | 11 + .../cold-start-vanilla.csv | 11 + .../env-injection-allowed.csv | 6 + .../env-injection-disallowed.csv | 6 + .../readiness-gates.csv | 11 + .../warm-pool-burst.csv | 26 + .../warm-pool-default.csv | 11 + experiments/results/README.md | 40 + .../results/events-prepulled-run-1.json | 524 ++++++++++ .../results/events-prepulled-run-10.json | 524 ++++++++++ .../results/events-prepulled-run-2.json | 556 ++++++++++ .../results/events-prepulled-run-3.json | 524 ++++++++++ .../results/events-prepulled-run-4.json | 524 ++++++++++ .../results/events-prepulled-run-5.json | 524 ++++++++++ .../results/events-prepulled-run-6.json | 556 ++++++++++ .../results/events-prepulled-run-7.json | 556 ++++++++++ .../results/events-prepulled-run-8.json | 524 ++++++++++ .../results/events-prepulled-run-9.json | 524 ++++++++++ experiments/results/events-vanilla-run-1.json | 201 ++++ .../results/events-vanilla-run-10.json | 168 ++++ experiments/results/events-vanilla-run-2.json | 168 ++++ experiments/results/events-vanilla-run-3.json | 168 ++++ experiments/results/events-vanilla-run-4.json | 168 ++++ experiments/results/events-vanilla-run-5.json | 168 ++++ experiments/results/events-vanilla-run-6.json | 168 ++++ experiments/results/events-vanilla-run-7.json | 168 ++++ experiments/results/events-vanilla-run-8.json | 168 ++++ experiments/results/events-vanilla-run-9.json | 168 ++++ .../results/events-wp-default-run-1.json | 588 +++++++++++ .../results/events-wp-default-run-10.json | 588 +++++++++++ .../results/events-wp-default-run-2.json | 556 ++++++++++ .../results/events-wp-default-run-3.json | 556 ++++++++++ .../results/events-wp-default-run-4.json | 556 ++++++++++ .../results/events-wp-default-run-5.json | 556 ++++++++++ .../results/events-wp-default-run-6.json | 556 ++++++++++ .../results/events-wp-default-run-7.json | 556 ++++++++++ .../results/events-wp-default-run-8.json | 556 ++++++++++ .../results/events-wp-default-run-9.json | 556 ++++++++++ experiments/sidecar/Dockerfile | 10 + experiments/sidecar/Makefile | 14 + experiments/sidecar/go.mod | 3 + experiments/sidecar/main.go | 46 + rfc/NNNN-warm-pool-feasibility/README.md | 454 +++++++++ rfc/NNNN-warm-pool-grpc-push/README.md | 371 +++++++ specs/001-warm-pool-feasibility/REVIEWERS.md | 128 +++ .../checklists/requirements.md | 35 + specs/001-warm-pool-feasibility/data-model.md | 75 ++ specs/001-warm-pool-feasibility/plan.md | 153 +++ specs/001-warm-pool-feasibility/quickstart.md | 89 ++ specs/001-warm-pool-feasibility/research.md | 60 ++ .../review-findings.md | 226 +++++ specs/001-warm-pool-feasibility/spec.md | 148 +++ specs/001-warm-pool-feasibility/tasks.md | 175 ++++ .../checklists/requirements.md | 36 + specs/002-warm-pool-grpc-push/data-model.md | 87 ++ specs/002-warm-pool-grpc-push/plan.md | 91 ++ specs/002-warm-pool-grpc-push/research.md | 47 + specs/002-warm-pool-grpc-push/spec.md | 147 +++ specs/002-warm-pool-grpc-push/tasks.md | 81 ++ 104 files changed, 24930 insertions(+), 1 deletion(-) create mode 100644 .specify/memory/constitution.md create mode 100644 brainstorm/00-overview.md create mode 100644 brainstorm/01-warm-pool-feasibility.md create mode 100644 brainstorm/02-cluster-setup.md create mode 100644 brainstorm/03-warm-pool-measurements.md create mode 100644 brainstorm/04-results-and-recommendations.md create mode 100644 brainstorm/05-warm-pool-grpc-push.md create mode 100644 brainstorm/06-warm-pool-grpc-poc.md create mode 100644 brainstorm/07-warm-pool-sandbox-profile.md create mode 100644 experiments/RESULTS.md create mode 100755 experiments/lib/common.sh create mode 100755 experiments/lib/wait-ready.sh create mode 100644 experiments/manifests/image-prepull-daemonset.yaml create mode 100644 experiments/manifests/readiness-gate-pod.yaml create mode 100644 experiments/manifests/sandbox-claim.yaml create mode 100644 experiments/manifests/sandbox-template.yaml create mode 100644 experiments/manifests/sidecar-readiness.yaml create mode 100644 experiments/manifests/warm-pool.yaml create mode 100755 experiments/measure-cold-start.sh create mode 100755 experiments/measure-combined.sh create mode 100755 experiments/measure-env-injection.sh create mode 100755 experiments/measure-readiness-gates.sh create mode 100755 experiments/measure-sidecar-readiness.sh create mode 100755 experiments/measure-warm-pool.sh create mode 100644 experiments/results/.gitignore create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/cold-start-noprepull.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/cold-start-prepulled.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/cold-start-vanilla.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/env-injection-allowed.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/env-injection-disallowed.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-1.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-10.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-2.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-3.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-4.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-5.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-6.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-7.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-8.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-9.json create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/readiness-gates.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/warm-pool-burst.csv create mode 100644 experiments/results/2026-07-09-rosa-hcp-422/warm-pool-default.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/cold-start-prepulled.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/cold-start-vanilla.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/env-injection-allowed.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/env-injection-disallowed.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/readiness-gates.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/warm-pool-burst.csv create mode 100644 experiments/results/2026-07-10-rosa-hcp-422/warm-pool-default.csv create mode 100644 experiments/results/README.md create mode 100644 experiments/results/events-prepulled-run-1.json create mode 100644 experiments/results/events-prepulled-run-10.json create mode 100644 experiments/results/events-prepulled-run-2.json create mode 100644 experiments/results/events-prepulled-run-3.json create mode 100644 experiments/results/events-prepulled-run-4.json create mode 100644 experiments/results/events-prepulled-run-5.json create mode 100644 experiments/results/events-prepulled-run-6.json create mode 100644 experiments/results/events-prepulled-run-7.json create mode 100644 experiments/results/events-prepulled-run-8.json create mode 100644 experiments/results/events-prepulled-run-9.json create mode 100644 experiments/results/events-vanilla-run-1.json create mode 100644 experiments/results/events-vanilla-run-10.json create mode 100644 experiments/results/events-vanilla-run-2.json create mode 100644 experiments/results/events-vanilla-run-3.json create mode 100644 experiments/results/events-vanilla-run-4.json create mode 100644 experiments/results/events-vanilla-run-5.json create mode 100644 experiments/results/events-vanilla-run-6.json create mode 100644 experiments/results/events-vanilla-run-7.json create mode 100644 experiments/results/events-vanilla-run-8.json create mode 100644 experiments/results/events-vanilla-run-9.json create mode 100644 experiments/results/events-wp-default-run-1.json create mode 100644 experiments/results/events-wp-default-run-10.json create mode 100644 experiments/results/events-wp-default-run-2.json create mode 100644 experiments/results/events-wp-default-run-3.json create mode 100644 experiments/results/events-wp-default-run-4.json create mode 100644 experiments/results/events-wp-default-run-5.json create mode 100644 experiments/results/events-wp-default-run-6.json create mode 100644 experiments/results/events-wp-default-run-7.json create mode 100644 experiments/results/events-wp-default-run-8.json create mode 100644 experiments/results/events-wp-default-run-9.json create mode 100644 experiments/sidecar/Dockerfile create mode 100644 experiments/sidecar/Makefile create mode 100644 experiments/sidecar/go.mod create mode 100644 experiments/sidecar/main.go create mode 100644 rfc/NNNN-warm-pool-feasibility/README.md create mode 100644 rfc/NNNN-warm-pool-grpc-push/README.md create mode 100644 specs/001-warm-pool-feasibility/REVIEWERS.md create mode 100644 specs/001-warm-pool-feasibility/checklists/requirements.md create mode 100644 specs/001-warm-pool-feasibility/data-model.md create mode 100644 specs/001-warm-pool-feasibility/plan.md create mode 100644 specs/001-warm-pool-feasibility/quickstart.md create mode 100644 specs/001-warm-pool-feasibility/research.md create mode 100644 specs/001-warm-pool-feasibility/review-findings.md create mode 100644 specs/001-warm-pool-feasibility/spec.md create mode 100644 specs/001-warm-pool-feasibility/tasks.md create mode 100644 specs/002-warm-pool-grpc-push/checklists/requirements.md create mode 100644 specs/002-warm-pool-grpc-push/data-model.md create mode 100644 specs/002-warm-pool-grpc-push/plan.md create mode 100644 specs/002-warm-pool-grpc-push/research.md create mode 100644 specs/002-warm-pool-grpc-push/spec.md create mode 100644 specs/002-warm-pool-grpc-push/tasks.md diff --git a/.github/workflows/sync-docs.yml b/.github/workflows/sync-docs.yml index 874189ed11..c1ca4b39be 100644 --- a/.github/workflows/sync-docs.yml +++ b/.github/workflows/sync-docs.yml @@ -93,7 +93,7 @@ jobs: path: docs-website - name: Install uv - uses: astral-sh/setup-uv@d31148d669074a8d0a63714ba94f3201e7020bc3 # v8.3.0 + uses: astral-sh/setup-uv@f98e06938123ccabd21905ea5d0069192241f9f1 # v8.3.1 with: version: "0.10.12" diff --git a/.gitignore b/.gitignore index 34b24bf768..93cfdebe37 100644 --- a/.gitignore +++ b/.gitignore @@ -229,3 +229,13 @@ scripts/lint-mermaid/node_modules/ # Nix /result /result-* + +# Warm pool experiment shared libraries (overrides Python lib/ pattern above) +!experiments/lib/ + +# spex: generated/local files (only constitution is committed) +# Use worktree-specific pattern to avoid shadowing root .claude/ tracked files +.claude/worktrees/*/.claude/ +**/.specify/** +!**/.specify/memory/ +!**/.specify/memory/constitution.md diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md new file mode 100644 index 0000000000..e29b2d2a62 --- /dev/null +++ b/.specify/memory/constitution.md @@ -0,0 +1,65 @@ + + +# OpenShell Constitution + +## Core Principles + +### I. Safe Arithmetic in Bash Loops + +All bash wait/polling loops MUST use `elapsed=$((elapsed + 1))` or +`(( ++elapsed ))` instead of `(( elapsed++ ))` when running under +`set -e` (or `set -euo pipefail`). + +**Rationale**: Bash `(( expr ))` returns exit code 1 when the expression +evaluates to 0. Post-increment (`elapsed++`) returns the value *before* +incrementing, so when `elapsed=0`, `(( elapsed++ ))` evaluates to 0, +producing exit code 1. Under `set -e`, this terminates the script on the +first loop iteration. The substitution form `$((elapsed + 1))` avoids +this because it is a value expansion, not a command whose exit code is +checked. Pre-increment (`++elapsed`) also works because it returns the +value *after* incrementing (1, not 0). + +### II. Non-Zero Exit on CLI Error + +Usage/help functions called from error paths (unknown options, invalid +arguments) MUST exit with non-zero status. The recommended pattern is +`exit "${1:-0}"` in the usage function body, with `usage 0` for help +requests and `usage 1` for error call sites. + +**Rationale**: When `usage` unconditionally calls `exit 0`, scripts that +receive invalid arguments appear to succeed. CI pipelines, wrapper +scripts, and experiment harnesses that check exit codes will miss the +failure, leading to silent misconfiguration. + +## Scope + +These principles apply to all bash scripts under `experiments/` and any +operator-facing shell tooling in the repository. They do not govern Rust, +Python, or other non-shell code. + +## Governance + +- This constitution supersedes conflicting practices within its scope. +- Amendments require documentation and a version bump. +- Version follows semantic versioning: MAJOR for removed/redefined + principles, MINOR for new principles, PATCH for clarifications. +- Bot review triage may surface recurring patterns that become candidate + principles (see triage workflow Step 14). + +**Version**: 1.0.0 | **Ratified**: 2026-07-10 | **Last Amended**: 2026-07-10 diff --git a/brainstorm/00-overview.md b/brainstorm/00-overview.md new file mode 100644 index 0000000000..a67ab92de3 --- /dev/null +++ b/brainstorm/00-overview.md @@ -0,0 +1,50 @@ +# Brainstorm Overview + +Last updated: 2026-07-11 + +## Sessions + +| # | Date | Topic | Status | Spec | Issue | +|---|------|-------|--------|------|-------| +| 01 | 2026-07-09 | warm-pool-feasibility | active | - | - | +| 02 | 2026-07-09 | cluster-setup | active | - | - | +| 03 | 2026-07-09 | warm-pool-measurements | active | - | - | +| 04 | 2026-07-09 | results-and-recommendations | active | - | - | +| 05 | 2026-07-10 | warm-pool-grpc-push | active | - | - | +| 06 | 2026-07-11 | warm-pool-grpc-poc | active | - | - | +| 07 | 2026-07-11 | warm-pool-sandbox-profile | active | - | - | + +## Structure + +01 is the parent document defining the overall feasibility study. 02-04 are execution phases: + +- **02** depends on: nothing (first step) +- **03** depends on: 02 (cluster must be running) +- **04** depends on: 03 (measurements must be complete) + +05 is the RFC brainstorm for always-on supervisor with gRPC-push identity binding. + +06-07 are prototype milestones incorporating NVIDIA feedback (2026-07-10): + +- **06** (Milestone 1): ActivateSandbox gRPC PoC with unidentified supervisor. Proves the claim-time flow end-to-end with manual pool setup. +- **07** (Milestone 2): SandboxProfile entity + workspace-scoped pool lifecycle. Builds on 06, makes pools mergeable upstream. Depends on Derek's workspace PR. + +## Open Threads + +- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs? (from #01, #02) +- Does env var injection at claim time trigger cold start? (from #01, #03) +- Is KEP-753 (native sidecars) available on the target OpenShift version? (from #01, #02) +- How does pool exhaustion behave (cold fallback vs. Pending)? (from #03) +- Should findings be posted to upstream agent-sandbox repo? (from #04) +- What is the exact proto definition for ActivateSandbox? (from #05, #06) +- How does the supervisor discover the gateway endpoint at activation time? (from #06) +- Should the supervisor support an activation timeout to prevent resource waste? (from #06) +- Coordinate with Craig's work on issue #1955 (legacy RPC cleanup) (from #06) +- What is the relationship between SandboxProfile and SandboxTemplate? (from #07) +- Does Derek's workspace PR define a workspace proto that references SandboxProfiles? (from #07) +- How to handle pool config for workspaces spanning multiple namespaces? (from #07) +- Pool utilization metrics: SandboxProfile status field vs. metric endpoint? (from #07) + +## Parked Ideas + +None. diff --git a/brainstorm/01-warm-pool-feasibility.md b/brainstorm/01-warm-pool-feasibility.md new file mode 100644 index 0000000000..11932c80f2 --- /dev/null +++ b/brainstorm/01-warm-pool-feasibility.md @@ -0,0 +1,67 @@ +# Brainstorm: Warm Pool Feasibility Study for OpenShell + +**Date:** 2026-07-09 +**Status:** active + +## Problem Framing + +OpenShell's Kubernetes driver creates a fresh `agents.x-k8s.io` Sandbox CR for every sandbox request. This cold-start path pays for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration on every create. Measured latency is 8-12 seconds. For agent harnesses like OpenClaw that create sandboxes per tool call or per sub-agent, this is unusable. + +The upstream Agent Sandbox project (kubernetes-sigs/agent-sandbox, v0.5.0, v1beta1 API) provides extension CRDs for warm pooling: SandboxTemplate, SandboxWarmPool, and SandboxClaim. OpenShell does not use any of these today. The Kubernetes driver has no awareness of the extension API group `extensions.agents.x-k8s.io`. + +This feasibility study measures whether warm pooling can reduce sandbox startup latency to under 2 seconds on OpenShift, and produces a recommendation for how OpenShell should integrate warm pooling into its architecture. + +## Approaches Considered + +### A: Measure Raw Agent Sandbox Warm Pooling (Without OpenShell) + +Deploy the Agent Sandbox extension CRDs on OpenShift, create warm pools with vanilla containers, and measure claim-to-ready latency. This isolates the Kubernetes layer from OpenShell overhead. + +- Pros: Fast to set up, answers the fundamental feasibility question, no code changes required +- Cons: Does not account for OpenShell supervisor, identity binding, or policy injection + +### B: Measure OpenShell End-to-End with Code Changes + +Modify the OpenShell Kubernetes driver to create SandboxClaims instead of direct Sandbox CRs, then measure end-to-end latency. + +- Pros: Realistic numbers that include all OpenShell overhead +- Cons: Requires significant code changes before any measurement, high risk of wasted effort if the raw K8s layer is already too slow + +### C: Layered Approach (A then B) + +Start with raw Agent Sandbox measurements (no OpenShell code changes), then layer on OpenShell-specific concerns (supervisor sidecar, identity injection, readiness patterns) incrementally. + +- Pros: Answers feasibility fast, builds understanding incrementally, each layer adds data +- Cons: More phases to execute + +## Decision + +**Approach C: Layered measurement.** Start with raw Agent Sandbox warm pooling on OpenShift to establish a baseline, then progressively add OpenShell-specific complexity. This avoids wasting effort on code changes if the underlying Kubernetes primitives can't deliver acceptable latency. + +## Key Requirements + +1. **Fresh ROSA HCP cluster** provisioned via the ROSA plugin for consistent, isolated measurements +2. **Red Hat build of Agent Sandbox operator** (tech preview) installed from OperatorHub for the extension CRDs +3. **OpenShell deployed via the OpenShift deploy chart** (github.com/2000krysztof/Openshell-Openshift-Deploy) for fast setup +4. **Baseline cold-start measurements** (vanilla sandbox creation, no pooling) as the control +5. **Warm pool measurements** with varying configurations (readiness probe intervals, Pod Readiness Gates, sidecar readiness patterns) +6. **Health check optimization experiments** including Knative-style readiness wrapping and KEP-580 Pod Readiness Gates +7. **Results document** with measured data, phase-by-phase breakdown, and architectural recommendations for OpenShell warm pool integration +8. **Scope: Kubernetes only.** Podman/Docker single-player warm pooling is out of scope. + +## Experiment Phases + +This study decomposes into three execution phases, each with its own brainstorm document: + +| Phase | Brainstorm | What it covers | +|-------|------------|----------------| +| 1 | 02-cluster-setup | ROSA cluster provisioning, Agent Sandbox operator, OpenShell deployment | +| 2 | 03-warm-pool-measurements | Cold-start baseline, warm pool experiments, health check optimization | +| 3 | 04-results-and-recommendations | Data synthesis, OpenShell architecture recommendations | + +## Open Questions + +- Does the Red Hat Agent Sandbox operator tech preview include the extension CRDs (SandboxWarmPool, SandboxClaim, SandboxTemplate), or only the core Sandbox CRD? +- Does env var injection at SandboxClaim time trigger a cold start, or can the `envVarsInjectionPolicy` on SandboxTemplate enable true warm adoption with claim-time injection? +- Is KEP-753 (native sidecar containers) available on the target OpenShift version (needs K8s 1.33+ / OpenShift 4.20+)? +- What is the minimum OpenShift version that supports both the Agent Sandbox operator tech preview and native sidecar containers? diff --git a/brainstorm/02-cluster-setup.md b/brainstorm/02-cluster-setup.md new file mode 100644 index 0000000000..af0580dc38 --- /dev/null +++ b/brainstorm/02-cluster-setup.md @@ -0,0 +1,78 @@ +# Brainstorm: Cluster Setup & OpenShell Deployment + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +Before any measurements can happen, we need an OpenShift cluster with OpenShell and the Agent Sandbox extension CRDs running. This phase covers provisioning, installation, and validation. + +Three components must work together: +1. A ROSA HCP cluster with a Kubernetes version that supports native sidecar containers (K8s 1.33+) +2. The Agent Sandbox operator with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) +3. OpenShell deployed and functional (can create cold-start sandboxes) + +## Approaches Considered + +### A: ROSA HCP with Upstream Agent Sandbox Manifests + +Provision ROSA, install agent-sandbox from upstream manifests (both `manifest.yaml` and `extensions.yaml`), deploy OpenShell with the OpenShell OpenShift deploy chart. + +- Pros: Uses upstream directly, most current version, full control over version +- Cons: No operator lifecycle management, manual CRD updates, no OperatorHub integration + +### B: ROSA HCP with Red Hat Agent Sandbox Operator (Tech Preview) + +Provision ROSA, install the Red Hat build of Agent Sandbox from OperatorHub, deploy OpenShell with the OpenShell OpenShift deploy chart. + +- Pros: Operator manages CRD lifecycle, OperatorHub integration, downstream supported path +- Cons: Tech preview may not include extension CRDs yet, version may lag upstream + +### C: Both Paths Available + +Install the Red Hat operator for the core Sandbox CRD, then layer upstream extension manifests on top if the operator doesn't include them. + +- Pros: Best of both worlds, flexibility +- Cons: Mixed provenance (downstream operator + upstream extensions), potential version conflicts + +## Decision + +**Approach C: Start with the Red Hat operator, fall back to upstream extensions.** Install the tech preview operator from OperatorHub first. If it includes the extension CRDs, use them. If not, apply upstream `extensions.yaml` on top. This gives us the downstream operator path for the core CRDs while ensuring we have warm pool primitives available. + +## Key Requirements + +1. **ROSA HCP cluster** via `rosa:create` with the AAET profile + - OpenShift version must support K8s 1.33+ for native sidecar containers (KEP-753 GA) + - Region: us-east-2 (matches AAET profile subnets) + - Worker nodes: at least 3 for realistic scheduling behavior + +2. **Agent Sandbox installation** + - Red Hat Agent Sandbox operator from OperatorHub (if available) + - Upstream extensions.yaml as fallback for SandboxTemplate/SandboxWarmPool/SandboxClaim + - Verify all four CRDs are served: `kubectl api-resources | grep agents` + +3. **OpenShell deployment** + - Clone github.com/2000krysztof/Openshell-Openshift-Deploy at a pinned commit/tag for reproducibility + - Run `deploy.sh` with default configuration + - Verify: `openshell status` shows Connected + - Verify: `openshell sandbox create --from base` succeeds (cold-start baseline works) + +4. **Image pre-pulling** + - Pre-pull the OpenShell base sandbox image and supervisor image on all worker nodes + - This removes image pull latency from warm pool measurements + - Use a DaemonSet with `initContainers` that pull and exit, or `oc debug node/` to pre-pull + +5. **Validation checklist** + - Gateway pod is Running + - Agent Sandbox controller is Running + - Extension CRDs are registered + - Cold-start sandbox creation works end-to-end + - Images are pre-pulled on all nodes + +## Open Questions + +- What OpenShift version does ROSA HCP currently offer that includes K8s 1.33+? Need to check `rosa list versions`. +- Does the Red Hat Agent Sandbox operator tech preview install from the default OperatorHub catalog, or does it require a custom CatalogSource? +- Does the deploy script handle OpenShift 4.20+ or does it need updates for newer SCC/security changes? +- How much cluster capacity do we need? The warm pool experiments will create 5-20 pre-provisioned pods simultaneously. diff --git a/brainstorm/03-warm-pool-measurements.md b/brainstorm/03-warm-pool-measurements.md new file mode 100644 index 0000000000..42b24cfef3 --- /dev/null +++ b/brainstorm/03-warm-pool-measurements.md @@ -0,0 +1,175 @@ +# Brainstorm: Baseline & Warm Pool Measurements + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +With the cluster running, we need a structured measurement plan that answers: how fast can we get a sandbox with warm pooling, and what are the bottlenecks? The measurements must cover cold-start baseline (control), raw Agent Sandbox warm pooling (no OpenShell), and progressively more realistic configurations that approximate what OpenShell would need. + +The key unknowns are: +- What is the actual cold-start latency breakdown on OpenShift? +- Does warm pooling deliver sub-2s claim-to-ready? +- Is the readiness probe interval the dominant bottleneck, and can Pod Readiness Gates (KEP-580) or Knative-style readiness wrapping eliminate it? +- Can env vars be injected at claim time without triggering cold start? + +## Approaches Considered + +### A: Simple Before/After + +Measure cold start, then warm pool, compare. Two data points. + +- Pros: Fastest to execute +- Cons: No insight into what drives the latency, can't identify optimization levers + +### B: Phase Breakdown with Configuration Matrix + +Measure cold start with per-phase timestamps. Then measure warm pooling across a matrix of configurations (probe intervals, readiness gates, sidecar patterns, env var injection). + +- Pros: Identifies specific bottlenecks, tests optimization hypotheses, actionable data +- Cons: More experiments to run, needs a measurement script + +### C: Full Benchmark Suite with Statistical Rigor + +N=50+ runs per configuration, p50/p90/p99, automated benchmark harness, CSV output for analysis. + +- Pros: Publication-quality data, statistically meaningful +- Cons: Overkill for a feasibility study, takes days + +## Decision + +**Approach B: Phase breakdown with configuration matrix.** N=10-20 runs per configuration is enough to see the pattern. We need per-phase timestamps to identify bottlenecks, and we need the configuration matrix to test our optimization hypotheses (readiness gates, sidecar patterns). A simple shell script with `kubectl` timestamps is sufficient. + +## Experiment Design + +### Experiment 1: Cold-Start Baseline (Control) + +Measure OpenShell sandbox creation latency without pooling. Captures the current state. + +**What to measure:** +- Total time: `openshell sandbox create` to sandbox Ready +- Phase breakdown using `kubectl get events` and pod timestamps: + - API call to pod Scheduled + - Scheduled to image pulled (should be 0 with pre-pulled images) + - Image pulled to init containers complete + - Init containers complete to supervisor Ready + - Supervisor Ready to SSH available + +**Runs:** N=10 with pre-pulled images, N=5 without (to quantify image pull impact) + +### Experiment 2: Vanilla Agent Sandbox Warm Pool (No OpenShell) + +Measure raw Agent Sandbox warm pooling without OpenShell. This isolates Kubernetes overhead from OpenShell overhead. + +**Setup:** +```yaml +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: base-template +spec: + sandbox: + spec: + containers: + - name: agent + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest +--- +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: base-pool +spec: + templateRef: + name: base-template + replicas: 5 +``` + +**What to measure:** +- Pool fill time: SandboxWarmPool created to all 5 replicas Ready +- Claim-to-ready time: SandboxClaim created to adopted Sandbox transitioning to Ready +- Claim-to-ready with default readiness probe (10s periodSeconds) +- Claim-to-ready with aggressive readiness probe (1s periodSeconds) + +**Runs:** N=10 per configuration + +### Experiment 3: Pod Readiness Gates (KEP-580) + +Test whether Pod Readiness Gates can replace polling-based readiness for faster claim-to-ready. + +**How it works:** +- Add a custom ReadinessGate condition (e.g., `sandbox.openshell.io/claimed`) to the pod template +- Warm-pooled pods start with the condition missing (defaults to False, pod is NotReady) +- On claim, an external controller sets the condition to True via a pod status patch +- Pod transitions to Ready immediately (no probe interval wait) + +**What to measure:** +- Time from condition patch to pod Ready status +- Compare with probe-based readiness at 1s and 10s intervals + +**Runs:** N=10 + +**KEP-580 status:** GA since Kubernetes 1.14. Available on all OpenShift versions. No feature gate needed. + +### Experiment 4: Sidecar Readiness Pattern + +Test the Knative-style pattern where a sidecar controls pod readiness. + +**How it works:** +- Define a supervisor-like sidecar (init container with `restartPolicy: Always`) +- Sidecar starts, runs a readiness HTTP endpoint that returns 503 (not ready) +- On claim, inject a signal (touch a file in a shared emptyDir, or set an env var) +- Sidecar detects the signal, flips readiness to 200 +- Pod transitions to Ready + +**What to measure:** +- Time from signal injection to sidecar readiness flip +- Time from sidecar readiness flip to pod Ready +- End-to-end claim-to-ready with sidecar pattern + +**Runs:** N=10 + +**KEP-753 status (native sidecars):** GA since Kubernetes 1.33 (April 2025). Requires OpenShift 4.20+. Sidecar readiness probes contribute to pod readiness. + +### Experiment 5: Env Var Injection at Claim Time + +Test whether SandboxClaim env var injection works without forcing cold start. + +**Setup:** +- SandboxTemplate with `envVarsInjectionPolicy: Allowed` +- SandboxClaim with env vars simulating OpenShell identity (OPENSHELL_SANDBOX_ID, OPENSHELL_ENDPOINT) + +**What to measure:** +- Does the claim adopt a warm sandbox, or does it trigger cold start? +- If warm adoption works: claim-to-ready latency +- If cold start is triggered: document this as a constraint + +**Runs:** N=5 + +### Experiment 6: Combined (Sidecar + Readiness Gates + Env Var Injection) + +Combine the best-performing readiness pattern with env var injection. This approximates what a real OpenShell integration would look like. + +**What to measure:** +- End-to-end claim-to-ready with the full stack +- Compare with cold-start baseline + +**Runs:** N=10 + +## Measurement Script + +A shell script that: +1. Creates a SandboxClaim with `kubectl apply` +2. Records the creation timestamp +3. Watches the pod status until Ready (or timeout) +4. Records the Ready timestamp +5. Calculates the delta +6. Collects pod events for phase breakdown +7. Outputs CSV: run_number, config, create_ts, ready_ts, delta_ms, phases + +## Open Questions + +- Can we use `kubectl wait --for=condition=Ready` with millisecond precision, or do we need a custom watcher? +- For the sidecar experiment, what's the simplest possible sidecar binary? A Go binary that listens on :8080 and watches for a file in /tmp/signal/? +- How does pool replenishment affect back-to-back claim latency? Should we measure burst patterns (claim 5 sandboxes simultaneously)? +- What happens when the pool is exhausted? Does SandboxClaim fall back to cold start or stay Pending? diff --git a/brainstorm/04-results-and-recommendations.md b/brainstorm/04-results-and-recommendations.md new file mode 100644 index 0000000000..7df557618c --- /dev/null +++ b/brainstorm/04-results-and-recommendations.md @@ -0,0 +1,113 @@ +# Brainstorm: Results Document & OpenShell Recommendations + +**Date:** 2026-07-09 +**Status:** active +**Parent:** 01-warm-pool-feasibility + +## Problem Framing + +After the measurements are complete, we need to synthesize findings into a document that serves two audiences: + +1. **The OpenShell core team**: What should OpenShell change to support warm pooling? Which approach from Issue #2157 is backed by data? What are the architectural constraints? +2. **The Red Hat integration team**: Is the upstream Agent Sandbox warm pooling viable for enterprise OpenShift deployments? What gaps exist in the Red Hat tech preview? + +The document should be concrete enough to inform Issue #2157's design decisions. + +## Approaches Considered + +### A: Internal Technical Report + +A detailed technical document with raw data, analysis, and recommendations. Shared internally and referenced in GitHub issue comments. + +- Pros: Complete record, referenceable, can share selectively +- Cons: May be too dense for quick consumption + +### B: GitHub Issue Comment + Summary Doc + +Post key findings as a comment on Issue #2157 with a link to a detailed report. The comment is the "executive summary," the report is the appendix. + +- Pros: Directly visible to the upstream community, invites discussion, keeps the issue alive +- Cons: GitHub comments are hard to update as understanding evolves + +### C: Both (Report + Issue Comment) + +Write the full report as a document, then distill key findings into a GitHub comment on #2157. + +- Pros: Best of both, full technical depth plus community visibility +- Cons: Two artifacts to maintain + +## Decision + +**Approach C: Full report plus GitHub comment.** The report is published as a standalone RFC in `rfc/` (see clarification in spec.md). A distilled comment on #2157 can be posted as a follow-up step to share findings with the upstream community. + +## Report Structure + +### 1. Executive Summary +- One paragraph: can warm pooling hit sub-2s on OpenShift? +- Key finding: what is the dominant bottleneck? +- Recommendation: which integration path for OpenShell? + +### 2. Experiment Setup +- Cluster configuration (OpenShift version, K8s version, node count, instance type) +- Agent Sandbox version and CRDs installed +- OpenShell version and deployment method +- Image pre-pull status + +### 3. Results + +#### Cold-Start Baseline +- Table: phase breakdown with p50/p90 latencies +- Chart: latency distribution + +#### Warm Pool Results +- Table: configuration matrix with claim-to-ready latencies +- Comparison: default probes vs. aggressive probes vs. readiness gates vs. sidecar pattern +- Finding: which configuration achieves the target? + +#### Health Check Analysis +- Readiness probe interval impact (10s vs. 1s) +- Pod Readiness Gates (KEP-580) performance +- Sidecar readiness pattern performance +- Knative-style comparison + +#### Env Var Injection +- Does claim-time injection work without cold start? +- Which injection policy is needed? +- What can be injected (env vars) vs. what needs another mechanism (TLS certs, files)? + +### 4. OpenShell Architecture Recommendations + +Map findings to OpenShell's architecture: + +- **Kubernetes driver changes:** What needs to change in `driver.rs` to support SandboxClaim creation? +- **Supervisor changes:** Should the supervisor become a native sidecar with late-bind identity? +- **Gateway store changes:** How should the gateway handle sandbox records for warm-pooled sandboxes? +- **Identity binding:** Which mechanism works (env var injection, volume projection, API call)? +- **Configuration surface:** Where should warm pool configuration live (driver config, workspace admin, operator)? +- **Issue #2157 recommendation:** Which of the four alternatives from the issue is best supported by data? +- **Issue #1447 comparison:** Native pool vs. upstream CRDs, with data backing + +### 5. Gaps and Risks +- Missing features in the Agent Sandbox extension API (e.g., volumeClaimTemplates, Issue #453) +- Red Hat tech preview coverage gaps +- KEP-753 availability on the target OpenShift version +- Pool replenishment under burst load +- Identity isolation between warm slot reuse + +### 6. Next Steps +- Concrete list of upstream contributions (issues, PRs, RFCs) +- Internal work items for the next sprint +- Follow-up actions for stakeholder discussions + +## Key Requirements + +1. **Report saved to the configured notes vault** with a date prefix +2. **RFC in `rfc/`** as the canonical, version-controlled results document +3. **No Google Drive links** in any public-facing artifacts +4. **Data tables with raw numbers**, not just qualitative assessments +5. **Clear recommendation** for the OpenShell core team, not just "it depends" + +## Open Questions + +- Should we also post findings to the upstream Agent Sandbox repo (e.g., as a discussion or issue comment on Issue #390)? +- Should the report include a proposed RFC outline for OpenShell warm pooling, or is that a separate step? diff --git a/brainstorm/05-warm-pool-grpc-push.md b/brainstorm/05-warm-pool-grpc-push.md new file mode 100644 index 0000000000..945656eb46 --- /dev/null +++ b/brainstorm/05-warm-pool-grpc-push.md @@ -0,0 +1,95 @@ +# Brainstorm: Warm Pool with Always-On Supervisor (gRPC-Push Identity) + +**Date:** 2026-07-10 +**Status:** active + +## Problem Framing + +The existing warm pool RFC (`rfc/NNNN-warm-pool-feasibility/`) proposes annotation-based identity binding: the supervisor starts at claim time, detects its identity via the Kubernetes downward API, and bootstraps normally. This approach delivers ~3-4s claim latency (2.3s claim + 1.5s supervisor startup). + +For workloads that need sub-1s sandbox provisioning, the supervisor startup at claim time is the bottleneck. The 1.5s breaks down as ~80ms for 8 gRPC calls and ~1.4s for process startup plus OPA policy compilation. + +An alternative architecture keeps the supervisor always running in the warm pod. Global OPA policies (applicable to every sandbox) are pre-compiled at pool time. At claim time, the gateway pushes the sandbox identity directly to the supervisor via gRPC, the supervisor compiles only sandbox-specific policies, and the session activates. This eliminates process startup entirely and reduces OPA compilation to the per-sandbox delta. + +This brainstorm scopes the alternative RFC, its relationship to the annotation RFC, and how gateway-managed pool lifecycle works across multiple sandbox images. + +## Approaches Considered + +### A: Two standalone RFCs with cross-references + +Create a new RFC (`NNNN-warm-pool-grpc-push`) that stands independently alongside the existing annotation RFC. Both get an "Alternative Approaches" section with a comparison table. No ordering or primary/secondary framing. + +- Pros: Each RFC is self-contained, reviewable independently. Maintainers can accept one, both, or neither. +- Cons: Some duplication in shared context (motivation, measurements). + +### B: Single RFC with two design variants + +Merge both approaches into one RFC with variant sections. + +- Pros: No duplication, single document. +- Cons: Very long. Harder to discuss one variant without the other. + +### C: Split RFC by concern (3 documents) + +Shared pool infrastructure RFC plus two identity binding RFCs. + +- Pros: Cleanest separation. +- Cons: Three documents is heavy to manage. + +## Decision + +**Approach A: Two standalone RFCs with cross-references.** The shared context is short enough to duplicate. The pool management section (gateway-managed pools, multi-image config) lives in the new gRPC-push RFC since it's the one proposing gateway-managed pools. The annotation RFC can reference it. + +## Key Requirements + +### gRPC-Push RFC content + +1. **Always-on supervisor in warm pods.** Supervisor starts at pool provisioning time. Pre-compiles global OPA policies. Exposes a new gRPC endpoint (`ActivateSandbox` or similar) for identity push. + +2. **Two-tier OPA compilation.** Global policies (applicable to all sandboxes) compile at pool time. Sandbox-specific policies (per-sandbox rules, provider-specific constraints) compile at claim time after identity push. The split must be explicit in the RFC. + +3. **Identity binding via gRPC push.** After the operator binds the claim, the gateway reads the pod IP from `.status.sandbox.podIPs`, connects directly to the supervisor's gRPC endpoint, and pushes sandbox identity (ID, name, policy config). No downward API, no annotation propagation delay. + +4. **Gateway-managed pool lifecycle.** The gateway owns SandboxTemplate + SandboxWarmPool resource creation, scaling, and cleanup. Gateway config defines which images get pools and their sizes. Gateway reconciles pool state against desired config. + +5. **Pool-per-image for known images.** Gateway config lists images with pool sizes. Example: + ```yaml + warmPools: + - image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + replicas: 5 + - image: ghcr.io/nvidia/openshell-community/sandboxes/ollama:latest + replicas: 2 + ``` + +6. **Unknown images always cold-start.** Custom Dockerfiles, arbitrary image refs, first-time images go through the existing cold-start path. No auto-promotion in the initial design. + +7. **Dynamic pool promotion as documented extension.** The RFC includes a "Future Extensions" section describing auto-promotion: gateway creates a pool for an image after seeing it N times. Not part of the initial design. + +### Cross-referencing and comparison + +8. **Both RFCs get an "Alternative Approaches" section.** Each links to the other with a comparison table covering: + - Claim-to-ready latency + - Supervisor complexity (idle-start vs always-on) + - Gateway complexity (claim-only vs pool reconciler + gRPC client) + - OPA compilation (full at claim vs pre-compiled global + delta) + - Security surface (downward API vs new gRPC endpoint on supervisor) + - Alignment with upstream (craig-kindo's validated approach vs novel) + - Resource cost of idle pools (sleeping process vs running supervisor) + +9. **Equal framing.** Neither RFC is "primary." Both are valid alternatives for different latency targets and complexity appetites. + +### Upstream references (shared) + +- [OpenShell#2157](https://github.com/NVIDIA/OpenShell/issues/2157): Feature issue for warm-pool provisioning +- [OpenShell#1447](https://github.com/NVIDIA/OpenShell/issues/1447): craig-kindo's annotation-based implementation (validates the other RFC) +- [agent-sandbox#1118](https://github.com/kubernetes-sigs/agent-sandbox/pull/1118): Operator adoption finalization improvement +- [agent-sandbox#384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384): Upstream file-based env injection tracking + +## Open Questions + +- What is the new gRPC endpoint name and proto definition for the supervisor identity push? +- How does the gateway discover pool pod IPs before the supervisor has registered? (likely from `.status.sandbox.podIPs` on the adopted Sandbox resource) +- What happens if the gRPC push fails (supervisor crashed, network partition)? Fall back to cold start or retry? +- How does gateway config validation prevent conflicting pool definitions (same image, different pool sizes)? +- Should the gateway pool reconciler run as a background loop or be event-driven (watch SandboxWarmPool status)? +- How does the always-on supervisor handle image updates? The supervisor process is running an older binary while the pool template references a newer image. diff --git a/brainstorm/06-warm-pool-grpc-poc.md b/brainstorm/06-warm-pool-grpc-poc.md new file mode 100644 index 0000000000..8b20cca09a --- /dev/null +++ b/brainstorm/06-warm-pool-grpc-poc.md @@ -0,0 +1,150 @@ +# Brainstorm: Warm Pool gRPC PoC (Milestone 1) + +**Date:** 2026-07-11 +**Status:** active + +## Problem Framing + +The feasibility study proved that claiming a pre-provisioned pod from a +SandboxWarmPool takes ~1.4s (operator reconciliation), compared to ~16.7s +for a full OpenShell cold start. The main blocker is identity binding: +`spec.env` on SandboxClaim bypasses the warm pool entirely. + +Two RFCs proposed solutions (annotation-based at ~3s, gRPC-push at sub-2s). +NVIDIA feedback from the Slack thread (2026-07-10) confirmed: + +- **+1 on gRPC-push** (Andrew Newberry): "This is the direction we want to + move the supervisor to." +- **Unidentified supervisor state** (Andrew): Warm pods should run with an + unidentified supervisor, identity arrives at claim time. No pre-compiled + global OPA (policies "may become out of date by the time the sandbox + comes online"). +- **Validation from internal POC** (Dhiraj Bokde): They've used pooled + pods for months, <1s claim latency confirmed, "pod has to be re-IDed + once claimed." + +This PoC proves the claim-time gRPC flow end-to-end before wiring the +entity model (see brainstorm 07). + +## Approaches Considered + +### A: ActivateSandbox with Unidentified Supervisor (chosen) + +Supervisor starts in an unidentified/unbound state in warm pool pods. +No gateway connection, no OPA compilation, no identity. At claim time, +the gateway discovers the pod IP from the SandboxClaim status, calls +`ActivateSandbox` with sandbox ID, JWT, name, and full policy config. +The supervisor bootstraps from there (OPA compile, gateway connect). + +- Pros: Aligns with NVIDIA direction (Andrew's unidentified state + proposal). Avoids stale global policy problem. Simpler supervisor + lifecycle (idle until activated). Clean separation between pool-time + and claim-time. +- Cons: All OPA compilation happens at claim time (~100-200ms estimated + for sandbox-specific policies without global pre-compilation). Slightly + higher claim latency than two-tier OPA, but simpler. + +### B: Two-Tier OPA (original RFC proposal) + +Supervisor starts at pool time, pre-compiles global OPA policies, waits +for ActivateSandbox with only the sandbox-specific delta. + +- Pros: Lowest possible claim-time latency (global policies already + compiled). +- Cons: Andrew explicitly pushed back: "policies may become out of date." + More complex supervisor lifecycle (must handle policy refresh, partial + state). Requires the supervisor to know what "global" means at pool + time without gateway context. + +### C: Annotation-based (companion RFC) + +Supervisor starts at claim time, detects identity via downward API +annotations. No new gRPC endpoint. + +- Pros: Simplest implementation (~18 story points). No new proto surface. +- Cons: ~3-4s latency (1.5s supervisor startup at claim time). NVIDIA + prefers gRPC-push direction. Downward API propagation adds 1-2s. + +## Decision + +**Approach A: ActivateSandbox with Unidentified Supervisor.** + +This aligns with NVIDIA's stated direction and Andrew's explicit +recommendation. The two-tier OPA optimization (Approach B) can be +re-evaluated later as a latency optimization if claim-time OPA +compilation becomes a bottleneck, but the unidentified state is the +right starting point. + +## Key Requirements + +### Supervisor Changes + +1. **Unidentified startup mode**: Supervisor process starts without + gateway connection, identity, or OPA policies. Listens on a gRPC + port and exposes `/readyz` once ready to receive activation. + +2. **ActivateSandbox gRPC endpoint**: New RPC on the supervisor that + receives sandbox identity (ID, name, JWT) and policy configuration. + On receipt, the supervisor: + - Stores identity + - Compiles OPA policies from the provided config + - Calls `IssueSandboxToken`, `GetSandboxConfig`, etc. against the + gateway + - Calls `ConnectSupervisor` to register the session + - Returns success/failure to the caller + +3. **Readiness signaling**: Pod readinessProbe checks `/readyz` which + returns 200 when the supervisor is listening for `ActivateSandbox` + (not when the sandbox is fully bootstrapped). + +### Gateway Changes (K8s Driver) + +4. **Claim-time activation flow**: After SandboxClaim binds and reports + Ready with a pod IP, the gateway: + - Reads pod IP from claim status + - Calls `ActivateSandbox` on the supervisor + - Waits for success response + - Returns sandbox as ready to the CLI + +5. **Pod IP discovery**: Read from `.status.sandbox.podIPs` on the + SandboxClaim (or `.status.sandbox.podIP`). + +6. **mTLS for ActivateSandbox**: Use the existing namespace mTLS + certificates for the gateway-to-supervisor channel. + +### Proto Changes + +7. **New ActivateSandbox RPC**: Added to the supervisor service proto. + Request carries sandbox ID, name, JWT, policy config, gateway + endpoint. Response carries success/failure and error details. + +### Pool Setup (Manual for PoC) + +8. **Manual pool creation**: For the PoC, pools are created manually + via kubectl (SandboxTemplate + SandboxWarmPool). No automated + lifecycle management yet (that's milestone 2). + +9. **Supervisor image in warm pods**: The warm pool SandboxTemplate + must include the supervisor container configured to start in + unidentified mode (new CLI flag or env var). + +### Cold-Start Fallback + +10. **Fallback path**: If no warm pool exists for the requested image, + or readyReplicas is 0, fall back to the existing cold-start path. + The two paths must coexist. + +## Open Questions + +- What is the exact proto definition for `ActivateSandbox`? Should it + be a new service or added to the existing supervisor service? +- How does the supervisor discover the gateway endpoint at activation + time? Passed in the ActivateSandbox request, or via env var at pool + provisioning time? +- Should the supervisor support a timeout for activation (e.g., kill + the pod if not activated within N minutes to prevent resource waste)? +- Related upstream work: issue #1955 (legacy RPC cleanup) may affect + where the new RPC is added. Coordinate with Craig's work. +- How to handle the case where ActivateSandbox fails (supervisor + crashes, OPA compilation error)? Should the gateway retry or fall + back to cold start? diff --git a/brainstorm/07-warm-pool-sandbox-profile.md b/brainstorm/07-warm-pool-sandbox-profile.md new file mode 100644 index 0000000000..bc29efc585 --- /dev/null +++ b/brainstorm/07-warm-pool-sandbox-profile.md @@ -0,0 +1,192 @@ +# Brainstorm: SandboxProfile + Workspace-Scoped Pool Lifecycle (Milestone 2) + +**Date:** 2026-07-11 +**Status:** active + +## Problem Framing + +Milestone 1 (brainstorm 06) proves the ActivateSandbox gRPC claim-time +flow with manually created pools. This milestone addresses the entity +model and lifecycle management that makes warm pools mergeable upstream. + +NVIDIA feedback (2026-07-10 Slack thread) established clear direction: + +- **Derek Carr**: Pools must be workspace-scoped, not global gateway + config. Workspaces map to namespaces. Different workspaces need + different pools (openclaw tool execution vs opencode on-demand). + "Prototyping with global config is fine for now, I just don't think + we will want to merge that." + +- **Andrew Newberry**: Use `SandboxProfile` as the entity, consistent + with Provider/ProviderProfile naming. Attach K8s-specific warm pool + config via the `driver_config` passthrough (RFC 0006, already + implemented). This avoids leaking K8s details to non-K8s drivers. + +- **Derek Carr**: "Maybe we expose certain entity model resources if + and only if OpenShell is using a particular driver." Also mentioned + that for non-K8s drivers, pools could signal image pre-pulls. + +## Approaches Considered + +### A: SandboxProfile with driver_config for pools (chosen) + +Introduce a `SandboxProfile` entity. The K8s driver's `driver_config` +block carries warm pool settings (image, replicas, readiness config). +The gateway reads SandboxProfile and reconciles SandboxWarmPool +resources in the workspace's namespace. + +``` +SandboxProfile + name: "fast-coding" + sandbox_template: "base" + driver_config: + kubernetes: + warm_pool: + replicas: 5 + readiness_timeout: 300s +``` + +Workspace references one or more SandboxProfiles. When a workspace is +created in a namespace, the gateway creates corresponding +SandboxWarmPool resources in that namespace. + +- Pros: Consistent with NVIDIA entity naming (Provider/ProviderProfile). + Uses existing RFC 0006 driver_config passthrough (no new API pattern). + Naturally workspace-scoped. Non-K8s drivers can ignore the warm_pool + block or use it for pre-pulls. +- Cons: Introduces a new entity (SandboxProfile). Requires workspace + entity to exist first (Derek's PR). + +### B: Workspace.spec.pools direct config + +Pool config lives directly in the workspace spec, no SandboxProfile +indirection: + +``` +Workspace + spec: + pools: + - image: "base:latest" + replicas: 5 +``` + +- Pros: Simpler (no new entity). Pool config is right where you see it. +- Cons: Workspace becomes K8s-aware (pool replicas are a K8s concept). + Can't reuse pool configs across workspaces. Doesn't follow the + ProviderProfile pattern. Andrew explicitly recommended SandboxProfile. + +### C: Gateway TOML with namespace mapping + +Keep pool config in gateway TOML but add namespace targeting: + +```toml +[[compute.warm_pools]] +image = "base:latest" +replicas = 5 +namespace = "workspace-alpha" +``` + +- Pros: No new entities. Works without workspace entity. +- Cons: Derek explicitly rejected global config ("we will not want to + merge that"). Doesn't scale to dynamic workspace creation. Static + config doesn't know about future namespaces. + +## Decision + +**Approach A: SandboxProfile with driver_config for pools.** + +This is the consensus from both Derek (workspace-scoped) and Andrew +(SandboxProfile entity, driver_config passthrough). The gateway TOML +approach (Approach C) is acceptable for the PoC in milestone 1 but +must not be the merge target. + +## Key Requirements + +### Entity Model + +1. **SandboxProfile proto definition**: New message in the OpenShell + proto. Contains a name, a reference to a SandboxTemplate (or inline + template spec), and a `driver_config` JSON block following RFC 0006. + +2. **K8s driver_config schema for warm pools**: + ```json + { + "warm_pool": { + "replicas": 5, + "readiness_timeout": "300s", + "max_surge": 1 + } + } + ``` + The K8s driver owns validation of this block. Other drivers ignore it + or interpret `warm_pool` as a pre-pull hint. + +3. **Workspace references SandboxProfile**: When a workspace is + configured, it references one or more SandboxProfiles by name. Each + profile maps to pool resources in the workspace's namespace. + +### Gateway Pool Lifecycle + +4. **Pool reconciliation per workspace**: When the gateway learns about + a workspace (via watch or startup scan), it reads the associated + SandboxProfiles, extracts the K8s driver_config, and creates/updates + SandboxTemplate + SandboxWarmPool resources in the workspace's + namespace. + +5. **Pool teardown**: When a workspace is deleted or a SandboxProfile + is removed from it, the gateway deletes the corresponding pool + resources. + +6. **Pool update**: When a SandboxProfile's driver_config changes + (replica count, image), the gateway updates the SandboxWarmPool. + Changes to image require draining the old pool and creating a new + one. + +### Non-K8s Driver Behavior + +7. **Docker/Podman**: The `warm_pool` driver_config block signals + which images to pre-pull. No container pooling (containers are + cheap to create locally). + +8. **VM driver**: Future consideration. Snapshot-based pooling is + possible but out of scope. + +### Integration with Milestone 1 + +9. **ActivateSandbox flow unchanged**: The claim-time gRPC flow from + milestone 1 is not affected by how pools are created. SandboxProfile + controls pool lifecycle; ActivateSandbox controls claim-time + activation. Clean separation. + +10. **Migration path**: PoC can start with gateway TOML config (M1), + then replace the pool creation source with SandboxProfile reads + (M2) without changing the claim-time path. + +### RBAC and Security + +11. **Workspace admin can configure pools**: SandboxProfile creation + and workspace association should be available to workspace admins, + not just cluster admins. + +12. **Pool resource limits**: Consider per-workspace quotas on total + warm pool replicas to prevent resource exhaustion. + +## Open Questions + +- What is the relationship between SandboxProfile and existing + SandboxTemplate? Is SandboxProfile a wrapper around SandboxTemplate + plus driver_config, or a separate entity that references a template? +- Does Derek's workspace PR define a workspace proto/CRD? If so, how + does it reference SandboxProfiles? By name, by label selector, or + inline? +- Should SandboxProfile be a K8s CRD or a gateway-internal entity + stored in the gateway's state? CRD is more natural for K8s but adds + RBAC complexity. +- How to handle pool config for images that don't yet exist in a + SandboxProfile? Fall back to cold start, or auto-create a profile? +- Pool utilization metrics: Derek raised the question of whether to + expose utilization. Should this be a SandboxProfile status field, + a separate metric endpoint, or both? +- What happens when a workspace spans multiple namespaces (Derek + mentioned this for operator workloads)? One pool per namespace, + or a pool that somehow spans namespaces? diff --git a/experiments/RESULTS.md b/experiments/RESULTS.md new file mode 100644 index 0000000000..a1afa0f7eb --- /dev/null +++ b/experiments/RESULTS.md @@ -0,0 +1,952 @@ +# Warm Pool Feasibility Study: Results + +## TL;DR + +**Warm pooling works.** Claiming a pre-provisioned pod from the Agent Sandbox operator's `SandboxWarmPool` takes **~1.4s** (operator reconciliation) compared to 16.7s for a full OpenShell cold start. This holds regardless of pod complexity (pause image vs full 3.2 GB sandbox with PVC, init container, and 4 volume mounts) because the claim only transfers ownership. With supervisor startup at claim time, estimated production latency is **~3s** (5-6x improvement). The sub-2s target looks achievable with the gRPC-push approach. + +Results validated across two independent ROSA HCP clusters (2026-07-09 and 2026-07-10). An independent implementation on GKE by @craig-kindo measured **1.9s p50** (including CLI overhead) using the same approach ([OpenShell#1447 comment](https://github.com/NVIDIA/OpenShell/issues/1447#issuecomment-4904470006)). + +| Scenario | Latency | vs cold start | +|----------|---------|---------------| +| OpenShell cold start (prepulled) | 16.7s (p50) | baseline | +| Warm pool claim (operator reconciliation) | ~1.4s | **~12x faster** | +| Warm pool claim (1s-poll measurement scripts) | 2.3s (p50) | 7x faster (includes polling overhead) | +| Claim with env vars | 15.2s (p50) | cold-start fallback (pool bypassed) | + +Note: The experiment scripts polled with a 1-second sleep interval, reporting 2.3s. Tight-polling (100ms interval) reveals the actual operator reconciliation time is ~1.4s (1,369-1,516ms across 5 runs). The difference (~0.9s) is client-side polling overhead. + +**The blocker is identity binding.** Env var injection (`spec.env`) forces a cold-start pod, bypassing the pool entirely. Two alternatives work: + +1. **Annotation-based** ([RFC](../rfc/NNNN-warm-pool-feasibility/README.md), verified in this study): inject sandbox ID via `additionalPodMetadata.annotations`, supervisor watches via downward API. ~1-2s propagation delay. +2. **Direct gRPC push** ([RFC](../rfc/NNNN-warm-pool-grpc-push/README.md), not yet tested): gateway pushes identity to the supervisor's pod IP after claim binds. Lowest latency, but requires a new supervisor endpoint and the gateway knowing the pod IP before registration. + +### Related upstream work + +- [OpenShell#2157](https://github.com/NVIDIA/OpenShell/issues/2157): Feature issue for warm-pool provisioning in the Kubernetes driver (spike-generated, covers all affected components) +- [OpenShell#1447](https://github.com/NVIDIA/OpenShell/issues/1447): Earlier warm pool feature request with @craig-kindo's validated implementation proposal +- [agent-sandbox#1118](https://github.com/kubernetes-sigs/agent-sandbox/pull/1118): Operator PR improving warm-pool adoption finalization (removes cache-lag requeue deferral) +- [agent-sandbox#384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384): Upstream tracking for file-based env injection as an alternative to `spec.env` + +The companion [RFC](../rfc/NNNN-warm-pool-feasibility/README.md) details all required changes per component. + +--- + +## Test Fixture + +| Parameter | Value | +|-----------|-------| +| Date | 2026-07-09 | +| Cluster | ROSA HCP (short-lived experiment cluster) | +| OpenShift version | 4.22.3 (Kubernetes 1.35.5) | +| Region | us-east-2 (US East, Ohio) | +| Worker nodes | 3x `m5.2xlarge` (8 vCPU, 32 GB RAM each) | +| Agent Sandbox operator | Red Hat build v0.9.0 (Tech Preview) | +| OpenShell gateway | v0.0.80 (Helm chart 0.0.73) | +| Sandbox image | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` (3.2 GB) | +| Supervisor image | `ghcr.io/nvidia/openshell/supervisor:latest` (15 MB) | +| Warm pool image | `ghcr.io/nvidia/openshell-community/sandboxes/base:latest` with `sleep infinity` entrypoint (see Methodology) | +| Warm pool size | 5 replicas | +| Network plugin | OVNKubernetes | +| Image pre-pull | DaemonSet on all worker nodes | +| AWS profile | (internal test account) | + +The cluster was provisioned as a short-lived experiment environment and torn down after data collection. All measurements were taken under idle cluster conditions with no competing workloads. + +## Validation Run (2026-07-10) + +A second run was conducted on a fresh ROSA HCP cluster to validate the original findings. The second cluster used the same OpenShift version (4.22.3) and operator version (v0.9.0) on Subnet 3 (the first run used Subnet 1). All experiment scripts were unchanged. + +### Validation Run: Test Fixture + +| Parameter | Run 1 (Jul 09) | Run 2 (Jul 10) | +|-----------|----------------|----------------| +| Cluster | warm-pool-study | warm-pool-rerun | +| OpenShift version | 4.22.3 | 4.22.3 | +| Workers | 3x m5.2xlarge | 3x m5.2xlarge | +| Operator | v0.9.0 | v0.9.0 | +| Helm chart | 0.0.73 | 0.0.73 | +| Region / subnet | us-east-2 / Subnet 1 | us-east-2 / Subnet 3 | + +### Validation Run: Results Comparison + +| Experiment | Run 1 Mean | Run 2 Mean | Delta | Verdict | +|------------|-----------|-----------|-------|---------| +| Warm pool default (10 runs) | 2,340 ms | 2,326 ms | -14 ms (0.6%) | Confirmed | +| Cold start prepulled (10 runs) | 18,262 ms | 20,643 ms | +2,381 ms (13%) | Consistent (see note) | +| Cold start vanilla (10 runs) | 2,788 ms | 2,923 ms | +135 ms (4.8%) | Confirmed | +| Env injection allowed (5 runs) | 15,754 ms | 14,041 ms | -1,713 ms (11%) | Confirmed (cold-start fallback) | +| Env injection disallowed | All timeout | All timeout | N/A | Confirmed | +| Readiness gates (10 runs) | 2,132 ms | 2,897 ms | +765 ms (36%) | Consistent (see note) | + +**Warm pool claim latency is reproducible.** The mean claim time of 2,326 ms (run 2) vs 2,340 ms (run 1) is within noise. The tight clustering (run 2: 2,270-2,382 ms) matches the first run (2,307-2,388 ms), confirming that the ~2.3s figure is a stable, repeatable measurement. + +**Cold start prepulled variance.** The 13% increase in run 2 mean is explained by one outlier (run 9: 65,024 ms, likely a node scale event during the experiment). Excluding that outlier, the run 2 mean drops to ~15,700 ms, within the first run's range. + +**Cold start vanilla improvement.** Run 2 measured 2,923 ms vs run 1's 2,788 ms (when excluding outliers from run 1's broader dataset). The run 1 report notes that some vanilla runs included create-failed and outlier runs that were excluded from the reported statistics. + +**Readiness gates variance.** The 36% increase is expected given the bimodal distribution (kubelet sync period). Both runs show the same pattern: roughly half the runs at ~1s and half at ~3s. The difference in means reflects which sync phase dominated more runs, not a behavioral change. + +**Burst behavior confirmed.** Run 2 burst claims show the same serialized pattern as run 1: each concurrent claim adds ~2s (claim 1: ~2.4s, claim 2: ~4.4s, claim 3: ~6.4s, claim 4: ~8.4s, claim 5: ~10.4s). All 25 claims across 5 rounds succeeded. + +**All critical findings confirmed:** + +1. Env var injection bypasses the warm pool (creates cold-start pod, pool stays at 5/5) +2. Annotation injection with `openshell.io/` domain works (warm pod adopted, no restart) +3. Env injection disallowed policy correctly rejects claims with clear error message +4. Pool replenishment between rounds works reliably + +### Validation Run: Setup Notes + +Two issues were encountered during the validation run that did not affect the original run: + +1. **Operator install mode.** The OperatorGroup with `targetNamespaces: [agent-sandbox-system]` (OwnNamespace mode) fails with "OwnNamespace InstallModeType not supported." The fix is to use an empty `spec: {}` (AllNamespaces mode). The reproduction instructions in this document have been corrected. + +2. **Image pre-pull DaemonSet.** The supervisor image (`ghcr.io/nvidia/openshell/supervisor:latest`) is a scratch image without `/bin/true`. The init container command `["/bin/true"]` fails. However, the image is still pulled successfully by the container runtime before the command fails, so the pre-pull objective is met. A fix for the DaemonSet manifest would use a command that exists in the supervisor image. + +## How to Reproduce + +### Prerequisites + +- AWS account with ROSA HCP provisioning rights +- `rosa`, `oc`, `kubectl`, and `openshell` CLIs installed +- Authentication to the AWS account (SAML, SSO, or IAM credentials) +- `jq` for JSON processing, `gdate` (macOS) or `date` (Linux) for timestamps + +### Cluster Setup + +```bash +# 1. Authenticate +rosa login --use-device-code --url=https://api.openshift.com + +# 2. Provision cluster (adjust profile values for your account) +rosa create cluster --sts --hosted-cp \ + --cluster-name= \ + --version=4.22.3 \ + --region= \ + --compute-machine-type=m5.2xlarge \ + --subnet-ids=, \ + --mode=auto \ + --oidc-config-id= \ + --role-arn= \ + --support-role-arn= \ + --worker-iam-role= + +# 3. Wait for workers (typically 20-30 min after control plane ready) +rosa list machinepools -c + +# 4. Create admin user and login +rosa create idp --cluster= --type=htpasswd \ + --name=htpasswd-admin --users 'admin:' -y +rosa grant user cluster-admin --cluster= --user=admin +oc login -u admin -p '' --insecure-skip-tls-verify + +# 5. Install Agent Sandbox operator from OperatorHub +oc apply -f - <Operator + participant Warm Pod + + Note over Warm Pod: idle process running,
no supervisor, no identity + + CLI->>Gateway: CreateSandbox + Gateway->>Gateway: Mint sandbox UUID + JWT + Gateway->>K8s API: Create SandboxClaim
(warmPoolRef, annotations) + K8s API->>Operator: Watch event + Operator->>Warm Pod: Patch annotations
(openshell.io/sandbox-id) + Operator->>K8s API: Set claim Ready + Note over Warm Pod: Downward API propagates
annotation to file (~1-2s) + Warm Pod->>Warm Pod: Detect identity, start supervisor + Warm Pod->>Gateway: IssueSandboxToken (31ms) + Gateway->>Warm Pod: Sandbox JWT + Warm Pod->>Gateway: GetSandboxConfig + 3 more (50ms) + Warm Pod->>Gateway: ReportPolicyStatus + ConnectSupervisor (30ms) + Note over Gateway: Sandbox marked Ready + Gateway->>CLI: WatchSandbox ready event + CLI->>Gateway: CreateSshSession + ForwardTcp (0.9s) +``` + +**Step 1: CLI requests sandbox creation** + +The user runs `openshell sandbox create`. The CLI calls the gateway's `CreateSandbox` gRPC endpoint. + +**Step 2: Gateway mints identity** + +The gateway generates a new sandbox UUID (`OPENSHELL_SANDBOX_ID`) and a sandbox JWT. It stores the sandbox in its internal sandbox store with status "pending". Today the gateway creates a Sandbox resource in K8s. With warm pooling, it creates a SandboxClaim instead: + +```yaml +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxClaim +metadata: + name: my-sandbox + namespace: openshell +spec: + warmPoolRef: + name: openshell-warm-pool + # NO spec.env (bypasses warm pool in v0.9.0) + # NO labels (rejected by operator's domain allowlist) + additionalPodMetadata: + annotations: + openshell.io/sandbox-id: "a1b2c3d4-..." + openshell.io/sandbox-name: "my-sandbox" +``` + +**Step 3: Agent Sandbox operator binds claim to warm pod** (~instant) + +The warm pool controller selects an available pod from the pool and patches its annotations with the values from `additionalPodMetadata`. The container is NOT restarted. The pod continues running with its original `sleep infinity` command. The claim status transitions to Ready. + +**Step 4: Supervisor detects activation** + +The supervisor (or its idle-mode replacement) watches for annotation changes via the Kubernetes downward API. The downward API projects pod annotations into a file at a known path (e.g., `/etc/podinfo/annotations`). When the `openshell.io/sandbox-id` annotation appears, the supervisor reads the sandbox ID and begins the activation sequence. + +``` +watch /etc/podinfo/annotations: + if openshell.io/sandbox-id is present and non-empty: + read sandbox ID from annotation value + start supervisor with identity +``` + +The kubelet propagates annotation changes to the projected volume within its sync period (~1-2 seconds with default settings). + +**Step 5: Supervisor authenticates** (~31ms) + +The supervisor calls `IssueSandboxToken` on the gateway, presenting its K8s ServiceAccount token from the projected volume at `/var/run/secrets/openshell/token`. + +The gateway performs a TokenReview, which returns the pod name and UID. The gateway then reads the pod's `openshell.io/sandbox-id` annotation (patched in Step 3) and verifies it matches a known sandbox in its store. It issues a sandbox-scoped JWT. + +**Step 6: Supervisor loads configuration** (~50ms) + +The supervisor makes 4 gRPC calls using the JWT: +- `GetSandboxConfig`: Retrieves sandbox settings +- `UpdateConfig`: Backfills policy from discovered rules +- `GetSandboxProviderEnvironment`: Fetches provider credentials +- `GetInferenceBundle`: Fetches inference routing configuration + +**Step 7: Supervisor applies policy and connects** (~30ms) + +- `ReportPolicyStatus`: Confirms OPA policy compiled and loaded +- `ConnectSupervisor`: Registers supervisor session with the gateway + +At this point the gateway marks the sandbox as Ready in its store. The CLI's `WatchSandbox` stream receives the ready event. + +**Step 8: CLI opens SSH session** (~0.9s) + +The CLI calls `CreateSshSession` and `ForwardTcp` to establish the SSH relay through the gateway. The user's terminal connects. + +### Total Identity Swap Timeline + +| Step | Duration | Cumulative | +|------|----------|------------| +| 1. CLI calls CreateSandbox | <1ms | 0s | +| 2. Gateway mints identity + creates SandboxClaim | ~20ms | 0.02s | +| 3. Agent Sandbox operator binds claim + patches annotations | ~0.3s | 0.3s | +| 4. Downward API propagates annotation to pod | ~1-2s | 1.3-2.3s | +| 5. Supervisor authenticates (IssueSandboxToken) | ~31ms | ~2.3s | +| 6. Supervisor loads config (4 gRPC calls) | ~50ms | ~2.4s | +| 7. Policy load + ConnectSupervisor | ~30ms | ~2.4s | +| OPA compilation overhead | ~0.9s | ~3.3s | +| 8. CLI opens SSH session | ~0.9s | ~4.2s | + +**Estimated total: ~3-4 seconds** from sandbox request to SSH ready. The dominant variable is the downward API propagation delay (step 4), which depends on the kubelet's sync frequency. + +### Upstream Dependency for Identity Binding + +The operator's env var injection (`spec.env` on SandboxClaim) bypasses the warm pool, making it unusable for identity binding. OpenShell must implement its own identity binding mechanism rather than relying on the operator's env var injection. Two approaches are explored in companion RFCs (see below). + +## Conclusions + +### Feasibility: Confirmed + +Warm pooling is technically feasible on OpenShift with the Agent Sandbox operator. The operator's claim-to-ready latency is ~1.4s (operator reconciliation), compared to 16.7s for a full cold start. The sub-2-second target is achievable. + +### Engineering Challenges + +The supervisor itself is fast (1.5s for 8 gRPC calls, of which only 80ms is network time). The challenges are elsewhere: + +1. **The sandbox image cannot idle.** It crashes without the supervisor. A warm pool pod needs a modified entrypoint that can wait for activation. + +2. **Identity is early-bound.** The supervisor reads identity from env vars set at pod creation time. Warm pooling requires late binding at claim time. + +3. **Per-sandbox credentials.** User-scoped credentials (OBO token exchange) and SPIFFE SVIDs cannot be pre-fetched because the user identity is unknown at pool provisioning time (see the [Per-Sandbox Identity section](#per-sandbox-identity-token-exchange-and-spiffe)). + +4. **The K8s lifecycle dominates cold start.** Scheduling (3.2s), volume attach (2.0s), and init containers (1.0s) account for 60% of cold-start time. Warm pooling eliminates all of this. + +### Two Approaches + +Both are documented as standalone RFCs with a side-by-side comparison. Neither is recommended over the other; the choice depends on the latency target and acceptable complexity. + +| | Annotation-based ([RFC](../rfc/NNNN-warm-pool-feasibility/README.md)) | gRPC-push ([RFC](../rfc/NNNN-warm-pool-grpc-push/README.md)) | +|---|---|---| +| **Supervisor at pool time** | Idle process (not running) | Always-on (global OPA pre-compiled) | +| **Identity delivery** | Downward API annotation projection | Direct gRPC call to pod IP | +| **Estimated latency** | ~3s | <2s | +| **User-scoped credentials** | Separate mechanism needed | Single `ActivateSandbox` call | +| **Effort (Phase 1)** | ~19 points | ~24 points | +| **Validated externally** | Yes (@craig-kindo, GKE) | Not yet | + +## Configuration Recommendations for Sandbox Tuning + +These settings are informed by the measurements and apply to both current (cold-start) and future (warm pool) deployments. + +### Image Pre-Pulling + +Deploy the image pre-pull DaemonSet (`experiments/manifests/image-prepull-daemonset.yaml`) on all clusters where sandboxes will be created. The 3.2 GB sandbox base image takes 20-30 seconds to pull on a cold node, which dwarfs all other latency contributors. With cached images, pull time drops to sub-second. The DaemonSet only pre-pulls the sandbox base image. The supervisor image (15 MB) pulls in under 1 second and does not need pre-pulling. Note: the supervisor image is a scratch image with no shell or utilities, so init containers using it cannot run commands like `/bin/true`. + +### Warm Pool Sizing + +The pool of 5 replicas handled all test scenarios without exhaustion, including burst claims of 5 simultaneous requests. For production, size the pool based on the expected burst rate of sandbox creation requests. A pool equal to the maximum expected concurrent sandbox creations per minute provides headroom for replenishment. + +Pool replenishment with the `pause` image takes under 10 seconds. With the full sandbox image (once the idle-mode architecture is implemented), replenishment will take approximately the same as a cold start (~16s). Plan pool size accordingly to absorb bursts during replenishment windows. + +### Resource Requests + +Warm pool pods with `pause` consume negligible resources (1m CPU, 4Mi RAM). With a real sandbox image in idle mode, resource requests should match the supervisor's baseline footprint. A reasonable starting point: + +```yaml +resources: + requests: + cpu: "100m" + memory: "128Mi" + limits: + cpu: "500m" + memory: "512Mi" +``` + +Idle-mode pods should use lower limits than active sandbox pods. The operator could increase limits at claim time if the SandboxTemplate supports resource overrides. + +### Namespace Isolation + +Deploy warm pools in the same namespace as the OpenShell gateway (`openshell` by default). The SandboxWarmPool controller watches for SandboxClaims in the same namespace as the pool. Cross-namespace claims are not supported. + +### Pool Lifecycle: Disposable Pods, Not Recycled + +The Agent Sandbox operator follows a **disposable pattern**: each warm pool pod serves exactly one sandbox session. Used pods are deleted, not returned to the pool. The pool controller replenishes by creating fresh pods. This is by design (see [Agent Sandbox lifecycle docs](https://agent-sandbox.sigs.k8s.io/docs/sandbox/lifecycle/) and [architecture overview](https://agent-sandbox.sigs.k8s.io/docs/getting_started/overview/)). + +The lifecycle for a single warm pod: + +``` +Pool creates fresh pod → Claim adopts pod → Agent session runs → Session ends → Pod deleted → Pool creates new fresh pod +``` + +**How the operator detects "finished":** The [SandboxClaim `lifecycle` field](https://agent-sandbox.sigs.k8s.io/docs/) provides three mechanisms: + +| Field | Type | How it works | +|-------|------|-------------| +| `lifecycle.shutdownTime` | RFC 3339 timestamp | Hard deadline. Pod is deleted at this time regardless of state. | +| `lifecycle.ttlSecondsAfterFinished` | integer | Auto-delete N seconds after the main container exits. The operator watches for container exit as the "finished" signal. | +| `lifecycle.shutdownPolicy` | `Delete` / `DeleteForeground` / `Retain` | What happens on shutdown. `Delete` removes the sandbox. `Retain` keeps the pod for debugging. | + +For OpenShell, the gateway would set `ttlSecondsAfterFinished` on the SandboxClaim. When the agent disconnects and the supervisor exits, the operator counts down and deletes the claim. The pool controller sees the pool is below its desired replica count and provisions a fresh pod. + +**No reclaim, no cleanup problem.** Because each session gets a fresh pod with a fresh PVC, there is no filesystem contamination between agent sessions. No cleanup hooks or workspace wipes are needed. The trade-off is replenishment time: creating a new pod with the full sandbox image takes ~16 seconds. This cost is hidden behind the pool if it is sized correctly. + +**Pool sizing for continuous availability:** The pool must absorb the replenishment delay. If agents use sandboxes for T minutes on average and replenishment takes R seconds: + +- **Steady state**: `pool_size >= concurrent_sessions + ceil(concurrent_sessions * R / (T * 60))` +- **Example**: 10 concurrent agents, 5-minute average sessions, 16-second replenishment: `10 + ceil(10 * 16 / 300) = 11` replicas + +With 11 replicas, the pool absorbs the replenishment delay and no agent ever hits a cold start. + +**OPA policy compilation at claim time.** Because each session gets a fresh pod, OPA policies must be compiled at claim time (policies are per-sandbox, not per-pool). The 1.4-second OPA compilation cost applies to every claim. Pre-compiling a base policy during pool provisioning would save time only if the claim-time policy is a subset or small delta of the base. This narrows the sub-1-second estimate: with per-session OPA compilation, the realistic floor for claim-time supervisor startup is ~1.5 seconds. + +**Update strategy.** The SandboxWarmPool supports two `updateStrategy` options: +- `Recreate`: Replace all pool pods when the template changes (disruptive but immediate). +- `OnReplenish`: Apply template changes only to newly provisioned pods (gradual, no disruption to running sessions). + +`OnReplenish` is the safer default for production: template updates (image version bumps, config changes) roll out naturally as pods are claimed and replenished. + +### Multi-Image Pool Management + +OpenShell supports multiple sandbox images. Users can create sandboxes from community images (`--from base`, `--from ollama`), custom Dockerfiles, or arbitrary container image references. Warm pooling must handle this diversity without requiring a separate pool for every possible image. + +**The problem:** A warm pool is tied to a single SandboxTemplate, which specifies one container image. If the user requests a sandbox with a different image, the pool cannot serve the request. Cold-start fallback is the only option for unknown images. + +**Gateway-managed pools.** The gateway should own the lifecycle of warm pools (creating, scaling, and deleting `SandboxTemplate` + `SandboxWarmPool` resources) rather than delegating this to the Helm chart or a static installation step. This allows the gateway to: + +- Manage multiple pools for different images from a single configuration +- Route sandbox requests to the correct pool based on the requested image +- Fall back to cold start when no pool matches +- Scale pools independently based on usage patterns +- Support future dynamic pool creation for unknown images without redeployment + +The gateway configuration defines which images get pools: + +```yaml +warmPools: + - image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + replicas: 5 + - image: ghcr.io/nvidia/openshell-community/sandboxes/ollama:latest + replicas: 2 +``` + +At startup, the gateway reconciles the cluster state against this configuration: creating pools that don't exist, updating replica counts that changed, and deleting pools that were removed. When a sandbox is requested, the gateway looks up the image in its pool registry and creates a `SandboxClaim` if a matching pool has available replicas. Otherwise, it falls back to cold start. + +**Custom/Dockerfile images** always cold-start on first use because the image doesn't exist until build time. + +**Future extension: dynamic pool promotion.** The gateway could create pools for previously unknown images after seeing them used repeatedly. This trades idle resource cost for latency reduction on subsequent requests. It is not part of the initial design but is enabled by the gateway-managed approach (the gateway already has the RBAC and reconciliation logic to create pools at runtime). + +**Cross-driver abstraction.** The pool management concept is Kubernetes-specific (SandboxWarmPool is a K8s CRD), but the optimization pattern applies to other drivers (Docker: pre-created containers, VM: pre-booted snapshots). The gateway should expose a driver-agnostic pool management interface so the same configuration and routing logic works regardless of the compute backend: + +```rust +trait WarmPoolManager { + async fn has_warm_instance(&self, image: &str) -> bool; + async fn claim_warm_instance(&self, image: &str, identity: &SandboxIdentity) -> Result; + async fn create_cold(&self, image: &str, identity: &SandboxIdentity) -> Result; +} +``` + +Each driver implements this trait differently. The Kubernetes driver creates `SandboxClaim` resources. The Docker driver starts pre-created containers. The VM driver restores snapshots. The gateway's sandbox creation path calls `has_warm_instance` first, then `claim_warm_instance` or `create_cold` depending on pool availability. This keeps pool management decisions in the gateway regardless of the underlying compute driver. + +## Outlook: Full OpenShell Integration + +This feasibility study measured the raw operator mechanics. A full OpenShell integration requires additional work that we could not test with the current architecture. + +### Next Steps for the Kubernetes Driver + +1. **Pool-aware provisioning path** in `crates/openshell-driver-kubernetes/`: When a pool is available, create a SandboxClaim instead of a Sandbox resource. Fall back to cold start when the pool is empty or unavailable. + +2. **Claim status monitoring**: Watch the SandboxClaim resource for Ready condition instead of polling the Sandbox directly. Use `.status.sandbox.name` (not `.status.sandboxRef.name`) to get the bound sandbox name. + +3. **Pool detection**: List SandboxWarmPool resources in the namespace to determine pool availability. Select the pool whose template matches the requested sandbox profile. + +### Next Steps for the Supervisor + +1. **Idle mode**: A new supervisor mode that starts the process, opens a health check port, and waits for an activation signal before initializing SSH and policy enforcement. + +2. **Identity injection**: Accept sandbox identity from claim-time env vars or from a gRPC push from the gateway, instead of reading static env vars at startup. + +3. **Hot reconfiguration**: Support re-reading policy configuration without a full restart, enabling claim-time policy specialization. + +### Next Steps for the Gateway + +1. **Warm pod pre-registration**: Track warm pool pods as "pending" entries in the sandbox store. Promote to active on claim. + +2. **Claim-time identity push**: Send sandbox ID, auth token, and policy configuration to the supervisor via gRPC after the operator binds the claim. + +### Upstream Contributions + +1. **Request `--no-connect` flag** on `openshell sandbox create` for programmatic use cases. + +2. **Report the watch stream crash** ([#2211](https://github.com/NVIDIA/OpenShell/issues/2211)). The gateway crashes when non-gateway Sandbox resources exist in the namespace. + +### Estimated Effort + +**Phase 1: Claim-time supervisor startup (~3-4s target)** + +Points use a normalized scale (1 = trivial, 2 = small, 3 = medium, 5 = large) consistent across both the annotation-based and gRPC-push RFCs for cross-approach comparison. + +| Work item | Points | Dependencies | +|-----------|--------|--------------| +| Supervisor idle mode + annotation activation | 5 | None | +| K8s driver claim path + cold-start fallback | 3 | Supervisor idle mode | +| Gateway pool-aware CreateSandbox | 2 | None | +| Helm chart (template, pool, RBAC) | 2 | None | +| Gateway config | 1 | None | +| CLI `--no-connect` flag | 1 | None | +| E2E testing on OpenShift | 3 | All above | +| Documentation | 1 | None | +| **Total** | **~19** | | + +**Phase 2: Optimized claim-time startup (~2s target)** + +| Work item | Points | Dependencies | +|-----------|--------|--------------| +| Overlap claim binding with supervisor startup | 2 | Phase 1 | +| OPA policy pre-compilation in warm pods | 3 | Phase 1 | +| CLI polling reduction (SSE or tighter interval) | 2 | Independent | +| **Total** | **~7** | Phase 1 complete | + +**Phase 3: Pre-started supervisor (sub-1s target, optional)** + +| Work item | Points | Dependencies | +|-----------|--------|--------------| +| Supervisor deferred identity mode (gRPC push) | 5 | Phase 1 | +| Gateway warm pod pre-registration | 3 | Phase 1 | +| Hot reconfiguration (policy delta at claim time) | 5 | Deferred identity | +| **Total** | **~13** | Phase 1 complete | + +The phased approach lets the team ship warm pooling incrementally. Phase 1 alone delivers a 4-5x improvement over cold start. Phase 2 reaches the sub-2s target with targeted optimizations. Phase 3 is optional and only needed for workloads requiring sub-second sandbox provisioning. + +## Tooling + +This feasibility study was conducted entirely through an agent-driven workflow. The entire process, from cluster provisioning through experiment execution, data analysis, and document authoring, ran inside a single Claude Code session with specialized plugins and skills. + +### Agent Harness + +[Claude Code](https://claude.ai/code) served as the agent harness for this study, running Claude Opus 4.6 (1M context). Claude Code orchestrated the full workflow: provisioning infrastructure, writing and debugging measurement scripts, running experiments, analyzing results, and authoring both the results document and RFC. The extended context window was essential for maintaining coherence across the multi-hour session spanning cluster setup, iterative experiment execution, and document refinement. + +### Specification Workflow + +[cc-spex](https://github.com/rhuss/cc-spex) is a Spec-Kit extension and workflow for Claude Code that implements Spec-Driven Development (SDD). SDD structures work around specifications rather than direct implementation, and this study demonstrates that the approach works beyond code: cc-spex guided the full research lifecycle from brainstorm through executable test plans, iterative experiment execution, and results analysis. The feasibility study started as a brainstorm document, was refined into a feature spec with user stories and acceptance scenarios, decomposed into a task plan with per-experiment scripts, and iterated on as findings invalidated earlier assumptions (the env var injection path, the supervisor startup estimate, the pause-vs-sandbox image comparison). cc-spex also provided the deep review skill that ran 5 parallel review agents (correctness, architecture, security, production readiness, test quality) against the experiment scripts before execution, and the PR triage skill that handled bot review comments from CodeRabbit, Copilot, and Devin. + +### Cluster Provisioning + +The [cc-rosa](https://gitlab.cee.redhat.com/rhuss/cc-rosa-rhoai) plugin for Claude Code automates ROSA HCP cluster lifecycle management (create, delete, status, GPU machinepools, operator installation). It handles AWS SAML authentication, OCM login, and cluster provisioning with pre-configured infrastructure profiles. This plugin is internal to Red Hat and requires access to a Red Hat AWS account with ROSA HCP provisioning rights. The cluster provisioned for this study used the AAET (Agentic & AI Engineering Tools) profile. + +### Prose Quality + +The [cc-prose](https://github.com/rhuss/cc-prose) plugin for Claude Code enforces human writing standards. Its humanizer skill is based on Wikipedia's [Signs of AI-generated content](https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI-generated_content) guide, which catalogs detectable patterns in AI-produced text (inflated symbolism, promotional language, AI vocabulary, em dash overuse, rule of three, and others). cc-prose also provides voice profiles for consistent tone and a pre-validator that scans for AI vocabulary, stoplist words, style compliance, and flow quality. The results document was validated through `/prose:check` before submission, scoring 88/100 with zero critical or high-severity AI pattern detections. + +### Other Tools + +| Tool | Purpose | +|------|---------| +| [OpenShell](https://github.com/NVIDIA/OpenShell) | The sandbox platform under study. CLI used for cold-start measurements. | +| [Agent Sandbox Operator](https://agent-sandbox.sigs.k8s.io/) | Red Hat build v0.9.0. Provides SandboxWarmPool CRD. | +| [OpenShell OpenShift Deploy](https://github.com/2000krysztof/Openshell-Openshift-Deploy) | Helm-based deployment wrapper for OpenShell on OpenShift. | +| `kubectl` / `oc` | Kubernetes API interaction, manifest application, event collection. | +| `jq` / `yq` | JSON and YAML processing for data extraction. | +| `gdate` (GNU coreutils) | Nanosecond-precision timestamps on macOS. | +| `bash` | All measurement scripts are POSIX-compatible shell scripts. | +| [CodeRabbit](https://coderabbit.ai/) | Automated code review (CLI version for local review). | diff --git a/experiments/lib/common.sh b/experiments/lib/common.sh new file mode 100755 index 0000000000..14e35eb777 --- /dev/null +++ b/experiments/lib/common.sh @@ -0,0 +1,258 @@ +#!/usr/bin/env bash +# Shared measurement functions for warm pool experiments. +# Source this file from experiment scripts: source "$(dirname "$0")/../lib/common.sh" + +set -euo pipefail + +EXPERIMENT_ID="${EXPERIMENT_ID:-unknown}" +RESULTS_DIR="${RESULTS_DIR:-experiments/results}" + +CLEANUP_RESOURCES=() +register_cleanup() { CLEANUP_RESOURCES+=("$1"); } +deregister_cleanup() { CLEANUP_RESOURCES=("${CLEANUP_RESOURCES[@]/$1}"); } +_cleanup_on_exit() { + for res in "${CLEANUP_RESOURCES[@]}"; do + [[ -z "$res" ]] && continue + log "Cleaning up: $res" + kubectl delete $res --ignore-not-found --wait=false 2>/dev/null || true + done +} +trap _cleanup_on_exit EXIT + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" +} + +ensure_results_dir() { + mkdir -p "$RESULTS_DIR" +} + +capture_timestamp() { + if command -v gdate &>/dev/null; then + gdate +%s%N + elif date +%s%N | grep -qv N; then + date +%s%N + else + # Fallback: seconds with 000000000 appended (no nanosecond support) + echo "$(date +%s)000000000" + fi +} + +write_csv_header() { + local file="$1" + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,init_ms,supervisor_ms,ssh_ms" > "$file" +} + +write_csv_row() { + local file="$1" + local run="$2" + local config="$3" + local create_ts="$4" + local ready_ts="$5" + local delta_ms="$6" + local pod="$7" + local row_status="$8" + local scheduled_ms="${9:-}" + local pulled_ms="${10:-}" + local init_ms="${11:-}" + local supervisor_ms="${12:-}" + local ssh_ms="${13:-}" + echo "${run},${EXPERIMENT_ID},${config},${create_ts},${ready_ts},${delta_ms},${pod},${row_status},${scheduled_ms},${pulled_ms},${init_ms},${supervisor_ms},${ssh_ms}" >> "$file" +} + +_iso_to_epoch() { + local ts="$1" + if [[ -z "$ts" || "$ts" == "null" ]]; then + echo "" + return + fi + if command -v gdate &>/dev/null; then + gdate -d "$ts" +%s 2>/dev/null || echo "" + else + date -d "$ts" +%s 2>/dev/null || echo "" + fi +} + +_event_ts() { + local events_json="$1" + local reason="$2" + local pick="${3:-first}" + local selector=".reason == \"$reason\"" + if [[ "$reason" == *"|"* ]]; then + local r1="${reason%%|*}" + local r2="${reason#*|}" + selector=".reason == \"$r1\" or .reason == \"$r2\"" + fi + if [[ "$pick" == "last" ]]; then + echo "$events_json" | jq -r "[.items[] | select($selector)] | last | (.lastTimestamp // .eventTime // null)" 2>/dev/null + else + echo "$events_json" | jq -r "[.items[] | select($selector)] | first | (.lastTimestamp // .eventTime // null)" 2>/dev/null + fi +} + +extract_phase_deltas() { + local events_json="$1" + local create_ts="$2" + local create_s=$((create_ts / 1000000000)) + + local sched_ts pull_ts init_ts start_ts + sched_ts=$(_event_ts "$events_json" "Scheduled" "first") + pull_ts=$(_event_ts "$events_json" "Pulled|Pulling" "last") + init_ts=$(_event_ts "$events_json" "Created" "first") + start_ts=$(_event_ts "$events_json" "Started" "last") + + local sched_ms="" pull_ms="" init_ms="" start_ms="" + + local epoch + epoch=$(_iso_to_epoch "$sched_ts") + [[ -n "$epoch" ]] && sched_ms=$(( (epoch - create_s) * 1000 )) + + epoch=$(_iso_to_epoch "$pull_ts") + [[ -n "$epoch" ]] && pull_ms=$(( (epoch - create_s) * 1000 )) + + epoch=$(_iso_to_epoch "$init_ts") + [[ -n "$epoch" ]] && init_ms=$(( (epoch - create_s) * 1000 )) + + epoch=$(_iso_to_epoch "$start_ts") + [[ -n "$epoch" ]] && start_ms=$(( (epoch - create_s) * 1000 )) + + echo "${sched_ms},${pull_ms},${init_ms},${start_ms}," +} + +collect_pod_events() { + local pod="$1" + local ns="${2:-${NAMESPACE:-openshell}}" + kubectl get events \ + --namespace="$ns" \ + --field-selector="involvedObject.name=$pod,involvedObject.kind=Pod" \ + --sort-by='.lastTimestamp' \ + -o json 2>/dev/null || echo '{"items":[]}' +} + +detect_adoption() { + local pod_name="$1" + local ns="${2:-${NAMESPACE:-openshell}}" + local claim_ts="${3:-}" + if [[ -z "$pod_name" ]]; then + echo "unknown" + return + fi + + local events_json + events_json=$(collect_pod_events "$pod_name" "$ns") + + if [[ -n "$claim_ts" ]]; then + local claim_s=$(( claim_ts / 1000000000 )) + local sched_ts pull_ts + sched_ts=$(_event_ts "$events_json" "Scheduled" "first") + pull_ts=$(_event_ts "$events_json" "Pulled|Pulling" "last") + + local sched_epoch pull_epoch + sched_epoch=$(_iso_to_epoch "$sched_ts") + pull_epoch=$(_iso_to_epoch "$pull_ts") + + local post_claim=false + [[ -n "$sched_epoch" ]] && (( sched_epoch >= claim_s )) && post_claim=true + [[ -n "$pull_epoch" ]] && (( pull_epoch >= claim_s )) && post_claim=true + + if [[ "$post_claim" == "true" ]]; then + echo "cold-fallback" + else + echo "warm-adopted" + fi + else + local scheduled_count + scheduled_count=$(echo "$events_json" | \ + jq '[.items[] | select(.reason == "Scheduled")] | length' 2>/dev/null || echo "0") + local pulled_count + pulled_count=$(echo "$events_json" | \ + jq '[.items[] | select(.reason == "Pulling" or .reason == "Pulled")] | length' 2>/dev/null || echo "0") + + if (( scheduled_count == 0 && pulled_count == 0 )); then + echo "warm-adopted" + else + echo "cold-fallback" + fi + fi +} + +wait_pool_replenished() { + local pool_name="${1:-$WARM_POOL_NAME}" + local ns="${2:-${NAMESPACE:-openshell}}" + local min_ready="${3:-1}" + local timeout="${4:-120}" + local elapsed=0 + + while (( elapsed < timeout )); do + local ready + ready=$(kubectl get sandboxwarmpool "$pool_name" -n "$ns" \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + if [[ -n "$ready" ]] && (( ready >= min_ready )); then + return 0 + fi + if (( elapsed % 10 == 0 && elapsed > 0 )); then + log "Waiting for pool replenishment... (${ready:-0} ready, need ${min_ready}, ${elapsed}s/${timeout}s)" + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + + log "WARN: Pool did not replenish to ${min_ready} replicas within ${timeout}s" + return 1 +} + +compute_stats() { + local csv_file="$1" + if [[ ! -f "$csv_file" ]]; then + log "ERROR: CSV file not found: $csv_file" + return 1 + fi + + awk -F',' ' + NR == 1 { next } + $8 != "ok" { skipped++; next } + $6 == "" { skipped++; next } + { + v = $6 + 0 + n++ + vals[n] = v + sum += v + if (n == 1 || v < min) min = v + if (n == 1 || v > max) max = v + } + END { + if (n == 0) { + print "No data rows found." + exit 1 + } + + # Sort values (insertion sort) + for (i = 2; i <= n; i++) { + key = vals[i] + j = i - 1 + while (j >= 1 && vals[j] > key) { + vals[j+1] = vals[j] + j-- + } + vals[j+1] = key + } + + p50_idx = int(n * 0.5 + 0.5) + p90_idx = int(n * 0.9 + 0.5) + if (p50_idx < 1) p50_idx = 1 + if (p90_idx < 1) p90_idx = 1 + if (p50_idx > n) p50_idx = n + if (p90_idx > n) p90_idx = n + + mean = sum / n + printf "\n--- Statistics (%d samples) ---\n", n + printf " min: %10.1f ms\n", min + printf " max: %10.1f ms\n", max + printf " mean: %10.1f ms\n", mean + printf " p50: %10.1f ms\n", vals[p50_idx] + printf " p90: %10.1f ms\n", vals[p90_idx] + printf "-------------------------------\n" + if (skipped > 0) printf " (%d non-ok rows excluded)\n", skipped + } + ' "$csv_file" +} diff --git a/experiments/lib/wait-ready.sh b/experiments/lib/wait-ready.sh new file mode 100755 index 0000000000..8433e536af --- /dev/null +++ b/experiments/lib/wait-ready.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +# Pod and resource readiness wait functions. +# Source this file from experiment scripts: source "$(dirname "$0")/../lib/wait-ready.sh" + +set -euo pipefail + +wait_for_ready() { + local resource="$1" + local timeout_s="${2:-120}" + kubectl wait --for=condition=Ready "$resource" --timeout="${timeout_s}s" 2>/dev/null +} + +wait_for_pod_ready() { + local pod="$1" + local ns="${2:-default}" + local timeout_s="${3:-120}" + local elapsed=0 + + while [[ $elapsed -lt $timeout_s ]]; do + local status + status=$(kubectl get pod "$pod" \ + --namespace="$ns" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + local phase + phase=$(kubectl get pod "$pod" \ + --namespace="$ns" \ + -o jsonpath='{.status.phase}' \ + 2>/dev/null) || true + + if [[ "$phase" == "Failed" || "$phase" == "Succeeded" ]]; then + echo "Pod $pod terminated with phase: $phase" >&2 + return 1 + fi + + if (( elapsed % 10 == 0 && elapsed > 0 )); then + echo "Waiting for pod $pod to be Ready... (${elapsed}s/${timeout_s}s, phase: ${phase:-unknown})" >&2 + fi + + sleep 1 + elapsed=$((elapsed + 1)) + done + + echo "Timeout: pod $pod not Ready after ${timeout_s}s" >&2 + return 1 +} + +wait_for_sandbox_ready() { + local name="$1" + local ns="${2:-default}" + local timeout_s="${3:-120}" + local elapsed=0 + + while [[ $elapsed -lt $timeout_s ]]; do + local status + status=$(kubectl get sandbox "$name" \ + --namespace="$ns" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + if (( elapsed % 10 == 0 && elapsed > 0 )); then + echo "Waiting for sandbox $name to be Ready... (${elapsed}s/${timeout_s}s)" >&2 + fi + + sleep 1 + elapsed=$((elapsed + 1)) + done + + echo "Timeout: sandbox $name not Ready after ${timeout_s}s" >&2 + return 1 +} diff --git a/experiments/manifests/image-prepull-daemonset.yaml b/experiments/manifests/image-prepull-daemonset.yaml new file mode 100644 index 0000000000..504f6177b2 --- /dev/null +++ b/experiments/manifests/image-prepull-daemonset.yaml @@ -0,0 +1,34 @@ +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: image-prepull + namespace: openshell + labels: + app.kubernetes.io/name: image-prepull + app.kubernetes.io/part-of: openshell-warm-pool +spec: + selector: + matchLabels: + app.kubernetes.io/name: image-prepull + template: + metadata: + labels: + app.kubernetes.io/name: image-prepull + spec: + initContainers: + - name: pull-sandbox + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + imagePullPolicy: IfNotPresent + command: ["/bin/true"] + containers: + - name: pause + image: registry.k8s.io/pause:3.10 + resources: + requests: + cpu: "1m" + memory: "4Mi" + limits: + cpu: "1m" + memory: "4Mi" + tolerations: + - operator: Exists diff --git a/experiments/manifests/readiness-gate-pod.yaml b/experiments/manifests/readiness-gate-pod.yaml new file mode 100644 index 0000000000..1c4e6097bd --- /dev/null +++ b/experiments/manifests/readiness-gate-pod.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Pod +metadata: + name: readiness-gate-test-PLACEHOLDER +spec: + readinessGates: + - conditionType: sandbox.openshell.io/claimed + containers: + - name: sandbox + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + command: ["sleep", "infinity"] + resources: + requests: + cpu: "100m" + memory: "64Mi" diff --git a/experiments/manifests/sandbox-claim.yaml b/experiments/manifests/sandbox-claim.yaml new file mode 100644 index 0000000000..8f612a63a4 --- /dev/null +++ b/experiments/manifests/sandbox-claim.yaml @@ -0,0 +1,17 @@ +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxClaim +metadata: + name: claim-PLACEHOLDER +spec: + warmPoolRef: + name: openshell-warm-pool + +# Variant with env var injection (requires envVarsInjectionPolicy: Allowed on template): +# spec: +# warmPoolRef: +# name: openshell-warm-pool +# env: +# - name: AGENT_ID +# value: "agent-001" +# - name: SESSION_TOKEN +# value: "tok-xxx" diff --git a/experiments/manifests/sandbox-template.yaml b/experiments/manifests/sandbox-template.yaml new file mode 100644 index 0000000000..4888965dd7 --- /dev/null +++ b/experiments/manifests/sandbox-template.yaml @@ -0,0 +1,98 @@ +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxTemplate +metadata: + name: openshell-warm +spec: + envVarsInjectionPolicy: Allowed + podTemplate: + spec: + automountServiceAccountToken: false + dnsPolicy: ClusterFirst + serviceAccountName: openshell-sandbox + securityContext: + fsGroup: 1000 + initContainers: + - name: workspace-init + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + command: + - sh + - -c + - | + if [ ! -f /workspace-pvc/.workspace-initialized ]; then + if [ -d /sandbox ]; then + tar -C /sandbox -cf - . | tar -C /workspace-pvc -xpf - + fi + touch /workspace-pvc/.workspace-initialized + fi + securityContext: + runAsUser: 0 + volumeMounts: + - mountPath: /workspace-pvc + name: workspace + containers: + - name: agent + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + command: ["sleep", "infinity"] + env: + - name: OPENSHELL_ENDPOINT + value: "https://openshell.openshell.svc.cluster.local:8080" + - name: OPENSHELL_SANDBOX_COMMAND + value: "sleep infinity" + - name: OPENSHELL_TELEMETRY_ENABLED + value: "true" + - name: OPENSHELL_SSH_SOCKET_PATH + value: "/run/openshell/ssh.sock" + - name: OPENSHELL_TLS_CA + value: "/etc/openshell-tls/client/ca.crt" + - name: OPENSHELL_TLS_CERT + value: "/etc/openshell-tls/client/tls.crt" + - name: OPENSHELL_TLS_KEY + value: "/etc/openshell-tls/client/tls.key" + - name: OPENSHELL_K8S_SA_TOKEN_FILE + value: "/var/run/secrets/openshell/token" + - name: OPENSHELL_SANDBOX_UID + value: "1000" + - name: OPENSHELL_SANDBOX_GID + value: "1000" + securityContext: + appArmorProfile: + type: Unconfined + capabilities: + add: ["SYS_ADMIN", "NET_ADMIN", "SYS_PTRACE", "SYSLOG"] + runAsUser: 0 + volumeMounts: + - mountPath: /etc/openshell-tls/client + name: openshell-client-tls + readOnly: true + - mountPath: /var/run/secrets/openshell + name: openshell-sa-token + readOnly: true + - mountPath: /opt/openshell/bin + name: openshell-supervisor-bin + readOnly: true + - mountPath: /sandbox + name: workspace + volumes: + - name: openshell-client-tls + secret: + defaultMode: 256 + secretName: openshell-client-tls + - name: openshell-sa-token + projected: + defaultMode: 256 + sources: + - serviceAccountToken: + audience: openshell-gateway + expirationSeconds: 3600 + path: token + - name: openshell-supervisor-bin + image: + reference: "ghcr.io/nvidia/openshell/supervisor:latest" + volumeClaimTemplates: + - metadata: + name: workspace + spec: + accessModes: ["ReadWriteOnce"] + resources: + requests: + storage: 2Gi diff --git a/experiments/manifests/sidecar-readiness.yaml b/experiments/manifests/sidecar-readiness.yaml new file mode 100644 index 0000000000..d096ffc607 --- /dev/null +++ b/experiments/manifests/sidecar-readiness.yaml @@ -0,0 +1,39 @@ +apiVersion: v1 +kind: Pod +metadata: + name: sidecar-readiness-test-PLACEHOLDER +spec: + initContainers: + - name: readiness-sidecar + image: registry.k8s.io/e2e-test-images/agnhost:2.53 + restartPolicy: Always + command: ["/agnhost"] + args: ["netexec", "--http-port=8080"] + ports: + - containerPort: 8080 + readinessProbe: + httpGet: + path: /healthz + port: 8080 + periodSeconds: 1 + volumeMounts: + - name: signal + mountPath: /tmp/signal + resources: + requests: + cpu: "10m" + memory: "16Mi" + containers: + - name: sandbox + image: ghcr.io/nvidia/openshell-community/sandboxes/base:latest + command: ["sleep", "infinity"] + volumeMounts: + - name: signal + mountPath: /tmp/signal + resources: + requests: + cpu: "100m" + memory: "64Mi" + volumes: + - name: signal + emptyDir: {} diff --git a/experiments/manifests/warm-pool.yaml b/experiments/manifests/warm-pool.yaml new file mode 100644 index 0000000000..d4491f10fe --- /dev/null +++ b/experiments/manifests/warm-pool.yaml @@ -0,0 +1,8 @@ +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxWarmPool +metadata: + name: openshell-warm-pool +spec: + sandboxTemplateRef: + name: openshell-warm + replicas: 5 diff --git a/experiments/measure-cold-start.sh b/experiments/measure-cold-start.sh new file mode 100755 index 0000000000..9f6ac449a4 --- /dev/null +++ b/experiments/measure-cold-start.sh @@ -0,0 +1,253 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="cold-start" +CONFIG="prepulled" +RUNS="" +NAMESPACE="${NAMESPACE:-openshell}" +SANDBOX_IMAGE="${SANDBOX_IMAGE:-ghcr.io/nvidia/openshell-community/sandboxes/base:latest}" + +usage() { + cat </dev/null || true + kubectl delete sandbox "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + ;; + vanilla) + kubectl delete sandbox "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + ;; + esac + # Wait for pod cleanup + local elapsed=0 + while (( elapsed < 30 )); do + if ! kubectl get sandbox "$name" -n "$NAMESPACE" &>/dev/null; then + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done +} + +get_sandbox_pod() { + local name="$1" + if kubectl get pod "$name" -n "$NAMESPACE" &>/dev/null; then + echo "$name" + else + kubectl get pods -n "$NAMESPACE" \ + -l "agents.x-k8s.io/sandbox-name-hash" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" + fi +} + +run_openshell_config() { + local config="$1" + local num_runs="$2" + local csv_file="${RESULTS_DIR}/cold-start-${config}.csv" + + log "Starting cold-start measurement: config=${config}, runs=${num_runs}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local sandbox_name="cold-${config}-run-${i}" + local status="ok" + + log "Run ${i}/${num_runs}: creating sandbox ${sandbox_name}" + + local create_ts + create_ts=$(capture_timestamp) + + if ! openshell sandbox create --name "$sandbox_name" --from base -- true 2>/dev/null; then + log "WARN: Failed to create sandbox ${sandbox_name}" + status="create-failed" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "$sandbox_name" "$status" + cleanup_sandbox "$sandbox_name" "sandbox" + continue + fi + + if wait_for_sandbox_ready "$sandbox_name" "$NAMESPACE" 120; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_sandbox_pod "$sandbox_name") + + local phase_deltas=",,,," + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-${config}-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + phase_deltas=$(extract_phase_deltas "$(cat "$events_file")" "$create_ts") + fi + + IFS=',' read -r sched_ms pull_ms init_ms sup_ms ssh_ms <<< "$phase_deltas" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "$ready_ts" "$delta_ms" "${pod_name:-unknown}" "$status" "$sched_ms" "$pull_ms" "$init_ms" "$sup_ms" "$ssh_ms" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown})" + else + status="timeout" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "$sandbox_name" "$status" + log "Run ${i}: TIMEOUT" + fi + + cleanup_sandbox "$sandbox_name" "sandbox" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_vanilla_config() { + local num_runs="$1" + local csv_file="${RESULTS_DIR}/cold-start-vanilla.csv" + + log "Starting cold-start measurement: config=vanilla, runs=${num_runs}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local sandbox_name="vanilla-run-${i}" + local status="ok" + + log "Run ${i}/${num_runs}: creating vanilla sandbox ${sandbox_name}" + + local create_ts + create_ts=$(capture_timestamp) + + if ! kubectl apply -n "$NAMESPACE" -f - < "$events_file" + phase_deltas=$(extract_phase_deltas "$(cat "$events_file")" "$create_ts") + fi + + IFS=',' read -r sched_ms pull_ms init_ms sup_ms ssh_ms <<< "$phase_deltas" + write_csv_row "$csv_file" "$i" "vanilla" "$create_ts" "$ready_ts" "$delta_ms" "${pod_name:-unknown}" "$status" "$sched_ms" "$pull_ms" "$init_ms" "$sup_ms" "$ssh_ms" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown})" + else + status="timeout" + write_csv_row "$csv_file" "$i" "vanilla" "$create_ts" "" "" "$sandbox_name" "$status" + log "Run ${i}: TIMEOUT" + fi + + cleanup_sandbox "$sandbox_name" "vanilla" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_config() { + local config="$1" + local num_runs="${RUNS:-$(default_runs_for_config "$config")}" + + case "$config" in + prepulled|noprepull) + run_openshell_config "$config" "$num_runs" + ;; + vanilla) + run_vanilla_config "$num_runs" + ;; + *) + log "ERROR: Unknown config: $config" + exit 1 + ;; + esac +} + +case "$CONFIG" in + all) + run_config "prepulled" + run_config "noprepull" + run_config "vanilla" + ;; + prepulled|noprepull|vanilla) + run_config "$CONFIG" + ;; + *) + log "ERROR: Invalid config: $CONFIG (must be prepulled, noprepull, vanilla, or all)" + exit 1 + ;; +esac + +log "Cold-start measurement complete." diff --git a/experiments/measure-combined.sh b/experiments/measure-combined.sh new file mode 100755 index 0000000000..4e07fdd251 --- /dev/null +++ b/experiments/measure-combined.sh @@ -0,0 +1,286 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="combined" +RUNS=10 +NAMESPACE="${NAMESPACE:-openshell}" +TEMPLATE_NAME="${TEMPLATE_NAME:-openshell-warm}" +WARM_POOL_NAME="${WARM_POOL_NAME:-openshell-warm-pool}" +READINESS_METHOD="probe-1s" +INJECT_ENV="true" + +usage() { + cat </dev/null || log "WARN: Could not patch template readiness probe" + ;; + probe-1s) + kubectl patch sandboxtemplate "$TEMPLATE_NAME" -n "$NAMESPACE" \ + --type merge -p '{ + "spec": { + "podTemplate": { + "spec": { + "containers": [{ + "name": "sandbox", + "readinessProbe": { + "tcpSocket": {"port": 2222}, + "initialDelaySeconds": 1, + "periodSeconds": 1 + } + }] + } + } + } + }' 2>/dev/null || log "WARN: Could not patch template readiness probe" + ;; + readiness-gate) + kubectl patch sandboxtemplate "$TEMPLATE_NAME" -n "$NAMESPACE" \ + --type merge -p '{ + "spec": { + "podTemplate": { + "spec": { + "readinessGates": [{"conditionType": "sandbox.openshell.io/claimed"}] + } + } + } + }' 2>/dev/null || log "WARN: Could not patch template readiness gates" + ;; + sidecar) + log "Sidecar readiness requires custom template with sidecar container." + log "Ensure your SandboxTemplate already includes the readiness sidecar." + ;; + esac +} + +patch_env_policy() { + if [[ "$INJECT_ENV" == "true" ]]; then + log "Enabling env var injection on template" + kubectl patch sandboxtemplate "$TEMPLATE_NAME" -n "$NAMESPACE" \ + --type merge \ + -p '{"spec":{"envVarsInjectionPolicy":"Allowed"}}' 2>/dev/null || \ + log "WARN: Could not patch env injection policy" + fi +} + +get_claim_pod() { + local claim_name="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "sandbox.agents.x-k8s.io/claim=$claim_name" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" +} + +wait_claim_ready() { + local name="$1" + local timeout="${2:-120}" + local elapsed=0 + while (( elapsed < timeout )); do + local status + status=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + local reason + reason=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.reason}{end}' \ + 2>/dev/null) || true + + if [[ "$reason" == "Failed" || "$reason" == "Error" ]]; then + return 1 + fi + + sleep 1 + elapsed=$((elapsed + 1)) + done + return 1 +} + +cleanup_claim() { + local name="$1" + kubectl delete sandboxclaim "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + local elapsed=0 + while (( elapsed < 30 )); do + if ! kubectl get sandboxclaim "$name" -n "$NAMESPACE" &>/dev/null; then + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + log "WARN: Claim $name still present after 30s cleanup timeout" + return 1 +} + +run_combined() { + local num_runs="$1" + local config_label="${READINESS_METHOD}" + if [[ "$INJECT_ENV" == "true" ]]; then + config_label="${config_label}+env" + fi + + local csv_file="${RESULTS_DIR}/combined.csv" + + log "Starting combined measurement: readiness=${READINESS_METHOD}, env=${INJECT_ENV}, runs=${num_runs}" + ensure_results_dir + + patch_template_for_readiness "$READINESS_METHOD" + patch_env_policy + + sleep 5 + log "Waiting for warm pool to stabilize after template patch..." + + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,adoption" > "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + if ! wait_pool_replenished "$WARM_POOL_NAME" "$NAMESPACE" 1 60; then + log "WARN: Pool not replenished before run ${i}, results may include cold-start fallback" + fi + + local claim_name="combined-run-${i}" + local status="ok" + local adoption="unknown" + + log "Run ${i}/${num_runs}: creating claim ${claim_name} (${config_label})" + + local create_ts + create_ts=$(capture_timestamp) + + local claim_yaml + if [[ "$INJECT_ENV" == "true" ]]; then + claim_yaml=$(cat </dev/null; then + log "WARN: Failed to create claim ${claim_name}" + status="create-failed" + echo "${i},${EXPERIMENT_ID},${config_label},${create_ts},,,,${status},${adoption}" >> "$csv_file" + cleanup_claim "$claim_name" + continue + fi + + if wait_claim_ready "$claim_name" 120; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_claim_pod "$claim_name") + + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-combined-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + adoption=$(detect_adoption "$pod_name" "$NAMESPACE" "$create_ts") + fi + + echo "${i},${EXPERIMENT_ID},${config_label},${create_ts},${ready_ts},${delta_ms},${pod_name:-unknown},${status},${adoption}" >> "$csv_file" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown}, adoption: ${adoption})" + else + status="timeout" + echo "${i},${EXPERIMENT_ID},${config_label},${create_ts},,,,${status},${adoption}" >> "$csv_file" + log "Run ${i}: TIMEOUT" + fi + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_combined "$RUNS" + +log "Combined measurement complete (${READINESS_METHOD}, env=${INJECT_ENV})." diff --git a/experiments/measure-env-injection.sh b/experiments/measure-env-injection.sh new file mode 100755 index 0000000000..e3fff07b17 --- /dev/null +++ b/experiments/measure-env-injection.sh @@ -0,0 +1,299 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="env-injection" +CONFIG="allowed" +RUNS=5 +NAMESPACE="${NAMESPACE:-openshell}" +TEMPLATE_NAME="${TEMPLATE_NAME:-openshell-warm}" +WARM_POOL_NAME="${WARM_POOL_NAME:-openshell-warm-pool}" + +usage() { + cat </dev/null || { + log "WARN: Could not patch template (may not exist yet or field unsupported)" + return 1 + } +} + +get_claim_pod() { + local claim_name="$1" + kubectl get pods -n "$NAMESPACE" \ + -l "sandbox.agents.x-k8s.io/claim=$claim_name" \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || echo "" +} + +wait_claim_ready() { + local name="$1" + local timeout="${2:-120}" + local elapsed=0 + while (( elapsed < timeout )); do + local status + status=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$status" == "True" ]]; then + return 0 + fi + + local reason + reason=$(kubectl get sandboxclaim "$name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.reason}{end}' \ + 2>/dev/null) || true + + if [[ "$reason" == "Failed" || "$reason" == "Error" ]]; then + log "Claim $name failed" + return 1 + fi + + sleep 1 + elapsed=$((elapsed + 1)) + done + + log "WARN: Claim $name did not become ready within ${timeout}s" + return 1 +} + +cleanup_claim() { + local name="$1" + kubectl delete sandboxclaim "$name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + local elapsed=0 + while (( elapsed < 30 )); do + if ! kubectl get sandboxclaim "$name" -n "$NAMESPACE" &>/dev/null; then + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + log "WARN: Claim $name still present after 30s cleanup timeout" + return 1 +} + +run_allowed() { + local num_runs="$1" + local csv_file="${RESULTS_DIR}/env-injection-allowed.csv" + + log "Starting env injection measurement: config=allowed, runs=${num_runs}" + ensure_results_dir + + patch_template_policy "Allowed" || true + + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,adoption" > "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + if ! wait_pool_replenished "$WARM_POOL_NAME" "$NAMESPACE" 1 60; then + log "WARN: Pool not replenished before run ${i}, results may include cold-start fallback" + fi + + local claim_name="envinj-allowed-run-${i}" + local status="ok" + local adoption="unknown" + + log "Run ${i}/${num_runs}: creating claim ${claim_name} with env vars" + + local create_ts + create_ts=$(capture_timestamp) + + if ! kubectl apply -n "$NAMESPACE" -f - <> "$csv_file" + cleanup_claim "$claim_name" + continue + fi + + if wait_claim_ready "$claim_name" 120; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_claim_pod "$claim_name") + + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-envinj-allowed-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + adoption=$(detect_adoption "$pod_name" "$NAMESPACE" "$create_ts") + fi + + echo "${i},${EXPERIMENT_ID},allowed,${create_ts},${ready_ts},${delta_ms},${pod_name:-unknown},${status},${adoption}" >> "$csv_file" + log "Run ${i}: ${delta_ms}ms (pod: ${pod_name:-unknown}, adoption: ${adoption})" + else + status="timeout" + echo "${i},${EXPERIMENT_ID},allowed,${create_ts},,,,${status},${adoption}" >> "$csv_file" + log "Run ${i}: TIMEOUT" + fi + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_disallowed() { + local num_runs="$1" + local csv_file="${RESULTS_DIR}/env-injection-disallowed.csv" + + log "Starting env injection measurement: config=disallowed, runs=${num_runs}" + ensure_results_dir + + patch_template_policy "Disallowed" || true + + echo "run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,behavior" > "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local claim_name="envinj-disallowed-run-${i}" + local behavior="unknown" + + log "Run ${i}/${num_runs}: creating claim ${claim_name} with env vars (policy=Disallowed)" + + local create_ts + create_ts=$(capture_timestamp) + + local apply_output + local apply_exit=0 + apply_output=$(kubectl apply -n "$NAMESPACE" -f - 2>&1 <> "$csv_file" + continue + fi + + sleep 3 + + local ready_reason + ready_reason=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.reason}{end}' \ + 2>/dev/null) || ready_reason="" + + local conditions + conditions=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.conditions[*].message}' 2>/dev/null) || conditions="" + + if [[ "$ready_reason" == "Failed" || "$ready_reason" == "Error" ]]; then + behavior="failed" + elif wait_claim_ready "$claim_name" 60; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_claim_pod "$claim_name") + local adoption + adoption=$(detect_adoption "${pod_name:-}" "$NAMESPACE" "$create_ts") + + if [[ "$adoption" == "cold-fallback" ]]; then + behavior="cold-fallback-with-env-stripped" + else + behavior="unexpected-warm-adopted" + fi + + echo "${i},${EXPERIMENT_ID},disallowed,${create_ts},${ready_ts},${delta_ms},${pod_name:-unknown},ok,${behavior}" >> "$csv_file" + log "Run ${i}: ${delta_ms}ms (behavior: ${behavior}, conditions: ${conditions})" + cleanup_claim "$claim_name" + continue + else + behavior="timeout" + fi + + log "Run ${i}: behavior=${behavior}, reason=${ready_reason}, conditions=${conditions}" + echo "${i},${EXPERIMENT_ID},disallowed,${create_ts},,,,${behavior},${behavior}" >> "$csv_file" + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + log "Review ${csv_file} for behavior observations under Disallowed policy." +} + +case "$CONFIG" in + allowed) + run_allowed "$RUNS" + ;; + disallowed) + run_disallowed "$RUNS" + ;; +esac + +log "Env injection measurement complete." diff --git a/experiments/measure-readiness-gates.sh b/experiments/measure-readiness-gates.sh new file mode 100755 index 0000000000..65d3f5fb6c --- /dev/null +++ b/experiments/measure-readiness-gates.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="readiness-gates" +RUNS=10 +NAMESPACE="${NAMESPACE:-openshell}" +MANIFEST="${SCRIPT_DIR}/manifests/readiness-gate-pod.yaml" +CSV_FILE="${RESULTS_DIR}/readiness-gates.csv" + +usage() { + cat </dev/null || echo "") + if [[ "$phase" == "Running" ]]; then + return 0 + fi + if [[ "$phase" == "Failed" || "$phase" == "Succeeded" ]]; then + log "Pod $pod terminated with phase: $phase" + return 1 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + log "Timeout: pod $pod not Running after ${timeout}s" + return 1 +} + +log "Starting readiness-gate measurement: runs=${RUNS}" +ensure_results_dir +write_csv_header "$CSV_FILE" + +for (( i=1; i<=RUNS; i++ )); do + POD="readiness-gate-run-${i}" + status="ok" + + log "Run ${i}/${RUNS}: creating pod ${POD}" + + sed "s/readiness-gate-test-PLACEHOLDER/${POD}/" "$MANIFEST" | \ + kubectl apply -n "$NAMESPACE" -f - + + if ! wait_pod_running "$POD" 120; then + status="not-running" + write_csv_row "$CSV_FILE" "$i" "readiness-gate" "" "" "" "$POD" "$status" + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + continue + fi + + patch_ts=$(capture_timestamp) + + kubectl patch pod "$POD" -n "$NAMESPACE" \ + --type=json --subresource=status \ + -p '[{"op":"add","path":"/status/conditions/-","value":{"type":"sandbox.openshell.io/claimed","status":"True","lastTransitionTime":"'"$(date -u +%Y-%m-%dT%H:%M:%SZ)"'"}}]' + + if wait_for_pod_ready "$POD" "$NAMESPACE" 60; then + ready_ts=$(capture_timestamp) + delta_ms=$(( (ready_ts - patch_ts) / 1000000 )) + + write_csv_row "$CSV_FILE" "$i" "readiness-gate" "$patch_ts" "$ready_ts" "$delta_ms" "$POD" "$status" + log "Run ${i}: ${delta_ms}ms" + else + status="timeout" + write_csv_row "$CSV_FILE" "$i" "readiness-gate" "$patch_ts" "" "" "$POD" "$status" + log "Run ${i}: TIMEOUT" + fi + + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + sleep 2 +done + +log "Results written to ${CSV_FILE}" +compute_stats "$CSV_FILE" +log "Readiness-gate measurement complete." diff --git a/experiments/measure-sidecar-readiness.sh b/experiments/measure-sidecar-readiness.sh new file mode 100755 index 0000000000..773f034228 --- /dev/null +++ b/experiments/measure-sidecar-readiness.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="sidecar-readiness" +RUNS=10 +NAMESPACE="${NAMESPACE:-openshell}" +MANIFEST="${SCRIPT_DIR}/manifests/sidecar-readiness.yaml" +CSV_FILE="${RESULTS_DIR}/sidecar-readiness.csv" + +usage() { + cat </dev/null || echo "") + if [[ -n "$running" ]]; then + return 0 + fi + sleep 1 + elapsed=$((elapsed + 1)) + done + log "Timeout: container $container in pod $pod not Running after ${timeout}s" + return 1 +} + +log "Starting sidecar-readiness measurement: runs=${RUNS}" +ensure_results_dir +write_csv_header "$CSV_FILE" + +for (( i=1; i<=RUNS; i++ )); do + POD="sidecar-readiness-run-${i}" + status="ok" + + log "Run ${i}/${RUNS}: creating pod ${POD}" + + sed "s/sidecar-readiness-test-PLACEHOLDER/${POD}/" "$MANIFEST" | \ + kubectl apply -n "$NAMESPACE" -f - + + if ! wait_container_running "$POD" "sandbox" 120; then + status="not-running" + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "" "" "" "$POD" "$status" + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + continue + fi + + signal_ts=$(capture_timestamp) + + if ! kubectl exec "$POD" -n "$NAMESPACE" -c sandbox -- touch /tmp/signal/ready 2>/dev/null; then + status="exec-failed" + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "" "" "" "$POD" "$status" + log "Run ${i}: kubectl exec failed" + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + continue + fi + + if wait_for_pod_ready "$POD" "$NAMESPACE" 60; then + ready_ts=$(capture_timestamp) + delta_ms=$(( (ready_ts - signal_ts) / 1000000 )) + + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "$signal_ts" "$ready_ts" "$delta_ms" "$POD" "$status" + log "Run ${i}: ${delta_ms}ms" + else + status="timeout" + write_csv_row "$CSV_FILE" "$i" "sidecar-readiness" "$signal_ts" "" "" "$POD" "$status" + log "Run ${i}: TIMEOUT" + fi + + kubectl delete pod "$POD" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true + sleep 2 +done + +log "Results written to ${CSV_FILE}" +compute_stats "$CSV_FILE" +log "Sidecar-readiness measurement complete." diff --git a/experiments/measure-warm-pool.sh b/experiments/measure-warm-pool.sh new file mode 100755 index 0000000000..fefe3e8abc --- /dev/null +++ b/experiments/measure-warm-pool.sh @@ -0,0 +1,400 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/lib/common.sh" +source "${SCRIPT_DIR}/lib/wait-ready.sh" + +EXPERIMENT_ID="warm-pool" +CONFIG="default" +RUNS="" +NAMESPACE="${NAMESPACE:-openshell}" +WARM_POOL_NAME="${WARM_POOL_NAME:-openshell-warm-pool}" +TEMPLATE_NAME="${TEMPLATE_NAME:-openshell-warm}" +CLAIM_TIMEOUT="${CLAIM_TIMEOUT:-60}" + +usage() { + cat </dev/null; then + log "ERROR: SandboxWarmPool '$WARM_POOL_NAME' not found in namespace '$NAMESPACE'" + log "Create it first: kubectl apply -f ${SCRIPT_DIR}/manifests/warm-pool.yaml" + exit 1 + fi + + local ready_replicas + ready_replicas=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + local desired_replicas + desired_replicas=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "0") + + if [[ "$ready_replicas" == "0" || -z "$ready_replicas" ]]; then + log "ERROR: SandboxWarmPool has no ready replicas (desired: ${desired_replicas})" + log "Wait for pool provisioning before running measurements" + exit 1 + fi + + log "Pool status: ${ready_replicas}/${desired_replicas} replicas ready" +} + + +get_claimed_sandbox_name() { + local claim_name="$1" + kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.sandbox.name}' 2>/dev/null || echo "" +} + +get_sandbox_pod_name() { + local sandbox_name="$1" + if kubectl get pod "$sandbox_name" -n "$NAMESPACE" &>/dev/null; then + echo "$sandbox_name" + else + echo "" + fi +} + +wait_claim_bound() { + local claim_name="$1" + local timeout="${2:-$CLAIM_TIMEOUT}" + local deadline=$(( $(date +%s) + timeout )) + + while (( $(date +%s) < deadline )); do + local ready_status + ready_status=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status}{end}' \ + 2>/dev/null) || true + + if [[ "$ready_status" == "True" ]]; then + return 0 + fi + + local sandbox_name + sandbox_name=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.sandbox.name}' 2>/dev/null) || true + if [[ -n "$sandbox_name" ]]; then + return 0 + fi + + sleep 1 + done + + log "WARN: Claim $claim_name not bound after ${timeout}s" + return 1 +} + +wait_claimed_sandbox_ready() { + local claim_name="$1" + local timeout="${2:-$CLAIM_TIMEOUT}" + + local sandbox_name + sandbox_name=$(get_claimed_sandbox_name "$claim_name") + if [[ -z "$sandbox_name" ]]; then + log "WARN: No sandbox assigned to claim $claim_name" + return 1 + fi + + wait_for_sandbox_ready "$sandbox_name" "$NAMESPACE" "$timeout" +} + +cleanup_claim() { + local claim_name="$1" + kubectl delete sandboxclaim "$claim_name" -n "$NAMESPACE" --ignore-not-found --wait=false 2>/dev/null || true +} + +create_claim_yaml() { + local claim_name="$1" + cat </dev/null || echo "10" +} + +run_single_config() { + local config="$1" + local num_runs="$2" + local csv_file="${RESULTS_DIR}/warm-pool-${config}.csv" + + log "Starting warm-pool measurement: config=${config}, runs=${num_runs}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( i=1; i<=num_runs; i++ )); do + local claim_name="wp-${config}-run-${i}" + local status="ok" + + # Wait for at least 1 replica before claiming + if ! wait_pool_replenished "$WARM_POOL_NAME" "$NAMESPACE" 1 60; then + log "WARN: Pool exhausted, recording run ${i} as pool-exhausted" + local ts + ts=$(capture_timestamp) + write_csv_row "$csv_file" "$i" "$config" "$ts" "" "" "" "pool-exhausted" + continue + fi + + log "Run ${i}/${num_runs}: creating claim ${claim_name}" + + local create_ts + create_ts=$(capture_timestamp) + + if ! create_claim_yaml "$claim_name" | kubectl apply -f - 2>/dev/null; then + log "WARN: Failed to create claim ${claim_name}" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "" "create-failed" + cleanup_claim "$claim_name" + continue + fi + + # Wait for claim to bind and sandbox to be ready + if wait_claim_bound "$claim_name" "$CLAIM_TIMEOUT"; then + local sandbox_name + sandbox_name=$(get_claimed_sandbox_name "$claim_name") + + if [[ -n "$sandbox_name" ]] && wait_for_sandbox_ready "$sandbox_name" "$NAMESPACE" "$CLAIM_TIMEOUT"; then + local ready_ts + ready_ts=$(capture_timestamp) + local delta_ms + delta_ms=$(( (ready_ts - create_ts) / 1000000 )) + + local pod_name + pod_name=$(get_sandbox_pod_name "$sandbox_name") + + if [[ -n "$pod_name" ]]; then + local events_file="${RESULTS_DIR}/events-wp-${config}-run-${i}.json" + collect_pod_events "$pod_name" "$NAMESPACE" > "$events_file" + fi + + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "$ready_ts" "$delta_ms" "${pod_name:-unknown}" "$status" + log "Run ${i}: ${delta_ms}ms (sandbox: ${sandbox_name}, pod: ${pod_name:-unknown})" + else + status="sandbox-not-ready" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "${sandbox_name:-unknown}" "$status" + log "Run ${i}: sandbox not ready (${sandbox_name:-unknown})" + fi + else + status="bind-timeout" + write_csv_row "$csv_file" "$i" "$config" "$create_ts" "" "" "" "$status" + log "Run ${i}: claim bind timeout" + fi + + cleanup_claim "$claim_name" + sleep 2 + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_burst_config() { + local num_runs="$1" + local burst_size=5 + local csv_file="${RESULTS_DIR}/warm-pool-burst.csv" + + log "Starting warm-pool burst measurement: runs=${num_runs}, burst_size=${burst_size}" + ensure_results_dir + write_csv_header "$csv_file" + + for (( round=1; round<=num_runs; round++ )); do + log "Burst round ${round}/${num_runs}: submitting ${burst_size} simultaneous claims" + + # Wait for pool to have enough replicas + if ! wait_pool_replenished "$WARM_POOL_NAME" "$NAMESPACE" "$burst_size" 120; then + log "WARN: Pool has fewer than ${burst_size} ready replicas for round ${round}" + fi + + local pool_ready_before + pool_ready_before=$(kubectl get sandboxwarmpool "$WARM_POOL_NAME" -n "$NAMESPACE" \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + log "Pool state before burst: ${pool_ready_before} ready" + + # Create all claims simultaneously + local create_ts + create_ts=$(capture_timestamp) + + declare -a claim_names=() + for (( j=1; j<=burst_size; j++ )); do + local claim_name="wp-burst-r${round}-c${j}" + claim_names+=("$claim_name") + create_claim_yaml "$claim_name" | kubectl apply -f - 2>/dev/null & + done + wait # Wait for all kubectl apply commands + + # Wait for each claim and record individual times + for (( j=0; j/dev/null || echo "unknown") + local claim_sandbox + claim_sandbox=$(kubectl get sandboxclaim "$claim_name" -n "$NAMESPACE" \ + -o jsonpath='{.status.sandbox.name}' 2>/dev/null || echo "") + if [[ -z "$claim_sandbox" ]]; then + status="pool-exhausted" + else + status="bind-timeout" + fi + write_csv_row "$csv_file" "$run_label" "burst" "$create_ts" "" "" "" "$status" + log " Claim ${claim_name}: ${status} (reason: ${claim_ready})" + fi + done + + # Clean up all claims from this round + for claim_name in "${claim_names[@]}"; do + cleanup_claim "$claim_name" + done + + # Wait for pool to replenish before next round + if (( round < num_runs )); then + log "Waiting for pool replenishment before next burst round..." + wait_pool_replenished "$WARM_POOL_NAME" "$NAMESPACE" "$burst_size" 180 || true + fi + + unset claim_names + done + + log "Results written to ${csv_file}" + compute_stats "$csv_file" +} + +run_config() { + local config="$1" + local num_runs="${RUNS:-$(default_runs_for_config "$config")}" + + case "$config" in + default) + run_single_config "default" "$num_runs" + ;; + aggressive) + local original_period + original_period=$(save_original_probe_period) + log "Saving original readinessProbe periodSeconds: ${original_period}" + + patch_readiness_probe 1 + trap 'patch_readiness_probe '"$original_period"'; _cleanup_on_exit' EXIT + trap 'patch_readiness_probe '"$original_period"'; _cleanup_on_exit; exit 130' INT TERM + + log "Waiting for pool to stabilize with aggressive probes..." + sleep 10 + wait_pool_replenished "$WARM_POOL_NAME" "$NAMESPACE" 1 120 || true + + run_single_config "aggressive" "$num_runs" + + log "Restoring original readinessProbe periodSeconds: ${original_period}" + patch_readiness_probe "$original_period" + trap _cleanup_on_exit EXIT + trap - INT TERM + ;; + burst) + run_burst_config "$num_runs" + ;; + *) + log "ERROR: Unknown config: $config" + exit 1 + ;; + esac +} + +# Main +check_prerequisites + +case "$CONFIG" in + all) + run_config "default" + run_config "aggressive" + run_config "burst" + ;; + default|aggressive|burst) + run_config "$CONFIG" + ;; + *) + log "ERROR: Invalid config: $CONFIG (must be default, aggressive, burst, or all)" + exit 1 + ;; +esac + +log "Warm-pool measurement complete." diff --git a/experiments/results/.gitignore b/experiments/results/.gitignore new file mode 100644 index 0000000000..9d5aa38cf8 --- /dev/null +++ b/experiments/results/.gitignore @@ -0,0 +1 @@ +# Track everything, including raw event JSON dumps for reproducibility diff --git a/experiments/results/2026-07-09-rosa-hcp-422/cold-start-noprepull.csv b/experiments/results/2026-07-09-rosa-hcp-422/cold-start-noprepull.csv new file mode 100644 index 0000000000..d7730e8937 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/cold-start-noprepull.csv @@ -0,0 +1,6 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,cold-start,noprepull,1783616392580032000,1783616430540589000,37960,unknown,ok,,,,, +2,cold-start,noprepull,1783616433972325000,1783616462593678000,28621,unknown,ok,,,,, +3,cold-start,noprepull,1783616466039834000,1783616478556702000,12516,unknown,ok,,,,, +4,cold-start,noprepull,1783616481977386000,1783616498498773000,16521,unknown,ok,,,,, +5,cold-start,noprepull,1783616501936618000,1783616518751389000,16814,unknown,ok,,,,, diff --git a/experiments/results/2026-07-09-rosa-hcp-422/cold-start-prepulled.csv b/experiments/results/2026-07-09-rosa-hcp-422/cold-start-prepulled.csv new file mode 100644 index 0000000000..708eb3a2b0 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/cold-start-prepulled.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,cold-start,prepulled,1783616150605823000,1783616187704642000,37098,unknown,ok,,,,, +2,cold-start,prepulled,1783616191154194000,1783616202492146000,11337,unknown,ok,,,,, +3,cold-start,prepulled,1783616205952992000,1783616221576068000,15623,unknown,ok,,,,, +4,cold-start,prepulled,1783616224984140000,1783616242597273000,17613,unknown,ok,,,,, +5,cold-start,prepulled,1783616246003680000,1783616262740392000,16736,unknown,ok,,,,, +6,cold-start,prepulled,1783616266147183000,1783616282825192000,16678,unknown,ok,,,,, +7,cold-start,prepulled,1783616286248671000,1783616302923812000,16675,unknown,ok,,,,, +8,cold-start,prepulled,1783616306354614000,1783616323050556000,16695,unknown,ok,,,,, +9,cold-start,prepulled,1783616326516505000,1783616344108164000,17591,unknown,ok,,,,, +10,cold-start,prepulled,1783616347541539000,1783616364112906000,16571,unknown,ok,,,,, diff --git a/experiments/results/2026-07-09-rosa-hcp-422/cold-start-vanilla.csv b/experiments/results/2026-07-09-rosa-hcp-422/cold-start-vanilla.csv new file mode 100644 index 0000000000..46e0e0b5e1 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/cold-start-vanilla.csv @@ -0,0 +1,17 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,cold-start,vanilla,1783617446006907000,1783617448803769000,2796,unknown,ok,,,,, +2,cold-start,vanilla,1783617452203498000,1783617454987662000,2784,unknown,ok,,,,, +3,cold-start,vanilla,1783617458389070000,1783617461158033000,2768,unknown,ok,,,,, +4,cold-start,vanilla,1783617464570137000,1783617467385803000,2815,unknown,ok,,,,, +5,cold-start,vanilla,1783617306690564000,1783617473516051000,166825,unknown,ok,,,,, +5,cold-start,vanilla,1783617470831135000,1783617473634198000,2803,unknown,ok,,,,, +6,cold-start,vanilla,1783617477040230000,,,vanilla-run-6,create-failed,,,,, +7,cold-start,vanilla,1783617478968604000,1783617481748488000,2779,unknown,ok,,,,, +8,cold-start,vanilla,1783617485213680000,1783617487988948000,2775,unknown,ok,,,,, +9,cold-start,vanilla,1783617491407715000,1783617494178541000,2770,unknown,ok,,,,, +10,cold-start,vanilla,1783617497583530000,1783617500379199000,2795,unknown,ok,,,,, +6,cold-start,vanilla,1783617476935510000,,,vanilla-run-6,timeout,,,,, +7,cold-start,vanilla,1783617668541518000,,,vanilla-run-7,timeout,,,,, +8,cold-start,vanilla,1783617844241742000,,,vanilla-run-8,timeout,,,,, +9,cold-start,vanilla,1783618021313839000,,,vanilla-run-9,timeout,,,,, +10,cold-start,vanilla,1783618197686499000,,,vanilla-run-10,timeout,,,,, diff --git a/experiments/results/2026-07-09-rosa-hcp-422/env-injection-allowed.csv b/experiments/results/2026-07-09-rosa-hcp-422/env-injection-allowed.csv new file mode 100644 index 0000000000..7c603cf5d2 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/env-injection-allowed.csv @@ -0,0 +1,6 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,adoption +1,env-injection,allowed,1783659298912519000,1783659313854067000,14941,unknown,ok,unknown +2,env-injection,allowed,1783659317825075000,1783659332837995000,15012,unknown,ok,unknown +3,env-injection,allowed,1783659336836643000,1783659353680868000,16844,unknown,ok,unknown +4,env-injection,allowed,1783659357672559000,1783659374476321000,16803,unknown,ok,unknown +5,env-injection,allowed,1783659378433728000,1783659393605611000,15171,unknown,ok,unknown diff --git a/experiments/results/2026-07-09-rosa-hcp-422/env-injection-disallowed.csv b/experiments/results/2026-07-09-rosa-hcp-422/env-injection-disallowed.csv new file mode 100644 index 0000000000..a7a39705e0 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/env-injection-disallowed.csv @@ -0,0 +1,4 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,behavior +1,env-injection,disallowed,1783620298476976000,,,,timeout,timeout +2,env-injection,disallowed,1783620419576074000,,,,timeout,timeout +3,env-injection,disallowed,1783620540710291000,,,,timeout,timeout diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-1.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-1.json new file mode 100644 index 0000000000..f40626c92f --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-1.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:43:17.992161Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147522", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-fg426 to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:17Z", + "name": "openshell-warm-pool-fg426.18c0d40b6de4fcd7", + "namespace": "openshell", + "resourceVersion": "147605", + "uid": "d4f9777f-7800-4c25-84eb-28a86f702626" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:22Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-48df1c5f-d314-4a4d-9de9-1c27954a50ad\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:22Z", + "name": "openshell-warm-pool-fg426.18c0d40c732fc152", + "namespace": "openshell", + "resourceVersion": "147665", + "uid": "79c11634-a739-42ce-8b32-4e6cf82bd2bb" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-fg426.18c0d40cffe6cecf", + "namespace": "openshell", + "resourceVersion": "147689", + "uid": "766f5ee8-8ff1-44c4-a136-0d1baff9d18e" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147608", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Add eth0 [10.131.0.130/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-fg426.18c0d40cede70d0b", + "namespace": "openshell", + "resourceVersion": "147678", + "uid": "97fa108d-50c4-420a-84cb-3e5fd08cf8a6" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 260ms (260ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-fg426.18c0d40cffdf3488", + "namespace": "openshell", + "resourceVersion": "147688", + "uid": "8433784e-625b-49c1-9d30-c9bed5122013" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-fg426.18c0d40d18d06d30", + "namespace": "openshell", + "resourceVersion": "147706", + "uid": "4ce6621f-3335-44fb-9180-f10926014e9d" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 283ms (283ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-fg426.18c0d40d10d40baa", + "namespace": "openshell", + "resourceVersion": "147697", + "uid": "0551e719-69ac-4b4a-b8d1-9b8cc7c39ed5" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-fg426.18c0d40d182fb89b", + "namespace": "openshell", + "resourceVersion": "147702", + "uid": "e3bf53fa-3b36-40f8-bbcd-7a267428f097" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 292ms (292ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-fg426.18c0d40d64ee6ec9", + "namespace": "openshell", + "resourceVersion": "147727", + "uid": "422115dc-02c3-4af0-b1f1-e26d5e0a7161" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-fg426.18c0d40d64f5b401", + "namespace": "openshell", + "resourceVersion": "147728", + "uid": "8dfeff78-0cea-4c65-9247-f85c72fb8656" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 294ms (294ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-fg426.18c0d40d76876f64", + "namespace": "openshell", + "resourceVersion": "147735", + "uid": "2a1e354e-0819-44bf-a37b-529990a88e7e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-fg426.18c0d40d7f893561", + "namespace": "openshell", + "resourceVersion": "147741", + "uid": "325fd61e-589a-4565-a5c4-3fc537d005fc" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-fg426.18c0d40d7ff7d067", + "namespace": "openshell", + "resourceVersion": "147742", + "uid": "9de3e1bf-8617-45f4-b327-2e6e86b69849" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 321ms (321ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-fg426.18c0d40da2e141e1", + "namespace": "openshell", + "resourceVersion": "147772", + "uid": "2e2d1077-8e30-40ec-ba87-d537cef6c87b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:28Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 281ms (281ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:28Z", + "name": "openshell-warm-pool-fg426.18c0d40ddc954435", + "namespace": "openshell", + "resourceVersion": "147791", + "uid": "d6e8ac62-3c6d-4d1b-b416-b154230c0245" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:52Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-fg426.18c0d40cf05c5433", + "namespace": "openshell", + "resourceVersion": "147891", + "uid": "680b85e8-f80a-42b1-80a8-bfe834493014" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:53Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-fg426", + "namespace": "openshell", + "resourceVersion": "147604", + "uid": "ad3fea50-5e7f-4947-aacc-1d6698ea4fc1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:53Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 270ms (270ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:53Z", + "name": "openshell-warm-pool-fg426.18c0d4139e947048", + "namespace": "openshell", + "resourceVersion": "147906", + "uid": "5c01d3fd-028e-46a7-8b54-7fcd61c30a2f" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-10.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-10.json new file mode 100644 index 0000000000..ea5ce02767 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-10.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:44:34.341223Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148415", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-klq2c to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:34Z", + "name": "openshell-warm-pool-klq2c.18c0d41d34a73ecc", + "namespace": "openshell", + "resourceVersion": "148455", + "uid": "79804f77-3fa3-4042-b335-10ba42582290" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:36Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:36Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-3dfcc231-7da7-4737-81ac-b29ae328eec4\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:36Z", + "name": "openshell-warm-pool-klq2c.18c0d41dbfae26b3", + "namespace": "openshell", + "resourceVersion": "148472", + "uid": "99c2b9af-ba66-4636-ad80-625b54638e09" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:37Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:37Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:37Z", + "name": "openshell-warm-pool-klq2c.18c0d41e0199b64a", + "namespace": "openshell", + "resourceVersion": "148510", + "uid": "05ea8dd9-c572-4916-a245-64515b980963" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:37Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148458", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:37Z", + "message": "Add eth0 [10.131.0.136/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:37Z", + "name": "openshell-warm-pool-klq2c.18c0d41deeda6baf", + "namespace": "openshell", + "resourceVersion": "148480", + "uid": "28ba8a1f-ecae-43d3-b90a-39f76d50a0ce" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:37Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:37Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 274ms (274ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:37Z", + "name": "openshell-warm-pool-klq2c.18c0d41e01921cdd", + "namespace": "openshell", + "resourceVersion": "148509", + "uid": "57a361de-2df9-4fc6-8105-3df7812ed10d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:38Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:38Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:38Z", + "name": "openshell-warm-pool-klq2c.18c0d41e17bae3f5", + "namespace": "openshell", + "resourceVersion": "148516", + "uid": "140303e2-87fb-4b62-ab3d-d88c61254db9" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:38Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:38Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 272ms (272ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:38Z", + "name": "openshell-warm-pool-klq2c.18c0d41e11dabd52", + "namespace": "openshell", + "resourceVersion": "148512", + "uid": "ee4e74eb-a52d-46d1-afa1-11e7f805be27" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:38Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:38Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:38Z", + "name": "openshell-warm-pool-klq2c.18c0d41e174fc696", + "namespace": "openshell", + "resourceVersion": "148515", + "uid": "1f47706b-6654-4efb-ab62-9a590b84773d" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:38Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:38Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 319ms (319ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:38Z", + "name": "openshell-warm-pool-klq2c.18c0d41e3b307931", + "namespace": "openshell", + "resourceVersion": "148529", + "uid": "1f612a6c-9dfa-4dbc-8908-96762dace357" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:38Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:38Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:38Z", + "name": "openshell-warm-pool-klq2c.18c0d41e3b389a6e", + "namespace": "openshell", + "resourceVersion": "148530", + "uid": "975ecf06-6d2f-4cc4-b5b1-4dbc7a4368b4" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:39Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:39Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 351ms (351ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:39Z", + "name": "openshell-warm-pool-klq2c.18c0d41e502dca1d", + "namespace": "openshell", + "resourceVersion": "148534", + "uid": "1818680e-aa60-420f-9694-9bee81ce47da" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:39Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:39Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:39Z", + "name": "openshell-warm-pool-klq2c.18c0d41e551ef566", + "namespace": "openshell", + "resourceVersion": "148535", + "uid": "6aeb3269-0237-4a89-98aa-a8745e32c402" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:39Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:39Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:39Z", + "name": "openshell-warm-pool-klq2c.18c0d41e5585f170", + "namespace": "openshell", + "resourceVersion": "148537", + "uid": "db04548b-d018-4a0d-a87e-948efabe970a" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:39Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:39Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 279ms (279ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:39Z", + "name": "openshell-warm-pool-klq2c.18c0d41e74d7b5e8", + "namespace": "openshell", + "resourceVersion": "148554", + "uid": "838d8977-1c3c-4a6b-aa6c-8b4cd3573820" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:40Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:40Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 292ms (292ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:40Z", + "name": "openshell-warm-pool-klq2c.18c0d41eb17df9aa", + "namespace": "openshell", + "resourceVersion": "148585", + "uid": "6f85ffc0-a5a3-4830-97d9-909e13737a11" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:37Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:50Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:37Z", + "name": "openshell-warm-pool-klq2c.18c0d41df13338fd", + "namespace": "openshell", + "resourceVersion": "148718", + "uid": "92309ad8-146a-40c8-818f-54e7cf654405" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:50Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-klq2c", + "namespace": "openshell", + "resourceVersion": "148454", + "uid": "a7332aa4-4fe5-4d58-8e51-75d3ecd57569" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:50Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 283ms (283ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:50Z", + "name": "openshell-warm-pool-klq2c.18c0d42101a607b9", + "namespace": "openshell", + "resourceVersion": "148732", + "uid": "ab0adf8f-cafc-43a5-a534-2d02fe84035f" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-2.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-2.json new file mode 100644 index 0000000000..16cd57d22b --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-2.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:43:18.048327Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147534", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-6z4hr to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:18Z", + "name": "openshell-warm-pool-6z4hr.18c0d40b713e0545", + "namespace": "openshell", + "resourceVersion": "147610", + "uid": "0ab4c35e-b48e-41d1-9850-57b7952c808f" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:22Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-58a9c2c8-9c57-4792-9c1b-db23ae9d6ba2\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:22Z", + "name": "openshell-warm-pool-6z4hr.18c0d40c71185048", + "namespace": "openshell", + "resourceVersion": "147662", + "uid": "c8035fe8-1560-4ab2-bb99-f7bc00c5de42" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d035f438a", + "namespace": "openshell", + "resourceVersion": "147693", + "uid": "437d104b-8720-46b0-9b13-e188e95feccd" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147617", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Add eth0 [10.131.0.131/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-6z4hr.18c0d40cf11db4a5", + "namespace": "openshell", + "resourceVersion": "147682", + "uid": "96b95cd4-9c2c-4519-a1fe-281fa3c407e3" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 267ms (268ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d03580275", + "namespace": "openshell", + "resourceVersion": "147692", + "uid": "9a0b682c-89a6-4a66-98e6-91e061d74bb6" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d19cc1daa", + "namespace": "openshell", + "resourceVersion": "147709", + "uid": "a9ada55a-03fb-4232-b36b-7839bcecb6ff" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 276ms (276ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d13e05049", + "namespace": "openshell", + "resourceVersion": "147699", + "uid": "0d6ba5d7-03bf-423f-8615-a7f8a706c3aa" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d193f1de9", + "namespace": "openshell", + "resourceVersion": "147708", + "uid": "6e50318d-49f2-447d-baae-1f7d6823edc4" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 295ms (295ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d6533c42f", + "namespace": "openshell", + "resourceVersion": "147729", + "uid": "b3abdc4b-1f87-446f-adf4-1b383e3053cc" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d653a0dfd", + "namespace": "openshell", + "resourceVersion": "147730", + "uid": "17ad70fc-23ae-4e94-8cec-018141f03b23" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 291ms (291ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d76949a1e", + "namespace": "openshell", + "resourceVersion": "147736", + "uid": "5d8737e6-8332-4791-a904-72d033d18c2a" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d7f7c9e98", + "namespace": "openshell", + "resourceVersion": "147738", + "uid": "d91b46b0-9cea-4429-916a-db33e82066da" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-6z4hr.18c0d40d7ff7eec4", + "namespace": "openshell", + "resourceVersion": "147743", + "uid": "aa827f5b-8f7f-40a3-a608-e26d8659053a" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 287ms (287ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-6z4hr.18c0d40da0edd734", + "namespace": "openshell", + "resourceVersion": "147769", + "uid": "aaaa7422-6892-46d4-a453-9fb646fa671c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:28Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 276ms (276ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:28Z", + "name": "openshell-warm-pool-6z4hr.18c0d40ddc501214", + "namespace": "openshell", + "resourceVersion": "147790", + "uid": "4ed8b86a-fa6c-48ea-b3de-d0e0577e7e03" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:59Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-6z4hr.18c0d40cf35e333e", + "namespace": "openshell", + "resourceVersion": "147961", + "uid": "6c0cfb03-f4f7-4038-9b95-5efa0c22ffb9" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:59Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-6z4hr", + "namespace": "openshell", + "resourceVersion": "147609", + "uid": "3a14133b-9e76-4b64-9278-d836e26f24dd" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:59Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 281ms (281ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:59Z", + "name": "openshell-warm-pool-6z4hr.18c0d41519c9d97d", + "namespace": "openshell", + "resourceVersion": "147975", + "uid": "9f75f172-efa5-4bac-a00b-8cf369a5c6ac" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-3.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-3.json new file mode 100644 index 0000000000..25c780516d --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-3.json @@ -0,0 +1,588 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:43:18.119737Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147544", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-2rk2j to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:18Z", + "name": "openshell-warm-pool-2rk2j.18c0d40b757fa418", + "namespace": "openshell", + "resourceVersion": "147621", + "uid": "3c7ec9e1-4f88-47ee-a24d-d461c75d6bea" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:22Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-8fdcccb8-f218-4883-bf8e-e8dbeb0c797b\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:22Z", + "name": "openshell-warm-pool-2rk2j.18c0d40c72edb23a", + "namespace": "openshell", + "resourceVersion": "147664", + "uid": "6966b0fe-ceb0-406c-a5fa-8b955a23d4a9" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d0ae701f8", + "namespace": "openshell", + "resourceVersion": "147696", + "uid": "76eb2176-ff5d-4661-8755-0a646960573e" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147624", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Add eth0 [10.131.0.132/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40ce2507d4f", + "namespace": "openshell", + "resourceVersion": "147672", + "uid": "c95590a4-25f2-4252-96d4-16a3c73ec3e9" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 272ms (272ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40cf508ea2a", + "namespace": "openshell", + "resourceVersion": "147685", + "uid": "061b3baf-1c23-4c62-934a-e18b2fcc0b3e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40cf51003b7", + "namespace": "openshell", + "resourceVersion": "147686", + "uid": "ceeedd22-55b6-4614-8b00-47c315c6dd4e" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 293ms (293ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d068f184f", + "namespace": "openshell", + "resourceVersion": "147694", + "uid": "f4827e83-3580-48ae-b715-ce9eb43a4456" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d0a7d4aad", + "namespace": "openshell", + "resourceVersion": "147695", + "uid": "d44b77b2-bdc9-41b2-838b-d9812155cf08" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 276ms (276ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d27e17833", + "namespace": "openshell", + "resourceVersion": "147711", + "uid": "1eb4e9bb-faa9-4b0a-a6c2-4b28c7fa6abe" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 331ms (331ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d671308ec", + "namespace": "openshell", + "resourceVersion": "147731", + "uid": "284eb836-c0f0-44d0-9b7d-0ffc5810e985" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d671d5472", + "namespace": "openshell", + "resourceVersion": "147732", + "uid": "316dbec5-5559-4e89-8c9f-80f8e4af3722" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 297ms (297ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d78dc22c8", + "namespace": "openshell", + "resourceVersion": "147737", + "uid": "43b2e4c9-a157-4344-8250-c62025972412" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d7f88785b", + "namespace": "openshell", + "resourceVersion": "147740", + "uid": "faf36aca-d35c-4d3b-86ab-72c13f4599e4" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-2rk2j.18c0d40d80057280", + "namespace": "openshell", + "resourceVersion": "147744", + "uid": "043d06fd-4240-41b3-a12f-c3af1fd674d1" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 289ms (289ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-2rk2j.18c0d40da133ad6f", + "namespace": "openshell", + "resourceVersion": "147770", + "uid": "211485de-e316-4c75-84e1-f9d3a6a0d27e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:28Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 298ms (298ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:28Z", + "name": "openshell-warm-pool-2rk2j.18c0d40dddaa96e8", + "namespace": "openshell", + "resourceVersion": "147792", + "uid": "5b4cb1f8-9c61-4a3d-8e77-69735648da90" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 6, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-2rk2j.18c0d40ce4cf84d8", + "namespace": "openshell", + "resourceVersion": "148027", + "uid": "6eb01ee2-8abf-4849-a495-8e08d5283212" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2rk2j", + "namespace": "openshell", + "resourceVersion": "147620", + "uid": "b3622085-4036-4cca-a53f-41fb3ca65dfb" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 296ms (296ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-2rk2j.18c0d41699d021d8", + "namespace": "openshell", + "resourceVersion": "148046", + "uid": "0fd7cc17-0cb9-4510-8b71-323bfaf25cc4" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-4.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-4.json new file mode 100644 index 0000000000..b372cd863f --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-4.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:43:18.186704Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147555", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-qhfwb to ip-10-0-1-179.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:18Z", + "name": "openshell-warm-pool-qhfwb.18c0d40b797d76d4", + "namespace": "openshell", + "resourceVersion": "147628", + "uid": "65156daf-25af-4115-886d-123461231634" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:21Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-28fd608e-7174-4aa2-98bc-cd0e6e2a7f06\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:21Z", + "name": "openshell-warm-pool-qhfwb.18c0d40c57ea8a01", + "namespace": "openshell", + "resourceVersion": "147659", + "uid": "ff4c1d73-f2e4-41be-a118-afb54c2a57d7" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:28Z", + "name": "openshell-warm-pool-qhfwb.18c0d40dc2531f3c", + "namespace": "openshell", + "resourceVersion": "147783", + "uid": "264c12a7-3856-4fbe-84e6-c3c242abe429" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147632", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Add eth0 [10.129.0.44/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-qhfwb.18c0d40d89012394", + "namespace": "openshell", + "resourceVersion": "147746", + "uid": "e1b76658-3066-4836-8e37-2153ffc22dd8" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 594ms (594ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-qhfwb.18c0d40daebe17b3", + "namespace": "openshell", + "resourceVersion": "147777", + "uid": "f8973451-b536-477c-922b-16c2684b3d2d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-qhfwb.18c0d40daec78f07", + "namespace": "openshell", + "resourceVersion": "147778", + "uid": "6feff849-23d0-4b97-8d4a-b216edcd0b84" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 254ms (254ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-qhfwb.18c0d40dbdfa73d7", + "namespace": "openshell", + "resourceVersion": "147781", + "uid": "0791ef1e-0d05-4861-b48f-8246f5ab6098" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:28Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:28Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:28Z", + "name": "openshell-warm-pool-qhfwb.18c0d40dc2c323e6", + "namespace": "openshell", + "resourceVersion": "147784", + "uid": "3cc9379d-f621-4400-b152-7b197cb6cb65" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:29Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:29Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 260ms (260ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:29Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e0423fa59", + "namespace": "openshell", + "resourceVersion": "147799", + "uid": "185345a3-1a44-4e9f-a3b6-2f7444a3bd45" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:29Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:29Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e042c15b3", + "namespace": "openshell", + "resourceVersion": "147800", + "uid": "29b51aeb-5961-4ff1-8b60-e4f77cf30ccf" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:29Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 265ms (265ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:29Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e13fd03b8", + "namespace": "openshell", + "resourceVersion": "147803", + "uid": "c5585d5a-e4a6-4d8a-ab09-984c03aeac6e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:29Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:29Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e1909f0dc", + "namespace": "openshell", + "resourceVersion": "147804", + "uid": "79a2f29d-2120-4f46-ad28-b24776e4544e" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:29Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:29Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e1976fc1d", + "namespace": "openshell", + "resourceVersion": "147805", + "uid": "55243fa0-1c8c-472a-9da5-e38efdee6779" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:30Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:30Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 265ms (265ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:30Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e4000c8da", + "namespace": "openshell", + "resourceVersion": "147813", + "uid": "33bdbc15-d88c-4761-b630-09fa70390b0d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:31Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:31Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 263ms (263ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:31Z", + "name": "openshell-warm-pool-qhfwb.18c0d40e7bd305ed", + "namespace": "openshell", + "resourceVersion": "147819", + "uid": "86002d4c-4d5c-45b4-ab49-da64aeb805b9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:12Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-qhfwb.18c0d40d8b511c2b", + "namespace": "openshell", + "resourceVersion": "148127", + "uid": "a02b9334-b519-4c0d-bdbb-f61df7e7b658" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:12Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-qhfwb", + "namespace": "openshell", + "resourceVersion": "147626", + "uid": "3c376c20-ce72-48a1-949d-d897f792bddc" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:12Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 264ms (264ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:12Z", + "name": "openshell-warm-pool-qhfwb.18c0d41819e80d6c", + "namespace": "openshell", + "resourceVersion": "148143", + "uid": "a29ebb07-ed88-4185-b628-23dc367541c7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-5.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-5.json new file mode 100644 index 0000000000..4c3926514a --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-5.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:43:18.255737Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147565", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-8bwz7 to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:18Z", + "name": "openshell-warm-pool-8bwz7.18c0d40b7d9ad600", + "namespace": "openshell", + "resourceVersion": "147634", + "uid": "d9cb5036-3d8f-4e43-9c25-f76715d1670c" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:21Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-d05e8305-1d7a-4b1d-9c6f-a24480dfc12d\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:21Z", + "name": "openshell-warm-pool-8bwz7.18c0d40c538adc7a", + "namespace": "openshell", + "resourceVersion": "147657", + "uid": "437c1c39-4918-4afb-ba37-23ba224f7c0f" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d00d4ca66", + "namespace": "openshell", + "resourceVersion": "147691", + "uid": "d80e690e-14f8-476b-9405-1bd7c6270cb6" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147642", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Add eth0 [10.131.0.133/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-8bwz7.18c0d40ceb76a8cf", + "namespace": "openshell", + "resourceVersion": "147676", + "uid": "474352e0-d774-4b4e-8dfb-a9d3c212b8dd" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 303ms (304ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d00cd22ec", + "namespace": "openshell", + "resourceVersion": "147690", + "uid": "3af05d80-bf2b-4b0b-a368-f60bb9caf070" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d191aaa05", + "namespace": "openshell", + "resourceVersion": "147707", + "uid": "53bf0832-e80a-4eba-b4f6-dface08c711c" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 291ms (291ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d123beb80", + "namespace": "openshell", + "resourceVersion": "147698", + "uid": "508b57c2-d4fe-4dc9-9122-e848b4ff0a87" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:25Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:25Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d18797a80", + "namespace": "openshell", + "resourceVersion": "147704", + "uid": "fecf66f7-c8f7-454f-b5bd-3eec5ac9a6ae" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 282ms (282ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d6445c618", + "namespace": "openshell", + "resourceVersion": "147725", + "uid": "23ad2806-f812-4c90-914d-fe81d7aaccdd" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d644d8487", + "namespace": "openshell", + "resourceVersion": "147726", + "uid": "5691fb0e-c919-43d8-9270-6cb67ea9379c" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 287ms (287ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d7576f2fb", + "namespace": "openshell", + "resourceVersion": "147734", + "uid": "090cabad-b4a9-4528-b9c8-4f36ed215b24" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d7f8726b1", + "namespace": "openshell", + "resourceVersion": "147739", + "uid": "30976c0d-62f4-4aba-9c2d-a57eb90d0e65" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:26Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:26Z", + "name": "openshell-warm-pool-8bwz7.18c0d40d80058da4", + "namespace": "openshell", + "resourceVersion": "147745", + "uid": "953fe98f-0831-43a4-a314-d92e5978be22" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 279ms (279ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:27Z", + "name": "openshell-warm-pool-8bwz7.18c0d40da03c6a5d", + "namespace": "openshell", + "resourceVersion": "147768", + "uid": "a7aa7fd7-d581-4d0d-b79f-26c90fe4d09d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:28Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 326ms (326ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:28Z", + "name": "openshell-warm-pool-8bwz7.18c0d40ddf481c6a", + "namespace": "openshell", + "resourceVersion": "147793", + "uid": "a6ddf9a4-51df-4eaf-bcab-2e456452cec6" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:24Z", + "name": "openshell-warm-pool-8bwz7.18c0d40ceeae0941", + "namespace": "openshell", + "resourceVersion": "148198", + "uid": "d51680df-ecb1-4598-8009-fd170dba49fe" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8bwz7", + "namespace": "openshell", + "resourceVersion": "147633", + "uid": "7b52f5cd-775a-4d19-9d56-5ec3f3602fd2" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 289ms (289ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-8bwz7.18c0d4199706d09d", + "namespace": "openshell", + "resourceVersion": "148213", + "uid": "6bbc817d-d664-4db5-b0c5-405bec09ab87" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-6.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-6.json new file mode 100644 index 0000000000..a713ac06ee --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-6.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:43:56.004130Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147897", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-s94bn to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:56Z", + "name": "openshell-warm-pool-s94bn.18c0d41447958dd1", + "namespace": "openshell", + "resourceVersion": "147929", + "uid": "0f7c5446-cc33-47a7-9785-8b6079c3240e" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:43:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:43:58Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-e85482be-e273-4fb0-9a72-95746a366e9f\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:43:58Z", + "name": "openshell-warm-pool-s94bn.18c0d414d364a039", + "namespace": "openshell", + "resourceVersion": "147948", + "uid": "d6038cb7-1db4-43af-9ff2-1dc9467b3e53" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d416919f6fd6", + "namespace": "openshell", + "resourceVersion": "148042", + "uid": "7d228d1b-2c04-463f-9607-3b67688fead7" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147932", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Add eth0 [10.131.0.134/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d416675de9ab", + "namespace": "openshell", + "resourceVersion": "148011", + "uid": "aaf791c7-0d07-47d3-a345-2fb2231a0a88" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 306ms (306ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d4167bd0c86d", + "namespace": "openshell", + "resourceVersion": "148014", + "uid": "47703a6a-4b12-4d54-82f8-8bd06806eac7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d4167bd85468", + "namespace": "openshell", + "resourceVersion": "148015", + "uid": "601f3ac9-eeea-453e-923c-85a99a17ef0d" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 280ms (280ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d4168c905af2", + "namespace": "openshell", + "resourceVersion": "148039", + "uid": "9c82bde2-0a75-4ad8-b3cb-4e4475854111" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d4169130bfcc", + "namespace": "openshell", + "resourceVersion": "148041", + "uid": "9b8a7b0c-71e9-456b-82da-0a8afd1f8d7e" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:06Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 280ms (281ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-s94bn.18c0d416be235227", + "namespace": "openshell", + "resourceVersion": "148054", + "uid": "2bbbcf12-9122-41dc-94e7-8eade6b47374" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:06Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-s94bn.18c0d416be2a1d3f", + "namespace": "openshell", + "resourceVersion": "148055", + "uid": "7cfdf35d-7988-4bf5-b195-efe0c9eb168b" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:06Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 305ms (305ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-s94bn.18c0d416d06462fe", + "namespace": "openshell", + "resourceVersion": "148059", + "uid": "99836228-51e7-4c1c-8de3-00e0870a684a" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:06Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-s94bn.18c0d416d5536419", + "namespace": "openshell", + "resourceVersion": "148060", + "uid": "9e11b734-ca5b-4988-8a71-19996775dc04" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:06Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-s94bn.18c0d416d5c140a1", + "namespace": "openshell", + "resourceVersion": "148061", + "uid": "4be132b4-e167-4e9b-abc7-c78aa8edf2d7" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:07Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:07Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 490ms (490ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:07Z", + "name": "openshell-warm-pool-s94bn.18c0d417068998a5", + "namespace": "openshell", + "resourceVersion": "148072", + "uid": "7ab797fa-4fb4-43e1-930e-bc92ce9b0ba5" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:08Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:08Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 262ms (262ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-s94bn.18c0d41734bd14a1", + "namespace": "openshell", + "resourceVersion": "148087", + "uid": "5f5bc6ba-b608-4943-b18b-3bf8cefdd8bc" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:24Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-s94bn.18c0d4166991fa36", + "namespace": "openshell", + "resourceVersion": "148291", + "uid": "945156d3-b2be-42bc-ae20-6807e43299a8" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:25Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-s94bn", + "namespace": "openshell", + "resourceVersion": "147928", + "uid": "83c833bc-b385-45ac-8081-6fbb659aa5a5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 272ms (272ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:25Z", + "name": "openshell-warm-pool-s94bn.18c0d41b0e309ed4", + "namespace": "openshell", + "resourceVersion": "148305", + "uid": "99b89335-19a9-44cd-b4a1-754a7ae4f423" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-7.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-7.json new file mode 100644 index 0000000000..9c03d2f66c --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-7.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:44:02.361095Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147967", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-wmgrv to ip-10-0-1-73.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:02Z", + "name": "openshell-warm-pool-wmgrv.18c0d415c27d453e", + "namespace": "openshell", + "resourceVersion": "147998", + "uid": "e68e9b0c-ab25-4bb2-b4e5-37a7ba5287df" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:05Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:05Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-c119990b-be6d-4ab2-869e-2df7db51a718\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:05Z", + "name": "openshell-warm-pool-wmgrv.18c0d41695ce2905", + "namespace": "openshell", + "resourceVersion": "148044", + "uid": "6b0edcce-4731-43f8-8484-e926e2b0a057" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "148001", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:06Z", + "message": "Add eth0 [10.128.0.131/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-wmgrv.18c0d416c59fdef4", + "namespace": "openshell", + "resourceVersion": "148056", + "uid": "37d39f77-3aea-43be-81ef-445a179c9ccc" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:07Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:07Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:07Z", + "name": "openshell-warm-pool-wmgrv.18c0d416fde804bf", + "namespace": "openshell", + "resourceVersion": "148070", + "uid": "dca3eb41-37b6-4d1e-a5e3-19374319e101" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:07Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:07Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 565ms (565ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:07Z", + "name": "openshell-warm-pool-wmgrv.18c0d416e9e23448", + "namespace": "openshell", + "resourceVersion": "148063", + "uid": "c79823f1-63b2-44d7-9c31-a00a94e5b592" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:07Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:07Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:07Z", + "name": "openshell-warm-pool-wmgrv.18c0d416e9ebfd2d", + "namespace": "openshell", + "resourceVersion": "148065", + "uid": "ea7ecf16-60b8-4742-a929-bb18a6a2041e" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:07Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:07Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 257ms (257ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:07Z", + "name": "openshell-warm-pool-wmgrv.18c0d416f9405716", + "namespace": "openshell", + "resourceVersion": "148068", + "uid": "2579af55-e3cc-432e-af02-92d996c1f7c0" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:07Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:07Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:07Z", + "name": "openshell-warm-pool-wmgrv.18c0d416fd7c0de9", + "namespace": "openshell", + "resourceVersion": "148069", + "uid": "ea74de07-1700-4b8c-98c6-c70c6a65938a" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:08Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:08Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-wmgrv.18c0d417225c5769", + "namespace": "openshell", + "resourceVersion": "148083", + "uid": "17253bd0-ce99-4452-b670-475ee70c61fd" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:08Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:08Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 266ms (266ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-wmgrv.18c0d4172253baff", + "namespace": "openshell", + "resourceVersion": "148082", + "uid": "ec093441-2464-42ff-983c-2d02c7cd1282" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:08Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:08Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 287ms (287ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-wmgrv.18c0d417337f6d47", + "namespace": "openshell", + "resourceVersion": "148086", + "uid": "918faef7-f01f-4777-b954-091ef61331d9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:08Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:08Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-wmgrv.18c0d41738f50847", + "namespace": "openshell", + "resourceVersion": "148093", + "uid": "02305250-b475-404b-8bef-943baaae5eb8" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:08Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:08Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-wmgrv.18c0d4173961333c", + "namespace": "openshell", + "resourceVersion": "148095", + "uid": "847bcf6f-635f-4c4c-b9d2-47543d465b5c" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:09Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:09Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 268ms (268ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:09Z", + "name": "openshell-warm-pool-wmgrv.18c0d4175e20d879", + "namespace": "openshell", + "resourceVersion": "148108", + "uid": "2a2b70f9-a87f-4ca0-905e-ae9f2535ef49" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:10Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:10Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 269ms (269ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:10Z", + "name": "openshell-warm-pool-wmgrv.18c0d4179a109eca", + "namespace": "openshell", + "resourceVersion": "148112", + "uid": "2ed2e7eb-6dec-4f6e-a03d-d99bc12f03e9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:06Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:31Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:06Z", + "name": "openshell-warm-pool-wmgrv.18c0d416c82ece5c", + "namespace": "openshell", + "resourceVersion": "148411", + "uid": "de166203-54fd-4ad9-a4e8-99ec7ec2302f" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:31Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wmgrv", + "namespace": "openshell", + "resourceVersion": "147997", + "uid": "5489101b-df32-42a2-aa2e-0a920dced15e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:31Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 285ms (285ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:31Z", + "name": "openshell-warm-pool-wmgrv.18c0d41c8da06b76", + "namespace": "openshell", + "resourceVersion": "148426", + "uid": "bae8e749-1745-4efd-97ff-5cca8edd34d4" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-73.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-73.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-8.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-8.json new file mode 100644 index 0000000000..f0bde37a61 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-8.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:44:08.751689Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148032", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-9bgrl to ip-10-0-1-179.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:08Z", + "name": "openshell-warm-pool-9bgrl.18c0d4173f65ebd8", + "namespace": "openshell", + "resourceVersion": "148097", + "uid": "8d364c7c-646e-465d-8083-fbc738f21221" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:11Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:11Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-ae7fa3dd-2066-450e-aa43-a5df61934701\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:11Z", + "name": "openshell-warm-pool-9bgrl.18c0d417c77356bf", + "namespace": "openshell", + "resourceVersion": "148114", + "uid": "8af69202-03d1-4c63-a146-2a704d775887" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d41999778295", + "namespace": "openshell", + "resourceVersion": "148216", + "uid": "bc732533-401d-4395-8d5e-bf1c8cf44497" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148102", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Add eth0 [10.129.0.45/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d4197179d6a4", + "namespace": "openshell", + "resourceVersion": "148180", + "uid": "f47d6123-c604-4f75-b025-a90a0539115e" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d41983f8743f", + "namespace": "openshell", + "resourceVersion": "148193", + "uid": "36ac4be4-7cbc-4623-ab9f-946a20ddeee2" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 280ms (280ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d41994ae9cce", + "namespace": "openshell", + "resourceVersion": "148211", + "uid": "a8c54644-3d3e-4b38-b0c6-c19f3d8db741" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d4199907a704", + "namespace": "openshell", + "resourceVersion": "148214", + "uid": "27a1030d-06d8-42eb-93ad-6ab094c24633" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:19Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 256ms (256ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:19Z", + "name": "openshell-warm-pool-9bgrl.18c0d419b56752ff", + "namespace": "openshell", + "resourceVersion": "148222", + "uid": "4f54fcc9-12ab-47b6-93e5-2b9bb3c32a56" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 272ms (272ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:20Z", + "name": "openshell-warm-pool-9bgrl.18c0d41a03037baa", + "namespace": "openshell", + "resourceVersion": "148232", + "uid": "1d9be543-8920-403b-b15c-3cea011da266" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 278ms (278ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:20Z", + "name": "openshell-warm-pool-9bgrl.18c0d419f2bacfee", + "namespace": "openshell", + "resourceVersion": "148229", + "uid": "f9395e7f-0848-4f08-8782-8e8d75766db7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:20Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:20Z", + "name": "openshell-warm-pool-9bgrl.18c0d419f2c230d4", + "namespace": "openshell", + "resourceVersion": "148230", + "uid": "bc204768-43d4-4fd2-ac22-c82029d605c9" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:20Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:20Z", + "name": "openshell-warm-pool-9bgrl.18c0d41a085f2502", + "namespace": "openshell", + "resourceVersion": "148233", + "uid": "cf364769-85c3-4ea1-aebb-ce195b6fa322" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:20Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:20Z", + "name": "openshell-warm-pool-9bgrl.18c0d41a08c9c4bf", + "namespace": "openshell", + "resourceVersion": "148234", + "uid": "afaa3374-8adb-4886-ad7d-6fd2405ad91c" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 272ms (272ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:21Z", + "name": "openshell-warm-pool-9bgrl.18c0d41a2e5eb760", + "namespace": "openshell", + "resourceVersion": "148251", + "uid": "90383e85-f12a-47ee-92d7-bbffb8b6b292" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 264ms (264ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:22Z", + "name": "openshell-warm-pool-9bgrl.18c0d41a69c5973b", + "namespace": "openshell", + "resourceVersion": "148269", + "uid": "c1358197-dadb-4fbc-806d-e44d3c9a43cf" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 2, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:37Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 271ms (271ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d41983f0d6ff", + "namespace": "openshell", + "resourceVersion": "148511", + "uid": "e7054a15-e245-4769-b83c-a21d186e661c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 6, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-9bgrl", + "namespace": "openshell", + "resourceVersion": "148096", + "uid": "31fcd461-f692-4fda-8c5c-9743e5c5b7e4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:37Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-9bgrl.18c0d41973c28704", + "namespace": "openshell", + "resourceVersion": "148494", + "uid": "03774c04-5191-40ac-8ea5-8a37d1381988" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-179.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-179.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-9.json b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-9.json new file mode 100644 index 0000000000..4c8badd5cd --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/events-wp-default-run-9.json @@ -0,0 +1,588 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T04:44:15.250494Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148134", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-w624n to ip-10-0-1-247.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:15Z", + "name": "openshell-warm-pool-w624n.18c0d418c2c1cd09", + "namespace": "openshell", + "resourceVersion": "148164", + "uid": "46c084d8-dd35-4614-be08-202c5be01241" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-688dfb9cc7-ml55z", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:18Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-75a64f14-de24-4803-af7c-d0245de990d0\" ", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:18Z", + "name": "openshell-warm-pool-w624n.18c0d419991dd47d", + "namespace": "openshell", + "resourceVersion": "148215", + "uid": "963360bb-5a4b-4941-8f4e-6ff57a993c02" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:25Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:25Z", + "name": "openshell-warm-pool-w624n.18c0d41b3da90f03", + "namespace": "openshell", + "resourceVersion": "148311", + "uid": "3dae9831-52c6-4f16-aa9b-4e69e89062ab" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:25Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148167", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:25Z", + "message": "Add eth0 [10.131.0.135/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:25Z", + "name": "openshell-warm-pool-w624n.18c0d41b2a199fe9", + "namespace": "openshell", + "resourceVersion": "148306", + "uid": "3280d210-1482-4edb-96e9-11d207b42a80" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:25Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 286ms (286ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:25Z", + "name": "openshell-warm-pool-w624n.18c0d41b3da21511", + "namespace": "openshell", + "resourceVersion": "148310", + "uid": "0f1d03b0-ce8e-4f97-a682-e3c44d9e36a9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:26Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:26Z", + "name": "openshell-warm-pool-w624n.18c0d41b5292b186", + "namespace": "openshell", + "resourceVersion": "148315", + "uid": "d3166e90-4e77-4a18-aa7c-eff5522245a4" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 283ms (283ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:26Z", + "name": "openshell-warm-pool-w624n.18c0d41b4e93da5d", + "namespace": "openshell", + "resourceVersion": "148313", + "uid": "a5795139-0979-49c5-b744-d6bf0dd256c3" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:26Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:26Z", + "name": "openshell-warm-pool-w624n.18c0d41b5226409c", + "namespace": "openshell", + "resourceVersion": "148314", + "uid": "c920f8b0-6397-4ec8-af0d-57e70381ec11" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 295ms (295ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:26Z", + "name": "openshell-warm-pool-w624n.18c0d41b6b6719d8", + "namespace": "openshell", + "resourceVersion": "148327", + "uid": "8b44f4c6-8db5-4211-bc25-b4fcbd5f9960" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:27Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 271ms (271ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:27Z", + "name": "openshell-warm-pool-w624n.18c0d41ba5e5387c", + "namespace": "openshell", + "resourceVersion": "148338", + "uid": "e76a6b48-504e-465b-86bc-ec38653bb5a9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:27Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:27Z", + "name": "openshell-warm-pool-w624n.18c0d41ba5edeb68", + "namespace": "openshell", + "resourceVersion": "148340", + "uid": "d21e6a29-d7fd-47de-889d-c382c045d448" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 285ms (286ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:27Z", + "name": "openshell-warm-pool-w624n.18c0d41bb6fa4dda", + "namespace": "openshell", + "resourceVersion": "148357", + "uid": "41741304-6bf6-4019-99cf-a0872e078ad2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:28Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:28Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:28Z", + "name": "openshell-warm-pool-w624n.18c0d41bbc1d3e66", + "namespace": "openshell", + "resourceVersion": "148360", + "uid": "7d8684eb-f2d1-4fc6-b6ba-7e929c6c0f7b" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:28Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:28Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:28Z", + "name": "openshell-warm-pool-w624n.18c0d41bbc8505bb", + "namespace": "openshell", + "resourceVersion": "148361", + "uid": "d3d9541a-177a-4716-8538-68a81310017a" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:28Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 297ms (297ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:28Z", + "name": "openshell-warm-pool-w624n.18c0d41be34b1d46", + "namespace": "openshell", + "resourceVersion": "148372", + "uid": "57686766-873a-4b0d-a724-7bf5057c01ce" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:29Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:29Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 306ms (306ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:29Z", + "name": "openshell-warm-pool-w624n.18c0d41c1f946e6e", + "namespace": "openshell", + "resourceVersion": "148383", + "uid": "5563c4b5-d5f3-41ae-9365-8ec2a89d2f2a" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 6, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:25Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:43Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:25Z", + "name": "openshell-warm-pool-w624n.18c0d41b2c91dcda", + "namespace": "openshell", + "resourceVersion": "148613", + "uid": "edd2d072-bf02-48e6-b38d-ca19bf052b6e" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T04:44:44Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-w624n", + "namespace": "openshell", + "resourceVersion": "148163", + "uid": "7027f0e0-2205-4cb8-87fa-f9bea7c6428f" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T04:44:44Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 256ms (256ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T04:44:44Z", + "name": "openshell-warm-pool-w624n.18c0d41f7ed48099", + "namespace": "openshell", + "resourceVersion": "148625", + "uid": "e2978c0b-2e6f-4bb1-be74-c4fec9b4de05" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-1-247.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-1-247.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/2026-07-09-rosa-hcp-422/readiness-gates.csv b/experiments/results/2026-07-09-rosa-hcp-422/readiness-gates.csv new file mode 100644 index 0000000000..66a0aa6c5b --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/readiness-gates.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,readiness-gates,readiness-gate,1783620006153548000,1783620008256954000,2103,readiness-gate-run-1,ok,,,,, +2,readiness-gates,readiness-gate,1783620014172659000,1783620015190020000,1017,readiness-gate-run-2,ok,,,,, +3,readiness-gates,readiness-gate,1783620020690305000,1783620023617311000,2927,readiness-gate-run-3,ok,,,,, +4,readiness-gates,readiness-gate,1783620029207572000,1783620030452205000,1244,readiness-gate-run-4,ok,,,,, +5,readiness-gates,readiness-gate,1783620035750071000,1783620038800880000,3050,readiness-gate-run-5,ok,,,,, +6,readiness-gates,readiness-gate,1783620044241508000,1783620045374404000,1132,readiness-gate-run-6,ok,,,,, +7,readiness-gates,readiness-gate,1783620050676861000,1783620053576329000,2899,readiness-gate-run-7,ok,,,,, +8,readiness-gates,readiness-gate,1783620058905548000,1783620061932566000,3027,readiness-gate-run-8,ok,,,,, +9,readiness-gates,readiness-gate,1783620067283234000,1783620070171525000,2888,readiness-gate-run-9,ok,,,,, +10,readiness-gates,readiness-gate,1783620075490132000,1783620076523557000,1033,readiness-gate-run-10,ok,,,,, diff --git a/experiments/results/2026-07-09-rosa-hcp-422/warm-pool-burst.csv b/experiments/results/2026-07-09-rosa-hcp-422/warm-pool-burst.csv new file mode 100644 index 0000000000..f95896d664 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/warm-pool-burst.csv @@ -0,0 +1,26 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1.1,warm-pool,burst,1783618653976897000,1783618656251026000,2274,openshell-warm-pool-mtk9l,ok,,,,, +1.2,warm-pool,burst,1783618653976897000,1783618658000982000,4024,openshell-warm-pool-wn5b5,ok,,,,, +1.3,warm-pool,burst,1783618653976897000,1783618659770299000,5793,openshell-warm-pool-wgm5p,ok,,,,, +1.4,warm-pool,burst,1783618653976897000,1783618661645201000,7668,openshell-warm-pool-km5rn,ok,,,,, +1.5,warm-pool,burst,1783618653976897000,1783618663413291000,9436,openshell-warm-pool-l97zl,ok,,,,, +2.1,warm-pool,burst,1783618667413108000,1783618669693510000,2280,openshell-warm-pool-7b7qs,ok,,,,, +2.2,warm-pool,burst,1783618667413108000,1783618671581225000,4168,openshell-warm-pool-8pddq,ok,,,,, +2.3,warm-pool,burst,1783618667413108000,1783618673328208000,5915,openshell-warm-pool-4tg8v,ok,,,,, +2.4,warm-pool,burst,1783618667413108000,1783618675070603000,7657,openshell-warm-pool-wg4pw,ok,,,,, +2.5,warm-pool,burst,1783618667413108000,1783618676817811000,9404,openshell-warm-pool-jlkxs,ok,,,,, +3.1,warm-pool,burst,1783618680722201000,1783618682985506000,2263,openshell-warm-pool-qt8hg,ok,,,,, +3.2,warm-pool,burst,1783618680722201000,1783618684741023000,4018,openshell-warm-pool-4tppv,ok,,,,, +3.3,warm-pool,burst,1783618680722201000,1783618686489840000,5767,openshell-warm-pool-4v94n,ok,,,,, +3.4,warm-pool,burst,1783618680722201000,1783618688252703000,7530,openshell-warm-pool-mmddh,ok,,,,, +3.5,warm-pool,burst,1783618680722201000,1783618689994908000,9272,openshell-warm-pool-bf5l9,ok,,,,, +4.1,warm-pool,burst,1783618693968594000,1783618696264535000,2295,openshell-warm-pool-r22sg,ok,,,,, +4.2,warm-pool,burst,1783618693968594000,1783618698024808000,4056,openshell-warm-pool-6pl8n,ok,,,,, +4.3,warm-pool,burst,1783618693968594000,1783618699778243000,5809,openshell-warm-pool-cp885,ok,,,,, +4.4,warm-pool,burst,1783618693968594000,1783618701546876000,7578,openshell-warm-pool-dg6gj,ok,,,,, +4.5,warm-pool,burst,1783618693968594000,1783618703303502000,9334,openshell-warm-pool-twrww,ok,,,,, +5.1,warm-pool,burst,1783618707168181000,1783618709451312000,2283,openshell-warm-pool-fl56f,ok,,,,, +5.2,warm-pool,burst,1783618707168181000,1783618711295028000,4126,openshell-warm-pool-22898,ok,,,,, +5.3,warm-pool,burst,1783618707168181000,1783618713140143000,5971,openshell-warm-pool-kf9ff,ok,,,,, +5.4,warm-pool,burst,1783618707168181000,1783618714885080000,7716,openshell-warm-pool-774vf,ok,,,,, +5.5,warm-pool,burst,1783618707168181000,1783618716632121000,9463,openshell-warm-pool-slh9g,ok,,,,, diff --git a/experiments/results/2026-07-09-rosa-hcp-422/warm-pool-default.csv b/experiments/results/2026-07-09-rosa-hcp-422/warm-pool-default.csv new file mode 100644 index 0000000000..8309c78225 --- /dev/null +++ b/experiments/results/2026-07-09-rosa-hcp-422/warm-pool-default.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,warm-pool,default,1783658631891629000,1783658634224797000,2333,openshell-warm-pool-fg426,ok,,,,, +2,warm-pool,default,1783658638280222000,1783658640588590000,2308,openshell-warm-pool-6z4hr,ok,,,,, +3,warm-pool,default,1783658644669403000,1783658647005891000,2336,openshell-warm-pool-2rk2j,ok,,,,, +4,warm-pool,default,1783658651135375000,1783658653493102000,2357,openshell-warm-pool-qhfwb,ok,,,,, +5,warm-pool,default,1783658657534501000,1783658659896792000,2362,openshell-warm-pool-8bwz7,ok,,,,, +6,warm-pool,default,1783658663836120000,1783658666150439000,2314,openshell-warm-pool-s94bn,ok,,,,, +7,warm-pool,default,1783658670286770000,1783658672594546000,2307,openshell-warm-pool-wmgrv,ok,,,,, +8,warm-pool,default,1783658676586189000,1783658678915653000,2329,openshell-warm-pool-9bgrl,ok,,,,, +9,warm-pool,default,1783658682920281000,1783658685308988000,2388,openshell-warm-pool-w624n,ok,,,,, +10,warm-pool,default,1783658689391486000,1783658691762469000,2370,openshell-warm-pool-klq2c,ok,,,,, diff --git a/experiments/results/2026-07-10-rosa-hcp-422/cold-start-prepulled.csv b/experiments/results/2026-07-10-rosa-hcp-422/cold-start-prepulled.csv new file mode 100644 index 0000000000..f03d0616be --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/cold-start-prepulled.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,cold-start,prepulled,1783666646834294000,1783666658190829000,11356,cold-prepulled-run-1,ok,4000,11000,8000,9000, +2,cold-start,prepulled,1783666662507817000,1783666678367951000,15860,cold-prepulled-run-2,ok,4000,15000,11000,13000, +3,cold-start,prepulled,1783666682739693000,1783666698328065000,15588,cold-prepulled-run-3,ok,4000,15000,12000,13000, +4,cold-start,prepulled,1783666702493130000,1783666718524604000,16031,cold-prepulled-run-4,ok,4000,15000,12000,13000, +5,cold-start,prepulled,1783666722746476000,1783666738661448000,15914,cold-prepulled-run-5,ok,4000,15000,12000,13000, +6,cold-start,prepulled,1783666742998751000,1783666759631752000,16633,cold-prepulled-run-6,ok,4000,16000,13000,14000, +7,cold-start,prepulled,1783666763997086000,1783666779706870000,15709,cold-prepulled-run-7,ok,4000,15000,12000,14000, +8,cold-start,prepulled,1783666784075155000,1783666799782968000,15707,cold-prepulled-run-8,ok,3000,14000,11000,13000, +9,cold-start,prepulled,1783666804192808000,1783666869217024000,65024,cold-prepulled-run-9,ok,3000,65000,60000,62000, +10,cold-start,prepulled,1783666873521349000,1783666892133155000,18611,cold-prepulled-run-10,ok,4000,18000,14000,16000, diff --git a/experiments/results/2026-07-10-rosa-hcp-422/cold-start-vanilla.csv b/experiments/results/2026-07-10-rosa-hcp-422/cold-start-vanilla.csv new file mode 100644 index 0000000000..3c57811f5a --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/cold-start-vanilla.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,cold-start,vanilla,1783666903289972000,1783666906563387000,3273,vanilla-run-1,ok,1000,2000,2000,2000, +2,cold-start,vanilla,1783666910585652000,1783666913459170000,2873,vanilla-run-2,ok,1000,1000,2000,2000, +3,cold-start,vanilla,1783666917471900000,1783666920300200000,2828,vanilla-run-3,ok,1000,1000,1000,1000, +4,cold-start,vanilla,1783666924323675000,1783666927264177000,2940,vanilla-run-4,ok,1000,1000,1000,1000, +5,cold-start,vanilla,1783666931307590000,1783666934177695000,2870,vanilla-run-5,ok,1000,1000,1000,1000, +6,cold-start,vanilla,1783666938230443000,1783666941106116000,2875,vanilla-run-6,ok,1000,1000,1000,1000, +7,cold-start,vanilla,1783666945165476000,1783666948100536000,2935,vanilla-run-7,ok,1000,1000,1000,1000, +8,cold-start,vanilla,1783666952178579000,1783666955071523000,2892,vanilla-run-8,ok,1000,1000,1000,1000, +9,cold-start,vanilla,1783666959125551000,1783666961989534000,2863,vanilla-run-9,ok,1000,1000,1000,1000, +10,cold-start,vanilla,1783666966078838000,1783666968958072000,2879,vanilla-run-10,ok,1000,1000,1000,1000, diff --git a/experiments/results/2026-07-10-rosa-hcp-422/env-injection-allowed.csv b/experiments/results/2026-07-10-rosa-hcp-422/env-injection-allowed.csv new file mode 100644 index 0000000000..869ab1de1d --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/env-injection-allowed.csv @@ -0,0 +1,6 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,adoption +1,env-injection,allowed,1783667286304290000,1783667297247287000,10942,unknown,ok,unknown +2,env-injection,allowed,1783667301207272000,1783667319680230000,18472,unknown,ok,unknown +3,env-injection,allowed,1783667323635709000,1783667340391980000,16756,unknown,ok,unknown +4,env-injection,allowed,1783667344370919000,1783667357170344000,12799,unknown,ok,unknown +5,env-injection,allowed,1783667361151936000,1783667372389368000,11237,unknown,ok,unknown diff --git a/experiments/results/2026-07-10-rosa-hcp-422/env-injection-disallowed.csv b/experiments/results/2026-07-10-rosa-hcp-422/env-injection-disallowed.csv new file mode 100644 index 0000000000..981d4f1036 --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/env-injection-disallowed.csv @@ -0,0 +1,6 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,behavior +1,env-injection,disallowed,1783667382443814000,,,,timeout,timeout +2,env-injection,disallowed,1783667506162860000,,,,timeout,timeout +3,env-injection,disallowed,1783667629428152000,,,,timeout,timeout +4,env-injection,disallowed,1783667753515090000,,,,timeout,timeout +5,env-injection,disallowed,1783667881036508000,,,,timeout,timeout diff --git a/experiments/results/2026-07-10-rosa-hcp-422/readiness-gates.csv b/experiments/results/2026-07-10-rosa-hcp-422/readiness-gates.csv new file mode 100644 index 0000000000..68a176ef0a --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/readiness-gates.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,readiness-gates,readiness-gate,1783667195595024000,1783667198605471000,3010,readiness-gate-run-1,ok,,,,, +2,readiness-gates,readiness-gate,1783667204048964000,1783667207302292000,3253,readiness-gate-run-2,ok,,,,, +3,readiness-gates,readiness-gate,1783667212936143000,1783667216139542000,3203,readiness-gate-run-3,ok,,,,, +4,readiness-gates,readiness-gate,1783667221676409000,1783667224718097000,3041,readiness-gate-run-4,ok,,,,, +5,readiness-gates,readiness-gate,1783667230083260000,1783667233161610000,3078,readiness-gate-run-5,ok,,,,, +6,readiness-gates,readiness-gate,1783667238673009000,1783667239783897000,1110,readiness-gate-run-6,ok,,,,, +7,readiness-gates,readiness-gate,1783667245195038000,1783667248373883000,3178,readiness-gate-run-7,ok,,,,, +8,readiness-gates,readiness-gate,1783667253868875000,1783667256852435000,2983,readiness-gate-run-8,ok,,,,, +9,readiness-gates,readiness-gate,1783667262444654000,1783667265479250000,3034,readiness-gate-run-9,ok,,,,, +10,readiness-gates,readiness-gate,1783667270830232000,1783667273908536000,3078,readiness-gate-run-10,ok,,,,, diff --git a/experiments/results/2026-07-10-rosa-hcp-422/warm-pool-burst.csv b/experiments/results/2026-07-10-rosa-hcp-422/warm-pool-burst.csv new file mode 100644 index 0000000000..d316ee83d1 --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/warm-pool-burst.csv @@ -0,0 +1,26 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1.1,warm-pool,burst,1783667096852006000,1783667099296580000,2444,openshell-warm-pool-btqt6,ok,,,,, +1.2,warm-pool,burst,1783667096852006000,1783667101253995000,4401,openshell-warm-pool-64dhs,ok,,,,, +1.3,warm-pool,burst,1783667096852006000,1783667103245356000,6393,openshell-warm-pool-5l9t7,ok,,,,, +1.4,warm-pool,burst,1783667096852006000,1783667105247873000,8395,openshell-warm-pool-72nd5,ok,,,,, +1.5,warm-pool,burst,1783667096852006000,1783667107237234000,10385,openshell-warm-pool-h2czr,ok,,,,, +2.1,warm-pool,burst,1783667115916330000,1783667118361667000,2445,openshell-warm-pool-wnzw9,ok,,,,, +2.2,warm-pool,burst,1783667115916330000,1783667120407752000,4491,openshell-warm-pool-4mkhf,ok,,,,, +2.3,warm-pool,burst,1783667115916330000,1783667122329346000,6413,openshell-warm-pool-mjd9f,ok,,,,, +2.4,warm-pool,burst,1783667115916330000,1783667124157857000,8241,openshell-warm-pool-lxt47,ok,,,,, +2.5,warm-pool,burst,1783667115916330000,1783667126203495000,10287,openshell-warm-pool-pt487,ok,,,,, +3.1,warm-pool,burst,1783667133546234000,1783667135973587000,2427,openshell-warm-pool-sd2fn,ok,,,,, +3.2,warm-pool,burst,1783667133546234000,1783667137946068000,4399,openshell-warm-pool-xv5w2,ok,,,,, +3.3,warm-pool,burst,1783667133546234000,1783667139992898000,6446,openshell-warm-pool-xtllc,ok,,,,, +3.4,warm-pool,burst,1783667133546234000,1783667142023273000,8477,openshell-warm-pool-7qt2n,ok,,,,, +3.5,warm-pool,burst,1783667133546234000,1783667143977281000,10431,openshell-warm-pool-sqfwz,ok,,,,, +4.1,warm-pool,burst,1783667148367564000,1783667150919699000,2552,openshell-warm-pool-pmjnm,ok,,,,, +4.2,warm-pool,burst,1783667148367564000,1783667153000621000,4633,openshell-warm-pool-nqspv,ok,,,,, +4.3,warm-pool,burst,1783667148367564000,1783667155078714000,6711,openshell-warm-pool-l6p8d,ok,,,,, +4.4,warm-pool,burst,1783667148367564000,1783667157103780000,8736,openshell-warm-pool-jkhkd,ok,,,,, +4.5,warm-pool,burst,1783667148367564000,1783667159127148000,10759,openshell-warm-pool-76k6v,ok,,,,, +5.1,warm-pool,burst,1783667167779018000,1783667170229892000,2450,openshell-warm-pool-wfdct,ok,,,,, +5.2,warm-pool,burst,1783667167779018000,1783667172198361000,4419,openshell-warm-pool-h6rrg,ok,,,,, +5.3,warm-pool,burst,1783667167779018000,1783667174259529000,6480,openshell-warm-pool-9s84q,ok,,,,, +5.4,warm-pool,burst,1783667167779018000,1783667176332160000,8553,openshell-warm-pool-z4f9r,ok,,,,, +5.5,warm-pool,burst,1783667167779018000,1783667178277483000,10498,openshell-warm-pool-mbg5b,ok,,,,, diff --git a/experiments/results/2026-07-10-rosa-hcp-422/warm-pool-default.csv b/experiments/results/2026-07-10-rosa-hcp-422/warm-pool-default.csv new file mode 100644 index 0000000000..f876d75e77 --- /dev/null +++ b/experiments/results/2026-07-10-rosa-hcp-422/warm-pool-default.csv @@ -0,0 +1,11 @@ +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +1,warm-pool,default,1783667022052469000,1783667024386431000,2333,openshell-warm-pool-2qbms,ok,,,,, +2,warm-pool,default,1783667028466534000,1783667030781082000,2314,openshell-warm-pool-tb5wn,ok,,,,, +3,warm-pool,default,1783667034875706000,1783667037257763000,2382,openshell-warm-pool-c49z8,ok,,,,, +4,warm-pool,default,1783667041323879000,1783667043673421000,2349,openshell-warm-pool-tp4jb,ok,,,,, +5,warm-pool,default,1783667047760215000,1783667050090514000,2330,openshell-warm-pool-wvkbd,ok,,,,, +6,warm-pool,default,1783667054198824000,1783667056543543000,2344,openshell-warm-pool-cp7hc,ok,,,,, +7,warm-pool,default,1783667060587897000,1783667062880277000,2292,openshell-warm-pool-842q6,ok,,,,, +8,warm-pool,default,1783667066942158000,1783667069254950000,2312,openshell-warm-pool-8ltck,ok,,,,, +9,warm-pool,default,1783667073291270000,1783667075561739000,2270,openshell-warm-pool-5m2m2,ok,,,,, +10,warm-pool,default,1783667079606913000,1783667081941015000,2334,openshell-warm-pool-gghg2,ok,,,,, diff --git a/experiments/results/README.md b/experiments/results/README.md new file mode 100644 index 0000000000..a688a1a8ac --- /dev/null +++ b/experiments/results/README.md @@ -0,0 +1,40 @@ +# Experiment Results + +Each subdirectory contains one complete test run, named by date and cluster +configuration. + +## Directory naming convention + +``` +YYYY-MM-DD-- +``` + +Example: `2026-07-09-rosa-hcp-422` is a run on ROSA HCP with OpenShift 4.22. + +## CSV files per run + +| File | Experiment | Description | +|------|-----------|-------------| +| `cold-start-prepulled.csv` | Phase 3 | OpenShell cold start, images pre-pulled | +| `cold-start-noprepull.csv` | Phase 3 | OpenShell cold start, no image pre-pull | +| `cold-start-vanilla.csv` | Phase 3 | Vanilla Agent Sandbox (pause image, K8s baseline) | +| `warm-pool-default.csv` | Phase 4 | Warm pool claim, default probes | +| `warm-pool-burst.csv` | Phase 4 | Warm pool burst (5 simultaneous claims) | +| `readiness-gates.csv` | Phase 5 | Readiness gate condition flip latency | +| `env-injection-allowed.csv` | Phase 6 | Env var injection with Allowed policy | +| `env-injection-disallowed.csv` | Phase 6 | Env var injection with Disallowed policy | + +## CSV columns + +``` +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status,scheduled_ms,pulled_ms,created_ms,started_ms,phase5_ms +``` + +- `delta_ms`: End-to-end latency from create to ready (milliseconds) +- `status`: `ok`, `timeout`, `create-failed`, `pool-exhausted`, `bind-timeout` +- Phase columns: Per-phase deltas from pod events (when available) + +## Reproducing a run + +See the [setup instructions](../RESULTS.md#how-to-reproduce) in the main +results document. diff --git a/experiments/results/events-prepulled-run-1.json b/experiments/results/events-prepulled-run-1.json new file mode 100644 index 0000000000..a5a2e80ccd --- /dev/null +++ b/experiments/results/events-prepulled-run-1.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:57:30.397104Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19223", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-1 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:30Z", + "name": "cold-prepulled-run-1.18c0db5e46a1713c", + "namespace": "openshell", + "resourceVersion": "19244", + "uid": "24391081-fc7a-485b-b6dd-813d431eb51c" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:32Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:32Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-e5cf0d61-11a3-4c63-a789-2d50a09b0122\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:32Z", + "name": "cold-prepulled-run-1.18c0db5ed024d8ae", + "namespace": "openshell", + "resourceVersion": "19257", + "uid": "bbb28b35-eed3-4a46-a7b0-a0034e355d7a" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:33Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:33Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:33Z", + "name": "cold-prepulled-run-1.18c0db5f1574d715", + "namespace": "openshell", + "resourceVersion": "19266", + "uid": "495cde4a-4c1e-4a41-af39-1f56e473390c" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:33Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19249", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:33Z", + "message": "Add eth0 [10.130.0.20/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:33Z", + "name": "cold-prepulled-run-1.18c0db5f017dc930", + "namespace": "openshell", + "resourceVersion": "19261", + "uid": "5589bd3c-016e-4891-afa2-3cc5ac91b94a" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:33Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:33Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 294ms (294ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:33Z", + "name": "cold-prepulled-run-1.18c0db5f156d0325", + "namespace": "openshell", + "resourceVersion": "19265", + "uid": "3d23d2fe-f4a1-48ac-a3a3-0de0ad6c06b2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:34Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:34Z", + "name": "cold-prepulled-run-1.18c0db5f2a9dad60", + "namespace": "openshell", + "resourceVersion": "19269", + "uid": "6ea8ad6e-09a4-4772-99fa-3861a3c33dce" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:34Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 286ms (286ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:34Z", + "name": "cold-prepulled-run-1.18c0db5f2684b89f", + "namespace": "openshell", + "resourceVersion": "19267", + "uid": "85038f8c-45b0-4b62-91b1-42a5be099b9c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:34Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:34Z", + "name": "cold-prepulled-run-1.18c0db5f2a2ecedc", + "namespace": "openshell", + "resourceVersion": "19268", + "uid": "a999cd5a-173d-42cd-9781-b888264aab1e" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:35Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:35Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 319ms (319ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:35Z", + "name": "cold-prepulled-run-1.18c0db5f5f959fe2", + "namespace": "openshell", + "resourceVersion": "19276", + "uid": "002cfe70-9640-4dda-ae81-089316c9eefc" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:35Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:35Z", + "name": "cold-prepulled-run-1.18c0db5f5f9ddcf7", + "namespace": "openshell", + "resourceVersion": "19277", + "uid": "c748a27c-e791-44f1-9c99-0f627e53e51b" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:35Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 281ms (281ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:35Z", + "name": "cold-prepulled-run-1.18c0db5f70687aed", + "namespace": "openshell", + "resourceVersion": "19278", + "uid": "8c8a5796-1402-40dd-83f9-ee06aaa08591" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:35Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:35Z", + "name": "cold-prepulled-run-1.18c0db5f755b652a", + "namespace": "openshell", + "resourceVersion": "19279", + "uid": "06d8173c-3a32-4097-975b-1a91c8393078" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:35Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:35Z", + "name": "cold-prepulled-run-1.18c0db5f75ca44dc", + "namespace": "openshell", + "resourceVersion": "19280", + "uid": "678ff81f-afb6-46ed-a54a-5e033e5932e6" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:33Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:36Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:33Z", + "name": "cold-prepulled-run-1.18c0db5f03daf161", + "namespace": "openshell", + "resourceVersion": "19288", + "uid": "ff91b25e-1066-4f2b-889c-c37bd6e0c1ee" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:36Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:36Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 310ms (310ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:36Z", + "name": "cold-prepulled-run-1.18c0db5f9af88a2a", + "namespace": "openshell", + "resourceVersion": "19284", + "uid": "04df9d44-2859-491a-a25b-c88d17b31a24" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:37Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-1", + "namespace": "openshell", + "resourceVersion": "19243", + "uid": "9c60fc6c-0294-4926-b5d2-3598e3f01b98" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:37Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 281ms (281ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:37Z", + "name": "cold-prepulled-run-1.18c0db5fd50e8392", + "namespace": "openshell", + "resourceVersion": "19289", + "uid": "1e23a846-59ed-44bf-8b9a-a5575b7eebb6" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-10.json b/experiments/results/events-prepulled-run-10.json new file mode 100644 index 0000000000..f8b13dd152 --- /dev/null +++ b/experiments/results/events-prepulled-run-10.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:01:17.145165Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20476", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-10 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:17Z", + "name": "cold-prepulled-run-10.18c0db9311de57cc", + "namespace": "openshell", + "resourceVersion": "20498", + "uid": "90f9eab7-deb9-45e1-84f8-b1d46d942bdf" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:19Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-0695cd62-4644-446b-8205-f9cf71d3c68c\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:19Z", + "name": "cold-prepulled-run-10.18c0db939e1431d2", + "namespace": "openshell", + "resourceVersion": "20509", + "uid": "d8774c36-555b-4a37-8545-91371c44d12e" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:26Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:26Z", + "name": "cold-prepulled-run-10.18c0db9558ad7e3c", + "namespace": "openshell", + "resourceVersion": "20533", + "uid": "27746b15-f376-4623-a26e-36731f7cf271" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20501", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:26Z", + "message": "Add eth0 [10.130.0.30/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:26Z", + "name": "cold-prepulled-run-10.18c0db9543e3428d", + "namespace": "openshell", + "resourceVersion": "20528", + "uid": "c6e4e265-e3ab-4ed0-94b6-ea2515b9c29a" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 308ms (308ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:26Z", + "name": "cold-prepulled-run-10.18c0db9558a5718b", + "namespace": "openshell", + "resourceVersion": "20532", + "uid": "1a9860d4-915a-4165-9765-27212de9a93b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:27Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:27Z", + "name": "cold-prepulled-run-10.18c0db959766551d", + "namespace": "openshell", + "resourceVersion": "20540", + "uid": "39a3fb65-f688-482b-8e25-80931dfe912a" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:27Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 985ms (985ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:27Z", + "name": "cold-prepulled-run-10.18c0db959365dfc8", + "namespace": "openshell", + "resourceVersion": "20537", + "uid": "a6ecf076-cc84-49bd-b5ec-3df17bb56a7d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:27Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:27Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:27Z", + "name": "cold-prepulled-run-10.18c0db9596f6245c", + "namespace": "openshell", + "resourceVersion": "20539", + "uid": "631b89a5-e020-4e16-a844-eea0e3731726" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:29Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:29Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 298ms (298ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:29Z", + "name": "cold-prepulled-run-10.18c0db95d93cf1d7", + "namespace": "openshell", + "resourceVersion": "20545", + "uid": "ce733630-2fb2-4905-8e8e-0073a833eeff" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:29Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:29Z", + "name": "cold-prepulled-run-10.18c0db95d944cfc8", + "namespace": "openshell", + "resourceVersion": "20546", + "uid": "69751f8f-5bac-41cc-8f96-81ea6bfb0a8f" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:29Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 299ms (299ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:29Z", + "name": "cold-prepulled-run-10.18c0db95eb1cc260", + "namespace": "openshell", + "resourceVersion": "20548", + "uid": "a82c9c58-ff01-4be4-81c2-843568ec86e8" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:29Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:29Z", + "name": "cold-prepulled-run-10.18c0db95efd41066", + "namespace": "openshell", + "resourceVersion": "20549", + "uid": "6a908a44-f143-4ec9-a67a-9cefc7190df2" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:29Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:29Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:29Z", + "name": "cold-prepulled-run-10.18c0db95f0427aba", + "namespace": "openshell", + "resourceVersion": "20550", + "uid": "529f7d7f-9285-42ba-bc5a-99128442ad8e" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:30Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:26Z", + "name": "cold-prepulled-run-10.18c0db954648ead2", + "namespace": "openshell", + "resourceVersion": "20558", + "uid": "77245c9a-4935-4c7f-b625-524b71c15390" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:30Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:30Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 346ms (346ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:30Z", + "name": "cold-prepulled-run-10.18c0db961812ec0d", + "namespace": "openshell", + "resourceVersion": "20555", + "uid": "ceedac93-3ca6-46d4-86f5-794f2227dfe3" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:31Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-10", + "namespace": "openshell", + "resourceVersion": "20497", + "uid": "49e5a3a6-54e8-4b25-8957-5a7274eb0ee5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:31Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 359ms (359ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:31Z", + "name": "cold-prepulled-run-10.18c0db9654b23a46", + "namespace": "openshell", + "resourceVersion": "20560", + "uid": "2e93b2ee-3e41-4128-8aed-d95f6c709533" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-2.json b/experiments/results/events-prepulled-run-2.json new file mode 100644 index 0000000000..1dd34052bc --- /dev/null +++ b/experiments/results/events-prepulled-run-2.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:57:46.076737Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19330", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-2 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:46Z", + "name": "cold-prepulled-run-2.18c0db61ed35af15", + "namespace": "openshell", + "resourceVersion": "19351", + "uid": "0c98455f-92fb-4471-9628-5aa5709bdd0b" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:49Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:49Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-51ee37ff-fa32-4fa5-8dcd-fd7093ae6a13\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:49Z", + "name": "cold-prepulled-run-2.18c0db62c2855c48", + "namespace": "openshell", + "resourceVersion": "19372", + "uid": "e499aa8b-c397-445d-b712-a72c448098f8" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:53Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db63bac20564", + "namespace": "openshell", + "resourceVersion": "19396", + "uid": "d1961581-664c-4aaa-abbf-e70afd3c37db" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19354", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:53Z", + "message": "Add eth0 [10.130.0.21/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db638fc70cb7", + "namespace": "openshell", + "resourceVersion": "19388", + "uid": "b5588aef-4828-43fa-a9e5-84a5f59f4689" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:53Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 304ms (304ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db63a51b581b", + "namespace": "openshell", + "resourceVersion": "19392", + "uid": "9f78c054-d5e2-4072-911f-a5226aadf417" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:53Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db63a522e8f0", + "namespace": "openshell", + "resourceVersion": "19393", + "uid": "776d00ce-2edf-4978-83b7-36dfcc6f59aa" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:53Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 281ms (281ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db63b5ece617", + "namespace": "openshell", + "resourceVersion": "19394", + "uid": "f44eb14e-cab5-49fd-a8d0-7a5eab2d9225" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:53Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db63ba52878d", + "namespace": "openshell", + "resourceVersion": "19395", + "uid": "f312ffea-07fa-4553-9697-2695eb25aaf1" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:54Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:54Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 285ms (285ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:54Z", + "name": "cold-prepulled-run-2.18c0db63ceaa43ca", + "namespace": "openshell", + "resourceVersion": "19400", + "uid": "eebba494-116f-4078-86c8-dfe3ad6c9b37" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:55Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:55Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 299ms (299ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:55Z", + "name": "cold-prepulled-run-2.18c0db640b792f96", + "namespace": "openshell", + "resourceVersion": "19405", + "uid": "ee86412d-70b6-4f3d-ad81-8cf98e65e0a6" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:55Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:55Z", + "name": "cold-prepulled-run-2.18c0db640b80dd96", + "namespace": "openshell", + "resourceVersion": "19406", + "uid": "efec3ebc-fc9f-4f28-90ea-f4f273775807" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:55Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 294ms (294ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:55Z", + "name": "cold-prepulled-run-2.18c0db641d14b20e", + "namespace": "openshell", + "resourceVersion": "19409", + "uid": "3b467c8f-d6d3-44b7-99fe-678c25c47cae" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:55Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:55Z", + "name": "cold-prepulled-run-2.18c0db64225226a3", + "namespace": "openshell", + "resourceVersion": "19410", + "uid": "38d47608-ed89-40f6-9dd5-28f0aa4fef43" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:55Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:55Z", + "name": "cold-prepulled-run-2.18c0db6422c28aac", + "namespace": "openshell", + "resourceVersion": "19411", + "uid": "c757fe06-6476-417f-8b51-40f2f010b14b" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:53Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:56Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:53Z", + "name": "cold-prepulled-run-2.18c0db6392fa9199", + "namespace": "openshell", + "resourceVersion": "19419", + "uid": "ec37bc80-a044-4835-8452-c64b7163af63" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:56Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:56Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 286ms (286ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:56Z", + "name": "cold-prepulled-run-2.18c0db6446992f99", + "namespace": "openshell", + "resourceVersion": "19416", + "uid": "dc642bd8-e677-4a4b-ad11-9f7f59f19147" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:57:57Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-2", + "namespace": "openshell", + "resourceVersion": "19350", + "uid": "795a41d7-ef29-4b96-ba40-1e6d8cdc09c1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:57:57Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 289ms (289ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:57:57Z", + "name": "cold-prepulled-run-2.18c0db64829856fd", + "namespace": "openshell", + "resourceVersion": "19420", + "uid": "d81f1889-9112-479c-af73-eafa984ce4d5" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-3.json b/experiments/results/events-prepulled-run-3.json new file mode 100644 index 0000000000..d10e41463e --- /dev/null +++ b/experiments/results/events-prepulled-run-3.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:58:06.373434Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19449", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-3 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:06Z", + "name": "cold-prepulled-run-3.18c0db66a6fcb270", + "namespace": "openshell", + "resourceVersion": "19470", + "uid": "f35a81f7-9f41-4259-ade8-e34335518eb5" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:08Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:08Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-7c346c92-a75d-4a67-bc40-640a2c55e5c6\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:08Z", + "name": "cold-prepulled-run-3.18c0db6732e62740", + "namespace": "openshell", + "resourceVersion": "19483", + "uid": "25767bcf-3e1d-4450-9baf-b1127fe204b2" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:13Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:13Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:13Z", + "name": "cold-prepulled-run-3.18c0db685f251424", + "namespace": "openshell", + "resourceVersion": "19514", + "uid": "b28b0f51-40fb-416a-9dd6-e29f94ceb5a4" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:13Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19473", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:13Z", + "message": "Add eth0 [10.130.0.22/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:13Z", + "name": "cold-prepulled-run-3.18c0db684a38959e", + "namespace": "openshell", + "resourceVersion": "19510", + "uid": "83f8f348-c951-4854-90c8-166da5814b70" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:13Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:13Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 309ms (309ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:13Z", + "name": "cold-prepulled-run-3.18c0db685f1daa5f", + "namespace": "openshell", + "resourceVersion": "19513", + "uid": "d4396c0b-be34-4984-804a-aa0cbf8f4faa" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:14Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:14Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:14Z", + "name": "cold-prepulled-run-3.18c0db68748f0795", + "namespace": "openshell", + "resourceVersion": "19520", + "uid": "d1a1f3b9-6cca-42a9-a685-200c6bf6b546" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:14Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:14Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 278ms (278ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:14Z", + "name": "cold-prepulled-run-3.18c0db686fc0dbcf", + "namespace": "openshell", + "resourceVersion": "19518", + "uid": "9a9c5a11-191f-41b3-b577-8d1246217d15" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:14Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:14Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:14Z", + "name": "cold-prepulled-run-3.18c0db68741f0295", + "namespace": "openshell", + "resourceVersion": "19519", + "uid": "5baad451-8934-481b-bf91-7472bce32c91" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:15Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:15Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 307ms (307ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:15Z", + "name": "cold-prepulled-run-3.18c0db68b988262f", + "namespace": "openshell", + "resourceVersion": "19525", + "uid": "25c738c9-5a0e-4e87-9156-5c082a5637fd" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:15Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:15Z", + "name": "cold-prepulled-run-3.18c0db68b98fd584", + "namespace": "openshell", + "resourceVersion": "19526", + "uid": "5bffba7f-610c-4551-9c20-c7945bc6cda0" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:15Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 339ms (339ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:15Z", + "name": "cold-prepulled-run-3.18c0db68cdd24ef7", + "namespace": "openshell", + "resourceVersion": "19527", + "uid": "b6712292-33f7-4a11-a239-545951d05fb9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:15Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:15Z", + "name": "cold-prepulled-run-3.18c0db68d299fa42", + "namespace": "openshell", + "resourceVersion": "19528", + "uid": "a9e945c9-b430-4b0e-b740-0388460576cf" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:15Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:15Z", + "name": "cold-prepulled-run-3.18c0db68d30c0013", + "namespace": "openshell", + "resourceVersion": "19529", + "uid": "13fc33a7-bca6-457d-8b22-ec5975461af6" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:13Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:16Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:13Z", + "name": "cold-prepulled-run-3.18c0db684ca7ab68", + "namespace": "openshell", + "resourceVersion": "19536", + "uid": "22747519-7c22-40aa-ad27-5304177333d6" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:16Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:16Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 341ms (341ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:16Z", + "name": "cold-prepulled-run-3.18c0db68f781b458", + "namespace": "openshell", + "resourceVersion": "19533", + "uid": "eb61fb5b-3edc-45fc-bc0a-98006d996ec0" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:17Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-3", + "namespace": "openshell", + "resourceVersion": "19469", + "uid": "9b007bfb-dc1d-4db9-9250-2d370066e38a" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:17Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 304ms (304ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:17Z", + "name": "cold-prepulled-run-3.18c0db6931297153", + "namespace": "openshell", + "resourceVersion": "19537", + "uid": "83c45ef2-f5b9-441d-891a-2dc5f138b636" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-4.json b/experiments/results/events-prepulled-run-4.json new file mode 100644 index 0000000000..343b352e9c --- /dev/null +++ b/experiments/results/events-prepulled-run-4.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:58:26.082359Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19563", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-4 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:26Z", + "name": "cold-prepulled-run-4.18c0db6b3dbb00d4", + "namespace": "openshell", + "resourceVersion": "19584", + "uid": "bc316927-ab26-4551-8f38-f4fd5f4b49a8" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:28Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-6e2534d4-7046-40bc-85fb-cb8962b060c8\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:28Z", + "name": "cold-prepulled-run-4.18c0db6bcb4de911", + "namespace": "openshell", + "resourceVersion": "19599", + "uid": "1bae0a6e-de05-4e37-bf9d-ee287e49bd2b" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:34Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d3ed00fe9", + "namespace": "openshell", + "resourceVersion": "19643", + "uid": "30f379a7-7507-4fa3-b7c8-82218d10d5bc" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19587", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:34Z", + "message": "Add eth0 [10.130.0.23/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d16aad6c5", + "namespace": "openshell", + "resourceVersion": "19633", + "uid": "8bbed02c-8049-4bd7-a23a-c7bdae1dbda0" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:34Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 281ms (281ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d29df3f16", + "namespace": "openshell", + "resourceVersion": "19637", + "uid": "0697da02-d2cb-4c85-8fcd-150f02b86a14" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:34Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d29e7526c", + "namespace": "openshell", + "resourceVersion": "19638", + "uid": "0a8f8e0d-ae49-48d0-84bf-fc562f23d2a5" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:34Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 276ms (276ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d3a5fda09", + "namespace": "openshell", + "resourceVersion": "19640", + "uid": "acb39492-5c07-4d2b-9b0d-12ca5ab6b88b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:34Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d3e66ce8e", + "namespace": "openshell", + "resourceVersion": "19642", + "uid": "a45ab017-936f-4f04-a10a-172c9f630ea1" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:35Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:35Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 275ms (275ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:35Z", + "name": "cold-prepulled-run-4.18c0db6d6431cc73", + "namespace": "openshell", + "resourceVersion": "19651", + "uid": "527db380-b8fd-4417-ace8-57cfaaf7aa62" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:35Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:35Z", + "name": "cold-prepulled-run-4.18c0db6d643970ad", + "namespace": "openshell", + "resourceVersion": "19652", + "uid": "6c6dfbb5-921a-4a5b-a63f-fce0b14f873c" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:35Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 328ms (328ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:35Z", + "name": "cold-prepulled-run-4.18c0db6d77d0fce8", + "namespace": "openshell", + "resourceVersion": "19653", + "uid": "959eb4c1-f96b-49cc-829b-03d459208cc2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:35Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:35Z", + "name": "cold-prepulled-run-4.18c0db6d7c795b22", + "namespace": "openshell", + "resourceVersion": "19654", + "uid": "9b48ab97-f400-4b57-a98d-1b6cee530ba7" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:35Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:35Z", + "name": "cold-prepulled-run-4.18c0db6d7cebe0be", + "namespace": "openshell", + "resourceVersion": "19655", + "uid": "b2c50f4b-fd1f-41fd-b867-d0714fe1bf60" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:36Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:36Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 289ms (289ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:36Z", + "name": "cold-prepulled-run-4.18c0db6da0fe1ba6", + "namespace": "openshell", + "resourceVersion": "19660", + "uid": "f1868789-3ccb-46d8-a098-61cfd5c0c0a7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:37Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:34Z", + "name": "cold-prepulled-run-4.18c0db6d191bbd53", + "namespace": "openshell", + "resourceVersion": "19663", + "uid": "2d9f6393-f88a-4095-af99-749db73f0349" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:37Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-4", + "namespace": "openshell", + "resourceVersion": "19583", + "uid": "6e009a8d-41b9-490d-8f7f-c9c1ec3dae97" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:37Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 287ms (287ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:37Z", + "name": "cold-prepulled-run-4.18c0db6ddcd3a96d", + "namespace": "openshell", + "resourceVersion": "19666", + "uid": "87089862-8a44-4781-b73f-ba437b9f219c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-5.json b/experiments/results/events-prepulled-run-5.json new file mode 100644 index 0000000000..5c964bc718 --- /dev/null +++ b/experiments/results/events-prepulled-run-5.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:58:46.341889Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19694", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-5 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:46Z", + "name": "cold-prepulled-run-5.18c0db6ff54ae642", + "namespace": "openshell", + "resourceVersion": "19713", + "uid": "89062763-0f58-4faf-ab5a-8886f37ec127" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:48Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:48Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-5f1785a9-54a4-46d3-8b77-0a35a48ed4c3\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:48Z", + "name": "cold-prepulled-run-5.18c0db7081c3f780", + "namespace": "openshell", + "resourceVersion": "19726", + "uid": "211f84d3-085f-4bf7-84d0-d15ca6787d6f" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:53Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19716", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:53Z", + "message": "Add eth0 [10.130.0.24/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:53Z", + "name": "cold-prepulled-run-5.18c0db71bcd3cbef", + "namespace": "openshell", + "resourceVersion": "19752", + "uid": "728934e7-5f59-4146-922b-d719439fc016" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:54Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:54Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:54Z", + "name": "cold-prepulled-run-5.18c0db71eb0bac27", + "namespace": "openshell", + "resourceVersion": "19761", + "uid": "cfaf015a-4b50-419e-8149-62acd4939fb6" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:54Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:54Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 357ms (357ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:54Z", + "name": "cold-prepulled-run-5.18c0db71d47f07b2", + "namespace": "openshell", + "resourceVersion": "19757", + "uid": "1319fd7e-3247-435c-875e-5dca6df1cb40" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:54Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:54Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:54Z", + "name": "cold-prepulled-run-5.18c0db71d486fa33", + "namespace": "openshell", + "resourceVersion": "19758", + "uid": "780db7be-dd94-41be-811a-bd30c51f50b9" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:54Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:54Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 303ms (303ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:54Z", + "name": "cold-prepulled-run-5.18c0db71e6a30fdc", + "namespace": "openshell", + "resourceVersion": "19759", + "uid": "04987c1e-2430-49b7-80a2-b44a72784fd8" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:54Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:54Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:54Z", + "name": "cold-prepulled-run-5.18c0db71ea9e8045", + "namespace": "openshell", + "resourceVersion": "19760", + "uid": "ee00a891-d2d3-4b6d-9106-16fb622cebae" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:55Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:55Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 309ms (309ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:55Z", + "name": "cold-prepulled-run-5.18c0db721389a135", + "namespace": "openshell", + "resourceVersion": "19770", + "uid": "0551814a-fb1f-4210-ab0f-64680c176074" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:55Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:55Z", + "name": "cold-prepulled-run-5.18c0db721390cf31", + "namespace": "openshell", + "resourceVersion": "19771", + "uid": "e9e9bc4c-fab7-4900-927a-c8c52a54bca6" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:55Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 340ms (340ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:55Z", + "name": "cold-prepulled-run-5.18c0db7227d57772", + "namespace": "openshell", + "resourceVersion": "19773", + "uid": "59f139ac-43d5-41ec-9d56-ba3da08171cf" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:55Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:55Z", + "name": "cold-prepulled-run-5.18c0db722cfc257b", + "namespace": "openshell", + "resourceVersion": "19774", + "uid": "e2458525-42c6-47e4-b5ba-7b95a957bd24" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:55Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:55Z", + "name": "cold-prepulled-run-5.18c0db722d77fa72", + "namespace": "openshell", + "resourceVersion": "19775", + "uid": "904d5619-7bea-481f-8e21-909b4eba66fb" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:56Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:56Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 354ms (354ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:56Z", + "name": "cold-prepulled-run-5.18c0db72522e4e6d", + "namespace": "openshell", + "resourceVersion": "19780", + "uid": "9a908642-b0d4-43f7-b050-2e7ed0624c16" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:54Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:57Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:54Z", + "name": "cold-prepulled-run-5.18c0db71bf2f63c2", + "namespace": "openshell", + "resourceVersion": "19784", + "uid": "637acaf0-921a-4f95-a571-fd5cf968edba" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:58:57Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-5", + "namespace": "openshell", + "resourceVersion": "19712", + "uid": "1dd11bb7-bb70-4ac8-9d25-3d47ed01b306" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:58:57Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 288ms (288ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:58:57Z", + "name": "cold-prepulled-run-5.18c0db728a4c0019", + "namespace": "openshell", + "resourceVersion": "19785", + "uid": "d8d1c6ee-ea53-440d-95f7-7955538cb82d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-6.json b/experiments/results/events-prepulled-run-6.json new file mode 100644 index 0000000000..fdc698d18f --- /dev/null +++ b/experiments/results/events-prepulled-run-6.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:59:06.674552Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19818", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-6 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:06Z", + "name": "cold-prepulled-run-6.18c0db74b136ba71", + "namespace": "openshell", + "resourceVersion": "19838", + "uid": "5867909d-730c-4210-ade1-1a52b6be31c1" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:09Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:09Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-42eb2a56-f623-4ff7-8974-8b7101c9816b\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:09Z", + "name": "cold-prepulled-run-6.18c0db753fd84149", + "namespace": "openshell", + "resourceVersion": "19849", + "uid": "61fe1910-d53c-4c11-842e-0e71f9dce6af" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:14Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:14Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:14Z", + "name": "cold-prepulled-run-6.18c0db76928316d8", + "namespace": "openshell", + "resourceVersion": "19881", + "uid": "efc19a50-55d8-4e2d-b41a-aa02060a4380" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:14Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19841", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:14Z", + "message": "Add eth0 [10.130.0.25/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:14Z", + "name": "cold-prepulled-run-6.18c0db7677beda23", + "namespace": "openshell", + "resourceVersion": "19876", + "uid": "e67ce9d0-0c62-4b27-a86b-16c856593915" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:14Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:14Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 411ms (411ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:14Z", + "name": "cold-prepulled-run-6.18c0db76927b552e", + "namespace": "openshell", + "resourceVersion": "19880", + "uid": "e827a85a-bdbb-4c27-b7c2-989c6da76fc7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:15Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:15Z", + "name": "cold-prepulled-run-6.18c0db76aa46c0bf", + "namespace": "openshell", + "resourceVersion": "19884", + "uid": "22343443-ef1d-46f7-82d9-d1809dc48604" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:15Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 327ms (327ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:15Z", + "name": "cold-prepulled-run-6.18c0db76a60445de", + "namespace": "openshell", + "resourceVersion": "19882", + "uid": "6f55d3b1-c487-459e-97fb-0ce7408bf07e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:15Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:15Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:15Z", + "name": "cold-prepulled-run-6.18c0db76a9da816a", + "namespace": "openshell", + "resourceVersion": "19883", + "uid": "917f5881-040e-4237-9e80-1564e17cfbf6" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:15Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:15Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 287ms (287ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:15Z", + "name": "cold-prepulled-run-6.18c0db76bfbb98e2", + "namespace": "openshell", + "resourceVersion": "19889", + "uid": "4c8ef720-8707-4d15-873c-a4d690f464b2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:16Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:16Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 307ms (307ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:16Z", + "name": "cold-prepulled-run-6.18c0db76fce6d6b9", + "namespace": "openshell", + "resourceVersion": "19892", + "uid": "083a6398-992a-4f9b-8908-3c7f375c8936" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:16Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:16Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:16Z", + "name": "cold-prepulled-run-6.18c0db76fceddece", + "namespace": "openshell", + "resourceVersion": "19893", + "uid": "345dd1e7-7d5a-4b6b-9b98-48f258abaccb" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:16Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:16Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 315ms (315ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:16Z", + "name": "cold-prepulled-run-6.18c0db770fc3615f", + "namespace": "openshell", + "resourceVersion": "19894", + "uid": "d6967824-048d-4403-af06-c0561a4f0052" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:16Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:16Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:16Z", + "name": "cold-prepulled-run-6.18c0db7714bac594", + "namespace": "openshell", + "resourceVersion": "19896", + "uid": "7193edb7-62c3-43e0-9e26-83e6e23bd836" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:16Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:16Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:16Z", + "name": "cold-prepulled-run-6.18c0db771526e795", + "namespace": "openshell", + "resourceVersion": "19897", + "uid": "a468fd1d-ff9e-4fda-aa66-708ff9b48d4f" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:17Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:17Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 301ms (301ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:17Z", + "name": "cold-prepulled-run-6.18c0db773883dc7a", + "namespace": "openshell", + "resourceVersion": "19904", + "uid": "046b2fd5-d920-49f4-bd71-f2687a0c82aa" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:14Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:18Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:14Z", + "name": "cold-prepulled-run-6.18c0db7679fb115f", + "namespace": "openshell", + "resourceVersion": "19907", + "uid": "224a8c8c-0c27-4f3d-96b7-96c8e21e4edf" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-6", + "namespace": "openshell", + "resourceVersion": "19837", + "uid": "f6a8630e-fca4-42ae-a91c-92eea4b33514" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:18Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 279ms (279ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:18Z", + "name": "cold-prepulled-run-6.18c0db77730ff212", + "namespace": "openshell", + "resourceVersion": "19908", + "uid": "e5d68e9a-c11c-4d33-a7bd-01560137444b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-7.json b/experiments/results/events-prepulled-run-7.json new file mode 100644 index 0000000000..f4647968c2 --- /dev/null +++ b/experiments/results/events-prepulled-run-7.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:59:27.560253Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19942", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-7 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:27Z", + "name": "cold-prepulled-run-7.18c0db798e193d41", + "namespace": "openshell", + "resourceVersion": "19965", + "uid": "baa3bc41-78ee-4345-b5a0-712bf7e1c54b" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:29Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:29Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-b78e740f-6c90-40b3-92c2-425cb055eeb2\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:29Z", + "name": "cold-prepulled-run-7.18c0db7a1833dddc", + "namespace": "openshell", + "resourceVersion": "19977", + "uid": "e51aa863-d473-4617-a0c2-cdcf6ff964a7" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:34Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:34Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:34Z", + "name": "cold-prepulled-run-7.18c0db7b44af54cf", + "namespace": "openshell", + "resourceVersion": "20006", + "uid": "f0d13db2-9e8a-4690-b834-45d53ac84b19" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19968", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:34Z", + "message": "Add eth0 [10.130.0.26/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:34Z", + "name": "cold-prepulled-run-7.18c0db7b314136a3", + "namespace": "openshell", + "resourceVersion": "20002", + "uid": "b7ea7425-0fde-4d3a-8c1b-909e1ddc5a85" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:34Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 283ms (283ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:34Z", + "name": "cold-prepulled-run-7.18c0db7b44a84185", + "namespace": "openshell", + "resourceVersion": "20005", + "uid": "c729bbfc-6ba3-4f1e-85c9-8b72ce43f746" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:35Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:35Z", + "name": "cold-prepulled-run-7.18c0db7b5b03ff2e", + "namespace": "openshell", + "resourceVersion": "20010", + "uid": "6119cde5-da6a-4933-9d36-0380dc0e9822" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:35Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 306ms (306ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:35Z", + "name": "cold-prepulled-run-7.18c0db7b56f0c9c4", + "namespace": "openshell", + "resourceVersion": "20008", + "uid": "5487c8a0-6c06-40ae-9851-004cfcf20d0c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:35Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:35Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:35Z", + "name": "cold-prepulled-run-7.18c0db7b5a96a832", + "namespace": "openshell", + "resourceVersion": "20009", + "uid": "97867279-7b4d-4d56-bb6d-8cae52d41037" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:35Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:35Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 352ms (352ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:35Z", + "name": "cold-prepulled-run-7.18c0db7b71233d07", + "namespace": "openshell", + "resourceVersion": "20015", + "uid": "55238ef3-5034-4df3-97ee-0cd26502a388" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:36Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:36Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 317ms (317ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:36Z", + "name": "cold-prepulled-run-7.18c0db7bab10a575", + "namespace": "openshell", + "resourceVersion": "20018", + "uid": "1684a74a-aef9-42af-9668-77c9fe1430e6" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:36Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:36Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:36Z", + "name": "cold-prepulled-run-7.18c0db7bab184076", + "namespace": "openshell", + "resourceVersion": "20019", + "uid": "af5e9f62-835a-4b62-a8ed-068cb0d5db98" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:36Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:36Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 279ms (279ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:36Z", + "name": "cold-prepulled-run-7.18c0db7bbbc21bf4", + "namespace": "openshell", + "resourceVersion": "20021", + "uid": "e871fb0c-1283-4890-9ccd-ec5d683534d5" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:36Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:36Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:36Z", + "name": "cold-prepulled-run-7.18c0db7bc089a41a", + "namespace": "openshell", + "resourceVersion": "20022", + "uid": "d063783d-c1e4-44bb-b0b0-59d69aaf49ac" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:37Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:37Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:37Z", + "name": "cold-prepulled-run-7.18c0db7bc0f8b8a7", + "namespace": "openshell", + "resourceVersion": "20023", + "uid": "b847b398-45b6-49bd-b231-9f1ef869bde4" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:37Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:37Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 303ms (303ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:37Z", + "name": "cold-prepulled-run-7.18c0db7be639bb1c", + "namespace": "openshell", + "resourceVersion": "20030", + "uid": "08dd04b4-1986-41d3-93eb-12c1ddc50d65" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:38Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:34Z", + "name": "cold-prepulled-run-7.18c0db7b33bc321f", + "namespace": "openshell", + "resourceVersion": "20033", + "uid": "ce5f7704-91ae-499c-aaa3-ce2fbde06568" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:38Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-7", + "namespace": "openshell", + "resourceVersion": "19964", + "uid": "1569dcf4-bb1f-4d72-896a-d2fb497e09ac" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:38Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 289ms (289ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:38Z", + "name": "cold-prepulled-run-7.18c0db7c21448985", + "namespace": "openshell", + "resourceVersion": "20034", + "uid": "c6e4fe2e-4a5b-4dff-ae63-61c5dbce4759" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-8.json b/experiments/results/events-prepulled-run-8.json new file mode 100644 index 0000000000..22278b31d9 --- /dev/null +++ b/experiments/results/events-prepulled-run-8.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T06:59:47.697314Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20061", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-8 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:47Z", + "name": "cold-prepulled-run-8.18c0db7e3e5c69a6", + "namespace": "openshell", + "resourceVersion": "20086", + "uid": "149e74a4-bace-4948-a1a1-42630485a68a" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:50Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:50Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-cfd9ae70-6c95-435f-87cd-31dbd3334b94\" ", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:50Z", + "name": "cold-prepulled-run-8.18c0db7ec91290de", + "namespace": "openshell", + "resourceVersion": "20100", + "uid": "6826bc37-2682-4ecc-a761-f5bdff9db0d2" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:55Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db801e760587", + "namespace": "openshell", + "resourceVersion": "20136", + "uid": "e2cd1b22-8cd6-49d2-9343-da354687d3d2" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20089", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:55Z", + "message": "Add eth0 [10.130.0.27/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db7ff309f0f2", + "namespace": "openshell", + "resourceVersion": "20124", + "uid": "76debe0e-6c34-405a-8a65-54ebd238735a" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:55Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 287ms (287ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db800677b88c", + "namespace": "openshell", + "resourceVersion": "20130", + "uid": "1dd5d20a-43c5-4b76-8170-845fcb996505" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:55Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db8006814b61", + "namespace": "openshell", + "resourceVersion": "20131", + "uid": "79270e80-27ac-4d34-a311-e51d012b0757" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:55Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 332ms (332ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db801a56c152", + "namespace": "openshell", + "resourceVersion": "20134", + "uid": "f54d6980-9f79-406f-9d69-ed4edae50c50" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:55Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db801e06d392", + "namespace": "openshell", + "resourceVersion": "20135", + "uid": "66374702-b806-41e2-a9cc-f7905b5fbb74" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:56Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:56Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 289ms (289ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:56Z", + "name": "cold-prepulled-run-8.18c0db8056e0bee0", + "namespace": "openshell", + "resourceVersion": "20142", + "uid": "f29ac695-00f8-4036-90a8-60706081a4c9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:56Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:56Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:56Z", + "name": "cold-prepulled-run-8.18c0db8056e8f44e", + "namespace": "openshell", + "resourceVersion": "20143", + "uid": "53bc618c-8092-47db-8016-15d9922e75bb" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:57Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:57Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 334ms (334ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:57Z", + "name": "cold-prepulled-run-8.18c0db806ade3aef", + "namespace": "openshell", + "resourceVersion": "20145", + "uid": "f3ee2659-c994-4280-9fcf-3a0974f0ef19" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:57Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:57Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:57Z", + "name": "cold-prepulled-run-8.18c0db806fcc5bf1", + "namespace": "openshell", + "resourceVersion": "20146", + "uid": "d75da731-0933-42c5-8128-e49c4369fac5" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:57Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:57Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:57Z", + "name": "cold-prepulled-run-8.18c0db8070396d44", + "namespace": "openshell", + "resourceVersion": "20147", + "uid": "1ac6db85-4a94-45b3-904d-64b1e2281c7c" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:57Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:57Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 300ms (300ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:57Z", + "name": "cold-prepulled-run-8.18c0db8093791162", + "namespace": "openshell", + "resourceVersion": "20152", + "uid": "b7310163-116b-43ca-a678-2e981c98b556" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:55Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:58Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:55Z", + "name": "cold-prepulled-run-8.18c0db7ff553e188", + "namespace": "openshell", + "resourceVersion": "20155", + "uid": "9a0b5032-aa49-4de9-b975-d4bb3bf9ce59" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T06:59:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-8", + "namespace": "openshell", + "resourceVersion": "20085", + "uid": "28a6d633-0e41-48a2-8f5c-c10fb2cdf06b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T06:59:58Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 314ms (314ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T06:59:58Z", + "name": "cold-prepulled-run-8.18c0db80d02a655e", + "namespace": "openshell", + "resourceVersion": "20156", + "uid": "ba62047a-d500-4166-bc61-91f006f085c7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-prepulled-run-9.json b/experiments/results/events-prepulled-run-9.json new file mode 100644 index 0000000000..e00d0be4a2 --- /dev/null +++ b/experiments/results/events-prepulled-run-9.json @@ -0,0 +1,524 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:00:07.767109Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20199", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/cold-prepulled-run-9 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:00:07Z", + "name": "cold-prepulled-run-9.18c0db82ea9d3086", + "namespace": "openshell", + "resourceVersion": "20230", + "uid": "4030bbf5-c88d-480a-a5fd-f0bd73f90848" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:00:10Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:00:10Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-9cff8802-2a69-4307-8987-6b670376eee5\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:00:10Z", + "name": "cold-prepulled-run-9.18c0db83788d3189", + "namespace": "openshell", + "resourceVersion": "20242", + "uid": "372fcca3-e113-4d1f-b51d-a3532392f324" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:00:15Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20233", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:00:15Z", + "message": "Add eth0 [10.130.0.29/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:00:15Z", + "name": "cold-prepulled-run-9.18c0db84b216c711", + "namespace": "openshell", + "resourceVersion": "20270", + "uid": "520943ae-d7a4-4988-bad2-566713f688b0" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:00:58Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:00:58Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:00:58Z", + "name": "cold-prepulled-run-9.18c0db8eba75fc23", + "namespace": "openshell", + "resourceVersion": "20407", + "uid": "dd2defdf-f7d8-4ea0-9a51-62698695cdda" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:00:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:00:58Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 43.051s (43.051s including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:00:58Z", + "name": "cold-prepulled-run-9.18c0db8eba6ea633", + "namespace": "openshell", + "resourceVersion": "20406", + "uid": "5515b7e5-f7a3-4fe9-a13a-3e399e4ca127" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:04Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:04Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:04Z", + "name": "cold-prepulled-run-9.18c0db900f34c88b", + "namespace": "openshell", + "resourceVersion": "20423", + "uid": "dd9905cc-4e2d-46fa-9f19-e9b4b9d9dcc8" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:04Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:04Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 5.638s (5.638s including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:04Z", + "name": "cold-prepulled-run-9.18c0db900a86009a", + "namespace": "openshell", + "resourceVersion": "20421", + "uid": "9fbfe9b6-80c6-4d7c-a77d-c74225a39a32" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:04Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:04Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:04Z", + "name": "cold-prepulled-run-9.18c0db900ec74dbc", + "namespace": "openshell", + "resourceVersion": "20422", + "uid": "bb697643-530f-4f0c-b02c-c313afa4c725" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:06Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:06Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 1.455s (1.455s including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:06Z", + "name": "cold-prepulled-run-9.18c0db90822f23ee", + "namespace": "openshell", + "resourceVersion": "20437", + "uid": "b9dfccce-8efe-4f35-8651-26cac70c4941" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:06Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:06Z", + "name": "cold-prepulled-run-9.18c0db908237f219", + "namespace": "openshell", + "resourceVersion": "20438", + "uid": "de1045b4-8b82-4d8b-b731-6779585cd1cb" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:06Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 309ms (309ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:06Z", + "name": "cold-prepulled-run-9.18c0db9094aa4623", + "namespace": "openshell", + "resourceVersion": "20440", + "uid": "8914a4a1-faca-482c-b364-760479b6a3ff" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:06Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:06Z", + "name": "cold-prepulled-run-9.18c0db90999a3c56", + "namespace": "openshell", + "resourceVersion": "20441", + "uid": "ddc20841-2e0d-406a-a5d9-241849931d6a" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:06Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:06Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:06Z", + "name": "cold-prepulled-run-9.18c0db909a0af5c0", + "namespace": "openshell", + "resourceVersion": "20442", + "uid": "c83f84e9-4988-4376-b036-6f20140a45a3" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 4, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:00:15Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:08Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:00:15Z", + "name": "cold-prepulled-run-9.18c0db84b4603040", + "namespace": "openshell", + "resourceVersion": "20452", + "uid": "33c1e597-4611-41b5-b7eb-dc837cfa95fa" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:08Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:08Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 1.723s (1.723s including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:08Z", + "name": "cold-prepulled-run-9.18c0db9109e6b879", + "namespace": "openshell", + "resourceVersion": "20451", + "uid": "c2a5f01c-8d27-480a-aba1-f0d0e1cff564" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:09Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "cold-prepulled-run-9", + "namespace": "openshell", + "resourceVersion": "20229", + "uid": "1928f5bc-1c8d-432c-8206-e4a3f6324675" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:09Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 345ms (345ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:09Z", + "name": "cold-prepulled-run-9.18c0db912f65e2d3", + "namespace": "openshell", + "resourceVersion": "20454", + "uid": "511a9b90-d6cf-42eb-9786-a76b676c5199" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-1.json b/experiments/results/events-vanilla-run-1.json new file mode 100644 index 0000000000..411207e719 --- /dev/null +++ b/experiments/results/events-vanilla-run-1.json @@ -0,0 +1,201 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:01:44.621839Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-1", + "namespace": "openshell", + "resourceVersion": "20614", + "uid": "0664b352-03d2-4bf8-969d-e69c1d86a3be" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-1 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:44Z", + "name": "vanilla-run-1.18c0db99779b0b9d", + "namespace": "openshell", + "resourceVersion": "20617", + "uid": "97a74daa-9821-4705-b0e4-47b0ce887b9b" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:45Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-1", + "namespace": "openshell", + "resourceVersion": "20621", + "uid": "0664b352-03d2-4bf8-969d-e69c1d86a3be" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:45Z", + "message": "Add eth0 [10.130.0.31/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:45Z", + "name": "vanilla-run-1.18c0db9991028cad", + "namespace": "openshell", + "resourceVersion": "20625", + "uid": "718bada0-377e-42d3-987b-bb8b5b1d1822" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:45Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-1", + "namespace": "openshell", + "resourceVersion": "20615", + "uid": "0664b352-03d2-4bf8-969d-e69c1d86a3be" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:45Z", + "message": "Pulling image \"registry.k8s.io/pause:3.10\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:45Z", + "name": "vanilla-run-1.18c0db99934c48de", + "namespace": "openshell", + "resourceVersion": "20627", + "uid": "6fca1482-0db6-4d71-aab8-ef635cab382a" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:45Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-1", + "namespace": "openshell", + "resourceVersion": "20615", + "uid": "0664b352-03d2-4bf8-969d-e69c1d86a3be" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:45Z", + "message": "Successfully pulled image \"registry.k8s.io/pause:3.10\" in 528ms (528ms including waiting). Image size: 742080 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:45Z", + "name": "vanilla-run-1.18c0db99b2c6fb61", + "namespace": "openshell", + "resourceVersion": "20629", + "uid": "67fb92ce-a64c-45cb-bd13-6df9d517eca4" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:45Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-1", + "namespace": "openshell", + "resourceVersion": "20615", + "uid": "0664b352-03d2-4bf8-969d-e69c1d86a3be" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:45Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:45Z", + "name": "vanilla-run-1.18c0db99b6564c9e", + "namespace": "openshell", + "resourceVersion": "20630", + "uid": "c5049102-b4ce-4700-ad13-24c1e395d285" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:45Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-1", + "namespace": "openshell", + "resourceVersion": "20615", + "uid": "0664b352-03d2-4bf8-969d-e69c1d86a3be" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:45Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:45Z", + "name": "vanilla-run-1.18c0db99b6c677a7", + "namespace": "openshell", + "resourceVersion": "20631", + "uid": "9dfb1f07-4a96-40bf-bd5d-640798f51db9" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-10.json b/experiments/results/events-vanilla-run-10.json new file mode 100644 index 0000000000..8892d33153 --- /dev/null +++ b/experiments/results/events-vanilla-run-10.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:47.014758Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-10", + "namespace": "openshell", + "resourceVersion": "21005", + "uid": "c5345490-4825-4116-ae2f-83095ce15eb3" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-10 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:47Z", + "name": "vanilla-run-10.18c0dba7fe83720c", + "namespace": "openshell", + "resourceVersion": "21008", + "uid": "739f0890-8bfd-47d1-9a57-444cd63b19d1" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:47Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-10", + "namespace": "openshell", + "resourceVersion": "21011", + "uid": "c5345490-4825-4116-ae2f-83095ce15eb3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:47Z", + "message": "Add eth0 [10.130.0.40/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:47Z", + "name": "vanilla-run-10.18c0dba817bb4312", + "namespace": "openshell", + "resourceVersion": "21016", + "uid": "fde2f471-16a3-4b5e-a13e-f700dd553741" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:47Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-10", + "namespace": "openshell", + "resourceVersion": "21006", + "uid": "c5345490-4825-4116-ae2f-83095ce15eb3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:47Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:47Z", + "name": "vanilla-run-10.18c0dba819f8c019", + "namespace": "openshell", + "resourceVersion": "21018", + "uid": "e96a162d-8294-4825-8f51-4a09869b7788" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:47Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-10", + "namespace": "openshell", + "resourceVersion": "21006", + "uid": "c5345490-4825-4116-ae2f-83095ce15eb3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:47Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:47Z", + "name": "vanilla-run-10.18c0dba81cd2b7a3", + "namespace": "openshell", + "resourceVersion": "21019", + "uid": "f959b3b3-cb3b-48cd-94d2-5c7e3e70d4c7" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:47Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-10", + "namespace": "openshell", + "resourceVersion": "21006", + "uid": "c5345490-4825-4116-ae2f-83095ce15eb3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:47Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:47Z", + "name": "vanilla-run-10.18c0dba81d42ac55", + "namespace": "openshell", + "resourceVersion": "21020", + "uid": "982038ef-515b-4c3f-8ce6-00a317df7595" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-2.json b/experiments/results/events-vanilla-run-2.json new file mode 100644 index 0000000000..53cfab6058 --- /dev/null +++ b/experiments/results/events-vanilla-run-2.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:01:51.497855Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-2", + "namespace": "openshell", + "resourceVersion": "20658", + "uid": "a3e81a6a-a644-4d77-987c-a39ee18bbd63" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-2 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:51Z", + "name": "vanilla-run-2.18c0db9b1172c133", + "namespace": "openshell", + "resourceVersion": "20661", + "uid": "a46e2a60-1260-4156-a685-9bfd6be52077" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:51Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-2", + "namespace": "openshell", + "resourceVersion": "20665", + "uid": "a3e81a6a-a644-4d77-987c-a39ee18bbd63" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:51Z", + "message": "Add eth0 [10.130.0.32/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:51Z", + "name": "vanilla-run-2.18c0db9b2aef7774", + "namespace": "openshell", + "resourceVersion": "20668", + "uid": "98273ecc-7502-487f-a6ff-cc3ee2f1e582" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:51Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-2", + "namespace": "openshell", + "resourceVersion": "20659", + "uid": "a3e81a6a-a644-4d77-987c-a39ee18bbd63" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:51Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:51Z", + "name": "vanilla-run-2.18c0db9b2d170e04", + "namespace": "openshell", + "resourceVersion": "20670", + "uid": "7774d4be-478b-48e2-9294-6eb635cb0074" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:52Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-2", + "namespace": "openshell", + "resourceVersion": "20659", + "uid": "a3e81a6a-a644-4d77-987c-a39ee18bbd63" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:52Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:52Z", + "name": "vanilla-run-2.18c0db9b30ba853c", + "namespace": "openshell", + "resourceVersion": "20671", + "uid": "d08df898-fd27-423e-8492-a6eab1fba199" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:52Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-2", + "namespace": "openshell", + "resourceVersion": "20659", + "uid": "a3e81a6a-a644-4d77-987c-a39ee18bbd63" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:52Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:52Z", + "name": "vanilla-run-2.18c0db9b3130d94a", + "namespace": "openshell", + "resourceVersion": "20673", + "uid": "4bba2cbe-02a5-4fa2-976c-f72589dca7de" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-3.json b/experiments/results/events-vanilla-run-3.json new file mode 100644 index 0000000000..a3ba574dc7 --- /dev/null +++ b/experiments/results/events-vanilla-run-3.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:01:58.367046Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-3", + "namespace": "openshell", + "resourceVersion": "20709", + "uid": "a06435ad-2d9b-4f3d-8652-9fd8d47ec82e" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-3 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:58Z", + "name": "vanilla-run-3.18c0db9caae2482d", + "namespace": "openshell", + "resourceVersion": "20712", + "uid": "8856ea20-de63-4395-a746-97efc5a83510" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-3", + "namespace": "openshell", + "resourceVersion": "20716", + "uid": "a06435ad-2d9b-4f3d-8652-9fd8d47ec82e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:58Z", + "message": "Add eth0 [10.130.0.33/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:58Z", + "name": "vanilla-run-3.18c0db9cc47fc3b3", + "namespace": "openshell", + "resourceVersion": "20717", + "uid": "d9b8c1c8-9d3a-49e3-9fef-9fab59f6aafb" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:58Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-3", + "namespace": "openshell", + "resourceVersion": "20710", + "uid": "a06435ad-2d9b-4f3d-8652-9fd8d47ec82e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:58Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:58Z", + "name": "vanilla-run-3.18c0db9cc6d9bcee", + "namespace": "openshell", + "resourceVersion": "20719", + "uid": "6f2b12b5-992e-4829-b71e-6ad53c91e901" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:58Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-3", + "namespace": "openshell", + "resourceVersion": "20710", + "uid": "a06435ad-2d9b-4f3d-8652-9fd8d47ec82e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:58Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:58Z", + "name": "vanilla-run-3.18c0db9cc9d6ee00", + "namespace": "openshell", + "resourceVersion": "20720", + "uid": "4a78e1eb-fb60-44e7-af3d-c20ff03fbeac" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:01:58Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-3", + "namespace": "openshell", + "resourceVersion": "20710", + "uid": "a06435ad-2d9b-4f3d-8652-9fd8d47ec82e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:01:58Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:01:58Z", + "name": "vanilla-run-3.18c0db9cca4849e2", + "namespace": "openshell", + "resourceVersion": "20721", + "uid": "e93c2695-cc1f-4d57-af70-da1723fc22ae" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-4.json b/experiments/results/events-vanilla-run-4.json new file mode 100644 index 0000000000..036f5999a9 --- /dev/null +++ b/experiments/results/events-vanilla-run-4.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:05.283833Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-4", + "namespace": "openshell", + "resourceVersion": "20758", + "uid": "f0121a7f-7bb0-4e56-ad02-577b17b4c295" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-4 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:05Z", + "name": "vanilla-run-4.18c0db9e47280e81", + "namespace": "openshell", + "resourceVersion": "20762", + "uid": "a5196523-b128-471b-a7b0-a0f36920dda6" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:05Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-4", + "namespace": "openshell", + "resourceVersion": "20767", + "uid": "f0121a7f-7bb0-4e56-ad02-577b17b4c295" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:05Z", + "message": "Add eth0 [10.130.0.34/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:05Z", + "name": "vanilla-run-4.18c0db9e616912b8", + "namespace": "openshell", + "resourceVersion": "20769", + "uid": "73e60ab2-1569-4530-8cda-a011b7ae5e1a" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-4", + "namespace": "openshell", + "resourceVersion": "20760", + "uid": "f0121a7f-7bb0-4e56-ad02-577b17b4c295" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:05Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:05Z", + "name": "vanilla-run-4.18c0db9e63974995", + "namespace": "openshell", + "resourceVersion": "20771", + "uid": "ac99bc04-9fe3-4ca5-98d7-cebac85e8f66" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-4", + "namespace": "openshell", + "resourceVersion": "20760", + "uid": "f0121a7f-7bb0-4e56-ad02-577b17b4c295" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:05Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:05Z", + "name": "vanilla-run-4.18c0db9e66715d9a", + "namespace": "openshell", + "resourceVersion": "20773", + "uid": "a4aed472-8e3b-4566-ad51-bbf02478292c" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:05Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-4", + "namespace": "openshell", + "resourceVersion": "20760", + "uid": "f0121a7f-7bb0-4e56-ad02-577b17b4c295" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:05Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:05Z", + "name": "vanilla-run-4.18c0db9e66e5241e", + "namespace": "openshell", + "resourceVersion": "20774", + "uid": "040ddd64-f393-44f9-b8d2-daf7a1b4acce" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-5.json b/experiments/results/events-vanilla-run-5.json new file mode 100644 index 0000000000..e7fec7c609 --- /dev/null +++ b/experiments/results/events-vanilla-run-5.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:12.245579Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-5", + "namespace": "openshell", + "resourceVersion": "20801", + "uid": "7b4d41df-4755-418f-b118-d3b26b818bf5" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-5 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:12Z", + "name": "vanilla-run-5.18c0db9fe61bdb61", + "namespace": "openshell", + "resourceVersion": "20804", + "uid": "4bab2a7c-86ad-4078-a2a3-82b19ba929b5" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:12Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-5", + "namespace": "openshell", + "resourceVersion": "20808", + "uid": "7b4d41df-4755-418f-b118-d3b26b818bf5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:12Z", + "message": "Add eth0 [10.130.0.35/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:12Z", + "name": "vanilla-run-5.18c0db9fff63d47f", + "namespace": "openshell", + "resourceVersion": "20810", + "uid": "0d2171e2-d73b-488d-a8d1-a5e6830e2542" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:12Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-5", + "namespace": "openshell", + "resourceVersion": "20802", + "uid": "7b4d41df-4755-418f-b118-d3b26b818bf5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:12Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:12Z", + "name": "vanilla-run-5.18c0dba0018a0b46", + "namespace": "openshell", + "resourceVersion": "20812", + "uid": "8573d7e5-7fda-4e58-bd04-86356420c282" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:12Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-5", + "namespace": "openshell", + "resourceVersion": "20802", + "uid": "7b4d41df-4755-418f-b118-d3b26b818bf5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:12Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:12Z", + "name": "vanilla-run-5.18c0dba00489db47", + "namespace": "openshell", + "resourceVersion": "20814", + "uid": "8fd174b5-53f5-496c-8814-2dc4eb91c5ea" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:12Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-5", + "namespace": "openshell", + "resourceVersion": "20802", + "uid": "7b4d41df-4755-418f-b118-d3b26b818bf5" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:12Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:12Z", + "name": "vanilla-run-5.18c0dba004f8ebe4", + "namespace": "openshell", + "resourceVersion": "20815", + "uid": "578b986c-4721-4ca6-96f2-58c5fb10082d" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-6.json b/experiments/results/events-vanilla-run-6.json new file mode 100644 index 0000000000..e7c765d0a5 --- /dev/null +++ b/experiments/results/events-vanilla-run-6.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:19.145216Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-6", + "namespace": "openshell", + "resourceVersion": "20841", + "uid": "2c000ba7-a697-491d-8c19-a8d1611fda74" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-6 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:19Z", + "name": "vanilla-run-6.18c0dba1815bf123", + "namespace": "openshell", + "resourceVersion": "20845", + "uid": "7e895baf-1ae8-4016-a617-6683e7eb1c97" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-6", + "namespace": "openshell", + "resourceVersion": "20848", + "uid": "2c000ba7-a697-491d-8c19-a8d1611fda74" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:19Z", + "message": "Add eth0 [10.130.0.36/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:19Z", + "name": "vanilla-run-6.18c0dba19af6e869", + "namespace": "openshell", + "resourceVersion": "20850", + "uid": "52b93783-f3d0-4f60-bfef-a3c8f3aa2cd9" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-6", + "namespace": "openshell", + "resourceVersion": "20842", + "uid": "2c000ba7-a697-491d-8c19-a8d1611fda74" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:19Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:19Z", + "name": "vanilla-run-6.18c0dba19d3732f2", + "namespace": "openshell", + "resourceVersion": "20852", + "uid": "6df5e44d-0a7f-4f2d-ae58-5d305a0b6e37" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-6", + "namespace": "openshell", + "resourceVersion": "20842", + "uid": "2c000ba7-a697-491d-8c19-a8d1611fda74" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:19Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:19Z", + "name": "vanilla-run-6.18c0dba19fddb4f3", + "namespace": "openshell", + "resourceVersion": "20853", + "uid": "a341044a-f9d7-4d2a-ae4a-abf9bff3800a" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-6", + "namespace": "openshell", + "resourceVersion": "20842", + "uid": "2c000ba7-a697-491d-8c19-a8d1611fda74" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:19Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:19Z", + "name": "vanilla-run-6.18c0dba1a04ee274", + "namespace": "openshell", + "resourceVersion": "20854", + "uid": "a8875032-fbc1-45f8-88e7-905161fbc9f1" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-7.json b/experiments/results/events-vanilla-run-7.json new file mode 100644 index 0000000000..eeee044ece --- /dev/null +++ b/experiments/results/events-vanilla-run-7.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:26.126976Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-7", + "namespace": "openshell", + "resourceVersion": "20890", + "uid": "3789727c-6b4b-4927-b4bc-46e944c50689" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-7 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:26Z", + "name": "vanilla-run-7.18c0dba3218145a0", + "namespace": "openshell", + "resourceVersion": "20893", + "uid": "629c37f7-fd5d-483b-b1c5-174af51ccbee" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-7", + "namespace": "openshell", + "resourceVersion": "20896", + "uid": "3789727c-6b4b-4927-b4bc-46e944c50689" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:26Z", + "message": "Add eth0 [10.130.0.37/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:26Z", + "name": "vanilla-run-7.18c0dba33b102956", + "namespace": "openshell", + "resourceVersion": "20900", + "uid": "3fe0c3cb-8464-4952-a41f-bd762075eec6" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-7", + "namespace": "openshell", + "resourceVersion": "20891", + "uid": "3789727c-6b4b-4927-b4bc-46e944c50689" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:26Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:26Z", + "name": "vanilla-run-7.18c0dba33d462141", + "namespace": "openshell", + "resourceVersion": "20902", + "uid": "6201ad6f-7095-4fdf-a626-5813f1513269" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-7", + "namespace": "openshell", + "resourceVersion": "20891", + "uid": "3789727c-6b4b-4927-b4bc-46e944c50689" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:26Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:26Z", + "name": "vanilla-run-7.18c0dba34018eb00", + "namespace": "openshell", + "resourceVersion": "20903", + "uid": "bf5b06ae-349e-4394-9356-1bcd9f4f98cb" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:26Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-7", + "namespace": "openshell", + "resourceVersion": "20891", + "uid": "3789727c-6b4b-4927-b4bc-46e944c50689" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:26Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:26Z", + "name": "vanilla-run-7.18c0dba3408a841b", + "namespace": "openshell", + "resourceVersion": "20904", + "uid": "5e341dc1-afa9-4dc8-a7c7-cb38dfffcac1" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-8.json b/experiments/results/events-vanilla-run-8.json new file mode 100644 index 0000000000..55db454ea1 --- /dev/null +++ b/experiments/results/events-vanilla-run-8.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:33.124436Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-8", + "namespace": "openshell", + "resourceVersion": "20930", + "uid": "551988be-e658-499e-8d02-8fc342a7257e" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-8 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:33Z", + "name": "vanilla-run-8.18c0dba4c295e587", + "namespace": "openshell", + "resourceVersion": "20933", + "uid": "f1e43af0-6bcd-4b08-80e4-4b49ec1da4f8" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:33Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-8", + "namespace": "openshell", + "resourceVersion": "20937", + "uid": "551988be-e658-499e-8d02-8fc342a7257e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:33Z", + "message": "Add eth0 [10.130.0.38/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:33Z", + "name": "vanilla-run-8.18c0dba4dc3b22f0", + "namespace": "openshell", + "resourceVersion": "20938", + "uid": "6ec03c91-33ce-4b8d-86ac-e4b12ee3b21f" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:33Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-8", + "namespace": "openshell", + "resourceVersion": "20931", + "uid": "551988be-e658-499e-8d02-8fc342a7257e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:33Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:33Z", + "name": "vanilla-run-8.18c0dba4dec084f8", + "namespace": "openshell", + "resourceVersion": "20940", + "uid": "da404f4e-877b-46db-b284-77cb83db0072" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:33Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-8", + "namespace": "openshell", + "resourceVersion": "20931", + "uid": "551988be-e658-499e-8d02-8fc342a7257e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:33Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:33Z", + "name": "vanilla-run-8.18c0dba4e215fd1a", + "namespace": "openshell", + "resourceVersion": "20941", + "uid": "02668d52-68aa-4a83-aff8-20551353d87b" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:33Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-8", + "namespace": "openshell", + "resourceVersion": "20931", + "uid": "551988be-e658-499e-8d02-8fc342a7257e" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:33Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:33Z", + "name": "vanilla-run-8.18c0dba4e2a298b3", + "namespace": "openshell", + "resourceVersion": "20942", + "uid": "d4fba61d-9cc2-43b1-b69a-c9c7423f88fa" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-vanilla-run-9.json b/experiments/results/events-vanilla-run-9.json new file mode 100644 index 0000000000..5f64487459 --- /dev/null +++ b/experiments/results/events-vanilla-run-9.json @@ -0,0 +1,168 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:02:40.053764Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-9", + "namespace": "openshell", + "resourceVersion": "20969", + "uid": "b79c9600-7d00-409c-8492-97eb955bc120" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/vanilla-run-9 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:40Z", + "name": "vanilla-run-9.18c0dba65f9b19f6", + "namespace": "openshell", + "resourceVersion": "20972", + "uid": "8b6c1694-641d-43c8-87bc-4cc87abb590a" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:40Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "vanilla-run-9", + "namespace": "openshell", + "resourceVersion": "20975", + "uid": "b79c9600-7d00-409c-8492-97eb955bc120" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:40Z", + "message": "Add eth0 [10.130.0.39/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:40Z", + "name": "vanilla-run-9.18c0dba6799915a7", + "namespace": "openshell", + "resourceVersion": "20977", + "uid": "9cff5a48-893b-4987-92c2-c6282b441eb2" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:40Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-9", + "namespace": "openshell", + "resourceVersion": "20970", + "uid": "b79c9600-7d00-409c-8492-97eb955bc120" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:40Z", + "message": "Container image \"registry.k8s.io/pause:3.10\" already present on machine and can be accessed by the pod", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:40Z", + "name": "vanilla-run-9.18c0dba67bd417ff", + "namespace": "openshell", + "resourceVersion": "20979", + "uid": "b31585b6-7c31-4aef-ad0b-07d8d2816c6c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:40Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-9", + "namespace": "openshell", + "resourceVersion": "20970", + "uid": "b79c9600-7d00-409c-8492-97eb955bc120" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:40Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:40Z", + "name": "vanilla-run-9.18c0dba67f48b185", + "namespace": "openshell", + "resourceVersion": "20980", + "uid": "9a5f989d-7ad2-4eb8-a20b-3776030fe448" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:02:40Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{sandbox}", + "kind": "Pod", + "name": "vanilla-run-9", + "namespace": "openshell", + "resourceVersion": "20970", + "uid": "b79c9600-7d00-409c-8492-97eb955bc120" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:02:40Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:02:40Z", + "name": "vanilla-run-9.18c0dba67fbbd53b", + "namespace": "openshell", + "resourceVersion": "20981", + "uid": "f4c42142-b322-44b6-90b5-4b550d7bd438" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-1.json b/experiments/results/events-wp-default-run-1.json new file mode 100644 index 0000000000..29634c4807 --- /dev/null +++ b/experiments/results/events-wp-default-run-1.json @@ -0,0 +1,588 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:15.465771Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21105", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-2qbms to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:15Z", + "name": "openshell-warm-pool-2qbms.18c0dbae9e537b84", + "namespace": "openshell", + "resourceVersion": "21190", + "uid": "498b3686-a59f-4156-8170-7ff0938ed594" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-ab4feb6d-4c1a-454f-a64e-651ed6d3efd1\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-2qbms.18c0dbaf9f839ac1", + "namespace": "openshell", + "resourceVersion": "21262", + "uid": "e91d315b-a017-4036-a058-88c8b2798924" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-2qbms.18c0dbafdcfc67ee", + "namespace": "openshell", + "resourceVersion": "21290", + "uid": "44e71484-fb2c-4a04-85a0-5f9060dbd23d" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21194", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Add eth0 [10.130.0.41/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-2qbms.18c0dbafc5febdb4", + "namespace": "openshell", + "resourceVersion": "21274", + "uid": "c73b114a-960a-45d5-a1bf-a59a8ffb6bec" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 341ms (341ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-2qbms.18c0dbafdcf49fe6", + "namespace": "openshell", + "resourceVersion": "21288", + "uid": "a0195af8-cb6a-441f-97f6-b8a04d416735" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-2qbms.18c0dbaff76966e2", + "namespace": "openshell", + "resourceVersion": "21305", + "uid": "961cb47a-63e1-4bce-8801-4d2d0acb3f6d" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 352ms (352ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-2qbms.18c0dbaff1ffe230", + "namespace": "openshell", + "resourceVersion": "21299", + "uid": "0e30b40c-c881-442f-8833-360c8c831155" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-2qbms.18c0dbaff6d1de63", + "namespace": "openshell", + "resourceVersion": "21304", + "uid": "0435dffd-8040-4d97-9b81-5f4c587bd8a7" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 421ms (421ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-2qbms.18c0dbb01157c98e", + "namespace": "openshell", + "resourceVersion": "21325", + "uid": "2de695af-f69b-4f0c-a7fd-0b273a953a2b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 360ms (360ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-2qbms.18c0dbb04a0d4580", + "namespace": "openshell", + "resourceVersion": "21347", + "uid": "89e895e9-3b57-4df5-8129-d58f515c934b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-2qbms.18c0dbb04a14f956", + "namespace": "openshell", + "resourceVersion": "21348", + "uid": "705b17a8-d656-4c32-8c6c-949dbc35079d" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 326ms (326ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-2qbms.18c0dbb05d8fd5ac", + "namespace": "openshell", + "resourceVersion": "21352", + "uid": "57e73a7d-2ec0-47ce-84ee-88d2521fa136" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-2qbms.18c0dbb064462285", + "namespace": "openshell", + "resourceVersion": "21354", + "uid": "69c5e182-8b33-4331-a00a-f42e8eb71afa" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-2qbms.18c0dbb064bacbee", + "namespace": "openshell", + "resourceVersion": "21357", + "uid": "0fc2771a-7858-4e19-8215-1ac775570e92" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 378ms (378ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-2qbms.18c0dbb0877fd810", + "namespace": "openshell", + "resourceVersion": "21379", + "uid": "01438c68-2abc-4bcf-8f98-c3f627db5beb" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 328ms (328ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:24Z", + "name": "openshell-warm-pool-2qbms.18c0dbb0c084e0e1", + "namespace": "openshell", + "resourceVersion": "21386", + "uid": "5d5e3dbf-942e-4f49-9a9d-af257fa30e6f" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 6, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:43Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-2qbms.18c0dbafc89f5685", + "namespace": "openshell", + "resourceVersion": "21444", + "uid": "4f75633c-1b31-4762-960c-1378f6624b5a" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:43Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-2qbms", + "namespace": "openshell", + "resourceVersion": "21189", + "uid": "d955e7c2-df5f-40c5-8269-6cba64248643" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:43Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 371ms (371ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:43Z", + "name": "openshell-warm-pool-2qbms.18c0dbb51e419fa5", + "namespace": "openshell", + "resourceVersion": "21456", + "uid": "86ca3f26-7ed0-4b44-a37c-65eb5f7595bd" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-10.json b/experiments/results/events-wp-default-run-10.json new file mode 100644 index 0000000000..a671cd6e6e --- /dev/null +++ b/experiments/results/events-wp-default-run-10.json @@ -0,0 +1,588 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:04:18.255114Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22131", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-gghg2 to ip-10-0-5-178.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:18Z", + "name": "openshell-warm-pool-gghg2.18c0dbbd3cdcd6fa", + "namespace": "openshell", + "resourceVersion": "22178", + "uid": "8539c2c0-2529-4929-ad58-a3a0b468722b" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-1d7716b4-0e92-47e8-9707-25671e1be028\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe10733c43", + "namespace": "openshell", + "resourceVersion": "22232", + "uid": "1c3c3462-0e59-4305-bbbe-7705d63a23ad" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe53f1f027", + "namespace": "openshell", + "resourceVersion": "22257", + "uid": "bc5c39c3-3011-4432-9e77-f3a9a82aa9ba" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22183", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Add eth0 [10.128.0.44/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe3f1a8740", + "namespace": "openshell", + "resourceVersion": "22244", + "uid": "333f038e-fe81-43aa-bf0c-5cedebe44498" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 307ms (307ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe53ea97fb", + "namespace": "openshell", + "resourceVersion": "22256", + "uid": "859f8aa8-b6d9-4d8d-9828-ffe4f1952bdd" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:23Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:23Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe6ceaac8e", + "namespace": "openshell", + "resourceVersion": "22263", + "uid": "c26a88d7-5713-4a5b-88aa-64ef42034346" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 331ms (331ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:23Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe67b41e4e", + "namespace": "openshell", + "resourceVersion": "22260", + "uid": "867da0af-43ea-43e2-a2d3-3cb5617186a1" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:23Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:23Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe6c79119d", + "namespace": "openshell", + "resourceVersion": "22262", + "uid": "71232db4-a47c-4574-a1bf-6cb28225553e" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:23Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 332ms (332ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:23Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe855695d7", + "namespace": "openshell", + "resourceVersion": "22274", + "uid": "a07da132-91f3-4844-b502-4df83d7c8828" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 348ms (348ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:24Z", + "name": "openshell-warm-pool-gghg2.18c0dbbec248d8c2", + "namespace": "openshell", + "resourceVersion": "22298", + "uid": "46185a15-e4c8-43e6-b7d5-337426138684" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:24Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:24Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:24Z", + "name": "openshell-warm-pool-gghg2.18c0dbbec2508dff", + "namespace": "openshell", + "resourceVersion": "22299", + "uid": "52e93049-187c-4a52-ae28-7c2488b17463" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 384ms (384ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:25Z", + "name": "openshell-warm-pool-gghg2.18c0dbbed93ee07c", + "namespace": "openshell", + "resourceVersion": "22301", + "uid": "757f476e-8752-462c-85e3-496996a23765" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:25Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:25Z", + "name": "openshell-warm-pool-gghg2.18c0dbbeded4a22b", + "namespace": "openshell", + "resourceVersion": "22302", + "uid": "9f03c5f6-a87e-49f4-815e-8b0a54a96e9f" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:25Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:25Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:25Z", + "name": "openshell-warm-pool-gghg2.18c0dbbedf406ed5", + "namespace": "openshell", + "resourceVersion": "22303", + "uid": "adb17da5-4a2b-40dd-bcf1-849a6b26384e" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:25Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:25Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 345ms (345ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:25Z", + "name": "openshell-warm-pool-gghg2.18c0dbbefe0fdb9f", + "namespace": "openshell", + "resourceVersion": "22309", + "uid": "3096b666-c309-47b5-9482-48511d81d961" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:26Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:26Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 295ms (295ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:26Z", + "name": "openshell-warm-pool-gghg2.18c0dbbf36fea89a", + "namespace": "openshell", + "resourceVersion": "22318", + "uid": "2b37966c-b8e3-4589-af2c-3c160c7e0ead" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 6, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:40Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-gghg2.18c0dbbe41950418", + "namespace": "openshell", + "resourceVersion": "22538", + "uid": "f59817ac-c1de-4c93-a40c-d090bcf49a8f" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:40Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-gghg2", + "namespace": "openshell", + "resourceVersion": "22177", + "uid": "a8252ec9-231a-4cda-9f00-24a93f4a08f8" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:40Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 314ms (314ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:40Z", + "name": "openshell-warm-pool-gghg2.18c0dbc28292c179", + "namespace": "openshell", + "resourceVersion": "22550", + "uid": "9997af2e-5844-40ca-9aa0-9ffea4ca1185" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-2.json b/experiments/results/events-wp-default-run-2.json new file mode 100644 index 0000000000..50cb981486 --- /dev/null +++ b/experiments/results/events-wp-default-run-2.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:15.507522Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21115", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-tb5wn to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:15Z", + "name": "openshell-warm-pool-tb5wn.18c0dbaea0d07b02", + "namespace": "openshell", + "resourceVersion": "21196", + "uid": "3d67b30d-f844-4091-8651-7dad3f1e5c00" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-ee8373ad-df80-4ff8-b17d-73b4160b8b17\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-tb5wn.18c0dbafa35d323c", + "namespace": "openshell", + "resourceVersion": "21263", + "uid": "6624867d-39f3-4cfc-9677-80567c0c75e8" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tb5wn.18c0dbafe087ab42", + "namespace": "openshell", + "resourceVersion": "21295", + "uid": "f9473ad6-7a62-4ed2-a97f-f0bd41978f46" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21205", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Add eth0 [10.130.0.42/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tb5wn.18c0dbafc81061f3", + "namespace": "openshell", + "resourceVersion": "21279", + "uid": "f4643b04-3c91-4ccf-ad75-98f842898d81" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 361ms (361ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tb5wn.18c0dbafe080ee7e", + "namespace": "openshell", + "resourceVersion": "21294", + "uid": "99067d14-5954-4a76-b043-593ec9e51f4f" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-tb5wn.18c0dbaffb65cae8", + "namespace": "openshell", + "resourceVersion": "21313", + "uid": "8bee415a-df11-4ef7-ae6a-e4f878d72cde" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 370ms (370ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-tb5wn.18c0dbaff69c372b", + "namespace": "openshell", + "resourceVersion": "21303", + "uid": "c51bb227-9d65-47a9-885a-01e13c97b280" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-tb5wn.18c0dbaffadf5f6b", + "namespace": "openshell", + "resourceVersion": "21311", + "uid": "3cd08a74-7f80-451f-a269-c8fb42c40e8f" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 433ms (433ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb04e41a29d", + "namespace": "openshell", + "resourceVersion": "21350", + "uid": "fd669874-96e2-4740-bc79-a8303b253190" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb04e485e32", + "namespace": "openshell", + "resourceVersion": "21351", + "uid": "75ffdbff-0997-45b0-af06-37b5bb7e1eb6" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 374ms (374ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb0649789da", + "namespace": "openshell", + "resourceVersion": "21356", + "uid": "daf2fe54-cc4b-42e1-863e-89fbb0bb185b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb069da29e6", + "namespace": "openshell", + "resourceVersion": "21359", + "uid": "75fc08d8-9c19-4a90-adda-71125cf83f83" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb06a458e8b", + "namespace": "openshell", + "resourceVersion": "21360", + "uid": "35796484-cc12-4a3d-a24d-5cc30b91120f" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 312ms (312ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb0835e3e73", + "namespace": "openshell", + "resourceVersion": "21377", + "uid": "ce27783e-bea3-45de-9361-6601835e4695" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 354ms (354ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:24Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb0c21be404", + "namespace": "openshell", + "resourceVersion": "21388", + "uid": "93b3a096-4709-48ab-bb85-427d19f215e1" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:49Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tb5wn.18c0dbafcafae8d5", + "namespace": "openshell", + "resourceVersion": "21515", + "uid": "5a335757-610e-47a9-abae-f7390e2c1ad6" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:49Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tb5wn", + "namespace": "openshell", + "resourceVersion": "21195", + "uid": "e5594b30-75c2-4a29-80ed-1bf5fe6350c4" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:49Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 311ms (311ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:49Z", + "name": "openshell-warm-pool-tb5wn.18c0dbb699ccf614", + "namespace": "openshell", + "resourceVersion": "21529", + "uid": "77778234-f4d1-492f-ae07-90fdf7bf3888" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-3.json b/experiments/results/events-wp-default-run-3.json new file mode 100644 index 0000000000..e244358a4b --- /dev/null +++ b/experiments/results/events-wp-default-run-3.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:15.563993Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21127", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-c49z8 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:15Z", + "name": "openshell-warm-pool-c49z8.18c0dbaea42e2fe5", + "namespace": "openshell", + "resourceVersion": "21207", + "uid": "3d2cea1a-1e96-4dc7-b4d7-2cb130019638" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-a136ca7d-d566-406a-8e0b-a140166afe82\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-c49z8.18c0dbafa932a2c0", + "namespace": "openshell", + "resourceVersion": "21265", + "uid": "bad98d5f-3f71-4474-963a-3bd27cd8d5a6" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21210", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Add eth0 [10.130.0.43/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-c49z8.18c0dbafdc27346a", + "namespace": "openshell", + "resourceVersion": "21287", + "uid": "9afa7957-66fd-4cb2-9480-0c05b8f1619d" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-c49z8.18c0dbb00d3b59df", + "namespace": "openshell", + "resourceVersion": "21324", + "uid": "5f9a9d94-663d-454c-bcf3-142b35b77f84" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 331ms (331ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-c49z8.18c0dbaff346ba5d", + "namespace": "openshell", + "resourceVersion": "21300", + "uid": "ee60d510-2227-49d2-b9af-2c6434140481" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-c49z8.18c0dbaff34ecbbb", + "namespace": "openshell", + "resourceVersion": "21301", + "uid": "bb5835f8-dc0a-4658-a4a2-6661c4eab6a9" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 340ms (340ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-c49z8.18c0dbb0079661f4", + "namespace": "openshell", + "resourceVersion": "21319", + "uid": "9a265e65-38b6-46e9-8e84-a2abeeed1331" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-c49z8.18c0dbb00cc5a8c5", + "namespace": "openshell", + "resourceVersion": "21323", + "uid": "3f2d6cd9-3699-4a4b-99db-f815a5c23cd9" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-c49z8.18c0dbb049fb6f7f", + "namespace": "openshell", + "resourceVersion": "21346", + "uid": "8f527701-63f8-4448-bf98-2f63e84eafcb" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 359ms (359ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-c49z8.18c0dbb049f40410", + "namespace": "openshell", + "resourceVersion": "21345", + "uid": "326c0a9c-4f00-4dcd-9d11-425ea8ca628b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 349ms (349ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-c49z8.18c0dbb05ed3dcde", + "namespace": "openshell", + "resourceVersion": "21353", + "uid": "3caf890d-b258-420f-9340-efdae746f935" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-c49z8.18c0dbb064521996", + "namespace": "openshell", + "resourceVersion": "21355", + "uid": "00f5261d-a4ed-47be-b430-d4ffd8e1c109" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-c49z8.18c0dbb064c83df1", + "namespace": "openshell", + "resourceVersion": "21358", + "uid": "8b14d18b-8009-4ed2-b1d0-d9ca910ef0f6" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 378ms (378ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-c49z8.18c0dbb08766bfc8", + "namespace": "openshell", + "resourceVersion": "21378", + "uid": "e68dfbcd-5cef-47eb-8850-89fa33522b90" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:24Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:24Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 323ms (323ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:24Z", + "name": "openshell-warm-pool-c49z8.18c0dbb0c03c53de", + "namespace": "openshell", + "resourceVersion": "21385", + "uid": "01d9b50a-d19e-481a-ba14-942a4b1f3ca8" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:55Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-c49z8.18c0dbafdf83d500", + "namespace": "openshell", + "resourceVersion": "21590", + "uid": "52b3af77-de79-415f-ad25-385946faee2c" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:56Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-c49z8", + "namespace": "openshell", + "resourceVersion": "21206", + "uid": "9733e581-e37a-48da-9fc4-217efea67e34" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:56Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 348ms (348ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:56Z", + "name": "openshell-warm-pool-c49z8.18c0dbb81d3794bb", + "namespace": "openshell", + "resourceVersion": "21603", + "uid": "05878c4a-a1af-4174-8bc9-6a478ddc66e9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-4.json b/experiments/results/events-wp-default-run-4.json new file mode 100644 index 0000000000..442224957f --- /dev/null +++ b/experiments/results/events-wp-default-run-4.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:15.627664Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21137", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-tp4jb to ip-10-0-5-57.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:15Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaea7f9c1d3", + "namespace": "openshell", + "resourceVersion": "21213", + "uid": "e2cbab95-8700-4b86-aa2d-5872ebe330cd" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:18Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-78318581-a18e-4e36-a15d-ae37d49a91d8\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:18Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf37a52930", + "namespace": "openshell", + "resourceVersion": "21241", + "uid": "8a25de52-35ad-46b4-87ec-4ef3b632c0ca" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21219", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:18Z", + "message": "Add eth0 [10.129.0.34/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:18Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf62fd0f58", + "namespace": "openshell", + "resourceVersion": "21245", + "uid": "d1530f1e-a31f-4fb6-8cbc-2cfb46659278" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf8fc539b7", + "namespace": "openshell", + "resourceVersion": "21261", + "uid": "e28caad3-aa46-4553-9b55-1699f3d5c120" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 348ms (348ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf7a00fb00", + "namespace": "openshell", + "resourceVersion": "21250", + "uid": "93b38f10-ea07-4119-abdb-17741a0f1bf7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf7a07dcf6", + "namespace": "openshell", + "resourceVersion": "21251", + "uid": "a6654c07-4bdc-4191-9123-5649764aca3f" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 295ms (295ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf8ba595fa", + "namespace": "openshell", + "resourceVersion": "21259", + "uid": "abdb629a-fee6-49ad-86d9-17eabf305bd9" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf8f5dcc8a", + "namespace": "openshell", + "resourceVersion": "21260", + "uid": "fc22379a-2155-43ba-998e-18c46f4b990d" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tp4jb.18c0dbafd5d3916d", + "namespace": "openshell", + "resourceVersion": "21284", + "uid": "485fb9ed-396e-4081-9e5d-043eb5969f16" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 337ms (337ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tp4jb.18c0dbafd5cd31bb", + "namespace": "openshell", + "resourceVersion": "21283", + "uid": "64f92bee-afef-40c5-a703-152d9425f16d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 295ms (295ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-tp4jb.18c0dbafe76fa4ea", + "namespace": "openshell", + "resourceVersion": "21296", + "uid": "4f6fb884-b0f3-4b1a-921c-722fe450fcc2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-tp4jb.18c0dbafec9515a1", + "namespace": "openshell", + "resourceVersion": "21297", + "uid": "413eabe6-2634-4d62-a6f9-4935ebb9bb92" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-tp4jb.18c0dbafecfaf3d5", + "namespace": "openshell", + "resourceVersion": "21298", + "uid": "ec350ae9-cd87-4129-a069-5478a1b8aed9" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 364ms (364ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-tp4jb.18c0dbb01309cdcc", + "namespace": "openshell", + "resourceVersion": "21326", + "uid": "9ffe1eae-506e-4dce-b569-882d2c042a20" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 310ms (310ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-tp4jb.18c0dbb04ba3e79d", + "namespace": "openshell", + "resourceVersion": "21349", + "uid": "3618297d-9dd8-43d8-a02d-c5b17468f29a" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:18Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:02Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:18Z", + "name": "openshell-warm-pool-tp4jb.18c0dbaf653a99ed", + "namespace": "openshell", + "resourceVersion": "21695", + "uid": "50969e4a-dba0-4285-8b13-7d2e528e6e0d" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:02Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-tp4jb", + "namespace": "openshell", + "resourceVersion": "21211", + "uid": "2648ee90-4b08-4dc9-ab8b-c7ded7d9d2d1" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:02Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 510ms (510ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:02Z", + "name": "openshell-warm-pool-tp4jb.18c0dbb9a519fc66", + "namespace": "openshell", + "resourceVersion": "21725", + "uid": "4fc5b96b-0e8f-4843-82af-b41139f31680" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-5.json b/experiments/results/events-wp-default-run-5.json new file mode 100644 index 0000000000..ce5b0cb4b5 --- /dev/null +++ b/experiments/results/events-wp-default-run-5.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:15.679838Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21147", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-wvkbd to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:15Z", + "name": "openshell-warm-pool-wvkbd.18c0dbaeab15d7f0", + "namespace": "openshell", + "resourceVersion": "21221", + "uid": "bd7b75a5-77ce-4c25-a040-9729a4abf055" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:19Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:19Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-7ec712dd-fcfa-4333-893a-34f3de1cf442\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:19Z", + "name": "openshell-warm-pool-wvkbd.18c0dbaf82ed5703", + "namespace": "openshell", + "resourceVersion": "21256", + "uid": "10dcc5f0-f16e-481c-b756-a5d4a7cec5f3" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbafdd898857", + "namespace": "openshell", + "resourceVersion": "21292", + "uid": "3ace2e37-17d9-4ef8-8647-ed36d349511a" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21224", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Add eth0 [10.130.0.44/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbaface52215", + "namespace": "openshell", + "resourceVersion": "21266", + "uid": "1ce1521a-a19d-41da-bccd-621c7a5b15dc" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 384ms (384ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbafc62393d0", + "namespace": "openshell", + "resourceVersion": "21275", + "uid": "3e89a705-1214-4258-981e-873fdf30c387" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbafc62f53e1", + "namespace": "openshell", + "resourceVersion": "21276", + "uid": "cc4b964c-3a9f-4d31-8219-d00fd8454b32" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 313ms (313ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbafd8d8481b", + "namespace": "openshell", + "resourceVersion": "21286", + "uid": "83f4a1bf-d5fc-4546-864e-ecd9b45fd84f" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:20Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbafdd19721b", + "namespace": "openshell", + "resourceVersion": "21291", + "uid": "3c5953d9-dc61-4b16-ad54-72e313d04a84" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 328ms (328ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb00bed7e59", + "namespace": "openshell", + "resourceVersion": "21321", + "uid": "b847553b-64fa-470a-8d41-0ffb72d1b480" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb00bf73b11", + "namespace": "openshell", + "resourceVersion": "21322", + "uid": "a32c887a-8a10-4c03-b2e8-a43005b97aed" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 313ms (313ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:21Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb01ea71dfd", + "namespace": "openshell", + "resourceVersion": "21327", + "uid": "50df1d0a-d24f-40d6-85df-743abce49967" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb023f71fc0", + "namespace": "openshell", + "resourceVersion": "21328", + "uid": "7cfe4aa8-6ee8-465a-aa45-c252b1b45ab1" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb02463dc8e", + "namespace": "openshell", + "resourceVersion": "21329", + "uid": "2c2f0363-7977-4671-8f7c-1f8c7855bd52" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 353ms (353ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:22Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb049c8953e", + "namespace": "openshell", + "resourceVersion": "21344", + "uid": "ef6ce390-b8ec-4236-a768-527e03ef3c2e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:23Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 407ms (407ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:23Z", + "name": "openshell-warm-pool-wvkbd.18c0dbb0893605b3", + "namespace": "openshell", + "resourceVersion": "21380", + "uid": "74ea3358-1eed-4c69-8b3b-95207858c5e2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:08Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:20Z", + "name": "openshell-warm-pool-wvkbd.18c0dbafaf32896c", + "namespace": "openshell", + "resourceVersion": "21945", + "uid": "02c6493f-0fdd-472b-ab6b-e82fda046004" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:09Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-wvkbd", + "namespace": "openshell", + "resourceVersion": "21220", + "uid": "431bc367-bf10-4202-b74d-abf58bbcb9b3" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:09Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 313ms (313ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:09Z", + "name": "openshell-warm-pool-wvkbd.18c0dbbb18eac1f8", + "namespace": "openshell", + "resourceVersion": "21968", + "uid": "aee1baa5-d4d1-4e19-b9d1-00c5081ce95d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-6.json b/experiments/results/events-wp-default-run-6.json new file mode 100644 index 0000000000..00456744d9 --- /dev/null +++ b/experiments/results/events-wp-default-run-6.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:52.504924Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21519", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-cp7hc to ip-10-0-5-57.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:52Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb73e0819a2", + "namespace": "openshell", + "resourceVersion": "21564", + "uid": "fe9dab5c-9ebc-4bb8-989a-b05c72fb87fa" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:54Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:54Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-370082a8-e0cd-4a7c-a73e-883049aa7563\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:54Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb7c5faf664", + "namespace": "openshell", + "resourceVersion": "21575", + "uid": "c4f8c9a7-5a44-4f05-8484-2f382cb9fe45" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:58Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:58Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:58Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8aefc4967", + "namespace": "openshell", + "resourceVersion": "21623", + "uid": "cf1cf56b-58f0-4cf0-be4d-c68727fd2a3d" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21567", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:58Z", + "message": "Add eth0 [10.129.0.35/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:58Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8993905a9", + "namespace": "openshell", + "resourceVersion": "21618", + "uid": "f295721b-79c7-43d1-81b8-32983803067c" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:58Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 317ms (317ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:58Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8aef5c689", + "namespace": "openshell", + "resourceVersion": "21622", + "uid": "d11b021f-61da-44df-93f7-a6ac1d54b1c6" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:59Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:59Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:59Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8ca9ed3cd", + "namespace": "openshell", + "resourceVersion": "21641", + "uid": "5f8ddfa5-2b1a-4b90-a14c-36d18403fd20" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:59Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:59Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 383ms (383ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:59Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8c5dd5f22", + "namespace": "openshell", + "resourceVersion": "21639", + "uid": "d4392eb7-9d97-4799-833c-aa85e1cb46f1" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:59Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:59Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:59Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8ca33d006", + "namespace": "openshell", + "resourceVersion": "21640", + "uid": "741096e0-a4bc-4e82-94ad-f6d9eeade106" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:59Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:59Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 347ms (347ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:59Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8f44df60a", + "namespace": "openshell", + "resourceVersion": "21647", + "uid": "ad66ba82-3020-4351-96e5-1e89acdc2b4e" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:59Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:59Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:59Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb8f4545ccb", + "namespace": "openshell", + "resourceVersion": "21648", + "uid": "e408a477-b59d-4714-a983-752af5b9ce3f" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:00Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:00Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 328ms (328ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:00Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb907e225c1", + "namespace": "openshell", + "resourceVersion": "21649", + "uid": "c9eb33c2-3bbd-414b-ad2c-2533625bb28b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:00Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:00Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:00Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb90cf5d075", + "namespace": "openshell", + "resourceVersion": "21650", + "uid": "cdd155f9-7416-44f8-be3b-f4098a04e89b" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:00Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:00Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:00Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb90d5aca48", + "namespace": "openshell", + "resourceVersion": "21651", + "uid": "4a7aaca6-b29d-4762-be80-4860f19ea6e1" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:00Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:00Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 469ms (469ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:00Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb93799502a", + "namespace": "openshell", + "resourceVersion": "21659", + "uid": "7931f1c1-29eb-4854-99e1-099353370635" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:01Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:01Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 324ms (324ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:01Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb96abc9222", + "namespace": "openshell", + "resourceVersion": "21664", + "uid": "314ba27e-6a58-45e9-8fc2-d44690ebc810" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:58Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:15Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:58Z", + "name": "openshell-warm-pool-cp7hc.18c0dbb89c07b20c", + "namespace": "openshell", + "resourceVersion": "22127", + "uid": "e1a7aac8-3bc9-4462-8353-db794f2e451f" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:15Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-cp7hc", + "namespace": "openshell", + "resourceVersion": "21563", + "uid": "2d0951ee-4170-4943-99b6-cf0ff4da9a6c" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:15Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 385ms (385ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:15Z", + "name": "openshell-warm-pool-cp7hc.18c0dbbc9cd0b8f3", + "namespace": "openshell", + "resourceVersion": "22142", + "uid": "abcd35db-c2e9-413b-9918-95182c8157b7" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-57.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-57.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-7.json b/experiments/results/events-wp-default-run-7.json new file mode 100644 index 0000000000..abda445a09 --- /dev/null +++ b/experiments/results/events-wp-default-run-7.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:03:46.080252Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21448", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-842q6 to ip-10-0-5-178.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:46Z", + "name": "openshell-warm-pool-842q6.18c0dbb5bf1760ea", + "namespace": "openshell", + "resourceVersion": "21478", + "uid": "c48fac03-b906-4445-9597-bbbdd10ce351" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:48Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:48Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-913a7e75-0206-4162-accc-7cce3dd85195\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:48Z", + "name": "openshell-warm-pool-842q6.18c0dbb64c799338", + "namespace": "openshell", + "resourceVersion": "21496", + "uid": "5fc90c1c-2841-4c18-932c-b882fe4dfa64" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:49Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:49Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 356ms (356ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:49Z", + "name": "openshell-warm-pool-842q6.18c0dbb6a4ccbb8a", + "namespace": "openshell", + "resourceVersion": "21530", + "uid": "6ed70056-b652-44b7-8d09-0bc8fcd5376b" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:49Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21481", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:49Z", + "message": "Add eth0 [10.128.0.43/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:49Z", + "name": "openshell-warm-pool-842q6.18c0dbb67a57a79f", + "namespace": "openshell", + "resourceVersion": "21498", + "uid": "33dc5137-16d7-447d-8a23-4ffd07827c35" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:49Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:49Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 312ms (312ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:49Z", + "name": "openshell-warm-pool-842q6.18c0dbb68f80de15", + "namespace": "openshell", + "resourceVersion": "21527", + "uid": "208f5a90-4a6a-4c67-8cd1-cdcf3ee4e089" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:49Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:49Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:49Z", + "name": "openshell-warm-pool-842q6.18c0dbb68f895e7a", + "namespace": "openshell", + "resourceVersion": "21528", + "uid": "fbdef5e0-fd71-4b60-bcc6-cdfd5348d33d" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:50Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:50Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:50Z", + "name": "openshell-warm-pool-842q6.18c0dbb6a93ffcb4", + "namespace": "openshell", + "resourceVersion": "21532", + "uid": "1b41fecf-f852-4044-91ad-f65779e57b3b" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:50Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:50Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:50Z", + "name": "openshell-warm-pool-842q6.18c0dbb6a8d0e39d", + "namespace": "openshell", + "resourceVersion": "21531", + "uid": "02c49a1b-a4bc-4b3f-9203-8e8cdcb74a3d" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:50Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:50Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 321ms (321ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:50Z", + "name": "openshell-warm-pool-842q6.18c0dbb6cc7100cc", + "namespace": "openshell", + "resourceVersion": "21538", + "uid": "e16e8799-4993-4a9a-9617-0638b52335b8" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:50Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:50Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:50Z", + "name": "openshell-warm-pool-842q6.18c0dbb6cc7876cd", + "namespace": "openshell", + "resourceVersion": "21539", + "uid": "4b3acb45-ab3a-4d52-b513-5626f6ef51a1" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:50Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:50Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 322ms (322ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:50Z", + "name": "openshell-warm-pool-842q6.18c0dbb6dfad1188", + "namespace": "openshell", + "resourceVersion": "21541", + "uid": "29faed30-0d68-4853-97f7-4be231d33767" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:51Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:51Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:51Z", + "name": "openshell-warm-pool-842q6.18c0dbb6e50dd7ba", + "namespace": "openshell", + "resourceVersion": "21542", + "uid": "574c5ab1-78ec-464b-9218-b4e54ff3c4c6" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:51Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:51Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:51Z", + "name": "openshell-warm-pool-842q6.18c0dbb6e5744b1f", + "namespace": "openshell", + "resourceVersion": "21543", + "uid": "2b5b9481-a789-4ef1-80e8-a9bbeea72caa" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:51Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:51Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 430ms (430ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:51Z", + "name": "openshell-warm-pool-842q6.18c0dbb70eed0982", + "namespace": "openshell", + "resourceVersion": "21553", + "uid": "d603e268-f1d0-404c-8de2-fb5c7fad872a" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:52Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:03:52Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 325ms (325ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:52Z", + "name": "openshell-warm-pool-842q6.18c0dbb7448d3fe3", + "namespace": "openshell", + "resourceVersion": "21571", + "uid": "b44794d0-605f-4e4c-ae41-a4e1a8aa99d0" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:03:49Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:03:49Z", + "name": "openshell-warm-pool-842q6.18c0dbb67ce1b4bf", + "namespace": "openshell", + "resourceVersion": "22217", + "uid": "3132f293-fa53-4779-8b1b-e61d8a1d9221" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-842q6", + "namespace": "openshell", + "resourceVersion": "21477", + "uid": "41d92aa0-7fe9-42de-bf94-83d826820765" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 297ms (297ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-842q6.18c0dbbe11a342a9", + "namespace": "openshell", + "resourceVersion": "22233", + "uid": "21648e1f-da2a-49d8-847b-8a6fce6d8d4d" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-178.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-178.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-8.json b/experiments/results/events-wp-default-run-8.json new file mode 100644 index 0000000000..ddfe262a9f --- /dev/null +++ b/experiments/results/events-wp-default-run-8.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:04:05.388915Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21701", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-8ltck to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:05Z", + "name": "openshell-warm-pool-8ltck.18c0dbba3dfa346d", + "namespace": "openshell", + "resourceVersion": "21819", + "uid": "e2f35e58-652a-4f50-923e-62367bdba52b" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:09Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:09Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-988e6f0d-b9ff-4687-9fae-0dafcb1d898b\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:09Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb163d42e2", + "namespace": "openshell", + "resourceVersion": "21962", + "uid": "4555bf28-03b4-40f3-8518-225ed5ca263a" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:10Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb7f466965", + "namespace": "openshell", + "resourceVersion": "22061", + "uid": "e832ef4f-fc1a-4860-a3ef-6e545d230bd4" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21822", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:10Z", + "message": "Add eth0 [10.130.0.45/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb51b94909", + "namespace": "openshell", + "resourceVersion": "22007", + "uid": "f9fd84ec-ddb7-4e37-8966-a9f43606f838" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:10Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 303ms (303ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb67140376", + "namespace": "openshell", + "resourceVersion": "22047", + "uid": "aeb983ef-b59f-4f60-bdcb-5c0c7ef72faf" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:10Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb671b3e19", + "namespace": "openshell", + "resourceVersion": "22048", + "uid": "88fe9da3-b9e2-43cb-b11f-b5a3d50fdf10" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:10Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 338ms (338ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb7b45cb5c", + "namespace": "openshell", + "resourceVersion": "22059", + "uid": "d5b0477a-b0f8-40d9-b005-abda022aa6d0" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:10Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb7ed97605", + "namespace": "openshell", + "resourceVersion": "22060", + "uid": "3fcc4fc1-33b8-4919-ae0e-c5e53096d88e" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:11Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:11Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 350ms (350ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:11Z", + "name": "openshell-warm-pool-8ltck.18c0dbbbbf96f44a", + "namespace": "openshell", + "resourceVersion": "22089", + "uid": "b407ed77-3961-48ee-94c4-07e0035ee157" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:11Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:11Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:11Z", + "name": "openshell-warm-pool-8ltck.18c0dbbbbf9dc32e", + "namespace": "openshell", + "resourceVersion": "22090", + "uid": "fc6bca7f-4cc1-4c2c-8b3d-c3d48193dde0" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:12Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:12Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 311ms (311ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:12Z", + "name": "openshell-warm-pool-8ltck.18c0dbbbd235d12a", + "namespace": "openshell", + "resourceVersion": "22093", + "uid": "7ef1a58e-b40d-44e1-9542-795b9ecaa6cf" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:12Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:12Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:12Z", + "name": "openshell-warm-pool-8ltck.18c0dbbbd77aadb4", + "namespace": "openshell", + "resourceVersion": "22094", + "uid": "011242c3-2ca5-4fab-836a-d15b76f75b29" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:12Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:12Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:12Z", + "name": "openshell-warm-pool-8ltck.18c0dbbbd7e57406", + "namespace": "openshell", + "resourceVersion": "22095", + "uid": "46be1290-3667-47f3-bf0b-1bc8e41da1dc" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:12Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:12Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 288ms (288ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:12Z", + "name": "openshell-warm-pool-8ltck.18c0dbbbf7d7fe87", + "namespace": "openshell", + "resourceVersion": "22102", + "uid": "866ef93d-94fa-47b9-b8a6-4d1c89f558f2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:13Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:13Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 351ms (352ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:13Z", + "name": "openshell-warm-pool-8ltck.18c0dbbc37815a10", + "namespace": "openshell", + "resourceVersion": "22107", + "uid": "c8a3f685-031d-4fdd-a64a-3546c0ed6238" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:10Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:27Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:10Z", + "name": "openshell-warm-pool-8ltck.18c0dbbb54ff81d6", + "namespace": "openshell", + "resourceVersion": "22334", + "uid": "18ba5342-c3af-4ea5-a112-c705ae535ffc" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:28Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-8ltck", + "namespace": "openshell", + "resourceVersion": "21818", + "uid": "c88bc6a1-61fa-4ac0-a4e4-8caae539750b" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:28Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 345ms (345ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:28Z", + "name": "openshell-warm-pool-8ltck.18c0dbbf90973fff", + "namespace": "openshell", + "resourceVersion": "22349", + "uid": "7342a4f0-d858-4597-bc26-d3390b666b99" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/results/events-wp-default-run-9.json b/experiments/results/events-wp-default-run-9.json new file mode 100644 index 0000000000..9730576555 --- /dev/null +++ b/experiments/results/events-wp-default-run-9.json @@ -0,0 +1,556 @@ +{ + "apiVersion": "v1", + "items": [ + { + "action": "Binding", + "apiVersion": "v1", + "eventTime": "2026-07-10T07:04:11.816947Z", + "firstTimestamp": null, + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "21952", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": null, + "message": "Successfully assigned openshell/openshell-warm-pool-5m2m2 to ip-10-0-5-254.us-east-2.compute.internal", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:11Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbbbd1e216d", + "namespace": "openshell", + "resourceVersion": "22082", + "uid": "94d567b8-0101-4dc6-a2d8-186de959c52b" + }, + "reason": "Scheduled", + "reportingComponent": "default-scheduler", + "reportingInstance": "default-scheduler-kube-scheduler-5d4d5bf5b-96lll", + "source": {}, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:14Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:14Z", + "message": "AttachVolume.Attach succeeded for volume \"pvc-60209930-072d-4a3a-8673-bd820e90dfe2\" ", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:14Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbc4635d06c", + "namespace": "openshell", + "resourceVersion": "22109", + "uid": "f17f0792-011c-45fa-bdfa-4b36870d89bc" + }, + "reason": "SuccessfulAttachVolume", + "reportingComponent": "attachdetach-controller", + "reportingInstance": "", + "source": { + "component": "attachdetach-controller" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:20Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:20Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:20Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbdcfd1ade4", + "namespace": "openshell", + "resourceVersion": "22201", + "uid": "6e687583-0b35-4b17-9ae8-26404191e330" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22088", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:20Z", + "message": "Add eth0 [10.130.0.46/23] from ovn-kubernetes", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:20Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbdb9702e4d", + "namespace": "openshell", + "resourceVersion": "22195", + "uid": "e6d26222-5119-4dff-bde4-ebfb7a9353fb" + }, + "reason": "AddedInterface", + "reportingComponent": "multus", + "reportingInstance": "", + "source": { + "component": "multus" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:20Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 335ms (335ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:20Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbdcfca9261", + "namespace": "openshell", + "resourceVersion": "22200", + "uid": "00a8d660-d432-43ac-ab28-33ec3d4ca9c5" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbde80fe4e8", + "namespace": "openshell", + "resourceVersion": "22204", + "uid": "a1f04e15-9b05-45b7-80eb-0e839de8c34a" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 339ms (339ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbde40aab4b", + "namespace": "openshell", + "resourceVersion": "22202", + "uid": "63c7ee1f-7846-4e24-ae35-eb073ffedbe2" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.initContainers{workspace-init}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbde7a14c20", + "namespace": "openshell", + "resourceVersion": "22203", + "uid": "2e1b5373-2cb9-4f32-9a4a-852cca212a9a" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 310ms (310ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe140fbf67", + "namespace": "openshell", + "resourceVersion": "22235", + "uid": "37d387bf-9efe-4878-8aed-c87799f49a5c" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:21Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:21Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:21Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe1416d3da", + "namespace": "openshell", + "resourceVersion": "22236", + "uid": "f561e0a4-6137-4201-9cee-01d2e2213efa" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell-community/sandboxes/base:latest\" in 409ms (409ms including waiting). Image size: 3422808342 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe2c7b9d94", + "namespace": "openshell", + "resourceVersion": "22237", + "uid": "005c4172-a0e1-4b5e-af7f-a130c01f0ba0" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Container created", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe32203192", + "namespace": "openshell", + "resourceVersion": "22238", + "uid": "150ee491-70d5-477b-b7eb-8e4ab548bfe0" + }, + "reason": "Created", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "fieldPath": "spec.containers{agent}", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Container started", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe328bac37", + "namespace": "openshell", + "resourceVersion": "22239", + "uid": "913fa3d7-e938-4c27-9b9d-2e0700684ce1" + }, + "reason": "Started", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:22Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:22Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 340ms (340ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:22Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe51bb058c", + "namespace": "openshell", + "resourceVersion": "22253", + "uid": "6977676a-36ca-4525-a580-d44006830efb" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:23Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:23Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 372ms (372ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:23Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbe8fa82d2e", + "namespace": "openshell", + "resourceVersion": "22277", + "uid": "2dc9927a-dcda-40a3-a464-12957690c1cd" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 5, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:20Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:34Z", + "message": "Pulling image \"ghcr.io/nvidia/openshell/supervisor:latest\"", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:20Z", + "name": "openshell-warm-pool-5m2m2.18c0dbbdbbcb2306", + "namespace": "openshell", + "resourceVersion": "22445", + "uid": "f5200160-9537-4deb-af39-895f8425deae" + }, + "reason": "Pulling", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + }, + { + "apiVersion": "v1", + "count": 1, + "eventTime": null, + "firstTimestamp": "2026-07-10T07:04:34Z", + "involvedObject": { + "apiVersion": "v1", + "kind": "Pod", + "name": "openshell-warm-pool-5m2m2", + "namespace": "openshell", + "resourceVersion": "22081", + "uid": "57b3d123-de2d-4d4f-a568-a6b5f93e0e00" + }, + "kind": "Event", + "lastTimestamp": "2026-07-10T07:04:34Z", + "message": "Successfully pulled image \"ghcr.io/nvidia/openshell/supervisor:latest\" in 290ms (290ms including waiting). Image size: 16599000 bytes.", + "metadata": { + "creationTimestamp": "2026-07-10T07:04:34Z", + "name": "openshell-warm-pool-5m2m2.18c0dbc106e0c4a8", + "namespace": "openshell", + "resourceVersion": "22459", + "uid": "a581306f-8afd-4a02-9e9d-7e16fcc622e0" + }, + "reason": "Pulled", + "reportingComponent": "kubelet", + "reportingInstance": "ip-10-0-5-254.us-east-2.compute.internal", + "source": { + "component": "kubelet", + "host": "ip-10-0-5-254.us-east-2.compute.internal" + }, + "type": "Normal" + } + ], + "kind": "List", + "metadata": { + "resourceVersion": "" + } +} diff --git a/experiments/sidecar/Dockerfile b/experiments/sidecar/Dockerfile new file mode 100644 index 0000000000..508c60c614 --- /dev/null +++ b/experiments/sidecar/Dockerfile @@ -0,0 +1,10 @@ +FROM golang:1.24-alpine AS builder +WORKDIR /src +COPY go.mod main.go ./ +RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /readiness-sidecar . + +FROM scratch +COPY --from=builder /readiness-sidecar /readiness-sidecar +USER 65532:65532 +EXPOSE 8080 +ENTRYPOINT ["/readiness-sidecar"] diff --git a/experiments/sidecar/Makefile b/experiments/sidecar/Makefile new file mode 100644 index 0000000000..bc2d6896c6 --- /dev/null +++ b/experiments/sidecar/Makefile @@ -0,0 +1,14 @@ +IMAGE ?= ghcr.io/nvidia/openshell/sidecar-readiness:latest + +.PHONY: all build image push + +all: build image push + +build: + CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o readiness-sidecar main.go + +image: + podman build -t $(IMAGE) . + +push: + podman push $(IMAGE) diff --git a/experiments/sidecar/go.mod b/experiments/sidecar/go.mod new file mode 100644 index 0000000000..d386c45aef --- /dev/null +++ b/experiments/sidecar/go.mod @@ -0,0 +1,3 @@ +module github.com/nvidia/openshell/experiments/sidecar + +go 1.24 diff --git a/experiments/sidecar/main.go b/experiments/sidecar/main.go new file mode 100644 index 0000000000..7f833bd75d --- /dev/null +++ b/experiments/sidecar/main.go @@ -0,0 +1,46 @@ +package main + +import ( + "fmt" + "log" + "net/http" + "os" + "sync/atomic" + "time" +) + +const signalPath = "/tmp/signal/ready" + +var ready atomic.Bool + +func main() { + go pollSignalFile() + + http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { + if ready.Load() { + w.WriteHeader(http.StatusOK) + fmt.Fprintln(w, "ready") + } else { + w.WriteHeader(http.StatusServiceUnavailable) + fmt.Fprintln(w, "not ready") + } + }) + + log.Println("readiness sidecar listening on :8080") + if err := http.ListenAndServe(":8080", nil); err != nil { + log.Fatalf("server failed: %v", err) + } +} + +func pollSignalFile() { + for { + if _, err := os.Stat(signalPath); err == nil { + if !ready.Load() { + ready.Store(true) + log.Println("signal file detected, transitioning to ready") + } + return + } + time.Sleep(100 * time.Millisecond) + } +} diff --git a/rfc/NNNN-warm-pool-feasibility/README.md b/rfc/NNNN-warm-pool-feasibility/README.md new file mode 100644 index 0000000000..55ace95d7d --- /dev/null +++ b/rfc/NNNN-warm-pool-feasibility/README.md @@ -0,0 +1,454 @@ +--- +authors: + - "@rhuss" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/2157 +--- + +# RFC NNNN: Warm Pool Integration for Kubernetes Sandboxes + +## Summary + +This RFC proposes integrating the Agent Sandbox operator's `SandboxWarmPool` CRD into OpenShell's Kubernetes driver to reduce sandbox startup latency from 16+ seconds to under 4 seconds (Phase 1) and potentially under 2 seconds (Phase 2). + +The proposal is based on a feasibility study conducted on a ROSA HCP 4.22.3 cluster with the Red Hat Agent Sandbox operator v0.9.0. Full measurement data, methodology, and limitations are documented in the companion [Results Document](../../experiments/RESULTS.md). + +## Motivation + +OpenShell's Kubernetes driver provisions a fresh sandbox pod for every `sandbox.create` request. On a typical cluster, this cold-start path takes 16-17 seconds (p50, with pre-pulled images), which is too slow for interactive agent workflows where sub-second tool calls are the norm. + +The feasibility study measured the following (see [Results Document](../../experiments/RESULTS.md) for full methodology and caveats): + +| Scenario | Image | p50 latency | +|----------|-------|-------------| +| Cold start (OpenShell) | sandbox (3.2 GB, PVC, init, supervisor) | 16,678 ms | +| Cold start (minimal) | pause (700 KB, no PVC, no init) | 2,784 ms | +| Warm pool claim | pause (pre-provisioned) | 2,271 ms | +| **Estimated production warm pool** | **sandbox (idle mode)** | **~3,200 ms** | +| Target | | < 2,000 ms | + +The warm pool claim was measured with a minimal `pause` image because the current sandbox image crashes without the supervisor. The estimated production latency of ~3.2s accounts for supervisor startup (1.5s) at claim time. The cold-start breakdown shows that the supervisor itself is fast (1.5 seconds, 8 gRPC calls); most of the 16.7s cold start comes from Kubernetes pod lifecycle (PVC provisioning, scheduling, init containers) which warm pooling eliminates entirely. + +## Design + +### Architecture Overview + +```mermaid +flowchart TD + CLI["CLI"] -->|CreateSandbox| Gateway["Gateway"] + Gateway --> Check{"Pool available?"} + Check -->|Yes| Claim["Create SandboxClaim"] + Check -->|No| Cold["Create Sandbox (today)"] + Claim --> Bind["Agent Sandbox operator\nbinds warm pod"] + Bind -->|.status.sandbox.name| Patch["Gateway patches pod annotation\nopenshell.io/sandbox-id"] + Patch --> Detect["Supervisor detects identity\nvia downward API"] + Detect -->|IssueSandboxToken\nConnectSupervisor| Ready["Sandbox Ready"] +``` + +### Key Constraint: Env Var Injection Bypasses Warm Pool + +The feasibility study discovered that when a SandboxClaim includes `spec.env` fields, the operator creates a new cold-start pod instead of adopting a warm pool pod. This means env var injection and warm pooling are mutually exclusive in operator v0.9.0. + +Identity must be bound through an alternative mechanism. This RFC recommends annotation-based activation via the Kubernetes downward API. + +See: [Upstream Issue #384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384) for the proposed file-based injection that would resolve this in a future operator release. + +## Changes Required + +### 1. Kubernetes Driver (`crates/openshell-driver-kubernetes/`) + +#### 1.1 Pool Detection + +The driver needs to discover available warm pools in the namespace. + +```rust +// New function: check for SandboxWarmPool resources +async fn find_warm_pool(&self, namespace: &str) -> Option { + // List SandboxWarmPool resources + // Return pool with readyReplicas > 0 + // Match pool's template to requested sandbox profile +} +``` + +The driver should cache pool availability with a TTL (~30 seconds) to avoid listing SandboxWarmPool resources on every sandbox creation. + +#### 1.2 Claim-Based Provisioning Path + +When a warm pool is available, the driver creates a SandboxClaim instead of a Sandbox resource. + +```yaml +apiVersion: extensions.agents.x-k8s.io/v1beta1 +kind: SandboxClaim +metadata: + name: + namespace: +spec: + warmPoolRef: + name: + # NO spec.env (would bypass warm pool) + # NO additionalPodMetadata (identity injected separately) +``` + +The claim must NOT include `spec.env` or the operator will fall back to cold-start provisioning. + +#### 1.3 Claim Status Monitoring + +After creating the SandboxClaim, the driver watches for the Ready condition: + +```rust +// Watch SandboxClaim status +// Ready when: .status.conditions[type=Ready].status == "True" +// Bound sandbox: .status.sandbox.name +// No .status.phase field exists in v0.9.0 +``` + +#### 1.4 Identity Injection After Claim + +After the claim binds and the sandbox name is known, the driver patches the pod's annotations: + +```rust +// Patch pod annotations with sandbox identity +kubectl.patch_pod_annotations(sandbox_name, namespace, { + "openshell.io/sandbox-id": sandbox_id, +}) +``` + +The supervisor detects this annotation via the downward API and activates. + +**User-scoped credentials and token exchange.** When sandboxes act on behalf of a specific user (OBO pattern), the gateway must exchange the user's token for a sandbox-scoped credential at claim time. Annotations are not suitable for carrying sensitive credential material. User-scoped credentials need a separate delivery mechanism: either a Kubernetes Secret created by the gateway and mounted into the pod, or a post-activation gRPC call from the supervisor to `GetSandboxProviderEnvironment` (which already delivers LLM provider credentials). This limitation does not apply to the [gRPC-push approach](../NNNN-warm-pool-grpc-push/README.md), which can deliver identity and user-scoped credentials in a single `ActivateSandbox` call. + +In SPIFFE/SPIRE deployments, the SPIRE agent may need to re-attest the warm pod after claim adoption because the pod's attestation properties (annotations, owner references) change. Re-attestation is fast (~10-20ms) but depends on the SPIRE agent detecting the change via the kubelet sync interval. + +#### 1.5 Fallback to Cold Start + +When the warm pool has zero ready replicas or the SandboxClaim stays Pending for longer than a configurable timeout (default: 5 seconds), the driver falls back to the current cold-start path (creating a Sandbox resource directly). + +```rust +match timeout(Duration::from_secs(5), wait_for_claim_ready(&claim)).await { + Ok(sandbox_name) => { /* warm path */ }, + Err(_) => { + delete_claim(&claim).await; + create_sandbox_cold_start(&sandbox).await; // existing path + } +} +``` + +#### 1.6 RBAC Requirements + +The gateway's ServiceAccount needs additional permissions: + +```yaml +# New RBAC rules for warm pool support +- apiGroups: ["extensions.agents.x-k8s.io"] + resources: ["sandboxclaims"] + verbs: ["create", "get", "watch", "delete"] +- apiGroups: ["extensions.agents.x-k8s.io"] + resources: ["sandboxwarmpools"] + verbs: ["list", "get", "watch"] +- apiGroups: [""] + resources: ["pods"] + verbs: ["patch"] # for annotation injection +``` + +### 2. Supervisor (`crates/openshell-sandbox/`) + +#### 2.1 Idle Mode + +The supervisor needs a new startup mode that waits for identity before proceeding with the normal initialization sequence. + +``` +Entrypoint logic: + +if OPENSHELL_SANDBOX_ID is set: + # Normal cold-start path (unchanged) + proceed with IssueSandboxToken, config fetch, policy load, connect + +else: + # Warm pool idle mode (new) + log "Supervisor starting in idle mode, waiting for identity" + start health check endpoint on :8080 (/healthz -> 200) + watch /etc/podinfo/annotations for openshell.io/sandbox-id + when annotation appears: + set OPENSHELL_SANDBOX_ID from annotation value + proceed with normal startup sequence +``` + +#### 2.2 Downward API Annotation Projection + +The supervisor reads its own pod's annotations via a projected volume: + +```yaml +# Added to the pod spec by the SandboxTemplate +volumes: +- name: podinfo + downwardAPI: + items: + - path: "annotations" + fieldRef: + fieldPath: metadata.annotations +``` + +The supervisor watches `/etc/podinfo/annotations` for changes using `inotify` (Linux) or periodic polling (fallback). When the `openshell.io/sandbox-id` key appears, the supervisor parses its value and begins the activation sequence. + +Kubelet propagates annotation changes to projected volumes within the `--sync-frequency` interval (default 60 seconds, configurable). For sub-5-second activation, the cluster should set `--sync-frequency=1s` or the supervisor should use polling with a 500ms interval as a fallback. + +#### 2.3 Health Check Endpoint + +In idle mode, the supervisor exposes a minimal HTTP health endpoint: + +``` +GET /healthz -> 200 OK (idle, waiting for identity) +GET /readyz -> 503 Service Unavailable (not yet activated) +``` + +After activation, `/readyz` transitions to `200 OK` once SSH is ready. The SandboxTemplate's readiness probe should target `/readyz` so the pool controller knows when the pod is ready for claiming. + +### 3. Gateway (`crates/openshell-server/`) + +#### 3.1 Pool-Aware Sandbox Creation + +The `CreateSandbox` gRPC handler needs a branching path: + +```rust +async fn create_sandbox(&self, request: CreateSandboxRequest) -> Result<...> { + let sandbox_id = Uuid::new_v4(); + let jwt = self.mint_sandbox_jwt(&sandbox_id); + + // Store sandbox as "pending" in the sandbox store + self.store.insert_pending(sandbox_id, &request.name); + + // Try warm pool first + if let Some(pool) = self.driver.find_warm_pool(&namespace).await { + match self.driver.create_claim(&request.name, &pool).await { + Ok(claim) => { + let sandbox_name = self.driver.wait_claim_bound(&claim).await?; + self.driver.inject_identity(&sandbox_name, &sandbox_id).await?; + // Supervisor will call back via ConnectSupervisor + return Ok(sandbox_id); + } + Err(_) => { /* fall through to cold start */ } + } + } + + // Cold-start fallback (existing path) + self.driver.create_sandbox(&sandbox_id, &request).await +} +``` + +#### 3.2 Sandbox Store Changes + +The sandbox store needs to track the warm pool lifecycle: + +```rust +enum SandboxState { + Pending, // Created, waiting for compute + WarmClaimed, // Claim bound, waiting for supervisor activation + Connected, // Supervisor called ConnectSupervisor + Ready, // SSH available, sandbox usable + // ... existing states +} +``` + +The `WarmClaimed` state is new and represents the window between claim binding and supervisor activation. + +### 4. Gateway-Managed Pool Lifecycle + +The gateway manages `SandboxTemplate` and `SandboxWarmPool` resources at runtime based on its configuration. This allows flexible multi-image pool management, independent scaling, and future dynamic pool creation without redeployment. + +#### 4.1 Gateway Configuration + +```toml +# gateway.toml additions +[compute.warm_pools] +enabled = false # Opt-in, disabled by default +fallback_timeout = "5s" # Seconds before falling back to cold start + +[[compute.warm_pools.pools]] +image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" +replicas = 3 + +[[compute.warm_pools.pools]] +image = "ghcr.io/nvidia/openshell-community/sandboxes/ollama:latest" +replicas = 2 +``` + +At startup, the gateway reconciles the cluster state against this configuration: creating `SandboxTemplate` + `SandboxWarmPool` pairs for each configured image, updating replica counts, and deleting pools that were removed from the config. + +#### 4.2 Helm Chart (RBAC only) + +The Helm chart provides the RBAC permissions the gateway needs to manage pool resources. It does not create the pool resources themselves. + +The ClusterRole for the gateway ServiceAccount needs the additional permissions from section 1.6 (create/get/watch/delete on `sandboxclaims`, `sandboxwarmpools`, `sandboxtemplates`). + +#### 4.3 Image Pre-Pull DaemonSet (Optional) + +An optional image pre-pull DaemonSet can be deployed alongside the gateway to cache sandbox images on all nodes. See `experiments/manifests/image-prepull-daemonset.yaml`. + +### 5. CLI (`crates/openshell-cli/`) + +#### 5.1 `--no-connect` Flag + +Add a `--no-connect` or `--detach` flag to `openshell sandbox create` that creates the sandbox and exits without opening an SSH session. This is needed for programmatic use and measurement scripts. + +#### 5.2 Pool Status in `openshell status` + +``` +Server Status + Gateway: k8s + Server: https://... + Status: Connected + Version: 0.0.80 + +Warm Pool + Pool: openshell-warm-pool + Ready: 3/3 + Template: openshell-warm +``` + +### 6. Gateway Configuration (`gateway.toml`) + +New configuration section for warm pool behavior: + +```toml +[compute.warm_pool] +# Enable warm pool claim path +enabled = true + +# Name of the SandboxWarmPool resource to use +pool_name = "openshell-warm-pool" + +# Seconds to wait for claim before falling back to cold start +fallback_timeout_secs = 5 + +# Seconds to cache pool availability checks +pool_cache_ttl_secs = 30 + +# Identity injection method: "annotation" (default) or "grpc" (future) +identity_method = "annotation" +``` + +### 7. Documentation (`docs/`) + +#### 7.1 Configuration Reference + +Add `[compute.warm_pool]` section to `docs/reference/gateway-config.mdx`. + +#### 7.2 Warm Pool Guide + +New page: `docs/guides/warm-pool.mdx` covering: +- Prerequisites (Agent Sandbox operator with extension CRDs) +- Enabling warm pool via gateway configuration +- Pool sizing guidance +- Monitoring pool health +- Troubleshooting (pool exhaustion, slow replenishment) + +## Implementation Plan + +### Phase 1: Claim-Time Supervisor Startup (~3-4s target) + +Points use a normalized scale consistent with the [gRPC-push RFC](../NNNN-warm-pool-grpc-push/README.md) so that effort can be compared across approaches: 1 = trivial, 2 = small, 3 = medium, 5 = large. + +| Work item | Points | Crate/Component | +|-----------|--------|-----------------| +| Supervisor idle mode + annotation activation | 5 | `openshell-sandbox` | +| K8s driver claim path + cold-start fallback | 3 | `openshell-driver-kubernetes` | +| Gateway pool-aware CreateSandbox | 2 | `openshell-server` | +| Gateway pool reconciler + config | 3 | `openshell-server` | +| Helm chart (RBAC for extension resources) | 1 | `deploy/` | +| CLI `--no-connect` flag | 1 | `openshell-cli` | +| E2E tests on OpenShift | 3 | `tests/` | +| Documentation | 1 | `docs/` | +| **Total** | **~19** | | + +### Phase 2: Optimized Startup (~2s target) + +| Work item | Points | Depends on | +|-----------|--------|------------| +| Overlap claim binding with supervisor startup | 2 | Phase 1 | +| OPA policy pre-compilation in warm pods | 3 | Phase 1 | +| CLI polling reduction (tighter interval or SSE) | 2 | Independent | +| **Total** | **~7** | Phase 1 | + +### Phase 3: Sub-Second Startup (optional) + +| Work item | Points | Depends on | +|-----------|--------|------------| +| Supervisor deferred identity mode (gRPC push) | 5 | Phase 1 | +| Gateway warm pod pre-registration | 3 | Phase 1 | +| Hot reconfiguration (policy delta at claim time) | 5 | Deferred identity | +| **Total** | **~13** | Phase 1 | + +## Risks and Mitigations + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Warm pool exhaustion under burst traffic | Latency spike (fallback to 16s cold start) | Configure pool size >= expected burst rate; driver falls back automatically | +| Kubelet annotation sync delay (up to 60s default) | Supervisor activation delayed | Set `--sync-frequency=1s` or use polling in supervisor | +| Operator upgrades change CRD schema | Driver breaks on field name changes | Abstract CRD field access behind a version-aware adapter | +| Idle pods consume cluster resources | Wasted capacity when sandboxes are not being created | Configurable pool size; can scale to 0 during off-hours | +| Stale warm pods (expired certs, outdated images) | Claimed sandbox has stale state | Operator handles pod rotation; supervisor validates cert freshness on activation | +| Red Hat operator removes/changes `additionalPodMetadata` | Identity injection path breaks | Upstream engagement to stabilize the API; fallback to ConfigMap-based injection | + +## Alternatives + +### Always-On Supervisor with gRPC-Push Identity Binding + +The companion [gRPC-push RFC](../NNNN-warm-pool-grpc-push/README.md) proposes keeping the supervisor running from pool provisioning time. Global OPA policies are pre-compiled at pool time. At claim time, the gateway pushes sandbox identity directly to the supervisor via a new `ActivateSandbox` gRPC endpoint, bypassing the downward API propagation delay entirely. Only sandbox-specific policies compile at claim time. + +This approach targets sub-1-second provisioning latency, compared to the ~3-4 seconds achievable with the annotation-based approach described in this RFC. Neither RFC is primary; the choice depends on the latency target and acceptable complexity. + +| Dimension | Annotation-based (this RFC) | gRPC-push (companion RFC) | +|-----------|----------------------------|---------------------------| +| **Claim-to-ready latency** | ~3-4s (claim + supervisor startup + downward API) | <1s target (claim + gRPC push + delta OPA) | +| **Supervisor complexity** | Idle-start: new mode that watches annotations, then runs normal startup | Always-on: runs from pool time, two-tier OPA, new gRPC endpoint | +| **Gateway complexity** | Claim creation + annotation patch | Claim creation + pool reconciler + gRPC client + pod IP discovery | +| **OPA compilation** | Full compilation at claim time (~1,400 ms) | Pre-compiled globals at pool time; sandbox-specific delta at claim time (<200 ms) | +| **Security surface** | Standard K8s downward API (no new endpoints) | New gRPC endpoint on supervisor (mTLS-secured) | +| **Upstream alignment** | Matches @craig-kindo's validated GKE implementation ([#1447](https://github.com/NVIDIA/OpenShell/issues/1447)) | Novel approach, not yet validated externally | +| **Resource cost (idle pools)** | Sleeping process (`sleep infinity`), minimal CPU/memory | Running supervisor process, ~100m CPU / 128Mi memory per pod | +| **User-scoped credentials (OBO)** | Separate mechanism needed (Secret mount or post-activation gRPC) | Single `ActivateSandbox` call carries identity + credentials | +| **Implementation effort (Phase 1)** | ~19 points (shippable with ~3-4s latency) | ~24 points (shippable with <1s latency) | + +**Incremental adoption path:** The annotation-based approach can be implemented first as Phase 1, with the gRPC-push approach added later as an optimization for workloads requiring sub-second provisioning. The gateway configuration's `identity_method` field can select between the two at runtime. + +### Do Nothing + +Keep the current cold-start path. Users accept 16+ second startup. This is unacceptable for interactive agent workflows. + +### Env Var Injection via SandboxClaim + +Use the operator's `envVarsInjectionPolicy: Allowed` to inject identity at claim time. **Rejected** because the operator v0.9.0 bypasses the warm pool when env vars are present, creating a cold-start pod instead. See [upstream Issue #384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384). + +### Container Checkpoint/Restore (CRIU) + +Snapshot a ready supervisor container and restore it on demand. Avoids pool management entirely. **Deferred** because CRIU support on OpenShift is limited, and the warm pool approach achieves comparable latency with standard Kubernetes primitives. + +### Client-Side Sandbox Reuse + +Reuse an existing sandbox across multiple tool invocations within the same agent session. Reduces cold-start frequency but does not eliminate it and changes the isolation model. **Complementary** to warm pooling, not a replacement. + +## Prior Art + +- **Agent Sandbox `SandboxWarmPool` CRD**: The operator's pool implementation, directly evaluated in this study. +- **Knative Cold-Start Mitigation**: Configurable minimum replica count for serverless workloads. Analogous pool pattern. +- **AWS Lambda SnapStart**: Pre-initializes function instances from snapshots. Similar intent, different mechanism. + +## Open Questions + +1. ~~Should the warm pool be created by the Helm chart (declarative) or by the gateway on first use (imperative)?~~ **Resolved: gateway-managed.** The gateway creates, scales, and deletes pool resources based on its configuration. This enables multi-image pools, independent scaling, and future dynamic pool creation. + +2. What is the minimum kubelet `--sync-frequency` that is safe for production clusters? The default of 60 seconds is too slow for annotation-based activation. + +3. Should the gateway support multiple warm pools with different resource profiles (e.g., CPU-only vs GPU-attached)? + +4. How should pool sizing be determined automatically? A fixed `replicas` count is simple but does not adapt to demand. HPA-based autoscaling of warm pools is a future consideration. + +5. ~~Does `additionalPodMetadata` actually patch annotations on an already-bound warm pod?~~ **Verified YES.** Annotations with custom domains (e.g., `openshell.io/sandbox-id`) are applied to the adopted warm pod without container restart. Labels and `agents.x-k8s.io/` domain annotations are rejected by the operator's allowlist. + +## References + +- [Feasibility Study Results](../../experiments/RESULTS.md): Full measurement data, methodology, cold-start breakdown, and identity binding analysis. +- [GitHub Issue #2157](https://github.com/NVIDIA/OpenShell/issues/2157): Warm pool integration feature request. +- [Agent Sandbox Issue #384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384): Upstream proposal for claim-time env var injection. +- [Agent Sandbox Documentation](https://agent-sandbox.sigs.k8s.io/docs/): CRD reference and getting started guides. diff --git a/rfc/NNNN-warm-pool-grpc-push/README.md b/rfc/NNNN-warm-pool-grpc-push/README.md new file mode 100644 index 0000000000..109d8b2818 --- /dev/null +++ b/rfc/NNNN-warm-pool-grpc-push/README.md @@ -0,0 +1,371 @@ +--- +authors: + - "@rhuss" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/2157 +--- + +# RFC NNNN: Warm Pool Integration with Always-On Supervisor (gRPC-Push) + +## Summary + +This RFC proposes an alternative warm pool integration for OpenShell's Kubernetes driver that keeps the supervisor process running in warm pool pods from provisioning time. At claim time, the gateway pushes sandbox identity directly to the supervisor via a new `ActivateSandbox` gRPC endpoint, bypassing the Kubernetes downward API propagation delay entirely. Global OPA policies are pre-compiled at pool time; only sandbox-specific policies compile at claim time. + +The target is sub-1-second sandbox provisioning (p50), compared to the ~3-4 seconds achievable with the companion [annotation-based RFC](../NNNN-warm-pool-feasibility/README.md). Both RFCs are equal alternatives for different latency targets and complexity appetites. + +## Motivation + +OpenShell's Kubernetes driver provisions a fresh sandbox pod for every `sandbox.create` request. On a typical cluster, this cold-start path takes 16-17 seconds (p50, with pre-pulled images), which is too slow for interactive agent workflows where sub-second tool calls are the norm. + +A feasibility study conducted on two independent ROSA HCP 4.22.3 clusters with the Red Hat Agent Sandbox operator v0.9.0 measured the following (see [Results Document](../../experiments/RESULTS.md) for full methodology and caveats): + +| Scenario | Image | p50 latency | +|----------|-------|-------------| +| Cold start (OpenShell) | sandbox (3.2 GB, PVC, init, supervisor) | 16,678 ms | +| Cold start (minimal) | pause (700 KB, no PVC, no init) | 2,784 ms | +| Warm pool claim | full sandbox spec (pre-provisioned) | 2,333 ms | +| Warm pool claim | pause (pre-provisioned) | 2,271 ms | + +The cold-start breakdown reveals that the supervisor itself is fast: 1.5 seconds for 8 sequential gRPC calls covering token issuance, configuration retrieval, policy loading, and session registration. Of that 1.5 seconds, ~80 ms is network time and ~1,400 ms is process startup plus OPA policy compilation. The remaining 15+ seconds comes from Kubernetes pod lifecycle (scheduling, PVC provisioning, init containers, image pulls) which warm pooling eliminates entirely. + +The annotation-based approach (see companion RFC) starts the supervisor at claim time and delivers ~3-4 seconds end-to-end. This RFC targets sub-1 second by eliminating supervisor startup from the claim-time critical path: the supervisor runs continuously in the warm pod, global policies are already compiled, and identity arrives via direct gRPC rather than through the Kubernetes control plane. + +## Non-goals + +- **Dynamic pool promotion.** Automatically creating warm pools for previously unseen images is a future extension. The initial design requires explicit pool configuration. +- **Non-Kubernetes compute drivers.** This RFC addresses the Kubernetes driver only. Analogous patterns for Docker (pre-created containers) and VM (snapshot restore) drivers are out of scope. +- **Code implementation.** This RFC describes the design. Implementation details (proto definitions, exact Rust API signatures) are deferred to the implementation phase. +- **Supervisor pre-connection to the gateway.** Pre-registering the supervisor session with the gateway before claim time would further reduce latency but introduces complex lifecycle management. It is excluded from the initial design. +- **Pool autoscaling.** HPA-based scaling of warm pools based on demand is a future optimization, not part of this proposal. + +## Proposal + +### Architecture Overview + +The gRPC-push architecture changes the claim-time flow by removing the Kubernetes downward API from the critical path and replacing it with a direct gRPC call from the gateway to the supervisor. + +```mermaid +sequenceDiagram + participant CLI + participant Gateway + participant K8s as K8s API + participant Operator as Agent Sandbox
Operator + participant Supervisor as Supervisor (warm pod) + + Note over Supervisor: Already running since pool time + Note over Supervisor: Global OPA policies compiled + + CLI->>Gateway: CreateSandbox + Gateway->>Gateway: Mint sandbox ID + JWT + Gateway->>K8s: Create SandboxClaim (no env vars) + K8s->>Operator: Reconcile claim + Operator->>Operator: Bind warm pod to claim + Operator-->>K8s: Update claim status (Ready) + K8s-->>Gateway: Claim Ready + pod IP + Gateway->>Supervisor: ActivateSandbox(id, name, policy) + Supervisor->>Supervisor: Compile sandbox-specific OPA + Supervisor->>Gateway: ActivateSandboxResponse(OK) + Supervisor->>Gateway: ConnectSupervisor + Gateway-->>CLI: Sandbox Ready + CLI->>Gateway: CreateSshSession +``` + +The key difference from the annotation-based approach: the supervisor is already running and has already compiled global OPA policies. The gateway pushes identity directly to the supervisor's pod IP (read from `.status.sandbox.podIPs` on the SandboxClaim), skipping the 1-2 second downward API annotation propagation delay and the 1.4 second process startup and OPA compilation overhead. + +### Always-On Supervisor with Two-Tier OPA Compilation + +In the gRPC-push model, the supervisor starts at pool provisioning time, not at claim time. This changes the supervisor lifecycle from a single startup sequence into a two-phase initialization: + +**Phase 1: Pool provisioning (before any claim)** + +The supervisor starts when the warm pool pod is created. It performs all initialization that does not depend on sandbox identity: + +1. Start process, initialize runtime +2. Compile global OPA policies (network rules, filesystem constraints applicable to all sandboxes) +3. Open the `ActivateSandbox` gRPC endpoint +4. Return 200 on `/readyz` to signal the pod is ready for claiming + +Global policies are stable across sandbox sessions. They include network egress rules, filesystem access constraints, and process execution policies that apply identically to every sandbox. Pre-compiling them at pool time removes the largest contributor to supervisor startup latency (approximately 1,400 ms of the 1,500 ms total). + +**Phase 2: Claim-time activation (after identity push)** + +When the gateway pushes sandbox identity via `ActivateSandbox`, the supervisor: + +1. Receives sandbox ID, name, and sandbox-specific policy configuration +2. Compiles only the sandbox-specific OPA policy delta (per-sandbox rules, provider-specific constraints) +3. Merges the delta with the pre-compiled global policy bundle +4. Calls `IssueSandboxToken`, `GetSandboxConfig`, and remaining gRPC calls (~80 ms) +5. Calls `ConnectSupervisor` to register the session with the gateway + +The sandbox-specific policy delta is expected to be small (a handful of rules compared to the full policy set), so its compilation time should be well under 200 ms. + +**Supervisor state machine:** + +```mermaid +stateDiagram-v2 + [*] --> INITIALIZING: Pod created + INITIALIZING --> IDLE: Global OPA compiled,\nActivateSandbox endpoint ready + IDLE --> ACTIVATING: ActivateSandbox received + ACTIVATING --> ACTIVE: Sandbox-specific OPA compiled,\nConnectSupervisor complete + ACTIVE --> TERMINATED: Session ends + TERMINATED --> [*]: Pod deleted by operator +``` + +| State | Description | `/readyz` | +|-------|-------------|-----------| +| INITIALIZING | Supervisor starting, global OPA compiling | 503 | +| IDLE | Global policies compiled, waiting for identity push | 200 | +| ACTIVATING | Received identity, compiling sandbox-specific policies | 200 | +| ACTIVE | Fully activated, SSH ready, serving session | 200 | +| TERMINATED | Session ended, pod will be deleted by operator | 503 | + +The `/readyz` endpoint returns 200 in both IDLE and ACTIVATING states so the pod remains Ready in the warm pool. The pool controller uses pool membership, not readiness, to determine claimability. + +### gRPC Identity Push Protocol + +After the operator binds a claim to a warm pod, the gateway reads the pod IP from `.status.sandbox.podIPs` on the adopted Sandbox resource and pushes sandbox identity to the supervisor. + +**Endpoint definition (conceptual):** + +```protobuf +service SandboxActivation { + rpc ActivateSandbox(ActivateSandboxRequest) returns (ActivateSandboxResponse); +} + +message ActivateSandboxRequest { + string sandbox_id = 1; + string sandbox_name = 2; + PolicyConfig policy_config = 3; + map provider_environment = 4; + // User-scoped credentials from OBO token exchange (when sandbox acts on behalf of a user) + UserCredentials user_credentials = 5; +} + +message ActivateSandboxResponse { + enum Status { + OK = 0; + FAILED = 1; + } + Status status = 1; + string error_message = 2; + google.protobuf.Timestamp ready_at = 3; +} +``` + +**Security:** The gRPC channel between gateway and supervisor uses the existing namespace mTLS certificates. The TLS client secret volume is already mounted in the warm pod template (the same volume mount used by the cold-start path). No new certificate infrastructure is needed. + +**Timeout and retry policy:** The gateway applies a 2-second timeout per `ActivateSandbox` attempt. If the first attempt fails (network error, supervisor crash, TLS handshake failure), the gateway retries once. If both attempts fail, the gateway falls back to cold-start provisioning. This ensures that a crashed warm pod never blocks sandbox creation. + +**Pod IP discovery:** The gateway reads the pod IP from the SandboxClaim's `.status.sandbox.podIPs` field after the claim reaches Ready status. This field is populated by the operator when the claim binds to a warm pod. The pod IP is routable from the gateway pod because both run in the same namespace with standard Kubernetes networking. + +**User-scoped credentials and token exchange.** When sandboxes act on behalf of a specific user (OBO pattern), the gateway exchanges the user's token for a sandbox-scoped credential at claim time. The `ActivateSandbox` call carries both sandbox identity and user-scoped credentials in a single message (`user_credentials` field), eliminating the need for a separate credential delivery mechanism. This is a key advantage over the annotation-based approach, where annotations cannot carry sensitive credential material and a separate Secret mount or post-activation gRPC call is needed. + +In SPIFFE/SPIRE deployments, the SPIRE agent may need to re-attest the warm pod after claim adoption because the pod's attestation properties change. Re-attestation is fast (~10-20ms) and can run in parallel with the `ActivateSandbox` call. + +### Gateway-Managed Pool Lifecycle + +The gateway owns the lifecycle of SandboxTemplate and SandboxWarmPool Kubernetes resources. Pool configuration is defined in the gateway config file, and the gateway reconciles the cluster state against this configuration at startup. + +**Configuration:** + +```toml +[[compute.warm_pools]] +image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" +replicas = 5 + +[[compute.warm_pools]] +image = "ghcr.io/nvidia/openshell-community/sandboxes/ollama:latest" +replicas = 2 +``` + +**Startup reconciliation:** + +When the gateway starts, it reads the `compute.warm_pools` configuration and reconciles against the namespace: + +1. For each configured pool entry, check if a matching SandboxTemplate and SandboxWarmPool exist. +2. If missing, create both resources with the specified image and replica count. +3. If present but with a different replica count, update the SandboxWarmPool's `replicas` field. +4. If a SandboxWarmPool exists in the namespace but has no matching config entry, delete both the pool and its template. + +This ensures the cluster state matches the gateway config after every restart. Operators change pool configuration by editing the gateway config and restarting, not by manually managing Kubernetes resources. + +**Validation:** The gateway validates the warm pool configuration at startup. Duplicate image entries are rejected. Replica counts must be positive integers. Invalid configurations cause the gateway to fail fast with a clear error message rather than starting with a broken pool state. + +**RBAC requirements:** + +```yaml +- apiGroups: ["extensions.agents.x-k8s.io"] + resources: ["sandboxclaims"] + verbs: ["create", "get", "watch", "delete"] +- apiGroups: ["extensions.agents.x-k8s.io"] + resources: ["sandboxwarmpools"] + verbs: ["create", "get", "list", "watch", "update", "delete"] +- apiGroups: ["extensions.agents.x-k8s.io"] + resources: ["sandboxtemplates"] + verbs: ["create", "get", "list", "update", "delete"] +``` + +### Multi-Image Pool Routing and Cold-Start Fallback + +The gateway maintains a pool registry mapping container images to their SandboxWarmPool resources. When a sandbox creation request arrives, the gateway looks up the requested image in this registry. + +**Pool selection:** + +``` +For each sandbox creation request: + 1. Resolve the requested image reference + 2. Look up the image in the pool registry + 3. If a pool exists AND has readyReplicas > 0: + → Create SandboxClaim against that pool (warm path) + 4. If a pool exists BUT has 0 readyReplicas: + → Fall back to cold-start provisioning + 5. If no pool exists for this image: + → Fall back to cold-start provisioning +``` + +**Cold-start fallback on pool exhaustion (FR-007):** When all replicas in a warm pool are claimed and the pool is temporarily empty, the gateway falls back to cold-start provisioning automatically. The user still gets a sandbox, just with higher latency (~16 seconds instead of sub-1 second). When the pool replenishes (the operator creates fresh pods to replace claimed ones), subsequent requests use warm pods again. + +**Pool exhaustion is expected under burst load.** The feasibility study confirmed that burst claims of 5 simultaneous requests against a pool of 5 replicas succeed, with the pool replenishing between rounds. If the burst exceeds the pool size, the gateway's fallback path ensures zero sandbox creation failures. + +**Unknown images always cold-start.** Custom Dockerfiles, arbitrary image references, and first-time images go through the existing cold-start path. Warm pooling requires a pre-built image with a matching pool configuration. + +### Comparison with Annotation-Based Approach + +Both this RFC and the [annotation-based RFC](../NNNN-warm-pool-feasibility/README.md) solve the same problem (warm pool integration for reduced sandbox startup latency) with different tradeoffs. Neither is primary. The choice depends on the latency target and acceptable complexity. + +| Dimension | Annotation-based (companion RFC) | gRPC-push (this RFC) | +|-----------|----------------------------------|----------------------| +| **Claim-to-ready latency** | ~3-4s (claim + supervisor startup + downward API) | <1s target (claim + gRPC push + delta OPA) | +| **Supervisor complexity** | Idle-start: new mode that watches annotations, then runs normal startup | Always-on: runs from pool time, two-tier OPA, new gRPC endpoint | +| **Gateway complexity** | Claim creation + annotation patch | Claim creation + pool reconciler + gRPC client + pod IP discovery | +| **OPA compilation** | Full compilation at claim time (~1,400 ms) | Pre-compiled globals at pool time; sandbox-specific delta at claim time (<200 ms) | +| **Security surface** | Standard K8s downward API (no new endpoints) | New gRPC endpoint on supervisor (mTLS-secured) | +| **Upstream alignment** | Matches @craig-kindo's validated GKE implementation ([#1447](https://github.com/NVIDIA/OpenShell/issues/1447)) | Novel approach, not yet validated externally | +| **Resource cost (idle pools)** | Sleeping process (`sleep infinity`), minimal CPU/memory | Running supervisor process, ~100m CPU / 128Mi memory per pod | +| **User-scoped credentials (OBO)** | Separate mechanism needed (Secret mount or post-activation gRPC) | Single `ActivateSandbox` call carries identity + credentials | +| **Implementation effort (Phase 1)** | ~19 points (shippable with ~3-4s latency) | ~24 points (shippable with <1s latency) | + +**When to choose annotation-based:** The annotation approach is simpler, aligns with an independently validated implementation, and delivers a 4-5x improvement over cold start. It is the lower-risk option for teams that do not need sub-second provisioning. + +**When to choose gRPC-push:** The gRPC-push approach targets sub-second provisioning by eliminating all claim-time startup overhead. It is the right choice for workloads where every second of sandbox startup latency matters (high-frequency agent tool calls, interactive development sessions with rapid sandbox cycling). + +**Incremental adoption:** The annotation-based approach can be implemented first (Phase 1 in the companion RFC), with the gRPC-push approach added later as an optimization. The gateway configuration's `identity_method` field can select between the two at runtime. + +## Implementation Plan + +The implementation is phased to deliver incremental value. Phase 1 delivers a shippable warm pool with the always-on supervisor and gRPC-push identity binding (pools created manually or via Helm). Phase 2 adds gateway-managed pool lifecycle automation and multi-image routing. + +Points use a normalized scale consistent with the [annotation-based RFC](../NNNN-warm-pool-feasibility/README.md) so that effort can be compared across approaches: 1 = trivial, 2 = small, 3 = medium, 5 = large. + +### Phase 1: Shippable Warm Pool with gRPC-Push + +| Work item | Points | Crate/Component | +|-----------|--------|-----------------| +| Supervisor always-on mode with two-tier OPA | 5 | `openshell-sandbox` | +| `ActivateSandbox` gRPC endpoint on supervisor | 3 | `openshell-sandbox`, `proto/` | +| Gateway gRPC client for identity push | 3 | `openshell-server` | +| K8s driver claim path + cold-start fallback | 3 | `openshell-driver-kubernetes` | +| Gateway pool-aware CreateSandbox | 2 | `openshell-server` | +| mTLS configuration for gRPC push channel | 1 | `openshell-sandbox`, `openshell-server` | +| Helm chart (template, pool, RBAC) | 2 | `deploy/` | +| Gateway config (`[compute.warm_pool]`) | 1 | `openshell-server` | +| E2E tests on OpenShift | 3 | `tests/` | +| Documentation | 1 | `docs/` | +| **Phase 1 total** | **~24** | | + +### Phase 2: Gateway-Managed Pool Automation + +| Work item | Points | Crate/Component | +|-----------|--------|-----------------| +| Gateway pool reconciler (create/update/delete pools) | 5 | `openshell-server` | +| Multi-image pool registry and routing | 2 | `openshell-server` | +| Cold-start fallback on pool exhaustion | 1 | `openshell-driver-kubernetes` | +| **Phase 2 total** | **~8** | | + +**Combined total: ~32 points.** Phase 1 is shippable with manually created or Helm-managed pools. Phase 2 adds gateway-driven pool lifecycle automation. + +## Risks + +**New gRPC endpoint surface area.** The `ActivateSandbox` endpoint on the supervisor is a new attack surface. A compromised pod in the same namespace could attempt to push a false identity to a warm pod. Mitigation: the endpoint is mTLS-secured using the existing namespace certificates, and the supervisor validates the caller's certificate against the gateway's identity. The endpoint only accepts calls before activation (IDLE state); once activated, subsequent calls are rejected. + +**Gateway complexity increase.** The gateway gains a pool reconciler (Kubernetes resource management), a gRPC client (identity push), and pod IP routing. This is significantly more complex than the annotation-based approach, which requires only a SandboxClaim and an annotation patch. Mitigation: the pool reconciler is a standard Kubernetes controller pattern, and the gRPC client reuses the existing mTLS infrastructure. Both components can be feature-flagged. + +**Unvalidated approach.** The annotation-based approach has been validated by @craig-kindo's independent GKE implementation ([#1447](https://github.com/NVIDIA/OpenShell/issues/1447)). The gRPC-push approach is novel and has not been validated externally. Mitigation: the feasibility study confirms the underlying mechanics (warm pool claim binding, pod IP availability). The gRPC-push itself is a straightforward RPC on a known pod IP. + +**Resource cost of idle pools.** Always-on supervisors consume more resources than sleeping processes. A pool of 5 replicas with running supervisors uses approximately 500m CPU and 640 Mi memory, compared to negligible resource usage with `sleep infinity`. Mitigation: resource requests for idle supervisors can be set low (100m CPU, 128 Mi memory) since the supervisor does minimal work in IDLE state. Active supervisors can request higher limits via resource overrides at claim time. + +**Outdated supervisor binary in warm pods.** If the supervisor image is updated but existing warm pods are not replaced, the running supervisor may have an outdated binary. Mitigation: the SandboxWarmPool's `OnReplenish` update strategy gradually replaces pods as they are claimed and replenished. New pods get the latest image. + +**Kubernetes API connectivity loss during pool reconciliation.** If the gateway loses connectivity to the Kubernetes API during startup reconciliation, pool state may be inconsistent. Mitigation: the gateway uses its last-known pool state and retries reconciliation on a backoff interval. Pool reconciliation failures are non-fatal; the gateway can still serve requests using existing pools. + +**mTLS certificate expiry.** If the namespace mTLS certificates expire between pool provisioning and claim time, the gRPC push will fail with a TLS handshake error. Mitigation: the same retry/fallback policy applies (2-second timeout, 1 retry, then cold-start fallback). Certificate rotation should be automated with cert-manager or equivalent. + +## Alternatives + +### Annotation-Based Identity Binding (Companion RFC) + +The [annotation-based RFC](../NNNN-warm-pool-feasibility/README.md) proposes starting the supervisor at claim time and injecting identity via Kubernetes pod annotations projected through the downward API. The supervisor detects its identity when the annotation file changes and proceeds with normal startup. + +This approach is simpler (no new gRPC endpoint, no gateway pool reconciler) and aligns with @craig-kindo's independently validated GKE implementation. The tradeoff is higher claim-to-ready latency (~3-4 seconds vs sub-1 second) due to supervisor startup and downward API propagation delay. See the [comparison table](#comparison-with-annotation-based-approach) for a detailed breakdown. + +### ConfigMap-Based Identity Injection + +The gateway creates a ConfigMap with sandbox identity before creating the SandboxClaim. The SandboxTemplate includes a volume mount for this ConfigMap path. After the claim adopts a warm pod, the Kubernetes projected volume propagates the ConfigMap contents into the running container. + +Rejected because ConfigMap propagation delay is 60-90 seconds with default kubelet settings, far too slow for interactive use. While configurable via `kubelet --sync-frequency`, this requires cluster-level configuration changes that most operators cannot make. + +### Gateway-Native Pool (No Operator Extension CRDs) + +The gateway manages its own pool of pre-provisioned Sandbox resources, bypassing the operator's SandboxWarmPool CRD. The gateway creates N sandbox pods at startup and assigns them to incoming requests. + +Rejected because it duplicates the Agent Sandbox operator's pool management logic (claim binding, pod rotation, replenishment), which the `SandboxWarmPool` CRD already handles. The Agent Sandbox operator owns the full warm pod lifecycle: it provisions replacement pods when claimed ones are consumed, applies update strategies (`Recreate` vs `OnReplenish`) for template changes, and enforces single-use semantics (each pod serves exactly one session). Re-implementing this in the gateway creates a maintenance burden that grows with every upstream operator release. + +### Do Nothing + +Keep the current cold-start path. Users accept 16+ second startup latency. This is unacceptable for interactive agent workflows where sub-second tool calls are the norm. + +## Prior art + +**Agent Sandbox `SandboxWarmPool` CRD.** The Agent Sandbox operator (kubernetes-sigs/agent-sandbox) provides the SandboxWarmPool extension CRD that this RFC builds on. The operator handles pool provisioning, claim binding, pod rotation, and replenishment. The feasibility study validated these mechanics on the Red Hat operator v0.9.0. + +**@craig-kindo's GKE warm pool implementation ([OpenShell#1447](https://github.com/NVIDIA/OpenShell/issues/1447)).** An independent implementation that validates the annotation-based approach on GKE, measuring 1.9s p50 claim latency. While this validates the companion RFC's approach rather than the gRPC-push approach, it confirms that the underlying warm pool mechanics work across cloud providers. + +**Agent Sandbox operator adoption improvements ([agent-sandbox#1118](https://github.com/kubernetes-sigs/agent-sandbox/pull/1118)).** This upstream PR removes cache-lag requeue deferral during warm pool adoption finalization, reducing claim binding latency. Both the annotation and gRPC-push approaches benefit from this improvement. + +**Feasibility study ([experiments/RESULTS.md](../../experiments/RESULTS.md)).** The study conducted on two independent ROSA HCP clusters provides the measurement data, cold-start breakdown, identity binding analysis, and architectural constraints that inform both this RFC and the companion RFC. Key findings: 2.3s claim latency (reproducible), env var injection bypasses warm pool, annotation injection works without container restart, supervisor startup is 1.5s (80ms network + 1,400ms OPA compilation). + +## Open questions + +1. **Proto definition for `ActivateSandbox`.** The exact protobuf message structure needs to be finalized. The `PolicyConfig` field in `ActivateSandboxRequest` must carry enough information for sandbox-specific OPA compilation without requiring the supervisor to call back to the gateway for additional policy data. + +2. **Global vs sandbox-specific OPA policy boundary.** Which OPA policies qualify as "global" (pre-compiled at pool time) and which are "sandbox-specific" (compiled at claim time)? The boundary must be well-defined so that global policy changes trigger pool replenishment while sandbox-specific rules can vary per session. + +3. **Image update handling for always-on supervisors.** When a new supervisor image is pushed and the pool template is updated, existing warm pods still run the old binary. The `OnReplenish` strategy handles this gradually, but there may be a window where claimed pods run outdated supervisors. Should the gateway track supervisor versions and prefer newer pods? + +4. **Pool reconciler design.** Should the gateway's pool reconciler run as a background loop (periodic reconciliation) or be event-driven (watch SandboxWarmPool status changes)? A background loop is simpler but may have delayed reactions to pool state changes. + +5. **Supervisor resource limits at activation.** Should the gateway patch the warm pod's resource limits when pushing identity (increasing from idle-mode limits to active-mode limits)? This would prevent resource contention between idle and active supervisors in the same pool, but adds a pod patch to the critical path. + +### Future Extensions + +**Dynamic pool promotion.** When the gateway creates a cold-start sandbox for an image that has no pool, it could optionally create a new SandboxTemplate and SandboxWarmPool for that image. This "promotes" a first-seen image into the pool system. A configuration flag would control this behavior: + +```toml +[compute.warm_pools] +auto_promote = true +auto_promote_min_replicas = 2 +max_pools = 10 +``` + +This trades idle resource cost for latency reduction on subsequent requests with the same image. It is intentionally excluded from the initial design to keep the pool management surface simple and predictable. + +## References + +- [Feasibility Study Results](../../experiments/RESULTS.md): Full measurement data, methodology, cold-start breakdown, and identity binding analysis. +- [Annotation-Based Warm Pool RFC](../NNNN-warm-pool-feasibility/README.md): Companion RFC proposing annotation-based identity binding with claim-time supervisor startup. +- [GitHub Issue #2157](https://github.com/NVIDIA/OpenShell/issues/2157): Warm pool integration feature request. +- [GitHub Issue #1447](https://github.com/NVIDIA/OpenShell/issues/1447): @craig-kindo's warm pool implementation proposal with GKE validation. +- [Agent Sandbox PR #1118](https://github.com/kubernetes-sigs/agent-sandbox/pull/1118): Operator adoption finalization improvement. +- [Agent Sandbox Issue #384](https://github.com/kubernetes-sigs/agent-sandbox/issues/384): Upstream proposal for claim-time env var injection. +- [Agent Sandbox Documentation](https://agent-sandbox.sigs.k8s.io/docs/): CRD reference and getting started guides. diff --git a/specs/001-warm-pool-feasibility/REVIEWERS.md b/specs/001-warm-pool-feasibility/REVIEWERS.md new file mode 100644 index 0000000000..ad7795cb41 --- /dev/null +++ b/specs/001-warm-pool-feasibility/REVIEWERS.md @@ -0,0 +1,128 @@ +# Review Guide: Warm Pool Feasibility Study + +**Generated**: 2026-07-09 | **Spec**: [spec.md](spec.md) + +## Why This Change + +OpenShell's Kubernetes driver creates a fresh Sandbox CR for every sandbox request, paying 8-12 seconds for pod scheduling, image pull, init container execution, supervisor startup, and gateway registration. For agent harnesses like OpenClaw that create sandboxes per tool call, this latency is unusable. The upstream Agent Sandbox project (v0.5.0) provides extension CRDs for warm pooling (SandboxTemplate, SandboxWarmPool, SandboxClaim), but OpenShell has no awareness of these. We need to know if warm pooling can deliver sub-2s sandbox startup on OpenShift before investing in integration work. + +## What Changes + +This study will create an `experiments/` directory with shell-based measurement scripts, Kubernetes manifests for warm pool configurations, and a minimal Go sidecar binary for readiness experiments. It will provision a short-lived ROSA HCP cluster, run 6 experiments measuring cold-start vs warm pool latency across readiness configurations, and produce an RFC in `rfc/` with data-backed architectural recommendations for OpenShell warm pool integration. No changes to OpenShell core code. This PR contains the specification artifacts only; implementation follows after spec approval. + +## How It Works + +The study uses a layered measurement approach across 4 execution phases: + +1. **Cluster Setup**: ROSA HCP with 3x m5.2xlarge workers, Red Hat Agent Sandbox tech preview operator (upstream fallback), OpenShell via the OpenShell OpenShift deploy chart, and image pre-pulling via DaemonSet. + +2. **Measurement Library**: Shared shell functions (`experiments/lib/common.sh`) for nanosecond timestamp capture, CSV output, pod event collection, and p50/p90 computation. Each experiment script wraps kubectl with these functions. + +3. **Experiments**: Six experiments progressively build understanding: + - Cold-start baseline (control measurement) + - Warm pool claim-to-ready with default and aggressive readiness probes + - Pod Readiness Gates (KEP-580) as probe-free alternative + - Sidecar readiness pattern (Knative-style, using KEP-753 native sidecars) + - Env var injection at claim time (identity binding feasibility) + - Combined best-configuration measurement + +4. **RFC**: Results compiled into a standalone RFC following the OpenShell RFC template, structured for later distillation into a GitHub Issue #2157 comment. + +## When It Applies + +**Applies when**: +- Evaluating whether OpenShell should adopt Agent Sandbox warm pooling +- Understanding sandbox startup latency bottlenecks on OpenShift +- Making architectural decisions about OpenShell's Kubernetes driver +- Informing Issue #2157 design discussions with measured data + +**Does not apply when**: +- Podman/Docker single-player warm pooling (explicitly out of scope) +- Production OpenShell code changes (this is measurement-only, no core changes) +- Non-OpenShift Kubernetes distributions (results are OpenShift-specific) + +## Key Decisions + +1. **Layered measurement (no OpenShell code changes)**: Start with raw Agent Sandbox measurements, then layer OpenShell-specific concerns. Rejected "modify the Kubernetes driver first" because if the raw K8s layer is too slow, code changes would be wasted. + +2. **Red Hat tech preview operator first, upstream fallback**: The tech preview may lack extension CRDs, but testing the downstream operator path is part of the study's value. Upstream manifests are applied on top only if needed. + +3. **Shell scripts over Go benchmark harness**: N=10-20 per configuration is enough for a feasibility study. Shell scripts with `date +%s%N` and `kubectl wait` provide sufficient precision without over-engineering. + +4. **Results as RFC (not Google Doc)**: The RFC template provides structured review and lives in the public repo. Google Docs are inaccessible to external contributors. + +5. **3x m5.2xlarge workers**: Enough capacity for warm pool replicas and burst tests without hitting resource limits that would skew measurements. Short-lived cluster to control cost. + +## Areas Needing Attention + +- **Readiness probe interval assumption**: The brainstorm identifies the readiness probe interval as the dominant latency bottleneck. If this assumption is wrong (e.g., kubelet overhead dominates), the Readiness Gate and sidecar experiments may not show the expected improvement. +- **Env var injection behavior is undocumented**: The SandboxClaim `envVarsInjectionPolicy` behavior is based on reading the CRD spec, not tested behavior. The actual controller implementation may differ. +- **Sidecar readiness binary**: Building and pushing a custom container image adds complexity. If the sidecar experiment is not needed (readiness gates are sufficient), this work could be skipped. +- **RFC number assignment**: The OpenShell RFC process requires maintainer-assigned numbers from an originating issue. A placeholder number is used until assignment. + +## Open Questions + +- Does the Red Hat Agent Sandbox operator tech preview include extension CRDs, or only the core Sandbox CRD? (Answered by T002/T003 during setup.) +- What happens when the warm pool is exhausted? Does SandboxClaim stay Pending or fall back to cold start? (Answered by T024 during experiments.) +- Is the sidecar experiment necessary if Pod Readiness Gates already eliminate the probe interval bottleneck? (Decided after T033 results.) + +## Review Checklist + +- [ ] Key decisions are justified +- [ ] Measurement methodology is sound (N=10+, per-phase timestamps, CSV output) +- [ ] Scope boundaries are clear (no OpenShell core changes, K8s only) +- [ ] Success criteria are measurable (sub-2s target, p50/p90 computation) +- [ ] Cluster sizing is appropriate for experiment scope +- [ ] RFC structure follows the OpenShell RFC template +- [ ] No Google Drive links referenced in public-facing artifacts + +## Revision History + +### Rev 1 (2026-07-09) - Address Devin and CodeRabbit review feedback + +**Trigger**: PR review feedback from [#2](https://github.com/rhuss/OpenShell/pull/2) (Devin, CodeRabbit) + +**Spec changes**: None (all findings were in non-spec files) + +**Non-spec fixes**: +- AGENTS.md: Restored original OpenShell agent instructions (was replaced entirely by spex plugin description) +- CLAUDE.md: Removed feature-specific SPECKIT block (should not be in committed version) +- quickstart.md: Removed hardcoded AWS account ID +- brainstorm/04: Replaced hardcoded Obsidian vault path with generic placeholder +- brainstorm/02: Added note about pinning deployment chart to a specific commit + +**Quality gates**: +- review-spec: skipped (no spec changes) +- review-plan: skipped (no plan changes) + +**Cascade impact**: +- plan.md: unchanged +- tasks.md: unchanged +- REVIEWERS.md: revision history appended + +### Rev 2 (2026-07-09) - Address Copilot and Devin re-review feedback + +**Trigger**: PR review feedback from [#2](https://github.com/rhuss/OpenShell/pull/2) (Copilot, Devin re-review) + +**Spec changes**: None + +**Non-spec fixes**: +- .gitignore: Narrowed `**/.claude/` to worktree-specific pattern to avoid blocking root `.claude/` tracked files +- quickstart.md: Fixed DaemonSet path to `experiments/manifests/`, clarified `rosa:create` as Claude Code skill wrapper with CLI equivalent +- tasks.md: Clarified T001 `rosa:create` wrapper reference +- checklists/requirements.md: Fixed self-assessment to accurately reflect technical spec content +- brainstorm/04: Aligned decision with RFC approach, removed individual names and meeting references +- REVIEWERS.md: Reworded "What Changes" to describe planned work (not present artifacts) + +**Quality gates**: +- review-spec: skipped (no spec changes) +- review-plan: skipped (no plan changes) + +**Cascade impact**: +- plan.md: unchanged +- tasks.md: T001 wording updated (no structural change) +- REVIEWERS.md: revision history appended, "What Changes" reworded + +--- + + diff --git a/specs/001-warm-pool-feasibility/checklists/requirements.md b/specs/001-warm-pool-feasibility/checklists/requirements.md new file mode 100644 index 0000000000..7db637f3dc --- /dev/null +++ b/specs/001-warm-pool-feasibility/checklists/requirements.md @@ -0,0 +1,35 @@ +# Specification Quality Checklist: Warm Pool Feasibility Study + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-09 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [ ] No implementation details (languages, frameworks, APIs) -- spec references CLI commands and K8s version constraints as domain context for the feasibility study +- [x] Focused on user value and business needs +- [ ] Written for non-technical stakeholders -- spec is technical by nature (feasibility study targeting engineers) +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [ ] No implementation details leak into specification -- CLI commands and K8s constraints are domain context, not implementation details + +## Notes + +- Items marked [ ] are intentional deviations: this is a technical feasibility study spec, not a product feature spec. CLI commands and K8s version constraints are domain context necessary for the study. +- The spec references specific Kubernetes CRD names and KEP numbers as domain terminology, not implementation choices. diff --git a/specs/001-warm-pool-feasibility/data-model.md b/specs/001-warm-pool-feasibility/data-model.md new file mode 100644 index 0000000000..03ea626306 --- /dev/null +++ b/specs/001-warm-pool-feasibility/data-model.md @@ -0,0 +1,75 @@ +# Data Model: Warm Pool Feasibility Study + +## Entities + +### MeasurementRun + +A single sandbox creation or claim with captured timestamps. + +| Field | Type | Description | +|-------|------|-------------| +| run_number | int | Sequential run number within an experiment | +| experiment_id | string | Experiment identifier (e.g., `cold-start`, `warm-pool-default`) | +| config_label | string | Configuration description (e.g., `probe-10s`, `readiness-gate`) | +| create_timestamp | int64 | Nanosecond epoch when kubectl apply was issued | +| ready_timestamp | int64 | Nanosecond epoch when pod/sandbox became Ready | +| delta_ms | float | Latency in milliseconds (ready - create) | +| phases | string | JSON-encoded phase breakdown from events | +| pod_name | string | Name of the pod that was created or claimed | +| status | enum | `success`, `timeout`, `error` | + +### ExperimentConfig + +A specific combination of settings being tested. + +| Field | Type | Description | +|-------|------|-------------| +| experiment_id | string | Unique identifier | +| description | string | Human-readable description | +| readiness_method | enum | `probe-default`, `probe-1s`, `readiness-gate`, `sidecar` | +| pool_size | int | Number of SandboxWarmPool replicas (0 for cold-start) | +| env_injection | bool | Whether SandboxClaim includes env vars | +| target_runs | int | Number of measurement runs to execute | + +### PhaseTimestamp + +Per-phase latency breakdown extracted from pod events. + +| Field | Type | Description | +|-------|------|-------------| +| run_number | int | FK to MeasurementRun | +| phase_name | string | `scheduled`, `image-pulled`, `init-complete`, `supervisor-ready`, `ssh-available` | +| timestamp | int64 | Nanosecond epoch of phase completion | +| delta_from_create_ms | float | Milliseconds from create to this phase | + +## Relationships + +``` +ExperimentConfig 1---* MeasurementRun +MeasurementRun 1---* PhaseTimestamp +``` + +## State Transitions + +### MeasurementRun Lifecycle + +``` +Created --> Running --> Success + \--> Timeout (>60s) + \--> Error +``` + +### Warm Pool Pod Lifecycle + +``` +Provisioned (warm, NotReady) --> Claimed (SandboxClaim created) + --> Ready (readiness condition met) + --> Terminated (cleanup) +``` + +## CSV Output Format + +```csv +run,experiment,config,create_ts,ready_ts,delta_ms,pod,status +1,cold-start,openshell-prepulled,1720000000000000000,1720000010500000000,10500.0,sandbox-abc123,success +``` diff --git a/specs/001-warm-pool-feasibility/plan.md b/specs/001-warm-pool-feasibility/plan.md new file mode 100644 index 0000000000..e503740b3c --- /dev/null +++ b/specs/001-warm-pool-feasibility/plan.md @@ -0,0 +1,153 @@ +# Implementation Plan: Warm Pool Feasibility Study + +**Branch**: `6111-warm-pool-feasibility` | **Date**: 2026-07-09 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `specs/001-warm-pool-feasibility/spec.md` + +## Summary + +Evaluate Kubernetes Agent Sandbox warm pooling on a ROSA HCP OpenShift cluster to determine whether sandbox startup latency can be reduced from 8-12 seconds to under 2 seconds. The study provisions a short-lived cluster, runs 6 experiments with shell-based measurement scripts, and produces an RFC in `rfc/` with latency data and architectural recommendations for OpenShell integration. + +## Technical Context + +**Language/Version**: Bash (measurement scripts), Go (sidecar readiness binary), Markdown (RFC) +**Primary Dependencies**: ROSA HCP (AAET profile), Agent Sandbox operator + extension CRDs, OpenShell Helm chart (the OpenShift deploy wrapper), kubectl, oc +**Storage**: CSV files for measurement data, RFC markdown for results +**Testing**: Manual validation of measurement scripts, N=10-20 runs per configuration +**Target Platform**: ROSA HCP, OpenShift 4.20+ (K8s 1.33+), us-east-2, 3x m5.2xlarge workers +**Project Type**: Feasibility study / spike +**Performance Goals**: Measure sandbox startup latency, warm pool claim-to-ready target <2s +**Constraints**: Short-lived cluster (tear down after experiments), no code changes to OpenShell core +**Scale/Scope**: 6 experiments, 4+ configurations, 10-20 runs each, ~100 total measurement runs + +## Constitution Check + +*No project constitution defined (template only). Gate passes trivially.* + +## Project Structure + +### Documentation (this feature) + +```text +specs/001-warm-pool-feasibility/ +├── plan.md # This file +├── spec.md # Feature specification +├── research.md # Phase 0: resolved unknowns +├── data-model.md # Phase 1: measurement data model +├── quickstart.md # Phase 1: setup and execution guide +└── tasks.md # Phase 2 output (generated by /speckit-tasks) +``` + +### Source Code (repository root) + +```text +experiments/ +├── measure-cold-start.sh # Experiment 1: cold-start baseline +├── measure-warm-pool.sh # Experiment 2: warm pool claim latency +├── measure-readiness-gates.sh # Experiment 3: Pod Readiness Gates +├── measure-sidecar-readiness.sh # Experiment 4: sidecar readiness pattern +├── measure-env-injection.sh # Experiment 5: env var injection +├── measure-combined.sh # Experiment 6: combined best configuration +├── lib/ +│ ├── common.sh # Shared measurement functions (timestamps, CSV output) +│ └── wait-ready.sh # Pod readiness wait with timeout +├── manifests/ +│ ├── sandbox-template.yaml # SandboxTemplate for warm pool experiments +│ ├── warm-pool.yaml # SandboxWarmPool configurations +│ ├── sandbox-claim.yaml # SandboxClaim templates +│ ├── readiness-gate-pod.yaml # Pod with custom ReadinessGate +│ ├── sidecar-readiness.yaml # Sidecar container with readiness endpoint +│ └── image-prepull-daemonset.yaml # DaemonSet for pre-pulling images +├── sidecar/ +│ ├── main.go # Readiness sidecar binary (HTTP 503/200) +│ ├── Dockerfile # Multi-stage build for scratch image +│ └── Makefile # Build and push sidecar image +└── results/ + └── *.csv # Raw measurement data (gitignored) + +rfc/NNNN-warm-pool-feasibility/ +└── README.md # Results RFC (number assigned by maintainers) +``` + +**Structure Decision**: Experiments live in `experiments/` at repo root for easy discovery. The sidecar binary is self-contained in `experiments/sidecar/`. Results CSV files are gitignored (raw data is synthesized into the RFC). The RFC follows the standard `rfc/NNNN-name/README.md` pattern. + +## Global Constraints + +These constraints from the spec apply to all tasks: + +- **Kubernetes version**: 1.33+ (required for KEP-753 native sidecar containers GA) +- **OpenShift version**: 4.20+ (maps to K8s 1.33) +- **Cluster sizing**: 3 worker nodes, m5.2xlarge (8 vCPU, 32 GB each), us-east-2 +- **Cluster lifecycle**: Short-lived, tear down after experiments +- **Operator preference**: Red Hat Agent Sandbox tech preview first, upstream v0.5.0 manifests as fallback +- **Measurement format**: Shell scripts producing CSV output +- **Scope**: Kubernetes-backed sandboxes only (no Podman/Docker) + +## Measurement Library Interfaces + +Functions exported by `experiments/lib/common.sh` (consumed by all experiment scripts): + +```bash +capture_timestamp() # Returns nanosecond epoch (date +%s%N) +write_csv_header(file) # Writes CSV header row to file +write_csv_row(file, run, config, create_ts, ready_ts, delta_ms, pod, status) +collect_pod_events(pod) # Returns JSON phase breakdown from kubectl events +compute_stats(csv_file) # Computes p50/p90 from CSV delta_ms column, prints summary +``` + +Functions exported by `experiments/lib/wait-ready.sh`: + +```bash +wait_for_ready(resource, timeout_s) # Blocks until resource is Ready or timeout +wait_for_pod_ready(pod, ns, timeout_s) # Blocks until pod condition Ready=True +``` + +## Execution Phases + +### Phase 1: Cluster Provisioning & Setup + +1. Provision ROSA HCP cluster (3x m5.2xlarge, OpenShift 4.20+) +2. Install Red Hat Agent Sandbox operator from OperatorHub (fallback: upstream manifests) +3. Verify extension CRDs: SandboxTemplate, SandboxWarmPool, SandboxClaim +4. Deploy OpenShell via the OpenShell OpenShift deploy wrapper +5. Pre-pull sandbox images on all worker nodes (DaemonSet approach) +6. Validate: cold-start sandbox creation works end-to-end + +### Phase 2: Measurement Script Development + +1. Build shared measurement library (`experiments/lib/common.sh`) +2. Build cold-start measurement script (Experiment 1) +3. Build warm pool measurement script (Experiments 2, 3) +4. Build sidecar readiness binary and container +5. Build sidecar/readiness-gate measurement scripts (Experiments 3, 4) +6. Build env var injection measurement script (Experiment 5) +7. Build combined measurement script (Experiment 6) + +### Phase 3: Experiment Execution + +1. Run Experiment 1: Cold-start baseline (N=10 pre-pulled, N=5 without) +2. Run Experiment 2: Warm pool claim latency (default probes, aggressive probes) +3. Run Experiment 3: Pod Readiness Gates +4. Run Experiment 4: Sidecar readiness pattern +5. Run Experiment 5: Env var injection +6. Run Experiment 6: Combined best configuration + +### Phase 4: Results & RFC + +1. Compile CSV data into summary tables (p50, p90 per configuration) +2. Write RFC following `rfc/0000-template/README.md` format +3. Include: executive summary, experiment setup, results tables, architecture recommendations, gaps/risks, next steps +4. Tear down cluster + +## Risk Mitigations + +| Risk | Mitigation | +|------|------------| +| Red Hat TP operator lacks extension CRDs | Fallback: upstream `extensions.yaml` on top of operator | +| OpenShift 4.20 not available on ROSA HCP | Check `rosa list versions`, use latest available with K8s 1.33+ | +| the OpenShell OpenShift deploy chart incompatible with OpenShift 4.20 | Adjust chart values; worst case deploy OpenShell manually | +| Sidecar readiness requires custom image | Build minimal Go binary, push to ghcr.io | +| Env var injection triggers cold start | Document as constraint, propose alternative identity binding | + +## Complexity Tracking + +No constitution violations to justify. diff --git a/specs/001-warm-pool-feasibility/quickstart.md b/specs/001-warm-pool-feasibility/quickstart.md new file mode 100644 index 0000000000..4e7e4f7e88 --- /dev/null +++ b/specs/001-warm-pool-feasibility/quickstart.md @@ -0,0 +1,89 @@ +# Quickstart: Warm Pool Feasibility Study + +## Prerequisites + +- AWS credentials for AAET profile +- `rosa` CLI authenticated +- `oc` / `kubectl` CLI +- `gh` CLI for GitHub operations +- `jq` for JSON processing + +## Step 1: Provision Cluster + +```bash +# Using the rosa Claude Code skill (cc-rosa-rhoai plugin): +# rosa:create warm-pool-study +# Or equivalently via the rosa CLI: +rosa create cluster --cluster-name warm-pool-study --sts --hosted-cp \ + --replicas 3 --compute-machine-type m5.2xlarge --region us-east-2 +``` + +3 worker nodes, m5.2xlarge, us-east-2. Wait for cluster to be Ready (~15 min). + +## Step 2: Install Agent Sandbox Operator + +```bash +# Check if Red Hat TP is on OperatorHub +oc get packagemanifests -n openshift-marketplace | grep sandbox + +# If available, install via Subscription +# If not, apply upstream manifests +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.0/manifest.yaml +kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/v0.5.0/extensions.yaml + +# Verify CRDs +kubectl api-resources | grep agents +``` + +## Step 3: Deploy OpenShell + +```bash +# Clone the OpenShell OpenShift deploy repo (pin to a specific commit for reproducibility) +git clone https://github.com/2000krysztof/Openshell-Openshift-Deploy +cd Openshell-Openshift-Deploy +./deploy.sh +openshell status +``` + +## Step 4: Pre-pull Images + +```bash +# DaemonSet that pre-pulls sandbox images on all nodes +kubectl apply -f experiments/manifests/image-prepull-daemonset.yaml +kubectl rollout status ds/image-prepull +``` + +## Step 5: Run Experiments + +```bash +# Cold-start baseline (N=10 pre-pulled, N=5 no pre-pull) +./experiments/measure-cold-start.sh + +# Warm pool with default probes +./experiments/measure-warm-pool.sh --config default + +# Warm pool with 1s probes +./experiments/measure-warm-pool.sh --config aggressive + +# Readiness gates +./experiments/measure-readiness-gates.sh + +# Sidecar readiness +./experiments/measure-sidecar-readiness.sh + +# Env var injection +./experiments/measure-env-injection.sh +``` + +## Step 6: Generate RFC + +Compile CSV data into the RFC at `rfc/NNNN-warm-pool-feasibility/README.md`. + +## Step 7: Tear Down + +```bash +# Using the rosa Claude Code skill: +# rosa:delete warm-pool-study +# Or equivalently: +rosa delete cluster --cluster warm-pool-study +``` diff --git a/specs/001-warm-pool-feasibility/research.md b/specs/001-warm-pool-feasibility/research.md new file mode 100644 index 0000000000..3f29fa77f8 --- /dev/null +++ b/specs/001-warm-pool-feasibility/research.md @@ -0,0 +1,60 @@ +# Research: Warm Pool Feasibility Study + +## R1: ROSA HCP OpenShift Version Availability (K8s 1.33+) + +**Decision**: Target OpenShift 4.20+ which ships K8s 1.33+, providing native sidecar containers (KEP-753 GA) and Pod Readiness Gates (KEP-580 GA since K8s 1.14). + +**Rationale**: K8s 1.33 is required for native sidecar containers (init containers with `restartPolicy: Always`), which is needed for Experiment 4 (sidecar readiness pattern). OpenShift 4.20 maps to K8s 1.33. ROSA HCP versions are checked at provisioning time via `rosa list versions`. + +**Alternatives considered**: +- OpenShift 4.19 (K8s 1.32): Missing native sidecar GA. Rejected because it would block Experiment 4. +- Kind/k3d local cluster: Faster but not representative of enterprise OpenShift. Rejected because the study specifically targets OpenShift compatibility. + +## R2: Red Hat Agent Sandbox Operator (Tech Preview) Availability + +**Decision**: Install the Red Hat Agent Sandbox tech preview from OperatorHub first. If extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim) are not included, apply upstream `extensions.yaml` from kubernetes-sigs/agent-sandbox v0.5.0. + +**Rationale**: The tech preview may only include the core Sandbox CRD. The extension CRDs are in a separate API group (`extensions.agents.x-k8s.io/v1beta1`). Verification: `kubectl api-resources | grep agents` should list all four CRD types. + +**Alternatives considered**: +- Upstream manifests only: Skips the operator path entirely. Rejected because testing the Red Hat operator is part of the study's value. +- Wait for Red Hat TP to include extensions: Unknown timeline, would block the study. Rejected. + +## R3: Measurement Script Design + +**Decision**: Shell scripts per experiment using `kubectl` with `date +%s%N` for nanosecond timestamps. Output to CSV with columns: `run,config,create_ts,ready_ts,delta_ms,phases`. + +**Rationale**: Shell scripts are sufficient for N=10-20 per config. `kubectl wait --for=condition=Ready` provides clean blocking semantics. Phase timestamps are extracted from `kubectl get events --field-selector involvedObject.name=$POD` post-hoc. + +**Alternatives considered**: +- Go benchmark harness with client-go watch: More precise but overkill for feasibility study. Rejected. +- Manual copy-paste: Error-prone with 10+ runs. Rejected. + +## R4: RFC Format and Number Assignment + +**Decision**: The results RFC will follow the OpenShell RFC template (`rfc/0000-template/README.md`). The RFC number must be assigned by maintainers from the originating GitHub issue. For this study, the RFC is created in `rfc/NNNN-warm-pool-feasibility/README.md` once a number is assigned. + +**Rationale**: Per `rfc/README.md`, RFCs require an originating GitHub issue and maintainer-assigned number. The study results naturally fit the RFC structure (Summary, Motivation, Design, Alternatives). + +**Alternatives considered**: +- Informal markdown report: Lacks the structured review process. Rejected because warm pool integration is a cross-cutting architectural decision. +- Google Doc: Prohibited from public repo references per CLAUDE.md. Rejected. + +## R5: Sidecar Readiness Binary + +**Decision**: Use a minimal Go binary for the sidecar readiness experiment. The binary serves HTTP on :8080, returns 503 until a signal file (`/tmp/signal/ready`) exists, then returns 200. Build as a static binary in a scratch container. + +**Rationale**: Go produces small static binaries. The signal file mechanism (shared emptyDir volume) is simple to test. This mirrors the Knative queue-proxy pattern. + +**Alternatives considered**: +- Python/bash sidecar: Requires larger base image (python/bash), slower startup. Rejected. +- gRPC readiness: More complex than needed for a readiness probe. Rejected. + +## R6: Pool Exhaustion Behavior + +**Decision**: Document the behavior when all warm pool replicas are claimed. Expected: SandboxClaim stays Pending until pool replenishes (based on upstream Agent Sandbox controller logic). + +**Rationale**: This is an open question in the spec. The experiment will observe actual behavior. If fallback to cold start is available, it would be a useful feature for production. + +**Alternatives considered**: +- Active testing with configured fallback: The v0.5.0 SandboxWarmPool spec may not support fallback configuration. Observation-only is the safe approach. diff --git a/specs/001-warm-pool-feasibility/review-findings.md b/specs/001-warm-pool-feasibility/review-findings.md new file mode 100644 index 0000000000..30ea9dd035 --- /dev/null +++ b/specs/001-warm-pool-feasibility/review-findings.md @@ -0,0 +1,226 @@ +# Deep Review Findings + +**Date:** 2026-07-09 +**Branch:** 6111-warm-pool-feasibility +**Rounds:** 2 +**Gate Outcome:** PASS +**Invocation:** manual + +## Summary + +| Severity | Found | Fixed | Remaining | +|----------|-------|-------|-----------| +| Critical | 1 | 1 | 0 | +| Important | 7 | 5 | 2 | +| Minor | 8 | - | 8 | +| Notable | 6 | - | 6 | +| **Total** | **22** | **6** | **16** | + +**Agents completed:** 5/5 (+ 0 external tools) +**Agents failed:** CodeRabbit (timed out after 5+ minutes) + +## Findings + +### FINDING-1 +- **Severity:** Critical +- **Confidence:** 95 +- **File:** experiments/lib/common.sh:63-91 (defined), experiments/measure-cold-start.sh (not called) +- **Category:** test-quality +- **Source:** test-quality-agent (also reported by: architecture-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +`extract_phase_deltas()` was defined in lib/common.sh but never called by any measurement script. The CSV header declares per-phase columns but all scripts passed only 8 positional arguments to `write_csv_row`, leaving phase columns empty. FR-004 requires per-phase timestamps. + +**Why this matters:** +The per-phase breakdown is the entire point of User Story 1. Without it, cold-start measurements produce only a single end-to-end delta_ms, making it impossible to identify whether scheduling, image pulling, or supervisor startup is the bottleneck. + +**How it was resolved:** +Added calls to `extract_phase_deltas()` in measure-cold-start.sh for both prepulled/noprepull and vanilla config paths. Results are split via `IFS=',' read` and passed as positional arguments 9-13 to write_csv_row. + +### FINDING-2 +- **Severity:** Important +- **Confidence:** 95 +- **File:** experiments/lib/common.sh:63-91 +- **Category:** correctness +- **Source:** correctness-agent (also reported by: architecture-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +`extract_phase_deltas()` jq timestamp conversion used incompatible reference frames. The `ts_to_epoch` function produced an approximate day-count value (`year*365*86400 + month*30*86400`), while `$cs` was a real Unix epoch. The subtraction produced nonsensical values (off by ~62 billion seconds for 2026 dates). It also ignored the time-of-day component entirely. + +**Why this matters:** +After FINDING-1 was fixed (wiring up the function), this would have written garbage values into the per-phase CSV columns, corrupting the feasibility study data. + +**How it was resolved:** +Rewrote `extract_phase_deltas()` to use shell-based ISO-to-epoch conversion via `gdate -d` (macOS) or `date -d` (Linux), producing correct epoch seconds for proper delta computation. + +### FINDING-3 +- **Severity:** Important +- **Confidence:** 95 +- **File:** experiments/lib/common.sh:141-143 +- **Category:** correctness +- **Source:** correctness-agent +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +`compute_stats` awk uses `vals[NR-1]` for array indexing, but `NR-1` creates non-contiguous indices when rows are skipped (non-ok status or empty delta). The insertion sort iterates `1..n` contiguously, reading uninitialized slots (treated as 0 by awk) while actual values at higher indices are orphaned. This corrupts p50/p90 percentile values. + +**Why this matters:** +When any measurement run produces a timeout or failed row, the percentile statistics in the summary output are wrong. Unset array slots read as 0, pulling percentiles down. + +**How it was resolved:** +Changed `vals[NR-1] = v; n++` to `n++; vals[n] = v` so the array is contiguously indexed by the sample counter, not the line number. + +### FINDING-4 +- **Severity:** Important +- **Confidence:** 92 +- **File:** experiments/measure-warm-pool.sh:374-384 +- **Category:** production-readiness +- **Source:** production-agent (also reported by: correctness-agent, architecture-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +The aggressive config's trap on line 374 replaced the common.sh `_cleanup_on_exit` EXIT trap. Line 384 (`trap - EXIT INT TERM`) permanently cleared ALL traps. When running `--config all`, the burst config ran with zero cleanup traps, meaning Kubernetes resources leaked on interrupt. + +**Why this matters:** +Resources registered via `register_cleanup` would not be cleaned up if the script crashed or was interrupted after the aggressive config completed. + +**How it was resolved:** +Changed the trap to chain both handlers: `trap 'patch_readiness_probe $period; _cleanup_on_exit' EXIT`. After aggressive run, restore only the original common.sh trap instead of clearing all traps. + +### FINDING-5 +- **Severity:** Important +- **Confidence:** 90 +- **File:** experiments/measure-combined.sh:165-184, experiments/measure-env-injection.sh:102-125 +- **Category:** architecture +- **Source:** architecture-agent (also reported by: test-quality-agent) +- **Round found:** 1 +- **Resolution:** fixed (round 1) + +**What is wrong:** +Two inconsistent `detect_adoption()` implementations: combined.sh checked only Scheduled events, env-injection.sh checked Scheduled AND Pulled/Pulling events. The combined.sh version would misclassify cold-fallback pods as "warm-adopted" when images were cached. + +**Why this matters:** +The combined experiment is designed to produce the "best case" measurement. Incorrect adoption detection means the RFC results could contain falsely inflated warm pool performance data. + +**How it was resolved:** +Extracted the more thorough version (checking Scheduled + Pulled/Pulling) to lib/common.sh as a shared function. Removed both local implementations. Updated all call sites to pass namespace. + +### FINDING-6 +- **Severity:** Important +- **Confidence:** 90 +- **File:** experiments/measure-env-injection.sh, experiments/measure-combined.sh +- **Category:** production-readiness +- **Source:** production-agent +- **Round found:** 1 +- **Resolution:** remaining (low risk) + +**What is wrong:** +Both scripts patch SandboxTemplate envVarsInjectionPolicy and readiness probes but never restore original values on interrupt. If interrupted mid-run, the template remains modified. + +### FINDING-7 +- **Severity:** Important +- **Confidence:** 85 +- **File:** experiments/measure-warm-pool.sh:302-339 +- **Category:** test-quality +- **Source:** test-quality-agent +- **Round found:** 1 +- **Resolution:** remaining (measurement artifact, documented) + +**What is wrong:** +Burst claim latencies are measured sequentially despite being submitted in parallel. All 5 burst claims share a single `create_ts`, but readiness checks run sequentially. Claim j's `ready_ts` is captured only after claims 1 through j-1 have been waited on, inflating later claims' measured latency. + +## Minor Findings + +### FINDING-8 (Minor, correctness) +- **File:** experiments/lib/common.sh:12 +- **Source:** correctness-agent (also: architecture-agent) +- **Description:** `deregister_cleanup()` uses bash pattern substitution `${array[@]/$1}` which does substring matching, not exact match. Currently unused by any script. + +### FINDING-9 (Minor, architecture) +- **File:** experiments/lib/wait-ready.sh:7-10 +- **Source:** architecture-agent +- **Description:** `wait_for_ready()` is defined but never called. Dead code. + +### FINDING-10 (Minor, architecture) +- **File:** experiments/measure-combined.sh, experiments/measure-env-injection.sh +- **Source:** architecture-agent +- **Description:** `wait_claim_ready()`, `cleanup_claim()`, `get_claim_pod()` remain duplicated across two scripts. Should be extracted to lib/. + +### FINDING-11 (Minor, architecture) +- **File:** experiments/measure-combined.sh, experiments/measure-env-injection.sh +- **Source:** architecture-agent +- **Description:** Both scripts bypass `write_csv_header()`/`write_csv_row()`, writing CSV rows directly with `echo` and custom headers (adding adoption/behavior columns). + +### FINDING-12 (Minor, security) +- **File:** experiments/lib/common.sh:17 +- **Source:** security-agent +- **Description:** `kubectl delete $res` in cleanup handler uses unquoted variable expansion. Low risk since resource names are controlled. + +### FINDING-13 (Minor, security) +- **File:** experiments/manifests/sandbox-claim.yaml:14-15 +- **Source:** security-agent +- **Description:** Commented-out example includes predictable synthetic token `tok-xxx`. Clearly test data, no real risk. + +### FINDING-14 (Minor, production-readiness) +- **File:** experiments/manifests/sidecar-readiness.yaml:6-19 +- **Source:** production-agent +- **Description:** Readiness sidecar init container has no resource requests or limits. + +### FINDING-15 (Minor, test-quality) +- **File:** experiments/measure-warm-pool.sh:370-378 +- **Source:** test-quality-agent +- **Description:** Aggressive probe config does not verify that warm pool pods actually have updated probe configuration before measuring. If the pool controller doesn't recycle pods on template changes, measurements may use old probes. + +## Notable Observations + +### NOTABLE-1 +- **File:** experiments/sidecar/main.go:35-46 +- **Category:** architecture +- **Source:** architecture-agent +- **Description:** Sidecar readiness binary exits polling loop after signal file detection, stays "ready" permanently even if file is deleted +- **Rationale:** Adequate for the experiment but worth noting if pattern is promoted to production + +### NOTABLE-2 +- **File:** experiments/measure-warm-pool.sh:303-305 +- **Category:** correctness +- **Source:** correctness-agent +- **Description:** Background `kubectl apply` failures in burst mode are silently swallowed by `wait` with no operands +- **Rationale:** Transient API server errors during burst would be misdiagnosed as bind timeouts + +### NOTABLE-3 +- **File:** experiments/measure-combined.sh:199-296 +- **Category:** test-quality +- **Source:** test-quality-agent +- **Description:** Combined script does not wait for pool replenishment between runs, unlike measure-warm-pool.sh +- **Rationale:** Later runs may measure cold-start fallback latency instead of warm pool claims + +### NOTABLE-4 +- **File:** rfc/NNNN-warm-pool-feasibility/README.md +- **Category:** architecture +- **Source:** architecture-agent +- **Description:** RFC document has TBD placeholders throughout all results and recommendation sections +- **Rationale:** Expected at this stage (pre-experiment), document structure is complete and ready to receive data + +### NOTABLE-5 +- **File:** experiments/manifests/sidecar-readiness.yaml, readiness-gate-pod.yaml +- **Category:** security +- **Source:** security-agent +- **Description:** Pod manifests lack securityContext (no runAsNonRoot, no dropped capabilities) +- **Rationale:** Acceptable for ephemeral experiment pods + +### NOTABLE-6 +- **File:** experiments/measure-cold-start.sh:85-138 +- **Category:** test-quality +- **Source:** test-quality-agent +- **Description:** The `noprepull` config uses the same code path as `prepulled` with no validation that images are actually uncached. If the prepull DaemonSet is still running, results will be indistinguishable. +- **Rationale:** Worth adding a pre-flight warning but not blocking + +## Test Suite Results + +No test command detected; post-fix test step was skipped. diff --git a/specs/001-warm-pool-feasibility/spec.md b/specs/001-warm-pool-feasibility/spec.md new file mode 100644 index 0000000000..ea0d1fd574 --- /dev/null +++ b/specs/001-warm-pool-feasibility/spec.md @@ -0,0 +1,148 @@ +# Feature Specification: Warm Pool Feasibility Study + +**Feature Branch**: `6111-warm-pool-feasibility` +**Created**: 2026-07-09 +**Status**: Draft +**Input**: Evaluate Kubernetes Agent Sandbox warm pooling on OpenShift to determine whether sandbox startup latency can be reduced from 8-12 seconds to under 2 seconds, and produce architectural recommendations for OpenShell integration. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Measure Cold-Start Baseline on OpenShift (Priority: P1) + +An engineer provisions a ROSA HCP cluster, deploys OpenShell with the Agent Sandbox operator, and measures the current cold-start sandbox creation latency with per-phase timestamps. This establishes the control measurement against which all warm pool experiments are compared. + +**Why this priority**: Without a reliable baseline, no warm pool improvement can be quantified. This is the foundation for every subsequent experiment. + +**Independent Test**: Can be tested by creating 10+ sandboxes on a clean cluster and recording timestamps from API call to SSH availability. Delivers a latency breakdown table. + +**Acceptance Scenarios**: + +1. **Given** a ROSA HCP cluster with OpenShell deployed via the OpenShell OpenShift deploy chart, **When** the engineer runs `openshell sandbox create --from base` 10 times with pre-pulled images, **Then** per-phase timestamps are captured (scheduled, image pulled, init complete, supervisor ready, SSH available) and p50/p90 latencies are computed. +2. **Given** the same cluster, **When** the engineer runs 5 sandbox creates without pre-pulled images, **Then** the image pull contribution to total latency is quantified separately. +3. **Given** the same cluster, **When** the engineer creates a vanilla Agent Sandbox (no OpenShell) 10 times, **Then** the OpenShell overhead vs. raw Kubernetes overhead is isolated. + +--- + +### User Story 2 - Measure Warm Pool Claim Latency (Priority: P1) + +An engineer creates a SandboxWarmPool with pre-provisioned pods, then measures the time from SandboxClaim creation to the claimed sandbox becoming Ready. Tests multiple readiness configurations to identify the dominant bottleneck. + +**Why this priority**: This is the core feasibility question. If raw warm pool claim-to-ready latency exceeds the target, no amount of OpenShell optimization will help. + +**Independent Test**: Can be tested by creating SandboxTemplate + SandboxWarmPool, then issuing SandboxClaims and measuring claim-to-ready timestamps. Delivers a configuration matrix with latency data. + +**Acceptance Scenarios**: + +1. **Given** a SandboxWarmPool with 5 pre-provisioned replicas using the base sandbox image, **When** a SandboxClaim is created, **Then** the claim-to-ready latency is measured with default readiness probe settings (10s periodSeconds). +2. **Given** the same pool, **When** a SandboxClaim is created with aggressive readiness probes (1s periodSeconds), **Then** the latency improvement over default probes is quantified. +3. **Given** the same pool, **When** 5 SandboxClaims are created simultaneously, **Then** burst claim behavior and pool replenishment time are measured. + +--- + +### User Story 3 - Test Health Check Optimization Patterns (Priority: P2) + +An engineer tests Pod Readiness Gates (KEP-580) and a Knative-style sidecar readiness pattern as alternatives to polling-based readiness probes, measuring whether they eliminate the probe interval bottleneck. + +**Why this priority**: Our research identified the readiness probe interval as the dominant latency bottleneck. These optimization patterns could eliminate the bottleneck entirely, but need validation on OpenShift. + +**Independent Test**: Can be tested by deploying pods with custom ReadinessGate conditions and sidecar containers, then measuring the time from condition/signal flip to pod Ready status. + +**Acceptance Scenarios**: + +1. **Given** a warm-pooled pod with a custom ReadinessGate condition (`sandbox.openshell.io/claimed`) set to False, **When** an external controller patches the condition to True, **Then** the time from patch to pod Ready is measured and recorded. +2. **Given** a warm-pooled pod with a sidecar container that controls readiness via an HTTP endpoint, **When** a signal file is created in a shared emptyDir volume, **Then** the sidecar flips its readiness response and the pod transitions to Ready within measured latency. +3. **Given** both patterns tested, **When** results are compared with probe-based readiness (1s and 10s), **Then** the relative improvement is quantified. + +--- + +### User Story 4 - Test Claim-Time Environment Injection (Priority: P2) + +An engineer tests whether SandboxClaim env var injection works without forcing a cold start, validating the mechanism OpenShell would use to bind identity to a pre-provisioned pod. + +**Why this priority**: OpenShell needs to inject sandbox identity (ID, endpoint, credentials) at claim time. If env var injection triggers cold start, the warm pool advantage is lost and a different identity binding mechanism is needed. + +**Independent Test**: Can be tested by creating a SandboxTemplate with `envVarsInjectionPolicy: Allowed` and a SandboxClaim with env vars, then observing whether the claim adopts a warm sandbox or creates a new one. + +**Acceptance Scenarios**: + +1. **Given** a SandboxTemplate with `envVarsInjectionPolicy: Allowed` and a SandboxWarmPool with 3 replicas, **When** a SandboxClaim is created with env vars (`OPENSHELL_SANDBOX_ID=test-123`), **Then** the claim adopts an existing warm sandbox (not cold start). +2. **Given** the same setup, **When** a SandboxClaim is created with env vars and `envVarsInjectionPolicy: Disallowed` on the template, **Then** the behavior (rejection or cold fallback) is documented. + +--- + +### User Story 5 - Produce Results Document with Recommendations (Priority: P1) + +An engineer compiles all measurement data into a structured report comparing cold-start vs. warm pool latencies across configurations, and produces architectural recommendations for how OpenShell should integrate warm pooling. + +**Why this priority**: The deliverable. Without this document, the experiments have no impact on the OpenShell project direction. + +**Independent Test**: Can be validated by checking that the document contains: raw data tables, per-configuration comparisons, a clear recommendation for Issue #2157, and concrete next steps for the OpenShell core team. + +**Acceptance Scenarios**: + +1. **Given** all experiments are complete, **When** the results RFC is written, **Then** it contains latency tables with p50/p90 for each configuration, a comparison chart, and a recommendation for the OpenShell integration approach. +2. **Given** the results RFC is complete, **Then** it is structured so a distilled summary can later be posted to GitHub Issue #2157 as a separate follow-up step. + +--- + +### Edge Cases + +- What happens when the warm pool is exhausted (all replicas claimed)? Does SandboxClaim fall back to cold start or stay Pending? +- What happens when a warm-pooled pod is on a node that goes NotReady during the experiment? +- How does pool replenishment interact with cluster autoscaler if nodes are at capacity? +- What if the Red Hat Agent Sandbox operator tech preview does not include extension CRDs? (Fallback: install upstream extensions.yaml manually) + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: Engineer MUST be able to provision a ROSA HCP cluster with a Kubernetes version supporting native sidecar containers (K8s 1.33+) +- **FR-002**: Engineer MUST be able to install the Agent Sandbox operator with both core and extension CRDs on the cluster +- **FR-003**: Engineer MUST be able to deploy OpenShell on the cluster using the OpenShell OpenShift deploy wrapper and create cold-start sandboxes +- **FR-004**: Measurement scripts MUST capture per-phase timestamps (API call, scheduled, image pulled, init complete, and for OpenShell experiments: supervisor ready, SSH available) +- **FR-005**: Shell-based measurement scripts MUST produce CSV output with run number, configuration label, and per-phase timestamps +- **FR-006**: Engineer MUST be able to create SandboxTemplate, SandboxWarmPool, and SandboxClaim resources for warm pool experiments +- **FR-007**: Engineer MUST be able to configure Pod Readiness Gates on warm-pooled pods and flip them via kubectl patch +- **FR-008**: Engineer MUST be able to deploy a sidecar container with a controllable readiness endpoint for sidecar readiness experiments +- **FR-009**: Results MUST be published as a standalone RFC in `rfc/` containing raw latency data, configuration matrix, and architectural recommendations +- **FR-010**: The RFC MUST be self-contained so it can later be distilled into a GitHub Issue #2157 comment (posting is a separate follow-up step, not part of this study) + +### Key Entities + +- **ROSA HCP Cluster**: The test environment. Provisioned via ROSA plugin, us-east-2 region, AAET profile. +- **Agent Sandbox CRDs**: Sandbox (core), SandboxTemplate, SandboxWarmPool, SandboxClaim (extensions). The primitives being evaluated. +- **Measurement Run**: A single sandbox creation or claim with captured timestamps. N=10-20 per configuration. +- **Configuration**: A specific combination of settings (probe interval, readiness gate, sidecar pattern, env var injection) being tested. +- **Results RFC**: The final deliverable in `rfc/`, synthesizing all measurements into architectural recommendations. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Cold-start baseline latency is measured with per-phase breakdown for at least 10 runs with pre-pulled images and 5 runs without +- **SC-002**: Warm pool claim-to-ready latency is measured across at least 4 configurations (default probes, aggressive probes, readiness gates, sidecar pattern) +- **SC-003**: Each configuration has at least 10 measurement runs with p50 and p90 latencies computed +- **SC-004**: The env var injection experiment conclusively determines whether claim-time injection triggers cold start or not +- **SC-005**: The results document contains a clear, data-backed recommendation for which warm pool integration approach OpenShell should pursue +- **SC-006**: The results RFC contains all required sections (data tables, configuration comparisons, architectural recommendation, next steps) and is structured for later distillation into a GitHub Issue #2157 comment + +## Clarifications + +### Session 2026-07-09 + +- Q: What ROSA HCP cluster sizing should be used for the experiments? → A: 3 worker nodes, m5.2xlarge (8 vCPU, 32 GB each). Cluster is short-lived (tear down after experiments). +- Q: Where should the full results document be stored? → A: Standalone RFC in `rfc/` directory of the OpenShell repo. +- Q: What level of measurement automation is expected? → A: Shell scripts per experiment that wrap kubectl with timestamp capture (CSV output). +- Q: Should the study prioritize the Red Hat Agent Sandbox tech preview or upstream manifests? → A: Red Hat tech preview first, upstream manifests only as fallback. + +## Assumptions + +- A ROSA HCP cluster with OpenShift 4.20+ (K8s 1.33+) is available for provisioning via the AAET AWS profile +- The cluster uses 3 worker nodes of type m5.2xlarge (8 vCPU, 32 GB each) in us-east-2, provisioned as a short-lived experiment cluster +- The Red Hat Agent Sandbox operator tech preview is installable from OperatorHub (if not, upstream manifests are used as fallback) +- the OpenShell OpenShift deploy wrapper works with the target OpenShift version (may need minor adjustments) +- Pre-pulling images on worker nodes is sufficient to eliminate image pull latency from measurements +- The upstream Agent Sandbox v0.5.0 extension CRDs (v1beta1) are compatible with the Red Hat operator build +- Pod Readiness Gates (KEP-580, GA since K8s 1.14) work on OpenShift without additional configuration +- Native sidecar containers (KEP-753, GA since K8s 1.33) are available on the target OpenShift version +- This study is scoped to Kubernetes-backed sandboxes only; Podman/Docker single-player warm pooling is out of scope diff --git a/specs/001-warm-pool-feasibility/tasks.md b/specs/001-warm-pool-feasibility/tasks.md new file mode 100644 index 0000000000..1df9348780 --- /dev/null +++ b/specs/001-warm-pool-feasibility/tasks.md @@ -0,0 +1,175 @@ +# Tasks: Warm Pool Feasibility Study + +**Input**: Design documents from `specs/001-warm-pool-feasibility/` +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, quickstart.md + +**Tests**: Not applicable (feasibility study with manual experiment validation). + +**Organization**: Tasks are grouped by user story from spec.md. US1 and US2 are P1 (core feasibility), US3-US4 are P2 (optimization experiments), US5 is P1 (deliverable). + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2) +- Include exact file paths in descriptions + +## Phase 1: Setup (Shared Infrastructure) + +**Purpose**: Cluster provisioning, operator installation, and shared tooling + +- [ ] T001 Provision ROSA HCP cluster with 3x m5.2xlarge workers using AAET profile (via `rosa create cluster` CLI or the `rosa:create` Claude Code skill) +- [ ] T002 Install Red Hat Agent Sandbox operator from OperatorHub (fallback: apply upstream `manifest.yaml` + `extensions.yaml` from kubernetes-sigs/agent-sandbox v0.5.0) +- [ ] T003 Verify all four CRDs are registered: Sandbox, SandboxTemplate, SandboxWarmPool, SandboxClaim via `kubectl api-resources | grep agents` +- [ ] T004 Deploy OpenShell on the cluster using the OpenShift deploy wrapper (github.com/2000krysztof/Openshell-Openshift-Deploy) +- [ ] T005 Validate cold-start sandbox creation works end-to-end via `openshell sandbox create --from base` +- [X] T006 Create image pre-pull DaemonSet to pre-pull sandbox images on all worker nodes in `experiments/manifests/image-prepull-daemonset.yaml` +- [ ] T007 Apply pre-pull DaemonSet and verify images are cached on all 3 nodes + +**Checkpoint**: Cluster running, operator installed, OpenShell functional, images pre-pulled + +--- + +## Phase 2: Foundational (Measurement Library) + +**Purpose**: Shared measurement infrastructure that all experiments depend on + +- [X] T008 Create `experiments/` directory structure per plan.md project structure +- [X] T009 Implement shared measurement functions (timestamp capture, CSV output, pod event collection) in `experiments/lib/common.sh` +- [X] T010 Implement pod readiness wait with configurable timeout in `experiments/lib/wait-ready.sh` +- [X] T011 [P] Create SandboxTemplate manifest for warm pool experiments in `experiments/manifests/sandbox-template.yaml` +- [X] T012 [P] Create SandboxWarmPool manifest (5 replicas, configurable readiness) in `experiments/manifests/warm-pool.yaml` +- [X] T013 [P] Create SandboxClaim manifest template (with/without env vars) in `experiments/manifests/sandbox-claim.yaml` + +**Checkpoint**: Measurement library ready, manifests templated. Experiment scripts can now be built. + +--- + +## Phase 3: User Story 1 - Measure Cold-Start Baseline (Priority: P1) MVP + +**Goal**: Establish cold-start sandbox creation latency with per-phase breakdown as the control measurement. + +**Independent Test**: Run 10+ sandboxes on the cluster and verify CSV output contains per-phase timestamps with computed p50/p90. + +- [X] T014 [US1] Implement cold-start measurement script that creates sandboxes via `openshell sandbox create --from base`, captures per-phase timestamps (scheduled, image pulled, init complete, supervisor ready, SSH available), and outputs CSV in `experiments/measure-cold-start.sh` +- [ ] T015 [US1] Run Experiment 1a: 10 cold-start runs with pre-pulled images, save results to `experiments/results/cold-start-prepulled.csv` +- [ ] T016 [US1] Run Experiment 1b: 5 cold-start runs without pre-pulled images, save results to `experiments/results/cold-start-noprepull.csv` +- [ ] T017 [US1] Run Experiment 1c: 10 vanilla Agent Sandbox creates (no OpenShell) to isolate OpenShell overhead, save results to `experiments/results/cold-start-vanilla.csv` +- [ ] T018 [US1] Compute p50/p90 latencies for all three cold-start configurations and create summary table + +**Checkpoint**: Cold-start baseline established with per-phase breakdown. + +--- + +## Phase 4: User Story 2 - Measure Warm Pool Claim Latency (Priority: P1) + +**Goal**: Measure raw warm pool claim-to-ready latency across probe configurations. Core feasibility answer. + +**Independent Test**: Create SandboxTemplate + SandboxWarmPool, issue SandboxClaims, verify claim-to-ready CSV output with latency data. + +- [X] T019 [US2] Implement warm pool measurement script that creates SandboxClaims against a running warm pool, captures claim-to-ready timestamps, and outputs CSV in `experiments/measure-warm-pool.sh` +- [ ] T020 [US2] Deploy SandboxWarmPool with 5 replicas and verify all replicas reach provisioned state +- [ ] T021 [US2] Run Experiment 2a: 10 claims with default readiness probes (10s periodSeconds), save results to `experiments/results/warm-pool-default.csv` +- [ ] T022 [US2] Run Experiment 2b: 10 claims with aggressive readiness probes (1s periodSeconds), save results to `experiments/results/warm-pool-aggressive.csv` +- [ ] T023 [US2] Run Experiment 2c: 5 simultaneous claims to measure burst behavior and pool replenishment, save results to `experiments/results/warm-pool-burst.csv` +- [ ] T024 [US2] Document pool exhaustion behavior: what happens when all replicas are claimed +- [ ] T025 [US2] Compute p50/p90 for all warm pool configurations and create comparison table vs cold-start + +**Checkpoint**: Core feasibility question answered. Warm pool claim-to-ready latency measured. + +--- + +## Phase 5: User Story 3 - Test Health Check Optimization Patterns (Priority: P2) + +**Goal**: Test Pod Readiness Gates and sidecar readiness as alternatives to polling-based probes. + +**Independent Test**: Deploy pods with ReadinessGate conditions and sidecar containers, measure condition-flip-to-Ready latency. + +- [X] T026 [P] [US3] Create pod manifest with custom ReadinessGate condition (`sandbox.openshell.io/claimed`) in `experiments/manifests/readiness-gate-pod.yaml` +- [X] T027 [P] [US3] Implement sidecar readiness binary (Go, HTTP 503/200, signal file watch) in `experiments/sidecar/main.go` +- [X] T028 [P] [US3] Create Dockerfile for sidecar binary (multi-stage, scratch base) in `experiments/sidecar/Dockerfile` +- [ ] T029 [US3] Build and push sidecar readiness image to ghcr.io via `experiments/sidecar/Makefile` +- [X] T030 [US3] Create sidecar readiness pod manifest (init container with restartPolicy: Always, shared emptyDir) in `experiments/manifests/sidecar-readiness.yaml` +- [X] T031 [US3] Implement readiness gate measurement script that patches ReadinessGate condition and measures flip-to-Ready latency in `experiments/measure-readiness-gates.sh` +- [X] T032 [US3] Implement sidecar readiness measurement script that triggers signal file and measures flip-to-Ready latency in `experiments/measure-sidecar-readiness.sh` +- [ ] T033 [US3] Run Experiment 3a: 10 readiness gate measurements, save to `experiments/results/readiness-gates.csv` +- [ ] T034 [US3] Run Experiment 3b: 10 sidecar readiness measurements, save to `experiments/results/sidecar-readiness.csv` +- [ ] T035 [US3] Create comparison table: default probes vs aggressive probes vs readiness gates vs sidecar pattern + +**Checkpoint**: Health check optimization patterns validated with latency data. + +--- + +## Phase 6: User Story 4 - Test Claim-Time Environment Injection (Priority: P2) + +**Goal**: Determine whether SandboxClaim env var injection triggers cold start or adopts warm sandbox. + +**Independent Test**: Create SandboxTemplate with `envVarsInjectionPolicy: Allowed`, issue SandboxClaim with env vars, observe warm adoption. + +- [X] T036 [US4] Update SandboxTemplate manifest with `envVarsInjectionPolicy: Allowed` variant in `experiments/manifests/sandbox-template.yaml` +- [X] T037 [US4] Implement env var injection measurement script in `experiments/measure-env-injection.sh` +- [ ] T038 [US4] Run Experiment 5a: 5 claims with env vars and Allowed policy, verify warm adoption, save to `experiments/results/env-injection-allowed.csv` +- [ ] T039 [US4] Run Experiment 5b: Document behavior with Disallowed policy (rejection vs cold fallback) +- [X] T040 [US4] Implement combined measurement script (best readiness pattern + env injection) in `experiments/measure-combined.sh` +- [ ] T041 [US4] Run Experiment 6: 10 combined measurements, save to `experiments/results/combined.csv` + +**Checkpoint**: Env var injection behavior conclusively documented. Combined best-case latency measured. + +--- + +## Phase 7: User Story 5 - Produce Results RFC (Priority: P1) + +**Goal**: Compile all measurement data into a structured RFC with architectural recommendations. + +**Independent Test**: RFC contains raw data tables, per-configuration comparisons, and a clear recommendation for OpenShell warm pool integration. + +- [X] T042 [US5] Create RFC directory structure `rfc/NNNN-warm-pool-feasibility/README.md` (number TBD by maintainers, use placeholder) +- [ ] T043 [US5] Write RFC Executive Summary section: can warm pooling hit sub-2s? What is the dominant bottleneck? +- [ ] T044 [US5] Write RFC Experiment Setup section: cluster config, operator version, OpenShell version, image pre-pull status +- [ ] T045 [US5] Write RFC Results section: compile all CSV data into p50/p90 tables per configuration with comparison charts +- [ ] T046 [US5] Write RFC Health Check Analysis section: probe interval impact, readiness gates performance, sidecar pattern performance +- [ ] T047 [US5] Write RFC Env Var Injection section: injection behavior, policy requirements, identity binding constraints +- [ ] T048 [US5] Write RFC Architecture Recommendations section: Kubernetes driver changes, supervisor changes, gateway store changes, identity binding mechanism, Issue #2157 recommendation +- [ ] T049 [US5] Write RFC Gaps and Risks section: missing Agent Sandbox extension features, Red Hat TP coverage gaps, pool replenishment under burst +- [ ] T050 [US5] Write RFC Next Steps section: upstream contributions, internal work items, concrete action items + +**Checkpoint**: RFC complete, ready for review. + +--- + +## Phase 8: Polish & Teardown + +**Purpose**: Cleanup, validation, and cluster teardown + +- [ ] T051 Validate all CSV result files exist and contain expected number of runs +- [ ] T052 Review RFC for completeness: all required sections present, data tables populated, recommendation clear +- [ ] T053 Tear down ROSA HCP cluster via `rosa:delete warm-pool-study` + +--- + +## Dependencies + +```text +Phase 1 (Setup) + └── Phase 2 (Measurement Library) + ├── Phase 3 (US1: Cold-Start Baseline) ─── MVP + │ └── Phase 4 (US2: Warm Pool Claims) + │ ├── Phase 5 (US3: Health Check Optimization) ──┐ + │ └── Phase 6 (US4: Env Var Injection) ──────────┤ + │ └── Phase 7 (US5: Results RFC) + │ └── Phase 8 (Teardown) +``` + +**Critical path**: Setup → Library → Cold-Start → Warm Pool → Health Check Optimization → RFC → Teardown + +**Parallel opportunities**: +- US3 and US4 can run in parallel after US2 completes +- Within US3: readiness gate manifest (T026), sidecar binary (T027), and sidecar Dockerfile (T028) can be built in parallel +- Within US2: warm pool deployment (T020) can begin while measurement script (T019) is being written + +## Implementation Strategy + +**MVP**: Phases 1-4 (Setup + Library + Cold-Start Baseline + Warm Pool Claims). This answers the core feasibility question: can warm pooling deliver sub-2s latency on OpenShift? + +**Full scope**: All 8 phases. The health check optimization and env var injection experiments refine the recommendation, and the RFC captures everything for the upstream community. + +**Incremental delivery**: Each phase produces independently verifiable output (CSV data files). If the warm pool latency exceeds target in Phase 4, Phases 5-6 may pivot to investigating why rather than optimizing further. diff --git a/specs/002-warm-pool-grpc-push/checklists/requirements.md b/specs/002-warm-pool-grpc-push/checklists/requirements.md new file mode 100644 index 0000000000..6ad5ed76de --- /dev/null +++ b/specs/002-warm-pool-grpc-push/checklists/requirements.md @@ -0,0 +1,36 @@ +# Specification Quality Checklist: Warm Pool gRPC-Push + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-07-10 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- SC-001 references "under 1 second" which is the target from the brainstorm. This is aspirational based on feasibility study measurements (2.3s claim + eliminated supervisor startup). Actual validation requires implementation. +- FR-001 names the gRPC endpoint "ActivateSandbox" as a working name from the brainstorm. The proto definition is a planning concern. +- The spec intentionally excludes dynamic pool promotion (auto-create pools for unknown images) per the brainstorm decision. This is documented as a future extension assumption. diff --git a/specs/002-warm-pool-grpc-push/data-model.md b/specs/002-warm-pool-grpc-push/data-model.md new file mode 100644 index 0000000000..49ca948675 --- /dev/null +++ b/specs/002-warm-pool-grpc-push/data-model.md @@ -0,0 +1,87 @@ +# Data Model: Warm Pool gRPC-Push RFC + +## Entities + +### WarmPoolConfig (Gateway Configuration) + +Gateway TOML configuration entry defining a warm pool for a specific sandbox image. + +| Field | Type | Description | +|-------|------|-------------| +| image | string | Container image reference (e.g., `ghcr.io/nvidia/openshell-community/sandboxes/base:latest`) | +| replicas | integer | Number of pre-provisioned pods to maintain in the pool | + +### ActivateSandboxRequest (gRPC Message) + +Message sent from gateway to supervisor after claim binding to inject sandbox identity. + +| Field | Type | Description | +|-------|------|-------------| +| sandbox_id | string (UUID) | Unique identifier for this sandbox session | +| sandbox_name | string | Human-readable sandbox name | +| policy_config | PolicyConfig | Sandbox-specific policy rules to compile at claim time | +| provider_environment | map | Provider credentials and environment variables | + +### ActivateSandboxResponse (gRPC Message) + +Response from supervisor confirming activation. + +| Field | Type | Description | +|-------|------|-------------| +| status | enum (OK, FAILED) | Whether activation succeeded | +| error_message | string | Error details if status is FAILED | +| ready_at | timestamp | When the supervisor became ready | + +### GlobalPolicyBundle + +Pre-compiled OPA policy set applicable to all sandboxes, compiled at pool provisioning time. + +| Field | Type | Description | +|-------|------|-------------| +| compiled_modules | bytes | Pre-compiled OPA modules (network rules, filesystem constraints) | +| version | string | Policy version hash for staleness detection | + +### SandboxPolicyDelta + +Sandbox-specific OPA rules compiled at claim time. + +| Field | Type | Description | +|-------|------|-------------| +| sandbox_rules | list | Per-sandbox rules (provider constraints, custom policies) | +| merged_with_global | string | Version of the global bundle this delta was merged with | + +## Relationships + +``` +WarmPoolConfig --[1:1]--> SandboxTemplate (K8s resource, gateway-managed) +WarmPoolConfig --[1:1]--> SandboxWarmPool (K8s resource, gateway-managed) +SandboxWarmPool --[1:N]--> Warm Pods (pre-provisioned, supervisor running) +SandboxClaim --[1:1]--> Warm Pod (adopted at claim time) +ActivateSandboxRequest --[1:1]--> Warm Pod (gRPC push after claim binding) +GlobalPolicyBundle --[1:1]--> Warm Pod (compiled at pool time) +SandboxPolicyDelta --[1:1]--> Activated Pod (compiled at claim time) +``` + +## State Transitions + +### Supervisor State Machine + +``` +INITIALIZING --> IDLE --> ACTIVATING --> ACTIVE --> TERMINATED +``` + +| State | Description | Health endpoint | +|-------|-------------|----------------| +| INITIALIZING | Supervisor starting, global OPA compiling | `/readyz` 503 | +| IDLE | Global policies compiled, waiting for identity push | `/readyz` 200 | +| ACTIVATING | Received identity, compiling sandbox-specific policies | `/readyz` 200 | +| ACTIVE | Fully activated, SSH ready, serving session | `/readyz` 200 | +| TERMINATED | Session ended, pod will be deleted by operator | `/readyz` 503 | + +### Pool Pod Lifecycle + +``` +Pool creates pod --> Supervisor reaches IDLE --> Pod marked Ready in pool +--> Claim adopts pod --> Gateway pushes identity --> Supervisor ACTIVE +--> Session ends --> Pod deleted --> Pool replenishes with fresh pod +``` diff --git a/specs/002-warm-pool-grpc-push/plan.md b/specs/002-warm-pool-grpc-push/plan.md new file mode 100644 index 0000000000..8f403690e8 --- /dev/null +++ b/specs/002-warm-pool-grpc-push/plan.md @@ -0,0 +1,91 @@ +# Implementation Plan: Warm Pool gRPC-Push RFC + +**Branch**: `6111-warm-pool-feasibility` | **Date**: 2026-07-10 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `specs/002-warm-pool-grpc-push/spec.md` + +## Summary + +Write a standalone RFC document proposing warm pool integration with an always-on supervisor and gRPC-push identity binding. The RFC covers gateway-managed pool lifecycle, two-tier OPA compilation, multi-image pool routing, and cold-start fallback. Add a cross-reference comparison section to both the new RFC and the existing annotation-based RFC (`rfc/NNNN-warm-pool-feasibility/`). + +## Technical Context + +**Language/Version**: Markdown (RFC document) +**Primary Dependencies**: Existing RFC template (`rfc/0000-template/`), existing annotation RFC (`rfc/NNNN-warm-pool-feasibility/`) +**Storage**: N/A (documents only) +**Testing**: N/A (no code) +**Target Platform**: GitHub rendered Markdown +**Project Type**: RFC documentation +**Performance Goals**: N/A +**Constraints**: Must follow OpenShell RFC template structure; must cross-reference the annotation RFC as an equal alternative +**Scale/Scope**: 2 files modified (existing RFC), 1 file created (new RFC) + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +Constitution is a template with no project-specific constraints filled in. No gates to evaluate. + +## Project Structure + +### Documentation (this feature) + +```text +specs/002-warm-pool-grpc-push/ +├── plan.md # This file +├── research.md # Research decisions +├── data-model.md # Entity and state model +├── checklists/ +│ └── requirements.md # Quality checklist +└── tasks.md # Task breakdown (generated by /speckit-tasks) +``` + +### Source (repository root) + +```text +rfc/ +├── NNNN-warm-pool-feasibility/ +│ └── README.md # Existing RFC (add Alternatives section) +└── NNNN-warm-pool-grpc-push/ + └── README.md # New RFC (this feature's primary output) +experiments/ +└── RESULTS.md # Update TL;DR comparison if needed +``` + +**Structure Decision**: This is a documentation-only feature. The output is RFC documents, not code. The existing RFC directory structure is followed. + +## Implementation Phases + +### Phase 1: Write the new gRPC-push RFC + +Create `rfc/NNNN-warm-pool-grpc-push/README.md` following the OpenShell RFC template with these sections: + +1. **Summary**: Always-on supervisor with gRPC-push identity, sub-1s target +2. **Motivation**: Same as annotation RFC (shared measurements from feasibility study) +3. **Non-goals**: Dynamic pool promotion, non-Kubernetes drivers, code implementation +4. **Proposal**: 6 subsections: + - Architecture overview (Mermaid diagram: gateway -> claim -> gRPC push -> supervisor) + - Always-on supervisor with two-tier OPA (global at pool time, sandbox-specific at claim time) + - gRPC identity push protocol (ActivateSandbox endpoint, mTLS, timeout/retry) + - Gateway-managed pool lifecycle (config-driven SandboxTemplate + SandboxWarmPool reconciliation) + - Multi-image pool routing (pool-per-image, cold-start fallback) + - Comparison with annotation-based approach (shared table) +5. **Implementation plan**: Phased work items +6. **Risks**: New gRPC endpoint surface, gateway complexity, unvalidated approach +7. **Alternatives**: Annotation-based (link to other RFC), ConfigMap-based, gateway-native pool +8. **Prior art**: Craig-kindo's GKE implementation, agent-sandbox#1118 +9. **Open questions**: Proto definition, image update handling, pool reconciler design + +### Phase 2: Add cross-reference to existing annotation RFC + +Update `rfc/NNNN-warm-pool-feasibility/README.md`: +- Add an "Alternatives" section (or update existing) with a link to the new RFC +- Add a comparison table covering latency, complexity, security, upstream alignment +- Keep equal framing (neither is primary) + +### Phase 3: Commit and push + +Commit both RFC changes together so the cross-references are consistent. + +## Complexity Tracking + +No constitution violations to justify. diff --git a/specs/002-warm-pool-grpc-push/research.md b/specs/002-warm-pool-grpc-push/research.md new file mode 100644 index 0000000000..116b7469a8 --- /dev/null +++ b/specs/002-warm-pool-grpc-push/research.md @@ -0,0 +1,47 @@ +# Research: Warm Pool gRPC-Push RFC + +## Existing Annotation-Based RFC Structure + +**Decision**: Follow the same RFC template structure as `rfc/NNNN-warm-pool-feasibility/README.md` (Summary, Motivation, Design, Implementation Plan, Risks, Alternatives, Prior Art, Open Questions). + +**Rationale**: Consistency with the existing RFC makes cross-referencing easier and reduces reviewer cognitive load. + +**Alternatives considered**: Combined RFC with variant sections (rejected per brainstorm decision for independent reviewability). + +## Identity Binding Mechanism + +**Decision**: gRPC push from gateway to supervisor pod IP. The gateway reads `.status.sandbox.podIPs` from the adopted Sandbox resource after claim binding. + +**Rationale**: Eliminates the 1-2s downward API propagation delay of the annotation approach. Direct network call is the lowest-latency option. The existing mTLS certificates already mounted in warm pods secure the channel. + +**Alternatives considered**: Annotation-based (covered by the other RFC), ConfigMap-based (60-90s propagation default), Kubernetes watch from supervisor (requires RBAC in the sandbox pod). + +## Two-Tier OPA Compilation + +**Decision**: Global policies compile at pool provisioning time. Sandbox-specific policies compile at claim time after identity push. + +**Rationale**: The 1.4s OPA compilation cost dominates supervisor startup. Global policies (network rules, filesystem constraints) are identical across sandboxes. Only per-sandbox rules (provider-specific constraints, custom policies) need claim-time compilation. + +**Alternatives considered**: Full compilation at pool time (impossible, sandbox-specific policies unknown until claim), full compilation at claim time (same as annotation RFC, no improvement). + +## Gateway Pool Management + +**Decision**: Gateway owns SandboxTemplate and SandboxWarmPool lifecycle. Config-driven, reconciled at startup. + +**Rationale**: Centralizes pool management in the gateway where image routing decisions are already made. Avoids requiring operators to manually manage Kubernetes extension resources. + +**Alternatives considered**: Operator-managed pools (simpler gateway, but pool management becomes a separate ops concern), Helm-managed pools (middle ground, but no runtime scaling). + +## Supervisor gRPC Endpoint + +**Decision**: New `ActivateSandbox` RPC on the supervisor's existing gRPC server. Accepts sandbox ID, name, and sandbox-specific policy configuration. + +**Rationale**: The supervisor already runs a gRPC server for health checks and internal communication. Adding an endpoint is simpler than introducing a new protocol. + +**Alternatives considered**: HTTP endpoint (different protocol from existing supervisor communication), Unix socket (not routable from gateway pod). + +## Upstream References + +**Decision**: Cross-reference OpenShell#2157, #1447, agent-sandbox#1118, #384 in both RFCs. + +**Rationale**: These issues track the upstream feature work and implementation proposals. Craig-kindo's GKE validation (#1447) confirms the annotation approach independently. PR #1118 improves operator adoption speed, benefiting both approaches. diff --git a/specs/002-warm-pool-grpc-push/spec.md b/specs/002-warm-pool-grpc-push/spec.md new file mode 100644 index 0000000000..0cdf4b004d --- /dev/null +++ b/specs/002-warm-pool-grpc-push/spec.md @@ -0,0 +1,147 @@ +# Feature Specification: Warm Pool Integration with Always-On Supervisor (gRPC-Push) + +**Feature Branch**: `6111-warm-pool-feasibility` +**Created**: 2026-07-10 +**Status**: Draft +**Input**: Alternative RFC for warm pool integration using always-on supervisor with gRPC-push identity binding + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - Sub-Second Sandbox Provisioning via Warm Pool (Priority: P1) + +An agent platform operator configures warm pools for known sandbox images in the gateway configuration. When an agent requests a sandbox, the gateway claims a pre-provisioned pod from the pool. The always-on supervisor in that pod receives the sandbox identity via a direct gRPC push from the gateway. The supervisor compiles only sandbox-specific OPA policies (global policies were pre-compiled at pool time) and activates the session. The agent gets an SSH-ready sandbox in under 1 second from request to ready. + +**Why this priority**: This is the core value proposition. The annotation-based alternative delivers 3-4s; the gRPC-push approach targets sub-1s by eliminating supervisor startup and downward API propagation delay. + +**Independent Test**: Can be tested by configuring a single warm pool for the base sandbox image, creating a sandbox, and measuring claim-to-ready latency. Expected: < 1s. + +**Acceptance Scenarios**: + +1. **Given** a warm pool with 5 ready replicas for the base sandbox image, **When** a user runs `openshell sandbox create`, **Then** the sandbox reaches Ready state in under 1 second. +2. **Given** a warm pool with available replicas, **When** a SandboxClaim is created, **Then** the gateway pushes identity to the supervisor via gRPC within 100ms of claim binding. +3. **Given** a supervisor that has pre-compiled global OPA policies at pool time, **When** it receives identity via gRPC push, **Then** it compiles only sandbox-specific policies and reports Ready within 500ms. + +--- + +### User Story 2 - Gateway-Managed Pool Lifecycle (Priority: P1) + +A platform operator defines warm pool configuration in the gateway config file, specifying which sandbox images should have pools and how many replicas each pool maintains. The gateway creates, scales, and deletes the corresponding SandboxTemplate and SandboxWarmPool Kubernetes resources. The operator does not need to manage these resources manually. + +**Why this priority**: Without gateway-managed pools, operators must manually create and maintain Kubernetes resources for each image. This is error-prone and blocks adoption. + +**Independent Test**: Can be tested by adding a warm pool entry to gateway config, restarting the gateway, and verifying that SandboxTemplate and SandboxWarmPool resources appear in the namespace with the correct image and replica count. + +**Acceptance Scenarios**: + +1. **Given** a gateway config with a warm pool entry for `base:latest` with 5 replicas, **When** the gateway starts, **Then** it creates a SandboxTemplate and SandboxWarmPool with the specified image and replica count. +2. **Given** a running gateway with a warm pool, **When** the operator changes the replica count in the config and restarts the gateway, **Then** the SandboxWarmPool is updated to the new count. +3. **Given** a running gateway with a warm pool, **When** the operator removes the pool entry from config and restarts the gateway, **Then** the SandboxWarmPool and SandboxTemplate are deleted. + +--- + +### User Story 3 - Multi-Image Pool Management (Priority: P2) + +A platform operator configures multiple warm pools for different sandbox images (e.g., base, ollama, custom-ml). Each image gets its own pool with independent replica counts. When a user creates a sandbox specifying an image that has a pool, the gateway routes the request to the matching pool. When the image has no pool, the gateway falls back to cold-start provisioning. + +**Why this priority**: Real deployments use multiple sandbox images. Without multi-image support, warm pooling only benefits a single image. + +**Independent Test**: Can be tested by configuring two pools (base with 5 replicas, ollama with 2 replicas), creating sandboxes with each image, and verifying pool routing. Then creating a sandbox with an unconfigured image and verifying cold-start fallback. + +**Acceptance Scenarios**: + +1. **Given** pools for `base:latest` (5 replicas) and `ollama:latest` (2 replicas), **When** a user creates a sandbox with `--from base`, **Then** the gateway claims from the base pool. +2. **Given** pools for `base:latest` and `ollama:latest`, **When** a user creates a sandbox with `--from ollama`, **Then** the gateway claims from the ollama pool. +3. **Given** no pool for `custom-ml:latest`, **When** a user creates a sandbox with that image, **Then** the gateway provisions via cold-start. + +--- + +### User Story 4 - Cold-Start Fallback on Pool Exhaustion (Priority: P2) + +When all replicas in a warm pool are claimed and the pool is temporarily empty, the gateway falls back to cold-start provisioning. The user still gets a sandbox, just with higher latency. When the pool replenishes, subsequent requests use warm pods again. + +**Why this priority**: Pool exhaustion is inevitable under burst load. Graceful fallback prevents failures. + +**Independent Test**: Can be tested by claiming all 5 replicas from a pool, then immediately requesting a 6th sandbox. The 6th should succeed via cold-start. After the pool replenishes, the next request should use a warm pod. + +**Acceptance Scenarios**: + +1. **Given** a warm pool with 0 ready replicas (all claimed), **When** a user creates a sandbox, **Then** the gateway provisions via cold-start and the sandbox reaches Ready. +2. **Given** a pool that was exhausted, **When** the pool controller replenishes replicas, **Then** the next sandbox request claims from the pool. + +--- + +### User Story 5 - Two-Tier OPA Policy Compilation (Priority: P2) + +The supervisor in a warm pool pod pre-compiles global OPA policies (network rules, filesystem constraints applicable to all sandboxes) at pool provisioning time. At claim time, only sandbox-specific policies (per-sandbox rules, provider-specific constraints) are compiled and merged with the pre-compiled global policies. This reduces claim-time policy overhead from ~1.4s to the delta compilation time. + +**Why this priority**: OPA compilation is the largest contributor to supervisor startup latency (1.4s of the 1.5s total). Splitting into pre-compiled globals and claim-time specifics is what makes sub-1s possible. + +**Independent Test**: Can be tested by measuring policy compilation time at pool provisioning (global) and at claim time (sandbox-specific delta). The claim-time compilation should be significantly faster than full compilation. + +**Acceptance Scenarios**: + +1. **Given** a warm pool pod with global policies pre-compiled, **When** it receives identity via gRPC push, **Then** it compiles only sandbox-specific policies. +2. **Given** a supervisor with pre-compiled global policies and a sandbox-specific network deny rule, **When** sandbox-specific policy compilation completes, **Then** the combined policy set blocks requests matching the sandbox-specific deny rule AND requests matching global deny rules. + +--- + +### Edge Cases + +- What happens when the gRPC push to the supervisor fails (pod crashed between claim and push)? The gateway retries once (2-second timeout per attempt), then falls back to cold-start provisioning. +- What happens when the gateway config defines duplicate pool entries for the same image? The gateway rejects the config at startup with a validation error. +- What happens when a warm pool pod's supervisor binary is outdated (image updated but pod not replaced)? The pool's `OnReplenish` update strategy gradually replaces pods as they are claimed and replenished. +- What happens when the gateway loses connectivity to the Kubernetes API during pool reconciliation? The gateway uses its last-known pool state and retries reconciliation on a backoff interval. +- What happens when mTLS certificates are missing or expired on the gRPC push channel? The gateway treats the push as failed (TLS handshake error), applies the same retry/fallback policy as FR-012 (2-second timeout, 1 retry, then cold-start fallback). + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: System MUST expose an `ActivateSandbox` gRPC endpoint on the supervisor that accepts sandbox identity (ID, name, sandbox-specific policy configuration). +- **FR-002**: System MUST start the supervisor process at pool provisioning time and pre-compile global OPA policies before the pod is marked as ready in the warm pool. +- **FR-003**: System MUST allow the gateway to push sandbox identity to the supervisor via direct gRPC connection to the pod IP after claim binding. +- **FR-004**: System MUST compile only sandbox-specific OPA policies at claim time, merging them with pre-compiled global policies. +- **FR-005**: System MUST allow the gateway to create, update, and delete SandboxTemplate and SandboxWarmPool Kubernetes resources based on gateway configuration. +- **FR-006**: System MUST support multiple warm pools, one per configured sandbox image, with independent replica counts. +- **FR-007**: System MUST fall back to cold-start provisioning when no warm pool exists for the requested image or when the matching pool has no available replicas. +- **FR-008**: System MUST read the adopted pod's IP from the SandboxClaim's `.status.sandbox.podIPs` field for gRPC push targeting. +- **FR-009**: System MUST NOT include `spec.env` fields on SandboxClaims, as this bypasses warm pool adoption in the Agent Sandbox operator v0.9.0. +- **FR-010**: System MUST validate gateway warm pool configuration at startup, rejecting duplicate image entries and invalid replica counts. +- **FR-011**: System MUST secure the gRPC push channel between gateway and supervisor using the existing namespace mTLS certificates (TLS client secret volume already mounted in the warm pod template). +- **FR-012**: System MUST apply a 2-second timeout with 1 retry for the gRPC identity push. If both attempts fail, the system MUST fall back to cold-start provisioning. +- **FR-013**: Supervisor MUST return 200 on `/readyz` in both idle (waiting for claim) and activated (serving session) states so the pod remains Ready in the warm pool. + +### Key Entities + +- **WarmPoolConfig**: Gateway configuration entry defining an image and its pool size. Maps to a SandboxTemplate + SandboxWarmPool pair. +- **ActivateSandboxRequest**: gRPC message carrying sandbox identity (ID, name, policy config) from gateway to supervisor. +- **GlobalPolicyBundle**: Pre-compiled OPA policy set applicable to all sandboxes, compiled at pool provisioning time. +- **SandboxPolicyDelta**: Sandbox-specific OPA rules compiled at claim time and merged with the global bundle. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Sandbox provisioning from warm pool reaches Ready state in under 1 second (p50) and under 2 seconds (p99), measured from `CreateSandbox` request to SSH-ready. +- **SC-002**: Gateway creates correct SandboxTemplate and SandboxWarmPool resources within 5 seconds of startup when warm pool config is present. +- **SC-003**: Sandbox-specific OPA policy compilation at claim time completes in under 200ms (compared to ~1,400ms for full compilation). +- **SC-004**: 100% of sandbox requests succeed when a warm pool is exhausted, via cold-start fallback. +- **SC-005**: Gateway correctly routes sandbox requests to the matching pool based on image, with zero misroutes across 100 consecutive requests. + +## Clarifications + +### Session 2026-07-10 + +- Q: How is the gRPC push channel between gateway and supervisor secured? → A: Use existing namespace mTLS certificates already mounted in the warm pod template (TLS client secret volume). No new certificate infrastructure needed. +- Q: What is the timeout and retry policy for gRPC push before cold-start fallback? → A: 2-second timeout with 1 retry. If both attempts fail, fall back to cold-start provisioning. +- Q: How does the supervisor's readiness probe distinguish between idle (ready for claim) and activated (serving a session)? → A: `/readyz` returns 200 in both states so the pod stays Ready in the pool. Activation state is tracked internally by the supervisor. The pool controller uses pool membership, not readiness, to determine claimability. + +## Assumptions + +- The Agent Sandbox operator v0.9.0 (or later) is installed with extension CRDs (SandboxTemplate, SandboxWarmPool, SandboxClaim). +- Env var injection on SandboxClaim bypasses warm pool adoption. This is a known operator constraint documented in the feasibility study. +- The gateway has Kubernetes RBAC permissions to create, update, and delete SandboxTemplate, SandboxWarmPool, and SandboxClaim resources. +- Pod IPs are routable from the gateway pod (same namespace, standard Kubernetes networking). +- Global OPA policies are stable across sandbox sessions. Policy updates require pool replenishment (new pods get new policies). +- This spec is an equal alternative to the annotation-based approach described in the feasibility study. Neither is primary. +- Dynamic pool promotion for unknown images is a future extension, not part of this initial design. diff --git a/specs/002-warm-pool-grpc-push/tasks.md b/specs/002-warm-pool-grpc-push/tasks.md new file mode 100644 index 0000000000..c5216e871b --- /dev/null +++ b/specs/002-warm-pool-grpc-push/tasks.md @@ -0,0 +1,81 @@ +# Tasks: Warm Pool gRPC-Push RFC + +**Feature**: Warm Pool Integration with Always-On Supervisor (gRPC-Push) +**Branch**: `6111-warm-pool-feasibility` +**Spec**: [spec.md](spec.md) | **Plan**: [plan.md](plan.md) + +## Phase 1: Setup + +- [X] T001 Create RFC directory and initialize from template at rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 2: Foundational (RFC Frontmatter and Shared Sections) + +- [X] T002 Write RFC frontmatter (authors, state, links) and Summary section in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T003 Write Motivation section with shared feasibility study measurements in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T004 Write Non-goals section (dynamic pool promotion, non-K8s drivers, code implementation) in rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 3: User Story 1 - Sub-Second Provisioning via gRPC-Push (P1) + +- [X] T005 [US1] Write Proposal subsection: Architecture Overview with Mermaid diagram (gateway -> claim -> gRPC push -> supervisor) in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T006 [US1] Write Proposal subsection: Always-On Supervisor with Two-Tier OPA (global at pool time, sandbox-specific at claim time) in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T007 [US1] Write Proposal subsection: gRPC Identity Push Protocol (ActivateSandbox endpoint, mTLS, 2s timeout, retry, fallback) in rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 4: User Story 2 - Gateway-Managed Pool Lifecycle (P1) + +- [X] T008 [US2] Write Proposal subsection: Gateway-Managed Pool Lifecycle (config-driven SandboxTemplate + SandboxWarmPool reconciliation, startup behavior, scaling, cleanup) in rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 5: User Story 3 - Multi-Image Pool Management (P2) + +- [X] T009 [US3, US4] Write Proposal subsection: Multi-Image Pool Routing (pool-per-image config, image matching, cold-start fallback for unknown images AND pool exhaustion per US4/FR-007) in rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 6: User Story 5 - Comparison and Cross-Reference (P2) + +- [X] T010 [US5] Write Proposal subsection: Comparison with Annotation-Based Approach (table covering latency, complexity, security, OPA, upstream alignment, resource cost) in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T011 [US5] Write Alternatives section (annotation-based link, ConfigMap-based, gateway-native pool, do nothing) in rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 7: Remaining RFC Sections + +- [X] T012 Write Implementation Plan section (phased work items) in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T013 Write Risks section (new gRPC surface, gateway complexity, unvalidated approach, resource cost, edge cases: outdated supervisor binary via OnReplenish, K8s API connectivity loss during reconciliation, mTLS cert expiry) in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T014 Write Prior Art section (craig-kindo GKE validation, agent-sandbox#1118, feasibility study) in rfc/NNNN-warm-pool-grpc-push/README.md +- [X] T015 Write Open Questions section and Future Extensions (dynamic pool promotion) in rfc/NNNN-warm-pool-grpc-push/README.md + +## Phase 8: Cross-Reference Update + +- [X] T016 Add Alternatives section to existing annotation RFC at rfc/NNNN-warm-pool-feasibility/README.md with link to gRPC-push RFC and comparison table + +## Phase 9: Polish + +- [X] T017 Review both RFCs for consistent terminology, fix cross-reference links, verify Mermaid diagrams render + +## Dependencies + +``` +T001 --> T002, T003, T004 (setup before content) +T002, T003, T004 --> T005..T009 (foundational before stories) +T005, T006, T007 --> T010, T011 (proposal before comparison) +T010, T011 --> T012..T015 (comparison before remaining sections) +T012..T015 --> T016 (new RFC complete before cross-referencing) +T016 --> T017 (cross-reference before polish) +``` + +## Parallel Opportunities + +- T002, T003, T004 can run in parallel (independent foundational sections) +- T005, T006, T007 can run in parallel (independent proposal subsections) +- T008, T009 can run in parallel (independent proposal subsections) +- T010, T011 can run in parallel (comparison and alternatives) +- T012, T013, T014, T015 can run in parallel (independent remaining sections) + +## Implementation Strategy + +**MVP**: T001-T007 (setup + summary/motivation + core proposal with architecture, supervisor, and gRPC protocol). This delivers the core RFC content that differentiates the gRPC-push approach. + +**Incremental delivery**: Each phase adds a complete RFC section. The RFC is reviewable after Phase 7. Phase 8-9 add cross-references and polish. + +## Summary + +- **Total tasks**: 17 +- **User Story tasks**: 7 (US1: 3, US2: 1, US3: 1, US5: 2) +- **Parallel opportunities**: 5 groups +- **Independent test**: Each phase produces a complete, reviewable RFC section