feat: kind: config embeds sandbox files in harness YAML

README.md

@@ -1,21 +1,34 @@
 # harness
 
-Deploy a fully sandboxed AI agent in one command.
+Declarative configuration harness for [OpenShell](https://github.com/NVIDIA/OpenShell) AI agent sandboxes.
 
 ```bash
 harness apply -f agent.yaml
 ```
 
-One file defines your agent, providers, gateway, and policy. The harness resolves credentials, deploys the gateway, registers providers, and launches a sandbox with deny-by-default L7 egress. Works on local Podman and remote Kubernetes/OpenShift with the same config.
+## OpenShell provides the runtime
 
-## What You Get
+[OpenShell](https://github.com/NVIDIA/OpenShell) runs AI agents in sandboxed containers with deny-by-default L7 network policy, credential proxy at the network boundary, Landlock filesystem isolation, and inference routing. The harness does not replace any of this.
 
-- **Sandbox isolation** -- every agent runs in a container with Landlock filesystem restrictions and deny-by-default network policy
-- **Credential proxy** -- secrets are resolved at the gateway boundary, never exposed inside the sandbox
-- **Multi-target** -- same agent YAML deploys to local Podman, kind, or OpenShift
-- **Declarative config** -- multi-document YAML bundles agent + providers + gateway + policy in one file
-- **Dry-run validation** -- `--dry-run` checks gateway, providers, env vars, and image before deploying
-- **Config inspection** -- `-o yaml` outputs the fully resolved harness config
+## The harness adds declarative configuration
+
+- **One-file agent definition** -- agent, providers, gateway, policy, and sandbox files in a single YAML
+- **Multi-document YAML** -- `kind: agent/provider/gateway/payload/policy` composed in one file
+- **Payload files** -- upload configs to sandbox paths without rebuilding the image
+- **Multi-target deploy** -- same YAML works on local Podman, kind, and OpenShift
+- **Dry-run validation** -- `--dry-run` checks everything before deploying
+- **Config inspection** -- `-o yaml` outputs the fully resolved config
+
+## Use OpenShell directly for runtime operations
+
+```bash
+openshell sandbox connect <name>     # interactive shell
+openshell sandbox exec <name> -- ... # run commands
+openshell sandbox logs <name>        # view logs
+openshell policy get <name>          # inspect policy
+```
+
+The harness handles setup. OpenShell handles the runtime.
 
 ## Install
 

TODO.md

@@ -9,68 +9,24 @@
 - Eliminates: openshell CLI binary dependency, output parsing fragility
 - Prerequisite: proto files stabilize (OpenShell is alpha)
 
-### Image registry as gateway config vs env override
-- `HARNESS_OS_IMAGE` env var overrides the version-based image resolution (for dev/CI)
-- Consider: gateway.yaml uses a `registry` field and images are relative to it
-
 ### registerProviders should filter by agent's provider list
 - `registerProviders()` in `cmd/providers.go` uses the gateway config's provider
   list, not the agent config's. When `gwCfg` is nil (common case), it tries to
   register all providers regardless of what the agent needs.
-- Why: confusing output — users see "skipped" messages for providers their
-  agent doesn't reference. No functional impact (missing credentials are
-  silently handled).
 - Fix: pass the agent's provider names to `registerProviders` and use them as
   a filter alongside (or instead of) the gateway config's list.
-- Files: `cmd/providers.go` (registerProviders signature), `cmd/up.go` (call site)
-
-## Config Format
-
-- [ ] Specify/document the YAML formats (agent config, provider profiles)
-- [ ] Document non-secret provider env vars (what `providers[].env` captures
-      and why it exists alongside secret credentials)
-
-## CLI — kubectl-style refactor
-
-Sequenced implementation plan. Each phase builds on the previous.
-
-### Phase 1: `apply` command (replaces `up` and `create`)
-- [ ] `cmd/apply.go` — unified deploy command, delegates to existing `upLocal()`
-- [ ] `-f` flag as primary interface (replaces `--agent-profile`)
-- [ ] `--attach` flag (default false) for interactive TTY (flips old `up` default)
-- [ ] `--dry-run` — resolve everything, report pass/fail per step, don't deploy
-- [ ] `-o yaml` — output fully resolved harness YAML with source annotations
-- [ ] `-o json` — machine-readable resolved config
-- [ ] Env var fallbacks: `OPENSHELL_GATEWAY`, `OPENSHELL_GATEWAY_ENDPOINT` (#1851)
-- [ ] Bare `harness apply` uses default agent config (same as old `harness up`)
-- [ ] Mark `up` and `create` as hidden deprecated aliases via cobra `Deprecated` field
-
-### Phase 2: `get` and `describe` commands (replaces `status`)
-- [ ] `cmd/get.go` — parent command with subcommands:
-  - `harness get agents` (aliases: `sandboxes`) — list running sandboxes
-  - `harness get providers` — list registered providers
-  - `harness get gateways` — list gateways
-- [ ] Shared `OutputFormat` type: `table|json|yaml` via `-o` flag on all `get` subcommands
-- [ ] Credential exclusion: `-o json/yaml` never includes secret values (#1830 pattern)
-- [ ] `cmd/describe.go` — `harness describe <name>` for detailed sandbox status
-- [ ] Mark `status` as hidden deprecated alias
-
-### Phase 3: `delete` command (replaces `teardown`)
-- [ ] `cmd/delete.go` — targeted + bulk deletion:
-  - `harness delete <name>` — delete specific sandbox
-  - `harness delete --all` — full teardown (sandboxes + providers + k8s)
-  - `harness delete --providers` — providers only
-  - `harness delete --k8s` — k8s resources only
-- [ ] Reuses existing `teardownSandboxes()`, `teardownProviders()`, `teardownK8s()`
-- [ ] Mark `teardown` as hidden deprecated alias
-
-### Phase 4: Integration + docs
-- [ ] Update `test/test-flow.sh` to use new verbs
-- [ ] Update SPEC.md command reference
-- [ ] Update README.md command reference
-- [ ] `render` becomes hidden alias for `apply -o yaml`
-- [ ] `deploy` stays as-is (infrastructure action)
-- [ ] `start`/`stop` stay as-is (lifecycle actions)
+- Files: `cmd/providers.go` (registerProviders signature), `cmd/executor.go` (call site)
+
+## CLI [DONE]
+
+kubectl-style refactor complete (PRs #67-#70):
+- [x] `harness apply` with `--dry-run`, `-o yaml|json`, `--attach`, `-f`
+- [x] `harness get agents|providers|gateways` with `-o table|json|yaml`
+- [x] `harness describe <name>`
+- [x] `harness delete <name>` with `--all`, `--sandboxes`, `--providers`, `--k8s`
+- [x] `harness deploy`, `start`, `stop` unchanged
+- [x] `teardown` and `status` as hidden deprecated aliases
+- [x] Old commands (`up`, `create`, `render`) removed (PR #68)
 
 ## Agent Config
 
@@ -79,45 +35,65 @@ Sequenced implementation plan. Each phase builds on the previous.
 - [x] `Harness` type with `ParseHarness`/`ParseHarnessFile`
 - [x] `RenderHarness` with built-in vs custom provider labeling
 - [x] Resolution: harness-local definitions > profiles/ tree > embedded defaults
-- [x] Backwards compat: single-doc agent YAMLs without `kind` still work
+
+### `kind: config` — embed sandbox files in harness YAML
+- [ ] `kind: config` documents for `claude.json`, `CLAUDE.md`, `mcp.json`,
+      `opencode.json`, `settings.json`, `policy.yaml`
+- [ ] Parsed by `ParseHarness` and stored as `Harness.Configs map[string][]byte`
+- [ ] Rendered to payload directory by `RenderPayload` instead of baking into image
+- [ ] Keeps sandbox image minimal — all agent-specific config in the harness YAML
+- [ ] Example:
+  ```yaml
+  ---
+  kind: config
+  name: claude.json
+  content: |
+    {"mcpServers": {"atlassian": {"command": "mcp-atlassian"}}}
+  ---
+  kind: config
+  name: CLAUDE.md
+  content: |
+    You are working inside an OpenShell sandbox.
+  ```
 
 ### Config reconciliation (`apply -o yaml`)
 - [ ] Resolves agent YAML against profiles/, defaults, and running gateway
 - [ ] Shows where each value came from (default, profile, harness file, env var)
 - [ ] Credentials rendered as `${VAR}` placeholders — shareable, replayable
 - [ ] Round-trip: `apply -o yaml > snapshot.yaml && apply -f snapshot.yaml`
-- [ ] `--dry-run` without `-o` reports pass/fail (gateway available? providers
-      resolvable? image exists? env vars resolved?)
-
-### `kind: config` — embed sandbox files in harness YAML (future)
-- [ ] `kind: config` documents for `claude.json`, `CLAUDE.md`, `mcp.json`, etc.
-- [ ] Rendered to payload directory instead of baking into sandbox image
-- [ ] Keeps sandbox image minimal — all agent-specific config in the harness YAML
 
 ### Provider abstraction layer
 - [ ] `kind: provider` targets `openshell provider create` today (imperative)
 - [ ] Abstraction supports future backends: gateway.toml (#1886), K8s CRDs (#1719)
-- [ ] Do not hard-code execution strategy — upstream is undecided
 
 ### Future fields
 - [ ] `description` — one line of human-readable context per agent config
 - [ ] `repo` — git URL to clone into the sandbox at start
 - [ ] `secrets` — non-provider secrets to inject
 
+## Testing [DONE]
+
+Config test suite (PR #71): 37 tests across 7 categories.
+- [x] Config parsing, output formats, env resolution, CLI flags
+- [x] Live sandbox lifecycle (create, describe, exec, env injection, delete)
+- [x] Provider registration (github, atlassian, vertex, gws, all-providers)
+- [x] Agent integration (claude inference, opencode inference, gh cli, jira mcp, gws gmail)
+- [x] CI: config-suite in CI workflow, test-suite-live in integration workflow
+
+### Known gaps
+- [ ] OpenCode + Vertex inference in sandbox blocked by policy fields not supported in 0.0.59
+      (`allow_encoded_slash`, custom policy sections crash supervisor)
+- [ ] Local `podman build` on macOS produces images that behave differently than CI `docker build`
+
 ## Upstream alignment
 
 ### Plugin compatibility (#1851)
 - [ ] Binary naming: plan for `openshell-harness` (PATH-based plugin discovery)
-- [ ] Dual invocation: standalone `openshell-harness` and plugin `openshell harness`
-- [ ] Env var fallbacks for gateway/endpoint/verbosity when running as plugin
-- [ ] Auth via `openshell-bootstrap` APIs (no token forwarding)
 - [ ] Status: #1851 is `question` label, not accepted. Design standalone first.
 
 ### Image building delegation
 - [ ] Evaluate delegating to `openshell-image-builder` for advanced image composition
-- [ ] Harness generates config (`.kaiden/workspace.json` + `config.toml`), builder consumes
 - [ ] Layered policy composition: base + agent-specific + user overlay
-- [ ] Programmatic Containerfile generation from config (stop maintaining static Dockerfiles)
 
 ### Upstream issues to track
 - #1719 — K8s Operator design (affects provider CRDs, declarative config)
@@ -126,15 +102,6 @@ Sequenced implementation plan. Each phase builds on the previous.
 - #1922 — Portable sandbox log collection (affects observability)
 - #1933 — Centralized audit/event log (affects run recorder)
 
-## Testing
-
-### Current coverage
-- Go unit tests across cmd/ and all internal/ packages (run in CI via `.github/workflows/ci.yml`)
-- Integration: local + kind + OCP via `make test-all`
-
-### Gaps
-- [ ] Integration test for `harness up --provider-refresh`
-
 ## Release
 
 - [x] Add CHANGELOG.md
@@ -143,51 +110,15 @@ Sequenced implementation plan. Each phase builds on the previous.
 
 ## Observability & Tracing
 
-Investigation (Jun 2026) validated two paths for capturing full agent session
-data (prompts, responses, tool calls, token counts, cost):
-
-### What works today
-- **OpenShell OCSF JSONL** (`ocsf_json_enabled` setting) captures network/process/policy
-  events inside the sandbox. Structured, OCSF v1.7.0 compliant. No conversation content.
-- **Claude Code OTel export** sends traces (span structure), logs (full API request/response
-  bodies), and metrics (token counts, cost) via standard OTLP env vars.
-- **Langfuse hooks plugin** (`langfuse-observability`) reads Claude Code transcript files
-  directly and creates Langfuse traces with full input/output. Best LLM-specific UI.
-  Setup: `docs/langfuse-setup.md`.
-- **MLflow** accepts OTel traces at `/v1/traces` (not logs). Proven working with Claude Code
-  via `OTEL_EXPORTER_OTLP_ENDPOINT`. Good for span structure + token counts but not
-  conversation content (that lives in the OTel logs signal, which MLflow doesn't ingest).
-
-### Integration options (pick one or combine)
-- [ ] **Langfuse (hooks)** -- full conversation content, best UI, self-hosted Docker.
-      No OTel plumbing needed. Plugin reads transcripts post-hoc.
-- [ ] **Langfuse (OTel)** -- span structure via OTLP/HTTP at `/api/public/otel`.
-      Prompts land in metadata (not input field) because Claude Code uses `user_prompt`
-      attribute, not `gen_ai.prompt`. Response content not captured via this path.
-- [ ] **MLflow (OTel)** -- traces only, no logs. Good for span structure + AI Gateway
-      (inference routing with budget/rate limiting). Self-hosted SQLite or Postgres.
-- [ ] **SigNoz (OTel)** -- accepts traces + logs + metrics on same OTLP endpoint.
-      Self-hosted Docker, 4GB RAM. Only backend that ingests all three Claude Code signals.
-- [ ] **OTel Collector fan-out** -- route traces to Langfuse/MLflow, logs to SigNoz/Loki,
-      metrics to SigNoz/Prometheus. Best-of-breed but more infrastructure.
+Investigation (Jun 2026) validated paths for capturing agent session data.
+Langfuse hooks plugin installed and working. MLflow spiked. SigNoz identified
+as strongest OTel backend for full signal coverage.
 
 ### Harness integration (future)
-- [ ] `harness deploy` starts observability backend (Langfuse/MLflow/SigNoz) if not running
-- [ ] `harness up` injects OTel env vars or configures hooks automatically
+- [ ] `harness apply` auto-injects OTel env vars or configures Langfuse hooks
 - [ ] `harness runs list/show` queries traces from the backend
 - [ ] Headless mode (`harness run --task '...'`) records automatically
 
-### Upstream to watch
-- OpenShell portable sandbox log collection (#1922) and centralized audit/event log (#1933)
-  are in early investigation. No concrete implementation yet -- this is where the harness
-  recorder fits.
-- AgentGateway (Linux Foundation) proposed for embedding in OpenShell (#998, rejected by
-  NVIDIA). Could be deployed as a companion proxy for LLM traffic observability with
-  token-level detail. See design doc.
-
-### Design doc
-`~/.gstack/projects/robbycochran-harness-openshell/rc-rc-nextnext-design-20260613-130837.md`
-
 ## Deferred (post-0.1)
 
 - [ ] Multi-agent workflow support (fleet.yaml / workflow.yaml)

cmd/apply.go

@@ -46,6 +46,11 @@ then deploy a sandbox. Use --dry-run to validate without deploying, or
 			agentCfg := harness.Agent
 			agentPath := resolveAgentPath(harnessDir, agentName, file)
 
+			// Print config path (skip for structured output -- goes to stderr)
+			if output == "" {
+				status.Infof("Config: %s", agentPath)
+			}
+
 			// Resolve output modes before touching the gateway
 			if output == "yaml" || output == "json" {
 				return renderOutput(harnessDir, harness, output)

cmd/deploy.go

@@ -139,7 +139,7 @@ func deployLocal(gw gateway.Gateway) error {
 func deployFromConfig(harnessDir string, gwCfg *gateway.GatewayConfig, gw gateway.Gateway, kc, clusterRunner k8s.Runner) (retErr error) {
 	defer func() {
 		if retErr != nil {
-			fmt.Fprintf(os.Stderr, "\nDeploy failed. Clean up with: harness teardown --k8s\n")
+			fmt.Fprintf(os.Stderr, "\nDeploy failed. Clean up with: harness delete --k8s\n")
 		}
 	}()
 	ctx := context.Background()

cmd/executor.go

@@ -82,6 +82,18 @@ func upLocal(opts upLocalOpts) error {
 		return fmt.Errorf("rendering payload: %w", err)
 	}
 
+	// Resolve payload entries into upload pairs
+	var extraUploads []gateway.Upload
+	if opts.harness != nil && len(opts.harness.Payloads) > 0 {
+		resolved, err := agent.ResolvePayloads(opts.harness.Payloads, opts.harnessDir, payloadDir)
+		if err != nil {
+			return fmt.Errorf("resolving payloads: %w", err)
+		}
+		for _, u := range resolved {
+			extraUploads = append(extraUploads, gateway.Upload{Src: u.Src, Dst: u.Dst})
+		}
+	}
+
 	status.Header("Sandbox")
 	var sandboxCmd []string
 	if noTTY {
@@ -100,6 +112,7 @@ func upLocal(opts upLocalOpts) error {
 		retrySleep: opts.retrySleep,
 		sandboxCmd: sandboxCmd,
 		payloadDir: payloadDir,
+		uploads:    extraUploads,
 		env:        agentCfg.BuildEnvMap(),
 	})
 }

cmd/get.go

@@ -175,9 +175,9 @@ func newGetGatewaysCmd(cli string) *cobra.Command {
 				if g.Active {
 					active = "*"
 				}
-				rows[i] = []string{active + g.Name, g.Endpoint}
+				rows[i] = []string{g.Name, g.Endpoint, active}
 			}
-			printTable([]string{"Name", "Endpoint"}, rows)
+			printTable([]string{"Name", "Endpoint", "Active"}, rows)
 			return nil
 		},
 	}

cmd/sandbox.go

@@ -16,15 +16,16 @@ import (
 type sandboxOpts struct {
 	harnessDir string
 	gw         gateway.Gateway
-	name       string            // sandbox name
-	image      string            // sandbox image ref or relative Dockerfile dir
-	providers  []string          // registered providers to attach
-	noTTY      bool              // true → TTY=false for the sandbox
-	retrySleep time.Duration     // pause between retry attempts
-	sandboxCmd []string          // command to run inside the sandbox
-	payloadDir string            // pre-rendered payload dir to upload
-	env        map[string]string // env vars injected via --env on sandbox create
-	onSuccess  func(name string) // called after successful creation (optional)
+	name       string              // sandbox name
+	image      string              // sandbox image ref or relative Dockerfile dir
+	providers  []string            // registered providers to attach
+	noTTY      bool                // true → TTY=false for the sandbox
+	retrySleep time.Duration       // pause between retry attempts
+	sandboxCmd []string            // command to run inside the sandbox
+	payloadDir string              // pre-rendered payload dir to upload
+	uploads    []gateway.Upload    // additional uploads (payloads)
+	env        map[string]string   // env vars injected via --env on sandbox create
+	onSuccess  func(name string)   // called after successful creation (optional)
 }
 
 // createSandbox resolves the image path, stages the payload directory,
@@ -55,16 +56,18 @@ func createSandbox(opts sandboxOpts) error {
 		return fmt.Errorf("staging payload: %w", err)
 	}
 
-	// Create sandbox with retry loop (up to 5 attempts).
-	for attempt := 1; attempt <= 5; attempt++ {
+	const maxRetries = 5
+	for attempt := 1; attempt <= maxRetries; attempt++ {
+		uploads := []gateway.Upload{{Src: uploadDir, Dst: "/sandbox/.config"}}
+		uploads = append(uploads, opts.uploads...)
+
 		err := opts.gw.SandboxCreate(gateway.SandboxCreateOpts{
 			Name:      opts.name,
 			From:      image,
 			Providers: opts.providers,
 			TTY:       !opts.noTTY,
 			Keep:      true,
-			UploadSrc: uploadDir,
-			UploadDst: "/sandbox/.config",
+			Uploads:   uploads,
 			Command:   opts.sandboxCmd,
 			Env:       opts.env,
 		})
@@ -75,10 +78,10 @@ func createSandbox(opts sandboxOpts) error {
 			return nil
 		}
 
-		status.Warnf("attempt %d: %v, retrying in 5s", attempt, err)
+		status.Warnf("attempt %d: %v, retrying in %s", attempt, err, opts.retrySleep)
 		opts.gw.SandboxDelete(opts.name) // best-effort cleanup
 
-		if attempt == 5 {
+		if attempt == maxRetries {
 			return fmt.Errorf("sandbox create failed after 5 attempts: %w", err)
 		}
 		time.Sleep(opts.retrySleep)