diff --git a/.agents/skills/validate/SKILL.md b/.agents/skills/validate/SKILL.md new file mode 100644 index 0000000..472f458 --- /dev/null +++ b/.agents/skills/validate/SKILL.md @@ -0,0 +1,95 @@ +--- +name: validate +description: Run the full test matrix for harness-openshell. Use when asked to "validate", "run tests", "check everything", or before any commit/push/PR. +--- + +# Validate + +Run the full test matrix. Skip steps that require unavailable infrastructure. + +## Steps + +Run each step sequentially. Report pass/fail for each. + +### 1. Build + +```bash +go build ./... +``` + +### 2. Unit tests + +```bash +CGO_ENABLED=0 go test ./... +``` + +### 3. Vet + +```bash +go vet ./... +``` + +### 4. Local integration (full providers) + +Requires: `openshell` running locally, provider credentials configured. + +```bash +make test-local +``` + +Skip if `openshell` is not on PATH or the gateway is not running. + +### 5. Local integration (CI mode) + +Requires: `openshell` running locally. No credentials needed. + +```bash +make test-local +``` + +Pass `--ci` to `test-flow.sh` (auto-detected when `CI=true`). + +### 6. OCP integration + +Requires: `KUBECONFIG` set, cluster accessible. + +```bash +make test-remote +``` + +Skip if `KUBECONFIG` is not set or `kubectl cluster-info` fails. + +### 7. Kind integration + +Requires: `kind` on PATH. + +```bash +make test-kind +``` + +Skip if `kind` is not on PATH. + +### 8. CI status + +Check if CI is green for the current branch: + +```bash +gh run list --branch $(git branch --show-current) --limit 3 +``` + +## Output + +Report a summary table: + +``` +Validation Results +────────────────── + Build: PASS + Unit tests: PASS (6 packages) + Vet: PASS + Local (full): PASS (22/22) + Local (CI): PASS (14/14) + OCP: PASS (10/10) + Kind: SKIP (kind not installed) + CI: GREEN (3/3 workflows) +``` diff --git a/AGENTS.md b/AGENTS.md index bdc3256..860cba8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,6 +62,56 @@ For multi-step tasks, state a brief plan: 3. **Upstream alignment** — OpenShell is alpha and developing quickly. Don't fight the framework. Use native patterns (providers v2, profiles, inference.local, policy composition) and adapt when upstream changes. +## Upstream Conventions + +Follow these conventions from the OpenShell ecosystem. Do not invent alternatives. + +### Output format: `-o table|json|yaml` +Every list/get command must support `-o` with three formats. `table` is the default +for human consumption, `json` and `yaml` for machine consumption. Define a shared +`OutputFormat` type used by all commands. Match the convention from OpenShell +issues #1745 and #1750. + +### Credential exclusion from structured output +Never serialize credential values into `-o json` or `-o yaml` output. Expose key +names only. This is a security invariant, not a nice-to-have. See OpenShell +PR #1830 `provider_to_json()` for the pattern. + +### Flag resolution order +Explicit flag > `OPENSHELL_*` env var > config file > default. This matches the +plugin host contract from issue #1851. The env vars `OPENSHELL_GATEWAY`, +`OPENSHELL_GATEWAY_ENDPOINT`, and verbosity flags propagate from the plugin host. + +### Policy schema +`kind: policy` documents must use the upstream OpenShell policy YAML schema +verbatim (from the `openshell-policy` crate). Do not invent a harness-specific +policy format. A policy written for the harness should be byte-compatible with +what `openshell-image-builder` generates. + +### Provider abstraction +`kind: provider` is an abstraction layer, not a thin wrapper around +`openshell provider create`. The backend may change to gateway.toml entries +(#1886) or K8s CRDs (#1719) as upstream settles. Implement the imperative +CLI backend today. Do not hard-code the execution strategy. + +### Plugin compatibility +The binary may eventually be discoverable as an OpenShell plugin via +`openshell-` PATH-based discovery (#1851). Design for standalone first. +Plugin compatibility (binary naming, env var consumption) is additive. Do not +depend on plugin host behavior that is not yet accepted upstream. + +### Do not cache or forward auth tokens +Issue #1851 explicitly prohibits token forwarding to plugins. The harness must +resolve credentials fresh via `openshell-bootstrap` or configured auth. Never +store, cache, or relay gateway auth tokens. + +### Delegate image building +Do not replicate Dev Container Feature fetching or complex image composition. +The `openshell-image-builder` handles agent installation, settings bake-in, +policy composition, and OCI artifact fetching. If the harness needs advanced +image support, generate config the image-builder consumes rather than +reimplementing in Go. + ## The harness should shrink, not grow This harness exists to bridge gaps in OpenShell's current capabilities. As OpenShell matures, custom code should be replaced by upstream features. Every workaround should reference the upstream issue that would eliminate it. diff --git a/TODO.md b/TODO.md index fa5fba0..2451506 100644 --- a/TODO.md +++ b/TODO.md @@ -30,76 +30,101 @@ - [ ] Document non-secret provider env vars (what `providers[].env` captures and why it exists alongside secret credentials) -## CLI - -- [ ] Flows that support agent.yaml (`create`, `up`) should also support - `--provider-profile` and provider config overrides +## CLI — kubectl-style refactor + +Sequenced implementation plan. Each phase builds on the previous. + +### Phase 1: `apply` command (replaces `up` and `create`) +- [ ] `cmd/apply.go` — unified deploy command, delegates to existing `upLocal()` +- [ ] `-f` flag as primary interface (replaces `--agent-profile`) +- [ ] `--attach` flag (default false) for interactive TTY (flips old `up` default) +- [ ] `--dry-run` — resolve everything, report pass/fail per step, don't deploy +- [ ] `-o yaml` — output fully resolved harness YAML with source annotations +- [ ] `-o json` — machine-readable resolved config +- [ ] Env var fallbacks: `OPENSHELL_GATEWAY`, `OPENSHELL_GATEWAY_ENDPOINT` (#1851) +- [ ] Bare `harness apply` uses default agent config (same as old `harness up`) +- [ ] Mark `up` and `create` as hidden deprecated aliases via cobra `Deprecated` field + +### Phase 2: `get` and `describe` commands (replaces `status`) +- [ ] `cmd/get.go` — parent command with subcommands: + - `harness get agents` (aliases: `sandboxes`) — list running sandboxes + - `harness get providers` — list registered providers + - `harness get gateways` — list gateways +- [ ] Shared `OutputFormat` type: `table|json|yaml` via `-o` flag on all `get` subcommands +- [ ] Credential exclusion: `-o json/yaml` never includes secret values (#1830 pattern) +- [ ] `cmd/describe.go` — `harness describe ` for detailed sandbox status +- [ ] Mark `status` as hidden deprecated alias + +### Phase 3: `delete` command (replaces `teardown`) +- [ ] `cmd/delete.go` — targeted + bulk deletion: + - `harness delete ` — delete specific sandbox + - `harness delete --all` — full teardown (sandboxes + providers + k8s) + - `harness delete --providers` — providers only + - `harness delete --k8s` — k8s resources only +- [ ] Reuses existing `teardownSandboxes()`, `teardownProviders()`, `teardownK8s()` +- [ ] Mark `teardown` as hidden deprecated alias + +### Phase 4: Integration + docs +- [ ] Update `test/test-flow.sh` to use new verbs +- [ ] Update SPEC.md command reference +- [ ] Update README.md command reference +- [ ] `render` becomes hidden alias for `apply -o yaml` +- [ ] `deploy` stays as-is (infrastructure action) +- [ ] `start`/`stop` stay as-is (lifecycle actions) ## Agent Config -### Self-contained agent YAML (multi-document, k8s-style) -- [ ] Support multi-document YAML (`---` separated) where all objects live in one file -- Agent still references everything by name (`profile: github`), but provider/gateway/policy - definitions can be co-located in the same file instead of separate files in profiles/ -- Parser reads all documents via `yaml.Decoder` loop, indexes by `kind`+`name`, resolves - references against local set first, falls back to profiles/ tree -- Composes naturally: split the file up and drop objects into `profiles/` to share across agents -- Example: - ```yaml - --- - kind: agent - name: my-agent - entrypoint: claude - gateway: local - providers: - - profile: github - - profile: vertex - env: - ANTHROPIC_BASE_URL: https://inference.local - --- - kind: provider - name: github - type: github - credentials: [GITHUB_TOKEN] - endpoints: - - { host: "api.github.com", port: 443 } - --- - kind: provider - name: vertex - type: google-vertex-ai - credentials: [GOOGLE_APPLICATION_CREDENTIALS] - --- - kind: gateway - name: local - type: local - insecure: true - --- - kind: policy - network_policies: - github: - endpoints: - - { host: "api.github.com", port: 443 } - ``` -- Goal: `harness up -f agent.yaml` with one file. Zero to working sandboxed agent. -- Existing single-document agent YAMLs (no `kind` field) continue to work unchanged - -### `harness render` as live config snapshot -- [ ] `harness render` queries the running gateway for effective state, not just YAML files -- Outputs what is actually configured: registered providers, active gateway, inference - config, sandbox policy, env structure -- Credentials replaced with `${VAR}` placeholders -- the snapshot is shareable -- Replay with different creds: `GITHUB_TOKEN=theirs harness up -f snapshot.yaml` -- Like `kubectl get -o yaml` -- captures the running shape, not the source config -- Round-trip: `harness render > snapshot.yaml && harness up -f snapshot.yaml` should - reproduce the same agent setup (with different credentials from env) -- [ ] `harness render preview -f harness.yaml` -- dry-run that resolves all references - (providers, gateway, env vars) and shows the fully resolved config without deploying. - Like `terraform plan` or `helm template`. Shows what `harness up` would do. +### Multi-document harness YAML [DONE] +- [x] `kind: agent/provider/gateway/policy` dispatch via `yaml.Decoder` loop +- [x] `Harness` type with `ParseHarness`/`ParseHarnessFile` +- [x] `RenderHarness` with built-in vs custom provider labeling +- [x] Resolution: harness-local definitions > profiles/ tree > embedded defaults +- [x] Backwards compat: single-doc agent YAMLs without `kind` still work + +### Config reconciliation (`apply -o yaml`) +- [ ] Resolves agent YAML against profiles/, defaults, and running gateway +- [ ] Shows where each value came from (default, profile, harness file, env var) +- [ ] Credentials rendered as `${VAR}` placeholders — shareable, replayable +- [ ] Round-trip: `apply -o yaml > snapshot.yaml && apply -f snapshot.yaml` +- [ ] `--dry-run` without `-o` reports pass/fail (gateway available? providers + resolvable? image exists? env vars resolved?) + +### `kind: config` — embed sandbox files in harness YAML (future) +- [ ] `kind: config` documents for `claude.json`, `CLAUDE.md`, `mcp.json`, etc. +- [ ] Rendered to payload directory instead of baking into sandbox image +- [ ] Keeps sandbox image minimal — all agent-specific config in the harness YAML + +### Provider abstraction layer +- [ ] `kind: provider` targets `openshell provider create` today (imperative) +- [ ] Abstraction supports future backends: gateway.toml (#1886), K8s CRDs (#1719) +- [ ] Do not hard-code execution strategy — upstream is undecided ### Future fields -- [ ] `description` -- one line of human-readable context per agent config -- [ ] `repo` -- git URL to clone into the sandbox at start -- [ ] `secrets` -- non-provider secrets to inject +- [ ] `description` — one line of human-readable context per agent config +- [ ] `repo` — git URL to clone into the sandbox at start +- [ ] `secrets` — non-provider secrets to inject + +## Upstream alignment + +### Plugin compatibility (#1851) +- [ ] Binary naming: plan for `openshell-harness` (PATH-based plugin discovery) +- [ ] Dual invocation: standalone `openshell-harness` and plugin `openshell harness` +- [ ] Env var fallbacks for gateway/endpoint/verbosity when running as plugin +- [ ] Auth via `openshell-bootstrap` APIs (no token forwarding) +- [ ] Status: #1851 is `question` label, not accepted. Design standalone first. + +### Image building delegation +- [ ] Evaluate delegating to `openshell-image-builder` for advanced image composition +- [ ] Harness generates config (`.kaiden/workspace.json` + `config.toml`), builder consumes +- [ ] Layered policy composition: base + agent-specific + user overlay +- [ ] Programmatic Containerfile generation from config (stop maintaining static Dockerfiles) + +### Upstream issues to track +- #1719 — K8s Operator design (affects provider CRDs, declarative config) +- #1851 — Plugin system (affects binary naming, env var contract) +- #1886 — Declarative provider config in gateway.toml (affects `kind: provider`) +- #1922 — Portable sandbox log collection (affects observability) +- #1933 — Centralized audit/event log (affects run recorder) ## Testing diff --git a/cmd/apply.go b/cmd/apply.go new file mode 100644 index 0000000..150fca7 --- /dev/null +++ b/cmd/apply.go @@ -0,0 +1,226 @@ +package cmd + +import ( + "encoding/json" + "fmt" + "os" + "time" + + "github.com/robbycochran/harness-openshell/internal/agent" + "github.com/robbycochran/harness-openshell/internal/gateway" + "github.com/robbycochran/harness-openshell/internal/status" + "github.com/spf13/cobra" +) + +func NewApplyCmd(harnessDir, cli string) *cobra.Command { + var ( + file string + agentName string + gatewayName string + gatewayProfile string + sandboxName string + attach bool + providerRefresh bool + dryRun bool + output string + ) + + cmd := &cobra.Command{ + Use: "apply [flags]", + Short: "Apply an agent configuration to create a sandbox", + Long: `Resolve an agent config against the profiles directory and running gateway, +then deploy a sandbox. Use --dry-run to validate without deploying, or +-o yaml to output the fully resolved configuration.`, + RunE: func(cmd *cobra.Command, args []string) error { + if gatewayName != "" && gatewayProfile != "" { + return fmt.Errorf("--gateway and --gateway-profile are mutually exclusive") + } + if len(args) > 0 && sandboxName == "" { + sandboxName = args[0] + } + + harness, err := resolveHarness(harnessDir, agentName, file) + if err != nil { + return err + } + agentCfg := harness.Agent + agentPath := resolveAgentPath(harnessDir, agentName, file) + + // Resolve output modes before touching the gateway + if output == "yaml" || output == "json" { + return renderOutput(harnessDir, harness, output) + } + + gw := gateway.New(cli) + if err := gw.CheckMinVersion("0.0.59"); err != nil { + status.Warn(fmt.Sprintf("OpenShell version: %v", err)) + } + + // Resolve gateway + var gwCfg *gateway.GatewayConfig + gwTarget := gatewayName + if gatewayProfile != "" { + gwCfg, err = resolveGatewayConfigFromFile(gatewayProfile) + if err != nil { + return err + } + gwTarget = gwCfg.Gateway.Type + } else { + if gwTarget == "" { + if agentCfg.Gateway != "" { + gwTarget = agentCfg.Gateway + } else { + gwTarget = "local" + } + } + gwCfg, _ = resolveGatewayConfigWithHarness(harnessDir, gwTarget, harness) + } + isRemote := gwTarget != "local" + + if dryRun { + return dryRunApply(gw, agentCfg, gwTarget, isRemote) + } + + return upLocal(upLocalOpts{ + harnessDir: harnessDir, + gw: gw, + gwCfg: gwCfg, + ensureLocal: !isRemote, + agentCfg: agentCfg, + agentPath: agentPath, + sandboxName: sandboxName, + noTTY: !attach, + providerRefresh: providerRefresh, + harness: harness, + retrySleep: 5 * time.Second, + }) + }, + } + + cmd.Flags().StringVarP(&file, "file", "f", "", "Path to harness/agent YAML file") + cmd.Flags().StringVar(&agentName, "agent", "default", "Agent config name (from profiles/)") + cmd.Flags().StringVar(&gatewayName, "gateway", envOr("OPENSHELL_GATEWAY", ""), "Gateway profile name (local, kind, ocp)") + cmd.Flags().StringVar(&gatewayProfile, "gateway-profile", "", "Path to gateway profile YAML") + cmd.Flags().StringVar(&sandboxName, "name", "", "Sandbox name (overrides agent config)") + cmd.Flags().BoolVar(&attach, "attach", false, "Attach TTY after creation (interactive mode)") + cmd.Flags().BoolVar(&providerRefresh, "provider-refresh", false, "Delete and recreate all providers") + cmd.Flags().BoolVar(&dryRun, "dry-run", false, "Validate configuration without deploying") + cmd.Flags().StringVarP(&output, "output", "o", "", "Output format: yaml or json") + + return cmd +} + +func renderOutput(harnessDir string, h *agent.Harness, format string) error { + builtinProviders := loadProviderProfiles(harnessDir) + + gwName := h.Agent.Gateway + if gwName == "" { + gwName = "local" + } + if len(h.Gateways) == 0 { + if gwData := loadGatewayProfile(harnessDir, gwName); gwData != nil { + h.Gateways[gwName] = gwData + } + } + + switch format { + case "yaml": + out, err := agent.RenderHarness(h, builtinProviders) + if err != nil { + return fmt.Errorf("rendering harness: %w", err) + } + fmt.Print(string(out)) + case "json": + data := map[string]any{ + "agent": h.Agent, + "gateways": mapKeys(h.Gateways), + "providers": mapKeys(h.Providers), + "hasPolicy": h.Policy != nil, + } + out, err := json.MarshalIndent(data, "", " ") + if err != nil { + return fmt.Errorf("marshaling json: %w", err) + } + fmt.Println(string(out)) + } + return nil +} + +func mapKeys(m map[string][]byte) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + return keys +} + +func dryRunApply(gw gateway.Gateway, agentCfg *agent.AgentConfig, gwTarget string, isRemote bool) error { + status.Header("Dry Run") + allPass := true + + // 1. Agent config + status.OKf("agent: %s (entrypoint: %s)", agentCfg.Name, agentCfg.EffectiveEntrypoint()) + + // 2. Image + image := resolveSandboxImage(agentCfg.Image) + status.OKf("image: %s", image) + + // 3. Gateway + if isRemote { + if gw.InferenceGet() != nil { + status.Failf("gateway: %s (not reachable)", gwTarget) + allPass = false + } else { + status.OKf("gateway: %s (reachable)", gwTarget) + } + } else { + status.OKf("gateway: %s (local)", gwTarget) + } + + // 4. Providers + for _, p := range agentCfg.Providers { + if gw.ProviderGet(p.Profile) == nil { + status.OKf("provider: %s (registered)", p.Profile) + } else { + status.Warnf("provider: %s (not registered, will be created)", p.Profile) + } + } + + // 5. Env vars + env := agentCfg.BuildEnvMap() + resolved := 0 + missing := 0 + for k, v := range env { + if v != "" { + resolved++ + } else { + status.Warnf("env: %s (empty)", k) + missing++ + } + } + if resolved > 0 { + status.OKf("env: %d vars resolved", resolved) + } + if missing > 0 { + status.Warnf("env: %d vars empty", missing) + } + + // 6. Task file + if agentCfg.Task != "" { + if _, err := os.Stat(agentCfg.Task); err != nil { + status.Failf("task: %s (not found)", agentCfg.Task) + allPass = false + } else { + status.OKf("task: %s", agentCfg.Task) + } + } + + fmt.Println() + if allPass { + status.OK("Ready to apply") + } else { + status.Fail("Issues found -- fix before applying") + return fmt.Errorf("dry-run failed") + } + return nil +} diff --git a/main.go b/main.go index dd2386a..567e5d3 100644 --- a/main.go +++ b/main.go @@ -60,17 +60,32 @@ func main() { } root.CompletionOptions.HiddenDefaultCmd = true + // Primary commands root.AddCommand( - cmd.NewUpCmd(harnessDir, cli), - cmd.NewCreateCmd(harnessDir, cli), + cmd.NewApplyCmd(harnessDir, cli), cmd.NewDeployCmd(harnessDir, cli), - cmd.NewTeardownCmd(harnessDir, cli), - cmd.NewStatusCmd(harnessDir, cli), cmd.NewStopCmd(harnessDir, cli), cmd.NewStartCmd(harnessDir, cli), - cmd.NewRenderCmd(harnessDir, cli), ) + // Deprecated aliases (hidden, print warning) + upCmd := cmd.NewUpCmd(harnessDir, cli) + upCmd.Hidden = true + upCmd.Deprecated = "use 'harness apply --attach' instead" + createCmd := cmd.NewCreateCmd(harnessDir, cli) + createCmd.Hidden = true + createCmd.Deprecated = "use 'harness apply' instead" + teardownCmd := cmd.NewTeardownCmd(harnessDir, cli) + teardownCmd.Hidden = true + teardownCmd.Deprecated = "use 'harness delete' instead" + statusCmd := cmd.NewStatusCmd(harnessDir, cli) + statusCmd.Hidden = true + statusCmd.Deprecated = "use 'harness get agents' instead" + renderCmd := cmd.NewRenderCmd(harnessDir, cli) + renderCmd.Hidden = true + renderCmd.Deprecated = "use 'harness apply -o yaml' instead" + root.AddCommand(upCmd, createCmd, teardownCmd, statusCmd, renderCmd) + if err := root.Execute(); err != nil { fmt.Fprintf(os.Stderr, "Error: %v\n", err) os.Exit(1) diff --git a/test/test-flow.sh b/test/test-flow.sh index e08adb7..0900796 100755 --- a/test/test-flow.sh +++ b/test/test-flow.sh @@ -34,12 +34,12 @@ AGENT_FLAG="--agent" if [[ "${CI:-}" == "true" ]]; then NO_PROVIDERS=true PROFILE="test/ci-agent.yaml" - AGENT_FLAG="--agent-profile" + AGENT_FLAG="--file" fi for arg in "$@"; do case "$arg" in - --ci) NO_PROVIDERS=true; PROFILE="test/ci-agent.yaml"; AGENT_FLAG="--agent-profile" ;; + --ci) NO_PROVIDERS=true; PROFILE="test/ci-agent.yaml"; AGENT_FLAG="--file" ;; --reuse-gateway) REUSE_GATEWAY=true ;; --no-providers) NO_PROVIDERS=true ;; --debug) DEBUG=true ;; @@ -179,7 +179,7 @@ summary() { test_errors() { echo "=== test: error scenarios ===" - step_fail "nonexistent profile" harness up --gateway local --agent nonexistent --no-tty + step_fail "nonexistent profile" harness apply --gateway local --agent nonexistent if $REUSE_GATEWAY; then step "teardown (first)" harness teardown --sandboxes --providers @@ -205,12 +205,12 @@ test_local() { # up auto-registers providers when missing local sandbox_name="test-agent" - step "sandbox create (up)" harness up --gateway local --name "$sandbox_name" $AGENT_FLAG "$PROFILE" --no-tty + step "sandbox create (up)" harness apply --gateway local --name "$sandbox_name" $AGENT_FLAG "$PROFILE" sandbox_verify "$sandbox_name" step "sandbox delete" "$CLI" sandbox delete "$sandbox_name" local create_name="test-create" - step "sandbox create (create)" harness create --name "$create_name" --agent-profile test/ci-agent.yaml + step "sandbox create (create)" harness apply --name "$create_name" --file test/ci-agent.yaml step "sandbox verify (create)" "$CLI" sandbox exec --name "$create_name" -- echo "hello" step "sandbox delete (create)" "$CLI" sandbox delete "$create_name" @@ -218,7 +218,7 @@ test_local() { echo "" echo "=== test: missing providers ===" step "teardown providers" harness teardown --providers - step "up with no providers" harness up --gateway local --name test-noprov --no-tty + step "up with no providers" harness apply --gateway local --name test-noprov step "cleanup" harness teardown --sandboxes fi @@ -264,7 +264,7 @@ test_kind() { step "gateway reachable" "$CLI" inference get local sandbox_name="test-kind" - step "sandbox create" harness up --gateway kind --name "$sandbox_name" $AGENT_FLAG "$PROFILE" --no-tty + step "sandbox create" harness apply --gateway kind --name "$sandbox_name" $AGENT_FLAG "$PROFILE" sandbox_verify "$sandbox_name" if ! $NO_PROVIDERS; then @@ -302,10 +302,10 @@ test_ocp() { local sandbox_name if $NO_PROVIDERS; then sandbox_name="test-ocp" - step "sandbox create" harness create --agent-profile=test/ci-agent.yaml --name "$sandbox_name" + step "sandbox create" harness apply -f test/ci-agent.yaml --name "$sandbox_name" else sandbox_name="agent" - step "sandbox create (up)" harness up --gateway ocp --name "$sandbox_name" --no-tty + step "sandbox create (up)" harness apply --gateway ocp --name "$sandbox_name" fi sandbox_verify "$sandbox_name"