Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/agent-task.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
You are reviewing the harness-openshell repository at /sandbox/harness-openshell.

Your job: identify exactly ONE improvement and implement it. Not a list — one concrete change with a clear diff.

Priority order (pick the first that applies):

1. **Spec drift** — a field, command, or behavior documented in SPEC.md that doesn't match the Go code, or code behavior not reflected in the spec
2. **Test gap** — a code path in cmd/ or internal/ that has no test coverage and is easy to test (one test function, not a test suite)
3. **Doc inaccuracy** — something in README.md or profiles/README.md that is wrong or misleading given the current code
4. **Code simplification** — dead code, redundant checks, or unnecessarily complex logic that can be simplified without changing behavior

Do NOT:
- Touch .github/workflows/ or .goreleaser.yaml
- Add new features or change behavior
- Refactor for style preferences
- Make multiple unrelated changes

After making the change, run `go build ./...` to verify compilation. If you added or changed Go code, run `go test ./...` to verify tests pass.

Output the change as a git diff:
```
cd /sandbox/harness-openshell
git diff
```

The last thing you output should be the diff and a one-line summary of what you changed and why.
4 changes: 4 additions & 0 deletions .github/agent.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
name: agent-improve
base_agent: default
repo: https://github.com/stackrox/harness-openshell
task: .github/agent-task.md
147 changes: 147 additions & 0 deletions .github/workflows/agent.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
name: Agent

on:
schedule:
- cron: "23 9 * * 1-5" # weekdays ~9:23am UTC
workflow_dispatch:

permissions:
contents: write
pull-requests: write
id-token: write # for Workload Identity Federation (future)
Comment on lines +8 to +11

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Remove OIDC token minting until WIF is actually enabled.

Line 11 grants id-token: write, but this workflow currently authenticates via static ADC JSON (Lines 35-44). That is avoidable privilege on a job that executes generated changes.

Suggested minimal hardening
 permissions:
   contents: write
   pull-requests: write
-  id-token: write  # for Workload Identity Federation (future)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 8 - 11, Remove the id-token: write
permission line from the permissions block in the workflow file, as it is not
currently needed. The workflow authenticates using static ADC JSON credentials
(as seen in lines 35-44) rather than Workload Identity Federation, so granting
id-token write privileges creates unnecessary security risk on a job that
executes generated changes. Keep only the contents: write and pull-requests:
write permissions in the permissions block.


jobs:
improve:
runs-on: ubuntu-latest
Comment on lines +3 to +15

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Add workflow concurrency to close the PR-throttle race window.

Two runs starting near-simultaneously can both pass the open-PR guard before either creates a PR, which defeats the throttle objective.

Suggested fix
 on:
   schedule:
     - cron: "23 9 * * 1-5"  # weekdays ~9:23am UTC
   workflow_dispatch:
+
+concurrency:
+  group: agent-improve
+  cancel-in-progress: false
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
on:
schedule:
- cron: "23 9 * * 1-5" # weekdays ~9:23am UTC
workflow_dispatch:
permissions:
contents: write
pull-requests: write
id-token: write # for Workload Identity Federation
jobs:
improve:
runs-on: ubuntu-latest
on:
schedule:
- cron: "23 9 * * 1-5" # weekdays ~9:23am UTC
workflow_dispatch:
concurrency:
group: agent-improve
cancel-in-progress: false
permissions:
contents: write
pull-requests: write
id-token: write # for Workload Identity Federation
jobs:
improve:
runs-on: ubuntu-latest
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 3 - 15, The workflow lacks
concurrency control, allowing multiple simultaneous runs to race past the
open-PR guard check before either creates a PR. Add a concurrency configuration
to the improve job (or at the workflow level) that groups runs by a consistent
key such as the branch or repository to ensure only one run executes at a time,
preventing the race condition that defeats the PR throttle objective.

steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
Comment on lines +17 to +19

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cat -n .github/workflows/agent.yml | head -150

Repository: stackrox/harness-openshell

Length of output: 5821


Pin all uses: actions to full commit SHAs.

These steps currently reference mutable tags (@v4, @v2, @v5). This leaves the workflow open to supply-chain drift in privileged CI execution.

Suggested hardening
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@<FULL_LENGTH_COMMIT_SHA>

-      - uses: google-github-actions/auth@v2
+      - uses: google-github-actions/auth@<FULL_LENGTH_COMMIT_SHA>

-      - uses: actions/setup-go@v5
+      - uses: actions/setup-go@<FULL_LENGTH_COMMIT_SHA>

-      - uses: actions/upload-artifact@v4
+      - uses: actions/upload-artifact@<FULL_LENGTH_COMMIT_SHA>

Also applies to: 37, 42, 139

🧰 Tools
🪛 zizmor (1.25.2)

[warning] 17-19: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false

(artipacked)


[error] 17-17: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 17 - 19, Replace all mutable
version tags in the uses statements throughout the workflow file with full
commit SHAs for security hardening. Specifically, update the actions/checkout@v4
reference on line 17 and any other actions referenced by version tags at lines
37, 42, and 139 to use their corresponding full commit SHA format instead (for
example, changing `@v4` to a full 40-character commit hash). This prevents
supply-chain drift attacks by ensuring the workflow always executes the exact
pinned version rather than a potentially updated mutable tag.

Source: Linters/SAST tools


- name: Check open agent PRs
id: guard
env:
GH_TOKEN: ${{ github.token }}
run: |
count=$(gh pr list --label agent --state open --json number --jq 'length')
echo "open_prs=$count"
if [ "$count" -ge 2 ]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "Skipping: $count agent PRs already open"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi

- name: Authenticate to GCP
if: steps.guard.outputs.skip != 'true'
run: |
echo '${{ secrets.GCP_ADC_JSON }}' > /tmp/adc.json
echo "GOOGLE_APPLICATION_CREDENTIALS=/tmp/adc.json" >> "$GITHUB_ENV"
# TODO: switch to Workload Identity Federation when ITPC registers stackrox org
# uses: google-github-actions/auth@v2
# with:
# workload_identity_provider: ${{ secrets.GCP_WORKLOAD_IDENTITY_PROVIDER }}
# service_account: ${{ secrets.GCP_SERVICE_ACCOUNT }}

- uses: actions/setup-go@v5
if: steps.guard.outputs.skip != 'true'
with:
go-version-file: go.mod

- name: Install openshell
if: steps.guard.outputs.skip != 'true'
run: curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
Comment on lines +51 to +53

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# Check if the file exists and read the relevant section
cat -n .github/workflows/agent.yml | head -60

Repository: stackrox/harness-openshell

Length of output: 2182


🏁 Script executed:

# Check the broader context of this workflow to understand when credentials are available
rg "REGISTRY_QUAY|secrets\." .github/workflows/agent.yml -A 2 -B 2

Repository: stackrox/harness-openshell

Length of output: 463


🏁 Script executed:

# Look for the job definition and understand the privilege level
rg "jobs:|permissions:" .github/workflows/agent.yml -A 5 -B 1

Repository: stackrox/harness-openshell

Length of output: 283


Pin and verify the openshell installer script to prevent CI compromise.

Line 49 executes a remote installer script from the main branch without verification. This workflow has elevated permissions (contents: write, id-token: write) and GCP credentials are authenticated at line 35 before this step runs, creating a direct CI compromise path if the upstream script is compromised.

Use a pinned commit hash instead of main, and add SHA256 verification:

Suggested fix
-      - name: Install openshell
+      - name: Install openshell (pinned + verified)
         if: steps.guard.outputs.skip != 'true'
-        run: curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
+        run: |
+          curl -fsSL -o /tmp/openshell-install.sh \
+            https://raw.githubusercontent.com/NVIDIA/OpenShell/<PINNED_COMMIT>/install.sh
+          echo "<EXPECTED_SHA256>  /tmp/openshell-install.sh" | sha256sum -c -
+          sh /tmp/openshell-install.sh
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- name: Install openshell
if: steps.guard.outputs.skip != 'true'
run: curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
- name: Install openshell (pinned + verified)
if: steps.guard.outputs.skip != 'true'
run: |
curl -fsSL -o /tmp/openshell-install.sh \
https://raw.githubusercontent.com/NVIDIA/OpenShell/<PINNED_COMMIT>/install.sh
echo "<EXPECTED_SHA256> /tmp/openshell-install.sh" | sha256sum -c -
sh /tmp/openshell-install.sh
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 47 - 49, The openshell installer
script in the "Install openshell" step is being fetched from the unpinned `main`
branch without verification, creating a CI compromise risk given the elevated
permissions (contents: write, id-token: write) and GCP authentication already
configured in the workflow. Replace the `main` branch reference in the curl
command with a pinned commit hash instead, and add SHA256 verification of the
downloaded script before executing it to ensure the script matches an expected
hash and prevent tampering.


- name: Wait for gateway
if: steps.guard.outputs.skip != 'true'
run: |
for i in $(seq 1 30); do
openshell inference get &>/dev/null && break
sleep 1
done
Comment on lines +55 to +61

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fail fast when gateway readiness never succeeds.

The loop at Line 54-57 exits after retries even if readiness never succeeds, and the step still returns success. That hides startup failures and shifts them downstream.

Suggested fix
       - name: Wait for gateway
         if: steps.guard.outputs.skip != 'true'
         run: |
+          ready=false
           for i in $(seq 1 30); do
-            openshell inference get &>/dev/null && break
+            if openshell inference get &>/dev/null; then
+              ready=true
+              break
+            fi
             sleep 1
           done
+          if [ "$ready" != "true" ]; then
+            echo "Gateway did not become ready within 30s"
+            exit 1
+          fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 51 - 57, The loop in the "Wait for
gateway" step completes all 30 iterations even when the openshell inference get
command never succeeds, causing the step to exit with success regardless of
actual gateway readiness. Add a verification check after the loop that runs the
openshell inference get command one more time, and if it fails (returns
non-zero), explicitly exit the step with a non-zero status code to fail the
workflow and prevent hiding startup failures.


- name: Build harness
if: steps.guard.outputs.skip != 'true'
run: CGO_ENABLED=0 go build -ldflags '-s -w -X main.version=ci' -o harness .

- name: Run agent
if: steps.guard.outputs.skip != 'true'
id: agent
env:
ANTHROPIC_VERTEX_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
CLOUD_ML_REGION: us-east5
run: |
SANDBOX_NAME="agent-improve-$(date +%s)"
HARNESS_OS_IMAGE=profiles/images/sandbox-default \
./harness apply -f .github/agent.yaml \
--name "$SANDBOX_NAME" \
--task @.github/agent-task.md

# Extract the diff from the sandbox
openshell sandbox exec --name "$SANDBOX_NAME" -- \
git -C /sandbox/harness-openshell diff > /tmp/agent.patch 2>/dev/null || true

# Extract the summary (last non-empty line of sandbox output)
openshell sandbox exec --name "$SANDBOX_NAME" -- \
git -C /sandbox/harness-openshell log --oneline -1 2>/dev/null \
| tail -1 > /tmp/agent-summary.txt || true
Comment on lines +73 to +87

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Summary extraction uses commit history, not the agent’s reported summary.

Line 81-83 pulls git log --oneline -1, which is just the repo’s latest commit message, not the summary requested in .github/agent-task.md. This can produce incorrect PR titles/body and commit messages.

Suggested fix
       - name: Run agent
         if: steps.guard.outputs.skip != 'true'
         id: agent
         env:
           ANTHROPIC_VERTEX_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
           CLOUD_ML_REGION: us-east5
         run: |
           SANDBOX_NAME="agent-improve-$(date +%s)"
           HARNESS_OS_IMAGE=profiles/images/sandbox-default \
             ./harness apply -f .github/agent.yaml \
               --name "$SANDBOX_NAME" \
-              --task @.github/agent-task.md
+              --task @.github/agent-task.md 2>&1 | tee /tmp/agent-run.log
@@
-          # Extract the summary (last non-empty line of sandbox output)
-          openshell sandbox exec --name "$SANDBOX_NAME" -- \
-            git -C /sandbox/harness-openshell log --oneline -1 2>/dev/null \
-            | tail -1 > /tmp/agent-summary.txt || true
+          # Extract the summary (last non-empty line of harness output)
+          awk 'NF { last=$0 } END { print last }' /tmp/agent-run.log > /tmp/agent-summary.txt || true
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 69 - 83, The summary extraction on
lines 81-83 incorrectly retrieves the repository's latest commit message using
git log instead of capturing the actual summary reported by the agent during
task execution. Replace the git log command that runs in the sandbox exec with a
method to capture the agent's actual summary output from the harness execution,
ensuring that the extracted summary reflects what the agent reported rather than
the repository's commit history.


openshell sandbox delete "$SANDBOX_NAME" || true

Comment on lines +74 to +90

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Guarantee sandbox deletion even on step failure.

With the default -e behavior, a failure before Line 85 skips deletion and can leave orphaned sandboxes/resources.

Suggested fix
         run: |
           SANDBOX_NAME="agent-improve-$(date +%s)"
+          cleanup() { openshell sandbox delete "$SANDBOX_NAME" >/dev/null 2>&1 || true; }
+          trap cleanup EXIT
@@
-          openshell sandbox delete "$SANDBOX_NAME" || true
+          trap - EXIT
+          cleanup
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
SANDBOX_NAME="agent-improve-$(date +%s)"
HARNESS_OS_IMAGE=profiles/images/sandbox-default \
./harness apply -f .github/agent.yaml \
--name "$SANDBOX_NAME" \
--task @.github/agent-task.md
# Extract the diff from the sandbox
openshell sandbox exec --name "$SANDBOX_NAME" -- \
git -C /sandbox/harness-openshell diff > /tmp/agent.patch 2>/dev/null || true
# Extract the summary (last non-empty line of sandbox output)
openshell sandbox exec --name "$SANDBOX_NAME" -- \
git -C /sandbox/harness-openshell log --oneline -1 2>/dev/null \
| tail -1 > /tmp/agent-summary.txt || true
openshell sandbox delete "$SANDBOX_NAME" || true
SANDBOX_NAME="agent-improve-$(date +%s)"
cleanup() { openshell sandbox delete "$SANDBOX_NAME" >/dev/null 2>&1 || true; }
trap cleanup EXIT
HARNESS_OS_IMAGE=profiles/images/sandbox-default \
./harness apply -f .github/agent.yaml \
--name "$SANDBOX_NAME" \
--task @.github/agent-task.md
# Extract the diff from the sandbox
openshell sandbox exec --name "$SANDBOX_NAME" -- \
git -C /sandbox/harness-openshell diff > /tmp/agent.patch 2>/dev/null || true
# Extract the summary (last non-empty line of sandbox output)
openshell sandbox exec --name "$SANDBOX_NAME" -- \
git -C /sandbox/harness-openshell log --oneline -1 2>/dev/null \
| tail -1 > /tmp/agent-summary.txt || true
trap - EXIT
cleanup
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/agent.yml around lines 70 - 86, The sandbox deletion
command using openshell sandbox delete for the "$SANDBOX_NAME" variable will not
execute if any preceding command fails due to bash's default error handling. To
guarantee cleanup even on step failure, use a bash trap to ensure the deletion
always runs before the script exits. Set up a trap function that calls openshell
sandbox delete "$SANDBOX_NAME" with error suppression, and register it at the
beginning of this script section to execute on EXIT or ERR to handle both
success and failure scenarios.

if [ -s /tmp/agent.patch ]; then
echo "has_diff=true" >> "$GITHUB_OUTPUT"
else
echo "has_diff=false" >> "$GITHUB_OUTPUT"
echo "Agent produced no diff"
fi

- name: Create PR
if: steps.guard.outputs.skip != 'true' && steps.agent.outputs.has_diff == 'true'
env:
GH_TOKEN: ${{ github.token }}
run: |
BRANCH="agent/improve-$(date +%Y%m%d-%H%M)"
SUMMARY=$(cat /tmp/agent-summary.txt 2>/dev/null || echo "Agent improvement")

git checkout -b "$BRANCH"
git apply /tmp/agent.patch
git add -A

git config user.name "harness-agent[bot]"
git config user.email "harness-agent[bot]@users.noreply.github.com"
git commit -m "fix: $SUMMARY"

git push origin "$BRANCH"

gh pr create \
--title "fix: $SUMMARY" \
--label agent \
--body "$(cat <<EOF
## Agent Improvement

This PR was created automatically by the harness self-improvement agent.

**Task:** Review the codebase and identify one concrete improvement.

**Change:** $SUMMARY

**Diff produced by:** \`harness apply -f .github/agent.yaml --task @.github/agent-task.md\`

---
Review and merge if the change is correct. Close if not useful.
EOF
)"

- name: Export logs
if: always() && steps.guard.outputs.skip != 'true'
run: |
mkdir -p /tmp/agent-logs
cp /tmp/agent.patch /tmp/agent-logs/ 2>/dev/null || true
cp /tmp/agent-summary.txt /tmp/agent-logs/ 2>/dev/null || true
journalctl --user -u openshell-gateway --no-pager > /tmp/agent-logs/gateway.log 2>/dev/null || true

- uses: actions/upload-artifact@v4
if: always() && steps.guard.outputs.skip != 'true'
with:
name: agent-logs
path: /tmp/agent-logs/
Loading