Skip to content

ci: split test matrix into privileged and non-privileged groups #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
skevetter merged 2 commits into from
May 13, 2026
Merged

ci: split test matrix into privileged and non-privileged groups #272

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
808aebb
ci: split test matrix into privileged and non-privileged groups
skevetter May 13, 2026
1660ec6
ci: move ide and machineprovider back to privileged group
skevetter May 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 66 additions & 30 deletions .github/workflows/pr-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,38 +119,92 @@ jobs:
name: devsy-${{ steps.os.outputs.runner_os }}
path: dist/devsy-${{ steps.os.outputs.runner_os }}_*/devsy-${{ steps.os.outputs.runner_os }}-*

integration-tests:
integration-tests-unprivileged:
name: Test ${{ matrix.label }} on ${{ matrix.runner }}
needs: [can-read-secret, build-cli]
needs: [build-cli]
strategy:
fail-fast: false
matrix:
include:
- label: context
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: ide
- label: readconfiguration
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: integration
- label: extends
runner: ubuntu-latest

- label: outdated
runner: ubuntu-latest

- label: machine
runner: ubuntu-latest

runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v6

- name: setup Go
uses: actions/setup-go@v6
with:
go-version-file: go.mod

- name: get operating system lowercase
id: os
shell: bash
run: |
OS=$(echo "${{ runner.os }}" | tr '[:upper:]' '[:lower:]')
echo "runner_os=$OS" >> "$GITHUB_OUTPUT"

- name: download CLI artifacts
uses: actions/download-artifact@v8
with:
pattern: devsy-${{ steps.os.outputs.runner_os }}
path: ${{ runner.temp }}/devsy-bin/
merge-multiple: true

- name: setup executable
shell: bash
working-directory: ./e2e
run: |
TEMP_DIR="${{ runner.temp }}"
ls -R "$TEMP_DIR/devsy-bin/"
mkdir -p ./bin/
find "$TEMP_DIR/devsy-bin/" -type f -name "devsy-*" -exec cp {} ./bin \;
find ./bin -name "devsy-*" -exec chmod +x {} \;
ls -R ./bin/

- name: run test
shell: bash
working-directory: ./e2e
env:
GH_USERNAME: ${{ github.repository_owner }}
GH_ACCESS_TOKEN: ${{ github.token }}
GH_CREDENTIAL_USERNAME: x-access-token
run: |
go test -v -ginkgo.v -timeout 1500s --ginkgo.label-filter="${{ matrix.label }}"

Comment on lines +122 to +187

@coderabbitai coderabbitai Bot May 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Description: Check git log for evidence of test execution and results for moved labels

echo "=== Checking commit messages for test execution evidence ==="
git log --oneline --grep="test\|fail\|pass" -i --since="2 weeks ago" | head -20

echo -e "\n=== Checking for test result artifacts or CI logs mentioning these labels ==="
for label in context readconfiguration extends outdated machine; do
  echo "Searching for: $label"
  rg -i "test.*$label|$label.*test" --type=yaml --type=md -C 2 || echo "  No matches found"
done

echo -e "\n=== Checking PR description or comments for verification notes ==="
gh pr view 272 --json body,comments --jq '.body, .comments[].body' | rg -i "verify|test.*without.*sudo|unprivileged" -C 2 || echo "  No verification notes found"

Repository: devsy-org/devsy

Length of output: 1565

Request explicit confirmation that the five unprivileged tests pass without sudo.

The PR summary indicates these tests are based on DEVSY-105 investigation, which found ~60% of test labels have zero Docker dependency. The selective movement of ide and machineprovider back to the privileged job confirms that tests are being executed and failures are detected. However, there is no explicit documentation confirming that context, readconfiguration, extends, outdated, and machine have been verified to pass without sudo.

Please confirm in a comment that these five tests execute successfully in the unprivileged job, or link to DEVSY-105 for reference.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/pr-ci.yml around lines 122 - 187, Add an explicit
confirmation that the five unprivileged test labels (context, readconfiguration,
extends, outdated, machine) were run and passed without sudo by updating the PR
description or adding a follow-up PR comment referencing the
integration-tests-unprivileged job; state that you executed the matrix job
locally or on CI, list each label as "passed (no sudo)", and if you used
DEVSY-105 as the source of truth include a direct reference/link to DEVSY-105
and briefly note the verification method (CI run ID or local reproduction) so
reviewers can trace the claim.

integration-tests:
name: Test ${{ matrix.label }} on ${{ matrix.runner }}
needs: [can-read-secret, build-cli]
strategy:
fail-fast: false
matrix:
include:
- label: ide
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: machine
- label: machineprovider
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: machineprovider
- label: integration
runner: ubuntu-latest
free-disk-space: false
install-kind: false
Expand Down Expand Up @@ -186,30 +240,12 @@ jobs:
install-kind: false
requires-secret: false

- label: readconfiguration
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: extends
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: extends-up
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: outdated
runner: ubuntu-latest
free-disk-space: false
install-kind: false
requires-secret: false

- label: down
runner: ubuntu-latest
free-disk-space: false
Expand Down
Loading