Skip to content

Split CGO unit tests into parallel shards #48275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pelikhan merged 3 commits into from
Jul 27, 2026
+42 −24
Merged

Split CGO unit tests into parallel shards #48275

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6f4bdd3
Shard CGO unit tests across jobs
Copilot Jul 27, 2026
1c343d5
Apply remaining changes
Copilot Jul 27, 2026
ea4dc82
Fix CGO shard edge cases
Copilot Jul 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/skills/agentic-workflows/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ Load these files from `github/gh-aw` (they are not available locally).
- `.github/aw/github-agentic-workflows.md`
- `.github/aw/github-mcp-server.md`
- `.github/aw/instructions.md`
- `.github/aw/linter-workflows.md`
- `.github/aw/llms.md`
- `.github/aw/loop.md`
- `.github/aw/lsp.md`
Expand Down
57 changes: 37 additions & 20 deletions .github/workflows/cgo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,12 +40,29 @@ on:
workflow_dispatch:
jobs:
test:
name: Unit tests (${{ matrix.name }})
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
strategy:
fail-fast: false
matrix:
include:
- name: A-C
pattern: ^Test[A-C]

@github-actions github-actions Bot Jul 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] The shard patterns ^Test[A-C], ^Test[D-L], ^Test[M-R], ^Test[S-Z] silently skip any test whose name does not start with an ASCII letter A–Z after Test — e.g. Test_Foo, Test0Bar are valid Go test names and will be dropped by every shard without a CI failure.

💡 Suggested fix

Add a catch-all shard or widen the last shard pattern:

- name: Other
  pattern: ^Test[^A-La-lM-Rm-rS-Zs-z]
  shard: other

Or change the final shard entry to act as a catch-all:

- name: S-Z (and other)
  pattern: ^Test([S-Z]|[^A-Ra-r])
  shard: s-z

Simplest option: drop the anchored character classes and use regex ranges that together cover ., adding Test[^A-Ra-r] as the last shard to capture digits, underscores, and everything outside A–R.

@copilot please address this.

Copilot AI Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ea4dc82. The final shard is now a catch-all with ^Test([S-Z]|[^A-R]), so valid non-alphabetic test names are still exercised by one shard instead of being dropped.

shard: a-c
- name: D-L
pattern: ^Test[D-L]
shard: d-l
- name: M-R
pattern: ^Test[M-R]
shard: m-r
- name: S-Z and other
pattern: ^Test([S-Z]|[^A-R])
shard: s-z
concurrency:
group: ci-${{ github.ref }}-test
group: ci-${{ github.ref }}-test-${{ matrix.shard }}
cancel-in-progress: true
steps:
- name: Checkout code
Expand Down Expand Up @@ -152,26 +169,16 @@ jobs:
run: |
set -o pipefail
# Run tests with JSON output for artifacts, but also show failures
go test -v -parallel=8 -timeout=3m -run='^Test' -tags '!integration' -coverprofile=coverage.out -json ./... | tee test-result-unit.json
go test -v -parallel=8 -timeout=3m -run='${{ matrix.pattern }}' -tags '!integration' -coverprofile=coverage-${{ matrix.shard }}.out -json ./... | tee test-result-unit-${{ matrix.shard }}.json

# Check if tests failed by looking at JSON output
if grep -q '"Action":"fail"' test-result-unit.json; then
if grep -q '"Action":"fail"' test-result-unit-${{ matrix.shard }}.json; then
echo "❌ Tests failed - see output above"
exit 1
fi

# Generate coverage HTML report
go tool cover -html=coverage.out -o coverage.html

- name: Validate pkg/cli and pkg/console tests executed
run: |
set -euo pipefail
for package in pkg/cli pkg/console; do
if ! grep -q "\"Action\":\"run\",\"Package\":\"github.com/github/gh-aw/${package}\"" test-result-unit.json; then
echo "Missing unit test execution records for ${package}" >&2
exit 1
fi
done
go tool cover -html=coverage-${{ matrix.shard }}.out -o coverage-${{ matrix.shard }}.html

- name: Report test failures
if: failure() && steps.run-unit-tests.outcome == 'failure'
Expand All @@ -182,7 +189,7 @@ jobs:
echo "" >> $GITHUB_STEP_SUMMARY

# Run the failure report script
if ./scripts/report-test-failures.sh test-result-unit.json | tee /tmp/failure-report.txt; then
if ./scripts/report-test-failures.sh test-result-unit-${{ matrix.shard }}.json | tee /tmp/failure-report.txt; then
echo "No failures detected in JSON output (unexpected - tests failed but no failure records found)" >> $GITHUB_STEP_SUMMARY
else
# Script found failures - add to summary
Expand All @@ -195,16 +202,16 @@ jobs:
- name: Upload coverage report
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: coverage-report
path: coverage.html
name: coverage-report-${{ matrix.shard }}
path: coverage-${{ matrix.shard }}.html
retention-days: 7

- name: Upload unit test results
if: always() # Upload even if tests fail so canary-go can track coverage
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: test-result-cgo-unit
path: test-result-unit.json
name: test-result-cgo-unit-${{ matrix.shard }}
path: test-result-unit-${{ matrix.shard }}.json
retention-days: 7

canary-go:
Expand All @@ -230,7 +237,7 @@ jobs:
uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4
with:
path: test-results
pattern: test-result-cgo-unit
pattern: test-result-cgo-unit-*
merge-multiple: false

- name: List downloaded artifacts
Expand All @@ -241,6 +248,16 @@ jobs:
echo ""
echo "Total JSON files: $(find test-results -type f -name \"*.json\" | wc -l)"

- name: Validate pkg/cli and pkg/console tests executed
run: |
set -euo pipefail
for package in pkg/cli pkg/console; do
if ! grep -R -q "\"Action\":\"run\",\"Package\":\"github.com/github/gh-aw/${package}\"" test-results; then
echo "Missing unit test execution records for ${package}" >&2
exit 1
fi
done

- name: Extract executed tests from artifacts
run: |
set -euo pipefail
Expand Down
7 changes: 3 additions & 4 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ CI_COVERAGE_ENABLED ?= 1
CI_COVERAGE_SOURCE_BRANCH ?= main
CI_RUN_ID ?=
CI_UNIT_WORKFLOW_FILE ?= cgo.yml
CI_UNIT_TEST_ARTIFACT_PATTERN ?= test-result-cgo-unit
CI_UNIT_TEST_ARTIFACT_PATTERN ?= test-result-cgo-unit-*
CI_UNIT_RUN_ID ?=
GO_IMPACTED_TEST_MAX_SECONDS ?= 60
GO_IMPACTED_TEST_PATTERN_MAX_CHARS ?= 8000
Expand Down Expand Up @@ -360,12 +360,11 @@ test-impacted-go:
rm -rf "$$UNIT_RESULT_DIR"; \
mkdir -p "$$UNIT_RESULT_DIR"; \
if gh run download "$$UNIT_RUN_ID" --pattern "$(CI_UNIT_TEST_ARTIFACT_PATTERN)" --dir "$$UNIT_RESULT_DIR" >/dev/null 2>&1; then \
UNIT_RESULT_FILE=$$(find "$$UNIT_RESULT_DIR" -type f -name '*.json' | head -n 1); \
if [ -n "$$UNIT_RESULT_FILE" ]; then \
if find "$$UNIT_RESULT_DIR" -type f -name '*.json' -print -quit | grep -q .; then \
IMPACTED_PACKAGE_FILE="$(CI_COVERAGE_DIR)/impacted-go-packages.txt"; \
printf '%s\n' "$$CHANGED_GO_PACKAGES" | sed 's|^\./|github.com/github/gh-aw/|' > "$$IMPACTED_PACKAGE_FILE"; \
IMPACTED_TEST_CANDIDATES="$(CI_COVERAGE_DIR)/impacted-go-test-candidates.tsv"; \
jq -r 'select(.Action == "pass" and .Package != null and .Test != null and (.Test | contains("/") | not) and .Elapsed != null) | [.Package, .Test, (.Elapsed | tostring)] | @tsv' "$$UNIT_RESULT_FILE" \
find "$$UNIT_RESULT_DIR" -type f -name '*.json' -exec jq -r 'select(.Action == "pass" and .Package != null and .Test != null and (.Test | contains("/") | not) and .Elapsed != null) | [.Package, .Test, (.Elapsed | tostring)] | @tsv' {} + \
| awk 'NR==FNR { pkgs[$$1] = 1; next } $$1 in pkgs { print }' "$$IMPACTED_PACKAGE_FILE" - \
| sort -u > "$$IMPACTED_TEST_CANDIDATES"; \
if [ -s "$$IMPACTED_TEST_CANDIDATES" ]; then \
Expand Down
1 change: 1 addition & 0 deletions pkg/cli/data/agentic_workflows_fallback_aw_files.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
"github-agentic-workflows.md",
"github-mcp-server.md",
"instructions.md",
"linter-workflows.md",
"llms.md",
"loop.md",
"lsp.md",
Expand Down
Loading