Skip to content

test: harden reliable test execution #699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
BigSimmo wants to merge 3 commits into from
Closed

test: harden reliable test execution #699

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b518c1d
test: harden reliable test execution
BigSimmo Jul 17, 2026
3091670
merge: resolve main conflicts for test reliability branch
Copilot Jul 17, 2026
6202835
merge: resolve main conflicts for test reliability branch
Copilot Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .github/workflows/ci-triage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,11 @@
# - main-side: the same job is also failing on the latest default-branch run
# (CI merges the PR branch with current main, so a main regression surfaces on
# every open PR — this has cost debugging time before).
# - possible known flake: a UI/Playwright job failed — check tests/flake-ledger.json.
# - needs investigation: everything else.
#
# SHIPPED INERT: this event-triggered workflow does nothing until the repo variable
# CI_TRIAGE_ENABLED == "true". It never runs PR-authored code — it only reads job
# metadata + the committed flake ledger via the trusted default-branch checkout, and
# metadata via the trusted default-branch checkout, and
# posts a comment with the built-in token.
name: CI Triage

Expand Down Expand Up @@ -45,7 +44,6 @@ jobs:
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const fs = require("fs");
const run = context.payload.workflow_run;

// Resolve the PR for this run (empty for fork PRs → skip quietly).
Expand Down Expand Up @@ -93,13 +91,11 @@ jobs:
core.info(`main-side check skipped: ${e.message}`);
}

const flakes = JSON.parse(fs.readFileSync("tests/flake-ledger.json", "utf8")).flakes || [];
const uiFlakeSpecs = flakes.map((f) => f.spec).filter((s) => /ui-/.test(s));
const looksUi = (name) => /ui|playwright|browser|e2e/i.test(name);

const lines = failed.map((name) => {
if (mainFailingJobs.has(name)) return `- \`${name}\` — **main-side**: also failing on the latest \`main\` run, likely not your change.`;
if (looksUi(name)) return `- \`${name}\` — **possible known flake**: UI/Playwright job; check \`tests/flake-ledger.json\`${uiFlakeSpecs.length ? ` (${uiFlakeSpecs.join(", ")})` : ""} and re-run before bisecting.`;
if (looksUi(name)) return `- \`${name}\` — **needs investigation**: use the uploaded JUnit classification and trace; job type alone is not evidence of a known flake.`;
return `- \`${name}\` — **needs investigation**.`;
});

Expand Down
Loading