Skip to content

fix(dev-lead): add concurrency queue to prevent batch API quota exhaustion#376

Merged
don-petry merged 4 commits into
mainfrom
dev-lead/issue-315-20260609-2205
Jun 16, 2026
Merged

fix(dev-lead): add concurrency queue to prevent batch API quota exhaustion#376
don-petry merged 4 commits into
mainfrom
dev-lead/issue-315-20260609-2205

Conversation

@don-petry

@don-petry don-petry commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Closes #315

Root Cause

All dev-lead failures in the monitored window (45.8%, 11/24 runs) were caused by Fleet Monitor batch-labeling multiple issues simultaneously, triggering concurrent runs that exhausted shared API quotas:

  • 2026-06-08T20:12–20:13 (5 failures): 6 concurrent runs hit GitHub secondary rate limit; the error JSON from the rate-limited gh api call ended up as the value of a bash integer variable, causing a comparison failure.
  • 2026-06-09T05:30–05:35 (6 failures): 6 concurrent runs exhausted both Claude daily quota and Gemini prepayment credits simultaneously; the openai/o4-mini fallback was also unavailable.

The statuses: read fix (PR #370) was already merged and did not prevent these failures.

Fix

  • dev-lead.yml: Add concurrency: { group: dev-lead, cancel-in-progress: false }. All dev-lead runs queue globally rather than running in parallel. Concurrent batch-label events now run sequentially and succeed rather than racing for API quota and failing.
  • scripts/apply-repo-settings.sh and scripts/setup-code-quality-ruleset.sh: Fix pre-existing SC2043 shellcheck warnings (single-item for loop → direct command -v check) so dev-lead-lint.sh passes.

Test Results

  • bash .dev-lead/scripts/dev-lead-lint.sh: all checks passed
  • npm run check: 0 errors (22 pre-existing warnings, unchanged)
  • npm test: 301/301 passed

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Updated GitHub Actions workflow concurrency configuration for improved job queuing.
    • Enhanced error handling in setup scripts to validate required tools availability.

…stion

All dev-lead workflow failures in the monitored window (45.8% failure
rate) were caused by Fleet Monitor batch-labeling multiple issues
simultaneously, triggering concurrent runs that exhausted Claude and
GitHub API quotas. Serialising all runs with a global concurrency group
prevents this without dropping any work.

Also fixes pre-existing SC2043 shellcheck warnings in scripts/ that
caused dev-lead-lint.sh to exit non-zero.

Closes #315

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@don-petry don-petry requested a review from a team as a code owner June 9, 2026 22:13
Copilot AI review requested due to automatic review settings June 9, 2026 22:13
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Adds a concurrency block to .github/workflows/dev-lead.yml using a fixed group dev-lead with cancel-in-progress: false. Simplifies the gh CLI preflight dependency check in scripts/apply-repo-settings.sh and scripts/setup-code-quality-ruleset.sh from a generic loop to a direct command -v gh call with a specific error message.

Changes

Workflow Concurrency and Script Preflight Fixes

Layer / File(s) Summary
Dev-lead workflow concurrency config
.github/workflows/dev-lead.yml
Adds a concurrency block with group dev-lead and cancel-in-progress: false, preventing overlapping runs from being cancelled.
Simplified gh CLI preflight checks
scripts/apply-repo-settings.sh, scripts/setup-code-quality-ruleset.sh
Replaces the for cmd in gh; do ... done loop pattern with a direct command -v gh check and a specific "Error: gh is required" stderr message in both scripts.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related issues

  • petry-projects — workflow failures detected 2026-06-15 .github-private#759 — The PR modifies dev-lead.yml and apply-repo-settings.sh, which are identified as failing in the workflow failure incident report; the concurrency and preflight fixes appear to address those failures.
  • petry-projects — workflow failures detected 2026-06-14 .github-private#693 — The PR targets dev-lead.yml (listed as a failing workflow) and improves preflight checks in scripts referenced by the apply-repo-settings workflow, aligning with the systemic workflow failures documented in that issue.
  • #315 — The dev-lead.yml concurrency change directly addresses the high cancellation rate (15 cancelled runs out of 42) and elevated p95 duration flagged in this fleet monitor warning.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Title clearly describes the main change: adding concurrency queue to the dev-lead workflow to prevent API quota exhaustion, which is the primary objective of the PR.
Linked Issues check ✅ Passed The PR directly addresses issue #315 by implementing the concurrency queue mechanism to prevent the workflow failures (11.9% failure rate) caused by concurrent API quota exhaustion documented in the issue.
Out of Scope Changes check ✅ Passed All changes are in-scope: concurrency config targets the workflow issue, and shellcheck warning fixes in scripts are pre-existing issues resolved as part of the same fix cycle.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev-lead/issue-315-20260609-2205

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

@don-petry don-petry enabled auto-merge (squash) June 9, 2026 22:14

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request simplifies the preflight checks in scripts/apply-repo-settings.sh and scripts/setup-code-quality-ruleset.sh by replacing the single-item loop for gh with a direct command check. The review feedback suggests improving error handling by redirecting the error messages to standard error (stderr) instead of standard output (stdout) in both scripts.

Comment thread scripts/apply-repo-settings.sh Outdated
Comment thread scripts/setup-code-quality-ruleset.sh Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR reduces dev-lead workflow failures caused by multiple concurrent runs exhausting shared external API quotas, by introducing a global concurrency queue for the workflow and cleaning up related lint warnings in supporting scripts.

Changes:

  • Add workflow-level concurrency to .github/workflows/dev-lead.yml to queue all dev-lead runs (no cancellation).
  • Fix SC2043 ShellCheck warnings by removing single-item for loops in two repo admin scripts.
  • Keep existing permissions and reusable-workflow wiring intact.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
.github/workflows/dev-lead.yml Queues dev-lead workflow runs to prevent concurrent executions from exhausting shared API quotas.
scripts/apply-repo-settings.sh Removes a single-item for loop in preflight to satisfy ShellCheck (SC2043).
scripts/setup-code-quality-ruleset.sh Removes a single-item for loop in preflight to satisfy ShellCheck (SC2043).

@don-petry don-petry disabled auto-merge June 9, 2026 22:15
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

CI Failure: SonarCloud Code Analysis

Step: SonarCloud Quality Gate
Root cause: Lint/style

SonarCloud scanned the changes in this PR and the quality gate did not pass. The diff modifies two shell scripts (apply-repo-settings.sh and setup-code-quality-ruleset.sh) by replacing a for cmd in gh; do preflight loop with a direct command -v gh check, and adds a concurrency block to the dev-lead workflow. SonarCloud likely flagged code smells or rule violations introduced in the shell script changes — common triggers include unquoted variables, missing set -e/set -u guards, or stylistic issues that push the project below its configured quality gate threshold.

Suggested fix: Open the SonarCloud dashboard for this PR (linked from the check details), review the "New Issues" tab to see the exact rules violated in the changed lines, and address each flagged finding in apply-repo-settings.sh or setup-code-quality-ruleset.sh before re-pushing.

View run logs

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

@don-petry don-petry enabled auto-merge (squash) June 9, 2026 22:16
@don-petry don-petry disabled auto-merge June 9, 2026 22:17
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

@don-petry don-petry enabled auto-merge (squash) June 9, 2026 22:18
@github-actions

Copy link
Copy Markdown
Contributor

Auto-rebase blocked — the base branch contains .github/workflows/ changes that require the workflows permission to merge into this branch, but the auto-rebase workflow's token does not have that permission.

Please rebase this branch manually:

git fetch origin
git rebase origin/main
git push --force-with-lease

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 13b17c3c5715c29c9cc062441852ec69b40c2c67
Review mode: triage-approved (single reviewer)

Summary

Serializes dev-lead workflow runs via a global concurrency group to stop concurrent batch-label runs from exhausting shared API quotas (issue #315), plus two SC2043 shellcheck lint fixes. Small, config/lint-only, CI green.

Linked issue analysis

Closes #315. The root cause — concurrent dev-lead runs racing for GitHub/Claude/Gemini API quota — is directly addressed by 'concurrency: { group: dev-lead, cancel-in-progress: false }', which queues all runs globally without dropping work. This is the correct mechanism for the failure mode documented in the PR body.

Findings

  • .github/workflows/dev-lead.yml: adds a static concurrency group; permissions: {} preserved, reusable-workflow wiring unchanged. No security smell, no injection vector. Sound fix.
  • scripts/apply-repo-settings.sh / setup-code-quality-ruleset.sh: replace single-item 'for' loop with a direct 'command -v gh' check (fixes SC2043) and redirect the missing-gh error to stderr. Behavior-preserving.
  • Nit (non-blocking): the adjacent 'gh auth status' error echo still writes to stdout, so the gemini stderr suggestion is only partially applied. Cosmetic, out of this fix's scope.

CI status

All green on 13b17c3: CodeQL (actions/javascript-typescript/python) SUCCESS, CodeRabbit SUCCESS, SonarCloud Quality Gate passed (0 new issues). Note: mergeStateStatus is BEHIND (branch needs a rebase before merge) — a merge-readiness item, not a review blocker.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

@sonarqubecloud

Copy link
Copy Markdown

@don-petry don-petry merged commit df9b87c into main Jun 16, 2026
23 of 26 checks passed
@don-petry don-petry deleted the dev-lead/issue-315-20260609-2205 branch June 16, 2026 23:55
@github-actions

Copy link
Copy Markdown
Contributor

CI Failure: SonarCloud Code Analysis

Step: SonarCloud Quality Gate
Root cause: Config error

SonarCloud is an external quality gate that analyzes code against configured rules and thresholds. The check failed on the main branch, which typically means the PR introduces code that violates quality gate conditions such as insufficient test coverage, new code smells, security hotspots, or reliability issues. The diff includes shell script changes and a workflow concurrency config addition, any of which could trigger SonarCloud rules (e.g. shellcheck-style issues or new uncovered logic).

Suggested fix: Review the SonarCloud dashboard at https://sonarcloud.io/dashboard?id=petry-projects_google-app-scripts&branch=main to identify the specific failing condition (coverage drop, new bug/smell/hotspot), then address the flagged issue in the PR branch before re-running the check.

View run logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fleet Monitor] petry-projects/google-app-scripts — dev-lead.yml

3 participants