Skip to content

ops: diagnose and bound organization GitHub Actions queue starvation #712

Description

@seonghobae

Operational gap

Required current-head checks across ContextualWisdomLab/fast-mlsirm and the organization .github repository are remaining in queued state while obsolete heads accumulate. This blocks policy-compliant auto-merge even after CodeRabbit succeeds and review threads are resolved.

Representative evidence observed on 2026-08-03:

  • fast-mlsirm#410 current-head CI/Security/SAST runs queued after prior head jobs were cancelled;
  • fast-mlsirm#391, #396, and #401 exact-head CI/Security/SAST runs queued;
  • .github#704 exact-head CodeQL, Security, SAST, OSV, Scorecard, SBOM, Secret Scan, and Python Security runs queued;
  • .github#650, which repairs Rust coverage provisioning for fast-mlsirm, is itself waiting on queued required checks.

The repositories must not solve this by bypassing required checks or force-merging stale evidence.

Required diagnosis

  1. Determine whether the queue is caused by hosted-runner concurrency, Actions billing/usage limits, organization policy, a reusable-workflow recursion, pending deployment/environment approval, or workflow-level concurrency keys.
  2. Produce a machine-readable health snapshot containing repository, workflow, run id, event, head SHA, queue age, concurrency group, and whether the run is obsolete.
  3. Distinguish a genuinely queued current-head run from a completed/cancelled job whose run-level status is stale or misleading.
  4. Identify and cancel only obsolete runs; never cancel the sole current-head required evidence.
  5. Add organization-level concurrency and close-event cancellation contracts where repository-local controls cannot operate before merge.
  6. Alert on current-head required checks that remain queued beyond a declared SLO.
  7. Preserve fail-closed merge policy: an unavailable runner is incomplete evidence, not success.

Acceptance

  • A deterministic script and scheduled workflow report queue health across allowlisted CWL repositories.
  • At most one pending/running instance exists per PR-number/workflow evidence lane, except explicitly documented matrix jobs.
  • Closed PRs and superseded heads do not retain expensive active jobs.
  • A current-head run can be traced from PR to workflow/job state without relying on contradictory run-level summaries.
  • The report identifies the external/manual action when billing, organization policy, or runner capacity cannot be fixed in code.
  • Tests, documentation, least-privilege permissions, and 100% coverage/docstrings are included.

Related work

  • fast-mlsirm#410: repository-local current-head concurrency and bounded review churn.
  • .github#704: hourly review-feedback repair scheduler.
  • .github#650: Rust coverage toolchain/offline dependency provisioning.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions