Fleet automation incident
Live protected main currently contains three independent Hourly Product Development defects that should be repaired together as one bounded automation slice, without inventing credentials or weakening fail-closed behavior.
1. Deterministic gates are incorrectly behind the NVIDIA credential
.github/workflows/hourly-product-development.yml checks NVIDIA_NIM_API_KEY before it inventories open pull requests or validates protected-main evidence. An hour with an open PR should terminate for open_pull_request even when no model credential exists; dry-run and deterministic evidence inspection also must not require a model key.
Required ordering:
open-PR / release-blocker / dry-run / protected-main deterministic evidence -> NVIDIA credential -> actual model-backed path
Keep NVIDIA_NIM_API_KEY only at the NVIDIA inference boundary.
2. Required-check evaluation currently accepts neutral/skipped as healthy
The protected-main check evaluator uses an accepted set equivalent to success, neutral, and skipped. Fleet policy requires required evidence to be completed successfully; neutral, skipped, pending, cancelled, absent, or stale-head evidence must not be promoted to success. Preserve explicitly optional checks only through an explicit policy contract rather than a blanket accepted-conclusion set.
3. Harden Runner has the same GitHub API alias exposure seen in ThreadWeave
The fail-closed egress allowlist includes api.github.com:443 but not the observed GitHub alias cafe.github.com:443 that caused ThreadWeave's scheduled API gate to fail. Revalidate against a current hosted run before changing the allowlist; if the same alias is observed, add only the narrowly evidenced endpoint while retaining egress-policy: block.
Acceptance criteria
- test-first contract proving deterministic stop gates execute and terminate before the optional NVIDIA credential check;
- adversarial regression proving removing a stop-gate
exit 0 fails the contract;
- exact policy test rejecting
neutral/skipped as required success unless an explicitly named optional-check policy says otherwise;
- exact Harden Runner endpoint-set contract and real
gh api command validation; no suffix/subdomain string tricks;
- no
COPILOT_GITHUB_TOKEN, guessed PAT, new secret alias, self-modifying repair workflow, or branch-protection/test weakening;
- exact-head CI, security, and review gates;
- normal protected-default-branch merge;
- a subsequent real scheduled/manual protected-main Hourly Product Development run must pass beyond the former failure point and produce the correct credential-independent no-op reason when PRs or deterministic gates block development.
This issue is part of the organization-wide Hourly Product Development fleet incident. Re-fetch all heads/workflow state before implementation; do not reuse identities from this report.
Fleet automation incident
Live protected
maincurrently contains three independent Hourly Product Development defects that should be repaired together as one bounded automation slice, without inventing credentials or weakening fail-closed behavior.1. Deterministic gates are incorrectly behind the NVIDIA credential
.github/workflows/hourly-product-development.ymlchecksNVIDIA_NIM_API_KEYbefore it inventories open pull requests or validates protected-main evidence. An hour with an open PR should terminate foropen_pull_requesteven when no model credential exists; dry-run and deterministic evidence inspection also must not require a model key.Required ordering:
open-PR / release-blocker / dry-run / protected-main deterministic evidence -> NVIDIA credential -> actual model-backed pathKeep
NVIDIA_NIM_API_KEYonly at the NVIDIA inference boundary.2. Required-check evaluation currently accepts neutral/skipped as healthy
The protected-main check evaluator uses an accepted set equivalent to
success,neutral, andskipped. Fleet policy requires required evidence to be completed successfully;neutral,skipped, pending, cancelled, absent, or stale-head evidence must not be promoted to success. Preserve explicitly optional checks only through an explicit policy contract rather than a blanket accepted-conclusion set.3. Harden Runner has the same GitHub API alias exposure seen in ThreadWeave
The fail-closed egress allowlist includes
api.github.com:443but not the observed GitHub aliascafe.github.com:443that caused ThreadWeave's scheduled API gate to fail. Revalidate against a current hosted run before changing the allowlist; if the same alias is observed, add only the narrowly evidenced endpoint while retainingegress-policy: block.Acceptance criteria
exit 0fails the contract;neutral/skippedas required success unless an explicitly named optional-check policy says otherwise;gh apicommand validation; no suffix/subdomain string tricks;COPILOT_GITHUB_TOKEN, guessed PAT, new secret alias, self-modifying repair workflow, or branch-protection/test weakening;This issue is part of the organization-wide Hourly Product Development fleet incident. Re-fetch all heads/workflow state before implementation; do not reuse identities from this report.