Skip to content

[Fleet incident] Repair hourly development credential and exact-check gates #73

Description

@seonghobae

Fleet automation incident

Live protected main currently contains three independent Hourly Product Development defects that should be repaired together as one bounded automation slice, without inventing credentials or weakening fail-closed behavior.

1. Deterministic gates are incorrectly behind the NVIDIA credential

.github/workflows/hourly-product-development.yml checks NVIDIA_NIM_API_KEY before it inventories open pull requests or validates protected-main evidence. An hour with an open PR should terminate for open_pull_request even when no model credential exists; dry-run and deterministic evidence inspection also must not require a model key.

Required ordering:

open-PR / release-blocker / dry-run / protected-main deterministic evidence -> NVIDIA credential -> actual model-backed path

Keep NVIDIA_NIM_API_KEY only at the NVIDIA inference boundary.

2. Required-check evaluation currently accepts neutral/skipped as healthy

The protected-main check evaluator uses an accepted set equivalent to success, neutral, and skipped. Fleet policy requires required evidence to be completed successfully; neutral, skipped, pending, cancelled, absent, or stale-head evidence must not be promoted to success. Preserve explicitly optional checks only through an explicit policy contract rather than a blanket accepted-conclusion set.

3. Harden Runner has the same GitHub API alias exposure seen in ThreadWeave

The fail-closed egress allowlist includes api.github.com:443 but not the observed GitHub alias cafe.github.com:443 that caused ThreadWeave's scheduled API gate to fail. Revalidate against a current hosted run before changing the allowlist; if the same alias is observed, add only the narrowly evidenced endpoint while retaining egress-policy: block.

Acceptance criteria

  • test-first contract proving deterministic stop gates execute and terminate before the optional NVIDIA credential check;
  • adversarial regression proving removing a stop-gate exit 0 fails the contract;
  • exact policy test rejecting neutral/skipped as required success unless an explicitly named optional-check policy says otherwise;
  • exact Harden Runner endpoint-set contract and real gh api command validation; no suffix/subdomain string tricks;
  • no COPILOT_GITHUB_TOKEN, guessed PAT, new secret alias, self-modifying repair workflow, or branch-protection/test weakening;
  • exact-head CI, security, and review gates;
  • normal protected-default-branch merge;
  • a subsequent real scheduled/manual protected-main Hourly Product Development run must pass beyond the former failure point and produce the correct credential-independent no-op reason when PRs or deterministic gates block development.

This issue is part of the organization-wide Hourly Product Development fleet incident. Re-fetch all heads/workflow state before implementation; do not reuse identities from this report.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions