fix(ci): harden hourly GitHub API gate - #74
Conversation
📝 WalkthroughWalkthrough시간별 제품 개발 워크플로에 Changes시간별 제품 개발 워크플로
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
@coderabbitai review Review exact current head |
|
@opencode-agent @cwl-noema-review Review-only request for exact current head |
🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
base='c8968ec1e68fab16d0ad8216fb5c8fd0b385e95f'
head='d1802174f73b5b2a3b7f8511e7816c8763d01ec0'
echo '== Commit identity =='
git show -s --format='BASE %H%n%P%n%s%n%ad' --date=iso-strict "$base"
git show -s --format='HEAD %H%n%P%n%s%n%ad' --date=iso-strict "$head"
echo
echo '== Base is ancestor of head =='
git merge-base --is-ancestor "$base" "$head" && echo yes
echo
echo '== Name-status diff =='
git diff --name-status "$base" "$head"
echo
echo '== Changed paths =='
git diff --name-only "$base" "$head"
echo
echo '== Outlines for changed workflow/test-like files =='
while IFS= read -r f; do
case "$f" in
*.yml|*.yaml|*.sh|*.bats|*.test.*|*test*|*spec*)
echo "--- $f ---"
if [[ "$f" == *.yml || "$f" == *.yaml ]]; then
ast-grep outline "$f" --items all 2>/dev/null || true
else
rg -n '(^[[:space:]]*(describe|it|test|function)[[:space:]]|\b(NVIDIA_NIM_API_KEY|cafe\.github\.com|api\.github\.com|egress-policy|exit[[:space:]]+[0-9]+|gh[[:space:]]+api)\b)' "$f" || true
fi
;;
esac
done < <(git diff --name-only "$base" "$head")Length of output: 2383 🏁 Script executed: #!/bin/bash
set -euo pipefail
base='c8968ec1e68fab16d0ad8216fb5c8fd0b385e95f'
head='d1802174f73b5b2a3b7f8511e7816c8763d01ec0'
workflow='.github/workflows/hourly-product-development.yml'
development_test='services/account_unification/tests/test_hourly_product_development.py'
incident_test='services/account_unification/tests/test_hourly_product_incident_contract.py'
echo '== Workflow outline at exact head =='
ast-grep outline "$workflow" --items all 2>/dev/null || true
echo '== Python test outlines at exact head =='
ast-grep outline "$development_test" --lang python --items all 2>/dev/null || true
ast-grep outline "$incident_test" --lang python --items all 2>/dev/null || true
echo '== Exact base-to-head diff =='
git diff --no-ext-diff --unified=80 "$base" "$head" -- \
"$workflow" "$development_test" "$incident_test"
echo '== Exact workflow at head, numbered =='
git show "$head:$workflow" | nl -ba
echo '== Incident contract test at head, numbered =='
git show "$head:$incident_test" | nl -ba
echo '== Development test at head, numbered =='
git show "$head:$development_test" | nl -baLength of output: 54048 🏁 Script executed: #!/bin/bash
set -euo pipefail
head='d1802174f73b5b2a3b7f8511e7816c8763d01ec0'
workflow='.github/workflows/hourly-product-development.yml'
echo '== GitHub API calls and enclosing workflow context at exact head =='
git show "$head:$workflow" > /tmp/hourly-product-development.yml
grep -n -E -C 12 '(^[[:space:]]+gh api|[[:space:]]gh api[[:space:]]|Harden runner|Harden independent verification|Harden publication|allowed-endpoints:|cafe\.github\.com:443|NIM_UPSTREAM_API_KEY|NVIDIA_NIM_API_KEY|DRY_RUN)' \
/tmp/hourly-product-development.yml
echo
echo '== Job / endpoint correspondence =='
python3 - <<'PY'
from pathlib import Path
import yaml
document = yaml.safe_load(Path("/tmp/hourly-product-development.yml").read_text())
for job_name, job in document["jobs"].items():
steps = job.get("steps", [])
gh_steps = [
step for step in steps
if isinstance(step, dict) and "gh api" in str(step.get("run", ""))
]
if not gh_steps:
continue
harden = next(
(
step for step in steps
if isinstance(step, dict)
and str(step.get("uses", "")).startswith("step-security/harden-runner@")
),
None,
)
endpoints = tuple(
line.strip()
for line in str(harden.get("with", {}).get("allowed-endpoints", "")).splitlines()
if line.strip()
) if harden else ()
print(f"{job_name}: gh-api-steps={len(gh_steps)}")
print(f" api.github.com:443={'api.github.com:443' in endpoints}")
print(f" cafe.github.com:443={'cafe.github.com:443' in endpoints}")
PY
echo
echo '== Workflow syntax / whitespace checks for exact diff =='
git diff --check c8968ec1e68fab16d0ad8216fb5c8fd0b385e95f "$head"
echo
echo '== Pull request and check state =='
gh pr view 74 --repo ContextualWisdomLab/keyverse \
--json number,url,headRefOid,baseRefOid,mergeStateStatus,reviews,statusCheckRollup
echo
gh api "repos/ContextualWisdomLab/keyverse/commits/$head/check-runs?per_page=100" \
--paginate --slurp \
--jq '[.[].check_runs[] | {name, status, conclusion, details_url}]'Length of output: 17144 🏁 Script executed: #!/bin/bash
set -euo pipefail
head='d1802174f73b5b2a3b7f8511e7816c8763d01ec0'
workflow='.github/workflows/hourly-product-development.yml'
git show "$head:$workflow" > /tmp/hourly-product-development.yml
echo '== Every job containing gh api and its exact GitHub endpoints =='
awk '
/^ [A-Za-z0-9_-]+:$/ {
if (job != "" && has_gh) {
printf "%s|gh_api=yes|api=%s|cafe=%s\n", job, api, cafe
}
job=$1; sub(/:$/, "", job)
has_gh=0; api=0; cafe=0
}
/gh api/ { has_gh=1 }
/^[[:space:]]*api\.github\.com:443[[:space:]]*$/ { api=1 }
/^[[:space:]]*cafe\.github\.com:443[[:space:]]*$/ { cafe=1 }
END {
if (job != "" && has_gh) {
printf "%s|gh_api=yes|api=%s|cafe=%s\n", job, api, cafe
}
}
' /tmp/hourly-product-development.yml
echo
echo '== Exact endpoint lists for GitHub API jobs =='
awk '
/^ [A-Za-z0-9_-]+:$/ {
job=$1; sub(/:$/, "", job)
in_endpoints=0
}
/^[[:space:]]*allowed-endpoints: \|[[:space:]]*$/ {
in_endpoints=1
next
}
in_endpoints && /^[[:space:]]{12}[[:graph:]].*:443[[:space:]]*$/ {
print job "|" $1
next
}
in_endpoints && !/^[[:space:]]*$/ {
in_endpoints=0
}
' /tmp/hourly-product-development.yml
echo
echo '== Credential references before broker step =='
awk '
/name: Start the loopback-only NIM credential broker/ { exit }
/NVIDIA_NIM_API_KEY|NIM_UPSTREAM_API_KEY/ { print NR ":" $0 }
' /tmp/hourly-product-development.yml
echo
echo '== Exact-head PR metadata, review state, and checks =='
gh pr view 74 --repo ContextualWisdomLab/keyverse \
--json number,url,headRefOid,baseRefOid,mergeStateStatus,reviews,statusCheckRollup
echo
gh api "repos/ContextualWisdomLab/keyverse/commits/$head/check-runs?per_page=100" \
--paginate --slurp \
--jq '[.[].check_runs[] | {name, status, conclusion, details_url}]'Length of output: 14767
I found no code-level issues in the requested scope. Verified:
The PR is not ready for merge under the stated closure criteria:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
services/account_unification/tests/test_hourly_product_development.py (1)
148-151: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win자격 증명 범위를 실제로 검증하도록 테스트를 강화하세요.
Line 151은 오류 문구의 존재만 확인합니다. 이 검사는
NVIDIA_NIM_API_KEY가 결정적 게이트 전에 읽히거나 검증되는 회귀를 탐지하지 못합니다. 테스트에서 NIM 단계의if: steps.gate.outputs.develop == 'true'조건과secrets.NVIDIA_NIM_API_KEY바인딩을 함께 확인하세요. 가능하면 단순 문자열 검색 대신 해당 YAML 단계 블록을 검증하세요.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@services/account_unification/tests/test_hourly_product_development.py` around lines 148 - 151, 테스트의 오류 문구 존재 확인을 강화해, _workflow_source()의 YAML에서 NIM 단계가 `if: steps.gate.outputs.develop == 'true'` 조건으로 보호되는지와 `secrets.NVIDIA_NIM_API_KEY`가 해당 단계에 바인딩되는지를 함께 검증하세요. 가능하면 전체 문자열 검색 대신 해당 YAML 단계 블록을 파싱하거나 추출해 두 조건이 같은 단계에 있는지 확인하십시오.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/hourly-product-development.yml:
- Line 58: Update all three allowed-endpoints declarations in the workflow to
use the folded, strip-chomping scalar syntax >- instead of the literal | syntax,
preserving the existing endpoint order and values.
- Around line 136-137: Update the GitHub inventory parsing flow around the
Python parser and its `if ! python3 ...` branch so unsupported response shapes
and JSON parsing errors return a distinct nonzero status, which the shell
reports with `::error::` and propagates via `exit 1`. Preserve the existing safe
skip (`exit 0`) only for missing or unhealthy evidence, and extend
`test_github_inventory_transport_failures_are_not_false_green` to verify the
malformed-response failure path.
---
Nitpick comments:
In `@services/account_unification/tests/test_hourly_product_development.py`:
- Around line 148-151: 테스트의 오류 문구 존재 확인을 강화해, _workflow_source()의 YAML에서 NIM 단계가
`if: steps.gate.outputs.develop == 'true'` 조건으로 보호되는지와
`secrets.NVIDIA_NIM_API_KEY`가 해당 단계에 바인딩되는지를 함께 검증하세요. 가능하면 전체 문자열 검색 대신 해당 YAML
단계 블록을 파싱하거나 추출해 두 조건이 같은 단계에 있는지 확인하십시오.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b508fc9e-ea2c-4ca4-873c-c5a9743193cd
📒 Files selected for processing (3)
.github/workflows/hourly-product-development.ymlservices/account_unification/tests/test_hourly_product_development.pyservices/account_unification/tests/test_hourly_product_incident_contract.py
|
@opencode-agent @cwl-noema-review Review-only request for exact current head |
|
@opencode-agent @cwl-noema-review Review-only request for exact current head |
|
Fleet-incident RCA handoff for exact unchanged head
Detection/control failure across all three: exact-head CI/security status is green because current workflow contracts cover endpoint serialization, transport/malformed evidence, and gate ordering, but not post-model secret authority, end-to-end runtime schedulability, or required-vs-optional check conclusions. Please repair these test-first under the existing Keyverse owner lease, one causal layer at a time; rerun focused/full exact-head CI, CodeQL, SAST, Security, review threads, and formal reviews after the final head. Do not merge or request another same-head review while these internal gates remain unclean. |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headf24d4b46e344f0f2c375c65fafcb3d9410284fa2. -
Head SHA:
f24d4b46e344f0f2c375c65fafcb3d9410284fa2 -
Workflow run: 31264416890
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: hourly-product-development.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: hourly-product-development.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Test (2 files)"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test (2 files)"]
R2 --> V2["targeted test run"]
OpenCode Review Overview
Pull request overviewOpenCode cannot approve yet because required coverage evidence did not pass. Review outcome1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
Coverage evidenceCoverage evidence job did not run or did not publish coverage evidence. Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: hourly-product-development.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: hourly-product-development.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Test (2 files)"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test (2 files)"]
R2 --> V2["targeted test run"]
|
|
@opencode-agent @cwl-noema-review Review-only request for exact current head |
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head2b572dd20c45e91853bdb9c20ba4f7b2bc6c625c. -
Head SHA:
2b572dd20c45e91853bdb9c20ba4f7b2bc6c625c -
Workflow run: 31266938607
-
Workflow attempt: 1
Coverage evidence
Coverage evidence job did not run or did not publish coverage evidence.
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow: hourly-product-development.yml"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow: hourly-product-development.yml"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Test (2 files)"]
S2 --> I2["regression suite"]
I2 --> R2["Review risk: Test (2 files)"]
R2 --> V2["targeted test run"]
1 participant
Incident
Protected-main Hourly product development reproduced a fail-closed Harden Runner egress defect: the GitHub API inventory call was blocked, but the gate converted that infrastructure failure into a warning plus
exit 0, so the workflow reported success while all development/reverification/publication work was skipped.A same-class protected-main ThreadWeave failure proved a second runtime boundary: newline-literal Harden Runner
allowed-endpointscan reach the installed agent as a malformed endpoint map with the GitHub API port collapsed to0. Endpoint membership alone is therefore insufficient; the serialized scalar must remain runtime-safe.Further exact-source RCA found three latent operational gaps that previous green checks did not cover: the 45-minute outer job could not accommodate three sequential 35-minute model fallbacks; the post-model packaging step rematerialized the raw NVIDIA secret even though the model only receives a loopback placeholder; and default-main check evidence treated
neutral/skippedconclusions as healthy.Repair
egress-policy: block;cafe.github.com:443alias alongsideapi.github.com:443;>-so the action receives one space-delimited scalar without losing ports;exit 0only for explicitly missing/unhealthy protected-main evidence;successfor default-main check evidence rather than acceptingneutralorskipped;NVIDIA_NIM_API_KEYmaterialization exclusively in the conditional loopback broker step and remove raw-secret materialization from credential-free packaging;Test-first and verification evidence
RED commit
3a274ab95bc98690689466d31dbd54e2717bde9aintroduced the original incident contracts. RED commitda5c8a5722e7e75de3fb82b486a0c1e087c2fb02added the runtime delimiter regression. RED commitd6eea5e47a8d7ac0f6f0f3fce3991c6fd1ed096dadded explicit contracts for success-only default-main check evidence, end-to-end fallback schedulability, and single-step NVIDIA secret materialization before the production repair.The production repair landed in
35975b79feef5bd6714013ae38bd584b4d9e18c8. CI then exposed two stale test assumptions (the old 45-minute deadline and deliberate post-model secret rematerialization);2b572dd20c45e91853bdb9c20ba4f7b2bc6c625caligned those tests with the stricter runtime boundary without weakening the new incident contracts.Exact current head:
2b572dd20c45e91853bdb9c20ba4f7b2bc6c625cagainst protected basec8968ec1e68fab16d0ad8216fb5c8fd0b385e95f.Exact-head hosted evidence:
cirun31265082573: success; locked install, Ruff, 100% docstring gate, complete pytest/100% production statement+branch coverage, realm/template validation and compose validation passed;CodeQLrun31265082570: success;SAST Semgreprun31265082568: success;Security Scanrun31265082593: success;The prior OpenCode
REQUEST_CHANGESreview is explicitly anchored to predecessor headf24d4b46e344f0f2c375c65fafcb3d9410284fa2and its failed review-workflow coverage evidence. A fresh exact-head review is required before this PR can be classified clean.Safety boundaries
No
COPILOT_GITHUB_TOKEN, no guessed credential, no widened egress policy, no temporary write-capable/self-modifying repair workflow, no review-agent identity/key-chain change, and no branch-protection bypass.Closure criteria
Do not merge unless the exact current head still has successful required checks, zero unresolved valid findings, and a qualifying independent non-author formal
APPROVEunder the automation governance policy. Merge alone is not operational closure: after protected-main merge, an actual scheduled/manual Hourly product development run must prove the GitHub API inventory gate works and either stops for the correct deterministic reason or reaches the bounded OpenCode/NVIDIA path.