Skip to content

fix(fork): push mirror fast-forwards with a deploy key - #27

Merged
NoahHendrickson merged 2 commits into
customfrom
fix/fork-sync-mirror-deploy-key
Jul 28, 2026
Merged

fix(fork): push mirror fast-forwards with a deploy key#27
NoahHendrickson merged 2 commits into
customfrom
fix/fork-sync-mirror-deploy-key

Conversation

@NoahHendrickson

Copy link
Copy Markdown
Owner

Problem

The hourly fork-sync-mirror job has failed on every run since ~2026-07-28T00:13Z. Upstream commit a148e0819 modified .github/workflows/release.yml, and the mirror pushes with the default GITHUB_TOKEN — a GitHub App token that can never hold the workflows permission (it is not grantable via the permissions: block). GitHub therefore rejected the fast-forward push and main went stale, 13 commits behind upstream.

Fix

Check out the mirror job with the new FORK_SYNC_PUSH_KEY deploy key (read-write, scoped to this repo only), so the push goes over SSH — SSH keys are exempt from the workflow-file token restriction. The drift job doesn't push and is unchanged.

Already done outside this PR:

  • main manually fast-forwarded to upstream 476d69cd1 (the exact operation the job performs), so the mirror is current again.
  • Deploy key fork-sync-mirror push key (id 158540815) added and its private key stored as the FORK_SYNC_PUSH_KEY Actions secret.
  • Verified end-to-end: the deploy key successfully pushed a commit modifying a workflow file to a throwaway branch (since deleted) — the exact operation that was being rejected.

Note

If the main branch ruleset from .fork/AGENTS.md (block direct pushes except the mirror) is ever configured, the deploy key must be on its bypass list.

🤖 Generated with Claude Code

The hourly fork-sync-mirror job has been failing since upstream commit
a148e08 modified .github/workflows/release.yml: the job pushed with
the default GITHUB_TOKEN, which can never hold the workflows permission,
so GitHub rejected any fast-forward range touching workflow files and
main went stale.

Check out the mirror job with the FORK_SYNC_PUSH_KEY deploy key
(read-write, repo-scoped) so the push goes over SSH, which is exempt
from that token restriction. Verified the key can push a
workflow-file-modifying commit before wiring it in.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 28, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XS labels Jul 28, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thermo-nuclear code quality review

No major structural issues found.

This is a direct, minimal credential-path fix: actions/checkout ssh-key for the mirror job is the canonical way around GITHUB_TOKEN’s hard workflows push restriction. No new branching, no abstraction churn, no file-size pressure (137 lines), and no missed code-judo simplification. Meets the approval bar.

Open in Web View Automation 

Sent by Cursor Automation: Thermo-nuclear PR review

Copy link
Copy Markdown
Owner Author

Review: aggressive pass

The diagnosis is right and the mechanism is the correct one — GITHUB_TOKEN is an App installation token, workflows is not grantable through the permissions: block, and SSH is genuinely exempt. A deploy key beats a fine-grained PAT here (no expiry, repo-scoped, no user identity). The inline comment explaining why is the kind of thing that stops this from being re-broken in six months. The end-to-end verification against a throwaway branch is the right proof.

But the change does something the description doesn't mention, and I think it blocks.


🔴 Blocking — this makes mirror pushes start triggering workflows

Pushes authenticated with GITHUB_TOKEN do not create workflow runs — GitHub's recursion guard. Pushes authenticated with a deploy key are not covered by that guard; using a deploy key or PAT is the documented workaround for when you want a bot push to trigger downstream workflows.

So today the hourly mirror push to main fires nothing. After this PR, every mirror push that advances main fires every workflow watching push: main. There are four:

workflow trigger what happens on the fork
deploy-relay.yml push: main (no path filter) production relay deploy
ci.yml push: main full CI, incl. mobile_native_static_analysis on macos-latest
issue-labels.yml push: main, paths .github/ISSUE_TEMPLATE/** fires when upstream edits templates
pr-vouch.yml push: main, paths .github/VOUCHED.td re-evaluates vouch labels on all open PRs

deploy-relay.yml is the one that matters. It checks out the pushed ref — raw, unreviewed upstream main — and runs vp run --filter t3code-relay deploy --stage prod --yes with CLOUDFLARE_API_TOKEN, PLANETSCALE_API_TOKEN, CLERK_SECRET_KEY, and APNS_PRIVATE_KEY. There is no github.repository gate on it, unlike release.yml.

What actually happens the first hour after merge is that it hangs, because deploy-relay.yml:20 is runs-on: blacksmith-8vcpu-ubuntu-2404 and per the ci-runners manifest entry the Blacksmith app "is installed on pingdotgg but not on this fork, so upstream's labels queue forever here." So the near-term damage is hourly hung runs plus real macOS-multiplier CI spend on a branch that is a pure upstream mirror — CI upstream already ran.

The long-term damage is the part I'd push back hardest on, because this repo has already written down the exact rule this breaks. From .fork/customizations.yaml#release-upstream-only:

Until this gate existed the runs did not publish only because they hung forever on Blacksmith runners the fork cannot schedule (see ci-runners) — an accident, not a safeguard. Keep the gate independent of the runner labels so retiring ci-runners can never start publishing from the fork.

deploy-relay.yml is protected by exactly two accidents: Blacksmith not being installed, and token loop-prevention. This PR retires the second one. ci-runners says the first "can be retired wholesale" the day Blacksmith is installed on the fork — and on that day, an hourly cron starts deploying unreviewed upstream code to production with live credentials. That is the failure mode release-upstream-only exists to prevent, and deploy-relay.yml never got the same treatment.

Suggested fix — apply the established pattern rather than inventing a new one:

# .github/workflows/deploy-relay.yml
jobs:
  deploy_relay:
    # fork:begin relay-upstream-only — see .fork/customizations.yaml#relay-upstream-only
    if: github.repository == 'pingdotgg/t3code'
    # fork:end relay-upstream-only

plus a manifest entry and a guard test alongside releaseUpstreamOnly.test.ts. Independent of runner labels, per the rule above.

I'd also drop main from ci.yml's push branches on the fork — main is a pure mirror, so that CI run is redundant by construction. Worth confirming whether any of these four are already disabled in the Actions UI; I can't see that from the repo, and it would change the urgency but not the gate.


🟡 The secret going missing silently re-creates this exact incident

If FORK_SYNC_PUSH_KEY is ever empty, deleted, or rotated badly, ssh-key evaluates to empty and actions/checkout silently falls back to token auth. The job then fails at git push with the same opaque workflow-permission error that just cost a day of debugging — with no hint that the cause is a missing secret. Worth three lines:

- name: Require the push key
  run: |
    if [ -z "${{ secrets.FORK_SYNC_PUSH_KEY }}" ]; then
      echo "::error::FORK_SYNC_PUSH_KEY is unset — the mirror push will be rejected on any range touching .github/workflows/*."
      exit 1
    fi

🟡 Nothing detects that this job is broken

The actual cost of this incident wasn't the token — it was main sitting 13 commits stale for a day before anyone noticed. This PR fixes the cause and adds no detection, and finding #1 makes detection worse by filling the Actions tab with hourly hung deploy-relay runs to scroll past.

The drift job already contains a find-or-create-issue github-script block. Copying it into an if: failure() step on mirror would close the loop cheaply.

🟢 contents: write is now dead — and dropping it hardens the failure

The push is over SSH now, so no job in this workflow needs write on the token: mirror pushes with the key, drift only reads. contents: read + issues: write is sufficient. Beyond least privilege, it makes the secret-missing case above fail unambiguously instead of falling back to a token that can almost-but-not-quite do the job.

🟢 Pin the invariant that keeps the key safe

The deploy key is repo-scoped read-write and now lives in a job whose workspace holds unreviewed upstream code at fetch-depth: 0. That's fine only because the mirror job runs nothing but git plumbing. Note that drift deliberately runs node .fork/detect-drift.mjs — from custom, and without the key. That separation is load-bearing and currently undocumented; one line saying "never add a step to this job that executes checked-out content" would keep a future npm ci from handing a write key to upstream-controlled code.

🟢 Move the ruleset note out of the PR body

The bypass-list caveat is real and will be needed at configuration time, months from now, by someone not reading merged PR descriptions. It belongs next to the ruleset requirement in .fork/AGENTS.md.


Verdict: the four lines are correct in isolation; ship them once deploy-relay.yml is gated. Everything below the blocking finding is cheap and optional, though the preflight check and the failure alarm both pay for themselves against this specific incident's history.


Generated by Claude Code

Address review on the deploy-key change. Deploy-key pushes, unlike
GITHUB_TOKEN pushes, trigger push workflows, so every hourly mirror
advance would have fired everything watching push: main.

- Gate deploy_relay on github.repository == 'pingdotgg/t3code'
  (fenced, manifest entry relay-upstream-only, guard test), matching
  release-upstream-only: independent of runner labels, so installing
  Blacksmith on the fork can never start deploying production.
- Drop main from ci.yml push triggers inside the ci-on-custom fence;
  main is a pure mirror upstream CI already validated. Guard extended
  to pin main's absence.
- Mirror job: preflight that FORK_SYNC_PUSH_KEY exists (checkout falls
  back to token auth silently otherwise), drop contents: write (SSH
  key does the push; token needs only read), report failures to a
  find-or-create [fork-sync] issue so a stale mirror is loud, and
  document the git-plumbing-only invariant that keeps the write key
  away from unreviewed upstream code.
- Document the deploy key and its ruleset bypass requirement in
  .fork/AGENTS.md and .fork/README.md §5 instead of the PR body.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@NoahHendrickson

Copy link
Copy Markdown
Owner Author

Addressed in 5afcc53 — point by point:

🔴 Deploy-key pushes trigger push: main workflows — gated. Confirmed the finding first: all four workflows are active in the Actions UI (nothing was disabled there), so the urgency stood. deploy_relay now carries the fenced if: github.repository == 'pingdotgg/t3code' gate, with a relay-upstream-only manifest entry modeled on release-upstream-only — same wording about runner labels being an accident, not a safeguard — and a relayUpstreamOnly.test.ts guard asserting every job in the file is gated (not just the one that exists today). Also dropped main from ci.yml's push triggers inside the existing ci-on-custom fence; that guard now pins both custom present and main absent. issue-labels.yml and pr-vouch.yml left alone: both are path-filtered to files upstream rarely touches, and neither holds deploy credentials — they'd fire occasionally and harmlessly.

🟡 Missing secret fails silently — preflight added. First step of mirror now fails with an explicit ::error:: naming FORK_SYNC_PUSH_KEY and pointing at .fork/README.md §5, before checkout can fall back to token auth.

🟡 No detection — failure alarm added. if: failure() step on mirror maintains a find-or-create [fork-sync] fork-sync-mirror is failing issue, reusing the drift job's github-script pattern. A stale mirror is now an open issue within the hour instead of a silent Actions-tab entry.

🟢 contents: write dropped to contents: read + issues: write, with a comment explaining that read-only is what makes the missing-key case fail unambiguously.

🟢 No-execution invariant pinned as a SECURITY INVARIANT comment on the mirror job (git plumbing only; anything executing repo content belongs in drift, which checks out custom and carries no key), and in .fork/README.md §5.

🟢 Ruleset note moved into .fork/AGENTS.md next to the ruleset requirement, plus a §5 write-up of the deploy key, why token pushes fail, and how to rotate the key.

Verification: full fork guard suite green (155 tests, including the two new/extended guards), vp fmt, web typecheck clean.

🤖 Generated with Claude Code

@github-actions github-actions Bot added size:L and removed size:XS labels Jul 28, 2026
@NoahHendrickson
NoahHendrickson merged commit 6d8324b into custom Jul 28, 2026
10 checks passed
@NoahHendrickson
NoahHendrickson deleted the fix/fork-sync-mirror-deploy-key branch July 28, 2026 02:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant