Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 20 additions & 12 deletions .github/workflows/turbo-nightly-sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ env:

jobs:
sync_source:
name: Rebase Turbo customizations
name: Merge upstream into Turbo
if: vars.TURBO_NIGHTLY_ENABLED == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 15
Expand Down Expand Up @@ -135,7 +135,7 @@ jobs:
--github-output

- id: rebase
name: Rebase in an isolated worktree
name: Merge in an isolated worktree
if: steps.resolve.outputs.has_update == 'true'
env:
PUBLISH_TAG: ${{ steps.resolve.outputs.tag }}
Expand Down Expand Up @@ -210,15 +210,24 @@ jobs:
git -C "$sync_worktree" switch -c turbo-sync-candidate
git -C "$sync_worktree" config user.name "t3-turbo-bot"
git -C "$sync_worktree" config user.email "t3-turbo-bot@users.noreply.github.com"
# The Turbo branch is not a linear stack above the recorded anchor: it carries merge
# commits of its own, and replaying it would re-litigate history those merges already
# reconciled. Upstream is merged in instead, and the manifest verifier below is what
# proves the registered fork behavior survived the ingestion.
source_date="$(git -C "$sync_worktree" show -s --format=%aI "$new_sha")"
# git merge names the conflicted files on stdout, so the collision report reads a
# combined log; splitting the streams would file an empty error section.
set +e
git -C "$sync_worktree" rebase --committer-date-is-author-date \
--onto "$new_sha" "$OLD_MAIN_SHA" \
> "$RUNNER_TEMP/rebase.stdout" 2> "$RUNNER_TEMP/rebase.stderr"
rebase_status=$?
GIT_AUTHOR_DATE="$source_date" GIT_COMMITTER_DATE="$source_date" \
git -C "$sync_worktree" merge --no-ff --no-edit \
-m "chore(turbo): merge upstream $UPSTREAM_BRANCH at ${new_sha:0:12}" \
"$new_sha" \
> "$RUNNER_TEMP/merge.log" 2>&1
merge_status=$?
set -e

echo "upstream_sha=$new_sha" >> "$GITHUB_OUTPUT"
if [[ $rebase_status -ne 0 ]]; then
if [[ $merge_status -ne 0 ]]; then
git -C "$sync_worktree" diff --name-only --diff-filter=U \
> "$RUNNER_TEMP/unmerged-paths.txt"
node scripts/turbo-nightly-sync.ts report \
Expand All @@ -229,9 +238,9 @@ jobs:
--upstream-paths "$RUNNER_TEMP/upstream-paths.txt" \
--customization-paths "$RUNNER_TEMP/customization-paths.txt" \
--unmerged-paths "$RUNNER_TEMP/unmerged-paths.txt" \
--rebase-error "$RUNNER_TEMP/rebase.stderr" \
--merge-error "$RUNNER_TEMP/merge.log" \
--output "$GITHUB_WORKSPACE/turbo-rebase-report.md"
git -C "$sync_worktree" rebase --abort || true
git -C "$sync_worktree" merge --abort || true
echo "conflicted=true" >> "$GITHUB_OUTPUT"
exit 0
fi
Expand All @@ -258,7 +267,6 @@ jobs:
'{ repository: $repository, branch: $branch, mainSha: $mainSha, nightlyTag: $nightlyTag, nightlySha: $nightlySha, version: $version, cutoffDate: $cutoffDate, cutoffInstant: $cutoffInstant }' \
> "$sync_worktree/.t3-turbo/upstream.json"
git -C "$sync_worktree" add .t3-turbo/upstream.json
source_date="$(git -C "$sync_worktree" show -s --format=%aI "$new_sha")"
GIT_AUTHOR_DATE="$source_date" GIT_COMMITTER_DATE="$source_date" \
git -C "$sync_worktree" commit -m "chore(turbo): track upstream main at ${new_sha:0:12}"
candidate_sha="$(git -C "$sync_worktree" rev-parse HEAD)"
Expand Down Expand Up @@ -294,11 +302,11 @@ jobs:
echo "- [Open this Actions run]($run_url)"
echo "- [Open the manual build page]($workflow_url)"
echo '- Create `repair/turbo-nightly-${{ github.run_id }}` from the unchanged `turbo` branch.'
echo '- Replay or amend the Turbo customization that collided, run the manifest verifier and its focused seam tests, then open a reviewed PR back to `turbo`.'
echo '- Resolve each collided file starting from the new upstream version, reapply only the fork behavior `SEAM.md` describes, run the manifest verifier and its focused seam tests, then open a reviewed PR back to `turbo`.'
echo '- If the workflow changes, land the identical workflow blob on `main` before retrying. Do not hand-edit the upstream checkpoint or publish partial artifacts.'
echo '- CLI retry: `gh workflow run turbo-nightly-sync.yml -R '"$GITHUB_REPOSITORY"' --ref main`'
} >> turbo-rebase-report.md
title="T3-Turbo rebase needs review: $NEW_TAG"
title="T3-Turbo merge needs review: $NEW_TAG"
issue_number="$(
gh issue list --state open --limit 100 --json number,title \
--jq ".[] | select(.title == \"$title\") | .number" | head -n 1
Expand Down
3 changes: 2 additions & 1 deletion .t3-turbo/customizations.json
Original file line number Diff line number Diff line change
Expand Up @@ -1070,7 +1070,8 @@
"timezone: \"America/New_York\"",
"Resolve the completed Eastern cutoff",
"-f until=\"$CUTOFF_INSTANT\"",
"Rebase in an isolated worktree",
"Merge in an isolated worktree",
"merge --no-ff --no-edit",
"apps/web/src/turbo/chatPanes/chatPaneResourcePolicy.test.ts",
"scripts/turbo-product-branding.test.ts",
"Record registered relay and portal branch state",
Expand Down
24 changes: 16 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,22 +20,30 @@
tracked in two places: `SEAM.md` (human conflict guidance per file) and
`.t3-turbo/customizations.json` (the machine-readable preservation contract, verified with
`pnpm --dir scripts turbo:customizations:verify`). If you modify an upstream-owned file,
register it in both. Keep Turbo-only work as a small, reviewable commit stack above the
upstream SHA recorded in `.t3-turbo/upstream.json`.
register it in both. The manifest — not the shape of the history — is what preserves fork
behavior across ingestion, so registering a change matters more than where its commit sits.
Keep Turbo-only work small and reviewable, and land it on `turbo` through normal reviewed
merges; `.t3-turbo/upstream.json` records the upstream SHA the branch has ingested, not a
base the branch is expected to sit linearly on top of.

3. **We ingest ALL upstream code daily.** The scheduled `turbo-nightly-sync.yml` workflow
(11:00 PM Eastern) reads official `main` from `pingdotgg/t3code` and rebases the Turbo commit
stack onto it. Upstream is strictly read-only from our side — **never push, open PRs, or
write anything to `pingdotgg/t3code`.** The only push target is `gfsaaser24/t3code`.
(11:00 PM Eastern) reads official `main` from `pingdotgg/t3code` and **merges** it into the
Turbo branch inside an isolated worktree. It merges rather than rebases because `turbo` is not
a linear stack: it carries merge commits of its own, and replaying it re-litigates conflicts
those merges already settled. Upstream is strictly read-only from our side — **never push,
open PRs, or write anything to `pingdotgg/t3code`.** The only push target is
`gfsaaser24/t3code`.

4. **We ingest both main commits and official Nightly releases.** The newest published official
Nightly source tag is the deterministic version anchor; ordinary commits pushed between
Nightlies come along with `main`. We never download or republish an official installer —
Turbo installers are always built from source in our own pipeline.

5. **Turbo changes always survive ingestion.** Every rebase candidate must pass the
customization manifest before it can be built. On conflict, automation aborts, uploads a
collision report, and opens an issue — it never chooses a resolution. When you resolve one:
5. **Turbo changes always survive ingestion.** Every merge candidate must pass the
customization manifest before it can be built — the manifest is the preservation contract, and
it is what proves the seams survived, since the merge itself makes no promise about history
shape. On conflict, automation aborts the merge, uploads a collision report, and opens an
issue — it never chooses a resolution. When you resolve one:
start from the new upstream file, reapply only the fork behavior described in `SEAM.md`, and
drop a fork hunk only when upstream now provides the equivalent. Never hand-edit
`.t3-turbo/upstream.json` and never delete checks to make a run green.
Expand Down
5 changes: 5 additions & 0 deletions SEAM.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,11 @@ fork's change.

## Nightly sync conflicts

Ingestion merges upstream `main` into `turbo`; it does not replay the fork's commits. A conflict is
therefore a single three-way merge to settle in one commit, not a collision to re-resolve at each
replayed commit — and the customization manifest, not the history shape, is what proves the seams
below survived.

Resolve against the new upstream file first, then reapply only the behavior above; never take the
fork's whole file over a newer upstream implementation. Drop a fork hunk when upstream now provides
equivalent gating or optionality. Preserve the PlanetScale fallback, the Cloudflare credential
Expand Down
39 changes: 21 additions & 18 deletions docs/internals/t3-turbo-nightly-inbound.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ which lets ordinary upstream changes coexist with the fork while still failing w
modules, assets, behavior markers, tests, or policy text disappear. Each seam is labeled
`implemented`, `policy`, or `planned`; a planned seam verifies its reviewed design contract and
does not claim that the feature has shipped. The same manifest registers infrastructure refs whose
state must be measured without bringing those branches into the product rebase. The
state must be measured without bringing those branches into the product ingest. The
`relay-portal` entry currently owns `refs/heads/infra/t3turbo-relay`.

## Recreating the pipeline in a fork
Expand All @@ -52,7 +52,7 @@ first scheduled run:
`.t3-turbo/upstream.json`).
2. A default `main` branch containing this workflow and a `turbo` branch containing the same
workflow file plus the Turbo customization stack. The workflow blobs on both branches must be
identical; the schedule is read from `main`, while inbound rebases carry the file forward on
identical; the schedule is read from `main`, while inbound merges carry the file forward on
`turbo`.
3. Every branch listed in `customizations.json`, including `infra/t3turbo-relay`. Registered
infrastructure branches use normal reviewed merges and are never rebased or force-pushed by
Expand All @@ -77,7 +77,7 @@ gh variable get TURBO_NIGHTLY_ENABLED --repo gfsaaser24/t3code

The two workflow blob hashes must match. The durable checkpoint must be committed on `turbo`; the
workflow updates it only inside the isolated candidate and publishes it with the candidate branch.
Never hand-edit `mainSha`, `nightlySha`, or `version` to bypass a failed rebase. If the checkpoint
Never hand-edit `mainSha`, `nightlySha`, or `version` to bypass a failed ingest. If the checkpoint
is wrong, stop the schedule, compare it with the last successful release, and repair the branch
as a reviewed commit.

Expand All @@ -100,7 +100,7 @@ Select main and the Nightly anchor at or before that instant
Validate forward-only history
|
v
Rebase Turbo commits temporarily
Merge upstream in temp worktree
| |
| conflict | clean
v v
Expand Down Expand Up @@ -139,10 +139,13 @@ claiming a future boundary. The repository variable `TURBO_NIGHTLY_ENABLED` is t
6. Git verifies that the new `main` descends from our recorded `mainSha`, that the Nightly commit
is contained in `main`, and that the official tag has not moved. A backward, divergent, or
rewritten update is rejected.
7. The workflow creates a temporary Git worktree and rebases our Turbo commits onto the new main
commit. It never edits the live `turbo` branch during this stage.
8. A clean rebase must pass the dependency-free customization manifest verifier inside the
7. The workflow creates a temporary Git worktree and merges the new `main` commit into our Turbo
branch. It never edits the live `turbo` branch during this stage. It merges rather than
rebases because `turbo` is not a linear stack above the recorded anchor — it carries merge
commits of its own, so replaying it would re-litigate conflicts those merges already settled.
8. A clean merge must pass the dependency-free customization manifest verifier inside the
temporary candidate worktree. Missing seams fail the source job before a bundle is created.
The manifest, not the shape of the history, is the preservation contract.
9. The verified candidate is bundled and handed to isolated Linux and Windows build jobs.
10. Linux builds the WSL `node-pty` native module. Windows builds the T3 Turbo NSIS installer and
update blockmap without signing or official-service credentials.
Expand Down Expand Up @@ -182,7 +185,7 @@ node scripts/turbo-customization-manifest.ts verify
vp test run scripts/turbo-nightly-sync.test.ts scripts/turbo-customization-manifest.test.ts
```

The source job runs the verifier before it bundles the rebased candidate. The Windows build job
The source job runs the verifier before it bundles the merged candidate. The Windows build job
runs the tooling tests plus the focused feature tests named directly in the workflow; marker
presence alone is not proof that a behavior still works. If upstream moves or replaces a seam,
update the manifest only in the same reviewed change that supplies and tests the replacement;
Expand All @@ -206,7 +209,7 @@ states the 11:00 PM Eastern ingestion cutoff. The updater assets retain their ve
architecture in their filenames because `nightly.yml` addresses them by those exact names.

The nightly cutoff imports the newest cumulative `main` commit at or before the recorded instant.
If several commits land during the day, all of them arrive together in one rebased Turbo build.
If several commits land during the day, all of them arrive together in one merged Turbo build.
Commits and Nightly releases after that instant wait for the following cutoff even when the runner
starts late.

Expand All @@ -228,14 +231,14 @@ mean it moved independently during the run; product ingestion still did not depl
Automation never chooses "ours" or "theirs." It:

1. records files changed by both upstream and Turbo;
2. records Git's genuinely unmerged paths and rebase error;
3. aborts the temporary rebase;
2. records Git's genuinely unmerged paths and the merge output;
3. aborts the temporary merge;
4. uploads the Markdown conflict report;
5. opens or updates a GitHub issue; and
6. leaves the current Turbo branch and installer release untouched.

The issue gives the exact recovery route: branch from the unchanged `turbo` tip, repair and test
the collided customization, and open a reviewed PR back to `turbo`. If the workflow changes, its
The issue gives the exact recovery route: branch from the unchanged `turbo` tip, resolve each
collided file starting from the new upstream version and reapplying only the `SEAM.md` behavior, and open a reviewed PR back to `turbo`. If the workflow changes, its
blob must also land identically on `main` before retry. The existing installation remains on the
last known-good release until that corrected stack is reviewed and pushed.

Expand Down Expand Up @@ -286,7 +289,7 @@ The nightly workflow uses standard GitHub-hosted runners (`ubuntu-24.04` for sou
dependency, publish, and notification jobs; `windows-2025` for the installer). The general CI
workflow keeps the upstream-only Blacksmith labels for `pingdotgg/t3code` and selects available
GitHub-hosted labels in this fork. A run that is queued indefinitely should be checked for a runner
label regression before changing the rebase logic.
label regression before changing the merge logic.

## Email and OpenClaw Telegram alerts

Expand All @@ -297,7 +300,7 @@ failed jobs, so `TURBO_NOTIFY_USER` is optional and only controls the issue assi
receives GitHub's normal issue notifications according to their account settings.

If an existing OpenClaw gateway already owns the Telegram channel, it can deliver the optional
intervention alerts. The workflow notification job runs only when the source rebase conflicts or a
intervention alerts. The workflow notification job runs only when the source merge conflicts or a
sync/build/publish job fails. It does not send success or no-update messages. Enable OpenClaw's
authenticated `/hooks/agent` endpoint and configure:

Expand Down Expand Up @@ -350,7 +353,7 @@ confirm-then-run link; never place a personal access token in an email or Telegr
## Current wiring and how to inspect it

The workflow file must remain identical on our `main` and `turbo` branches: `main` activates the
GitHub schedule, while `turbo` carries the workflow forward during rebases.
GitHub schedule, while `turbo` carries the workflow forward during merges.

Useful checks:

Expand All @@ -370,7 +373,7 @@ uses the green no-build path.

Routine forward updates require nothing from us. Human review is needed only when:

- Git cannot rebase a Turbo customization cleanly;
- Git cannot merge upstream into a Turbo customization cleanly;
- upstream history moves backward or becomes unrelated;
- manifest, focused seam, icon, native build, packaging, or publication checks fail; or
- we intentionally change the workflow and must copy the same YAML to both `main` and `turbo`.
Expand All @@ -383,7 +386,7 @@ Until the issue is resolved, the current branch and release remain the last know
| ---------------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| No sync job appears | `TURBO_NIGHTLY_ENABLED`, Actions enabled, 11 PM Eastern schedule | A disabled variable causes the job to be skipped. |
| Green run, no installer | `sync_source` outputs and checkpoint | The exact workflow run/version was already published; inspect if this was not a retry. |
| Conflict issue/artifact | Issue body, `turbo-rebase-report.md`, unmerged paths | Resolve the Turbo commit stack, then rerun; the prior release is safe. |
| Conflict issue/artifact | Issue body, `turbo-rebase-report.md`, unmerged paths | Resolve the merge conflict, then rerun; the prior release is safe. |
| Build or seam failure | Repair issue plus the first failed job log | The candidate was not published; follow the documented repair PR path. |
| No Telegram/Slack notice | notification job, variable/secrets names, hook reachability | OpenClaw delivery is optional and non-blocking; inspect the gateway separately. |
| Run queued indefinitely | job runner label and Actions runner availability | Check for an accidental private runner label; do not rewrite history to work around it. |
Expand Down
Loading
Loading