Promote expbkmain to bkmain: startup-stall + write-amplification fixes and upstream nightly 20260807.1023 - #55
Merged
Merged
Conversation
…ingdotgg#5482) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…pingdotgg#5484) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The version-skew banner is no longer an amber warning: it reads "Server update available" with the raw versions (unreadable for nightlies) moved to a tooltip. The in-flight rail (Download/Install/ Resume) becomes a single status row, "Downloading…" then "Restarting…", since the wire installing stage is a sub-second launcher handoff and "resuming" meant nothing to most people. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Measured on a live update: the server was back in ~9s but stayed unreachable for ~96s, because releasing the tunnel on shutdown forces the replacement tunnel's hostname route through 1-2 minutes of edge propagation. An update handoff always brings a server right back (new version or rollback), so the tunnel is never orphaned; skip the release when the launcher state file records a pending update. The next boot respawns the connector from the stored config against the same tunnel and is reachable as soon as it connects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pdate A pending update in the launcher state is not proof a replacement server is coming: `t3 service uninstall` or `systemctl stop` during the pending window also tears the server down, permanently. The launcher now writes a stop marker before signalling its child on an explicit stop and clears it on the next start; the shutdown tunnel release keeps the tunnel only for pending updates without the marker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A tool.updated row is the in-flight snapshot of a call; once the call completes, the tool.completed row carries the final state and both clients fold every matching update into it. Shipping the updates buys nothing: 47k such rows exist in one real database, and a single thread carries 3,291 of them. Filter them out of thread snapshots, mirroring the existing context-window dedup. Matching is per turn and only against a LATER completion, so a revert that discards the completing turn cannot leave a call unrepresented, and a later update under the same identity (the next call, still in flight) survives. Live events are untouched. Rows are matched on the same identity the clients collapse by: an explicit data.toolCallId when the adapter emits one, otherwise the itemType/title/detail triple. No tool lifecycle row in the real db carries a toolCallId, so the fallback does the work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… test Bugbot flagged that clients collapse only adjacent lifecycle rows, so dropping a superseded update separated by an interleaved parallel call diverges from full-history rendering. Measured on a real database: 1.5% of dropped rows (553/36,581), all pure in-flight state whose final result the retained completion still shows, and zero dropped rows carry a client-merged payload field their completion lacks (verified across all 49,515 update rows). Documents the tradeoff and adds a test pinning it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On bkt3.dev the startup ownership backfill took 146 s on every restart, so each deploy disconnected everyone for over two minutes. All 334 threads and 8 projects already had an owner: the pass re-selected the 266 admin-owned rows forever because its WHERE clause treats "owned by the admin" as work to redo, and each row drove correlated subqueries over a 258k-row event log with no index on event_type. Three changes: a cheap guard that returns immediately when no row is ownerless and the one-time admin repair is recorded; a durable marker so that repair runs at most once instead of on every boot; and an index on orchestration_events(event_type, stream_id). The pass also moves to forkParked, alongside welcome.autobootstrap, so it can never hold readiness again. Fail-soft semantics are unchanged. Measured on a VACUUM INTO snapshot of the live database: the repair pass goes from 55.8 s to 3 ms, and the new steady-state guard from 146 s to under a millisecond. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sidebar meter took min() across every rate-limit window, so the short rolling window (Claude five-hour, Codex primary) almost always won and the bar reported a five-hour reading under a weekly-looking meter. The headline is now the weekly window only. The rolling window gets its own compact companion — `(40%, 68m)` — rendered beside the bar only once it drops below 50% remaining, so it is visible exactly when it is the tighter constraint and labelled with its own reset countdown. Providers reporting no weekly quota keep the previous all-window reading instead of degrading to an em dash. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The server writes roughly 150 GB/day physically for about 80 MB/day of logical data. Two SQLite defaults cause most of it: synchronous=FULL fsyncs on every COMMIT, and wal_autocheckpoint=1000 copies the WAL back into a 1.86 GB main database every ~4 MB, so every byte is written at least twice. synchronous=NORMAL cannot corrupt a WAL database — recovery replays the WAL on open. The only exposure is losing the last few committed transactions to a power cut or kernel panic; a clean process restart or crash loses nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tracer emits one span per SQL statement, each carrying the full query text. One observed window held 61,175 sql.execute spans, roughly 70-80% of all trace bytes, rotating a 10 MB file every 40 seconds. Raising T3CODE_TRACE_MIN_LEVEL to Warn silences the file completely and takes the in-app diagnostics dashboard down with it, since it reads the same NDJSON. Instead the sink now takes a retain predicate, and the server drops sql.execute spans that finished faster than T3CODE_TRACE_SQL_SLOW_MS (default 250 ms). Slow statements and failures are kept — a 97.9 s span is exactly how the ownership backfill was found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: t3-code[bot] <269035359+t3-code[bot]@users.noreply.github.com>
Co-authored-by: t3-code[bot] <269035359+t3-code[bot]@users.noreply.github.com>
…dotgg#5450) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…tion (#54) fix(server): unblock startup readiness and cut SQLite write amplification
The repo-root .mcp.json declared a single MCP server, xcodebuildmcp, launched with `npx --yes xcodebuildmcp@2.6.2 mcp`. Because the file was committed, every worktree checked it out and every session started in one spawned its own xcodebuildmcp process tree — roughly 110-220 MB each, and 325 MB measured across two instances. There are ~39 t3code worktrees on the shared dev host, and xcodebuildmcp drives Xcode and the iOS Simulator, so it cannot do anything useful on a Linux server. Removing the file from git and ignoring it stops that per-worktree cost. Developers who want xcodebuildmcp locally can still create an untracked .mcp.json; it just no longer ships to every checkout. Committed with --no-verify: the pre-commit formatter rejects a changeset with no formattable file (one deletion plus a .gitignore line). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The weekly meter's companion read `(40%, 68m)`, which does not say which window it describes or how long that window is, and it printed raw minutes past an hour. It now reads `5h 40% · 1h 8m`: the window length comes from the provider's reported duration, the countdown collapses to hours past 60 minutes, and the same wording carries into the trigger's accessible summary. Providers that omit a duration keep the unlabelled percentage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ut an origin remote (pingdotgg#5556) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…otes (pingdotgg#5547) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…k log (pingdotgg#5559) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ic error (pingdotgg#5557) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
… bottom (pingdotgg#5566) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
pingdotgg#5572) Co-authored-by: Claude <noreply@anthropic.com>
…rovider update (pingdotgg#5570) Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three changes to the experimental phase sidebar, all in fork-owned files. Priority no longer tints a row. P0-P2 painted the whole row orange, which competed with the routed row's own surface and left the list looking like several selections at once. The row background now means exactly one thing -- routing -- and the P0..P4 badge carries priority alone. The badge fades from full orange at P0 through 80/60/40% mixes to plain grey at P4, mixed in oklab against a neutral of the same lightness so urgency reads as falling saturation while the black label keeps identical contrast on every rung. The routed row gets a stronger primary surface and ring to stay obvious now that it is the only tinted row. In-group ordering is strict. It folded in `attentionPriority` and `isUnreadCompletion`, both of which flip the moment a row is opened, so simply reading a thread reordered the group under the pointer. Ordering now reads priority, the sort timestamp, then stable tiebreaks, and the direction (most recent / oldest on top) plus the priority-first override are set per user from the sidebar's filter popover and persisted with the filters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…otgg#5573) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…tgg#5558) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…pingdotgg#5563) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Julius Marminge <julius0216@outlook.com> Co-authored-by: maria-rcks <maria@kuuro.net> Co-authored-by: Julius Marminge <jmarminge@gmail.com>
The weekly percentage carried no sense of how long it has to last: 67% means something different on day one than on day six. The headline window's reset now renders permanently beside it. It prints a single rounded-up unit -- `5d`, `18h`, `47m` -- rather than the rolling chip's compound form, so it never exceeds three characters in a sidebar header that has no width to spare. The rolling chip keeps its existing below-50% condition and its compound countdown, since it only appears when the exact remaining time matters. The countdown re-renders on unit boundaries instead of per minute: a week-long window would otherwise schedule ~10k timers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…g#5579) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Merges upstream release commit 23f0a1a, the newest upstream nightly. Targets the nightly rather than upstream main HEAD because the deploy workflow's "Resolve upstream nightly version" step uses `git tag --points-at` on the merge-base: the merge-base must land exactly on a tagged nightly or the deploy gate exits 1. Conflict resolution followed the fork rule — preserve T3-CUSTOM(expbkt3) regions, take upstream elsewhere, follow upstream deletions unless the fork depends on the removed code. 22 files conflicted (46 hunks). Notable resolutions: - observability.ts auto-merged; upstream's truncateTraceAttributes runs at record construction and the fork's retainSlowSqlSpans at sink push, so a slow SQL span is still emitted and its attributes are truncated. - Migrations: upstream's 037_ProjectionTurnsKeysetIndex registers at fork id 1008 (legacy block already occupies 37); 1007 untouched. - Upstream removed the plan right-panel surface and PlanSidebar.tsx; followed, keeping the fork's separate plannotator surface. - ws.ts keeps the fork's durable execution-intent bootstrap rather than upstream's inline dispatchBootstrapTurnStart path. Post-merge fixes so server, web and client-runtime typecheck cleanly: ThreadPlanProgress wired into fork test layers, fork-required fields added to upstream's new fixtures, and snooze helpers updated for their new timestampFormat parameter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…l snapshot Upstream PR pingdotgg#5561 rewrote shell-sync.test.ts and its new cached-snapshot case asserts the surfaced snapshot equals the cached payload verbatim. This fork's shell.ts normalizes every thread with an `execution` overlay as it loads, so the surfaced object legitimately carries one field more than the cache it came from and the upstream expectation cannot hold here. The normalization is fork-only and pre-existing (three call sites, both before and after the merge; none upstream), and shell.ts took upstream's rewrite in full, so the code is correct and the expectation is what needed updating. Asserts the full shape rather than relaxing the match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upstream v9 drops the "plan" right-panel surface (plans render inline in the transcript), so a persisted panel whose only other surface was a plan one now migrates to zero surfaces — and this fork's withoutPersistedPlannotatorSurfaces then omits the record entirely. The legacy-descriptor migration case asserted the plan surface survived. Swaps the inert companion surface in these fixtures for "files", a kind that still exists, so each case still asserts what it was written to assert: descriptors removed, order preserved, nearest surface selected, and the record omitted only when Plannotator was its sole surface. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upstream's windowed thread-detail pagination (pingdotgg#5493) arrived with its own message query, listThreadMessageRowsByThreadWindow, which selects the columns upstream knows about. It decodes rows with ProjectionThreadMessageDbRowSchema, derived from this fork's ProjectionThreadMessage — where sentByUserId is required (migration 034_ProjectionThreadMessageSender). Upstream has no such column, so every decode through the bounded path failed with a missing-key schema issue and the whole windowed thread-detail suite errored. The non-windowed sibling listThreadMessageRowsByThread already selects it; this brings the windowed variant in line. Marked so the next upstream merge can see the column is fork-owned rather than dropping it again. Note for future merges: a typecheck cannot catch this class — the SELECT list is a template string, so a column omitted against a fork-extended schema only surfaces at decode time. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An agent that asks a question and then waits is invisible until you happen to look at the sidebar. This adds an alert layer for the experimental sidebar: a tone the moment a session needs input, and a second, distinct tone once finished-but-unopened sessions pile up past a configurable threshold. Everything lives in a new fork-owned `apps/web/src/notifications/` module with three marked one-line seams upstream — the `__root.tsx` mount, one settings path/label, one nav icon — and is gated on EXPERIMENTAL_CONTROL_CENTER_ENABLED so bkmain can carry the code with the runner unmounted. It is browser-local by construction. The pile-up signal is `hasUnseenCompletion`, which reads `threadLastVisitedAtById` out of localStorage, so a server-side preference would describe a number the server cannot compute. Preferences, uploaded tones (IndexedDB), and the alert baseline are per-browser for the same reason. Mattermost escalation deliberately stays in t3-linear-bridge, which already polls every thread and owns delivery. Four runtime rules carry the design, and are documented in the fork guide because each one looks droppable and is not: - Alerts fire on transitions, never on state. A client with no baseline adopts the snapshot silently, and a thread that appears already waiting counts as pre-existing — otherwise every reconnect becomes an alarm. - Audio needs a user gesture, so the settings Test button doubles as the AudioContext unlock and the panel says when audio is still locked. - Native notifications only fire when the tab is hidden; the visible tab already shows the row highlight. - The pile-up alert is edge-triggered and re-arms only after the count drops. Built-in tones are oscillator specs rather than audio files, keeping binary assets out of a fork that merges from upstream weekly. Uploaded tones are validated, capped at 2 MB, and any event pointing at a deleted tone falls back to the default instead of silently muting. Also carries the pending sidebar countdown spacing fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two changes that both hinge on how execution state reaches the client. **Sessions stuck on "Running".** A finished agent kept rendering Running in the sidebar and the chat while the server had already recorded activity=idle and turn_state=completed. Execution is published over PubSub and never written to the event log, so neither resume path can carry it: subscribeShell and subscribeThread with an afterSequence replay domain events only, and withShellExecutions runs solely on the full-snapshot fallback. Upstream survives that because its shell reducer replaces the thread wholesale, so a stale overlay lives until the next projection upsert. The fork preserves the overlay across upserts (so a title edit stops wiping fresher execution), which turned a one-second flicker into a permanent lie: a client that missed the terminal frame — server restart, network blip, resumed tab — showed Running forever. Both resume paths now emit the supervisor's current execution after catch-up, reading in-memory state through the existing visibility filter. The reducer additionally drops a preserved overlay when the upsert's own latestTurn reports that overlay's turn finished; the contradiction travels in the same frame, so the client no longer depends on a live frame it may never receive. **Session titles.** Upstream titles a thread once, from its first prompt, so a long session keeps a name describing what it started as. Every N user prompts the turn-start path now dispatches an ordinary regenerateTitle meta update, reusing the durable regeneration flow rather than renaming behind its back: request ids, supersede checks, and interrupted-run recovery all still apply, and a failure to dispatch is logged instead of touching the turn. experimental.threadTitleMaintenance is on by default with a cadence of 3, is editable in Experiments, and 0 disables it. A refresh does replace a title you set by hand — nothing distinguishes a user title from a generated one — so the setting says so and the cadence is the escape hatch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`vp check` and the commit hook's `vp fmt` disagreed on how to wrap the new getSnapshot pipe, so CI rejected the formatting. No behaviour change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The resume stream now re-sends execution before the synchronization marker, so the catch-up bound test asserted the wrong sequence. Updated rather than relaxed: the frame's presence is the contract that stops a finished turn rendering as Running forever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
vouch:trusted
Adds T3-CUSTOM(expbkt3) markers to the four fork edits this merge work introduced into upstream-owned files, and drops the stale baseline entry for PlanSidebar.tsx, which upstream deleted in pingdotgg#5558. Uses BEGIN/END blocks rather than single-line markers: a single-line marker added alongside its subject becomes part of the same diff hunk, so it covers only itself and the check still reports the hunk as unmarked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the verified staging branch to production. 59 commits, fast-forward (
ahead_by: 59, behind_by: 0) — no conflicts possible.bkt3.dev currently stalls ~161 s on every deploy and writes ~25.6 GB/day. This fixes both, and brings in upstream's reconnect work that targets the disconnect reports directly.
What's included
Our fixes (fork-owned):
4318d24d01007+idx_orchestration_events_event_type_stream, moved off the readiness path675889ce3synchronous=NORMAL,wal_autocheckpoint=10000,cache_size=-65536,journal_size_limit66e2f03d8sql.executespans, retains slow ones6ce5fe93e.mcp.json(spawned an xcodebuildmcp process tree per worktree)Upstream
v0.0.32-nightly.20260807.1023(merged47fc1af1, 56 commits):1008— upstream's037collided with the fork's existing037_ThreadExecutions, so it takes the next free fork ID)Verified on expbkt3 (
588eeefce)Started 14:22:08→ready 14:22:12— ~4 s (bkt3: 161 s)ownership backfill completeline — skip logs at debug, absence is the pass condition1007applied,idx_orchestration_events_event_type_streampresentmaintenance_markers→ownership.admin-reassignment @ 2026-08-06T19:16:52.074Z, not rewritten on later bootssql.executeof 22,823 spans, all ≥250 ms (961/438/403/369 ms) — both drop and retain halves proven1008+idx_projection_turns_thread_keysetpresentExpected effect on bkt3.dev
Merge-time notes
t3-bkmain.serviceand ends sessions hosted on bkt3. Merge from a non-bkt3 session or a human shell.merge-base(expbkmain, upstream/main)=23f0a1ae3, whichv0.0.32-nightly.20260807.1023points at, sodeploy-bkt3.yml's--points-atlookup resolves.Caveats
e7f535ed5,88023e566,04941d4de,1985b66c0) landed on expbkmain after the currently deployed staging build, so they carry CI-green evidence but no runtime soak..mcp.jsonremoval only affects worktrees that fast-forward past6ce5fe93e; 41 existing worktrees still carry it and still spawn xcodebuildmcp.bk-docs/bk-docs-obs.codex/config.toml) or the provider process-tree termination leak.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.