fix(store): bound the WAL a killed session leaves behind - #184
Merged
Conversation
A session killed mid-write leaves an uncheckpointed `-wal` sidecar, and SQLite keeps that file at its high-water mark forever: passive checkpoints fold the frames back but never shrink the file. Measured on 0.42.1, a 9 204 112-byte residue stayed at exactly that size through a sync touching one line, and six interrupted rounds ratcheted 5 702 112 -> 6 439 592 -> 6 760 952 bytes with no path that ever reclaimed it. - `journal_size_limit` is now set on both write-connection configuration points, from the same `wal_valve_threshold_bytes()` the active valve already uses, so the next WAL-resetting write clips the file instead of preserving the mark. One concept, one env var: `CODEGRAPH_WAL_VALVE_MB` drives the pragma, the valve, and the status warning. The rebuild path re-validates its exclusive lease before the new pragma exactly as it does for the others, and threshold resolution moved to a pure resolver with `checked_mul` plus an `i64` bound so an oversized override falls back rather than overflowing SQLite. - `sync` and `index` now fold a leftover WAL before their ordinary writer opens the database. This is not upstream's fire-and-forget-from-`open()`: our fold needs proof the previous daemon owner is dead AND a bounded exclusive lease, so it stays synchronous, stays on the mutation entry points, and reports a heal only when the fold actually completed. A live or contended owner is never folded underneath. - `status` reports the WAL. A leftover sidecar blocks the strict Current read gate, so instead of querying rows it cannot corroborate, `status` degrades to a read-only diagnostic: the typed refusal, the paths, and the DB and WAL sizes, without counts or `journalMode`. It never checkpoints or deletes anything. `walSizeBytes` appears only when a WAL exists, so a healthy index's JSON keys are unchanged, and a WAL larger than both the database and the configured limit prints how to recover safely. Upstream also extended a liveness watchdog and stamped daemon.log lines; neither applies here. Our only watchdog watches the parent pid and holds no database handle, and every tracing sink already carries an RFC3339 timestamp. The three threshold tests run in child processes rather than mutating the environment of a multi-threaded test binary: making `Store::open` read the variable widened the read surface past what a test-local mutex can guard.
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (94.59%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage. @@ Coverage Diff @@
## main #184 +/- ##
==========================================
+ Coverage 95.12% 95.13% +0.01%
==========================================
Files 136 136
Lines 71294 71419 +125
==========================================
+ Hits 67818 67947 +129
+ Misses 3476 3472 -4
... and 1 file with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Batch 3 of the upstream v1.5.0 sync, porting
02c0e2c. A session killedmid-write leaves an uncheckpointed
-walsidecar, and SQLite keeps that file atits high-water mark forever — passive checkpoints fold the frames back but never
shrink the file.
Measured on released
0.42.1: a 9 204 112-byte residue stayed at exactly thatsize across 8 samples during a sync touching one line, and six interrupted
rounds ratcheted 5 702 112 → 6 439 592 → 6 760 952 bytes with no path that ever
reclaimed it.
journal_size_limiton both write-connection configuration points, resolvedfrom the same
wal_valve_threshold_bytes()the active valve already uses, so thenext WAL-resetting write clips the file. One concept, one env var:
CODEGRAPH_WAL_VALVE_MBdrives the pragma, the valve, and the status warning.The rebuild path re-validates its exclusive lease before the new pragma exactly
as it does for the others. Threshold resolution moved to a pure resolver with
checked_mulplus ani64bound, so an oversized override falls back instead ofoverflowing SQLite's signed integer.
syncandindexfold a leftover WAL before their ordinary writer opens thedatabase. This is deliberately not upstream's fire-and-forget from
open():our fold needs proof the previous daemon owner is dead AND a bounded exclusive
lease, so it stays synchronous, stays on the mutation entry points, and reports a
heal only when the fold actually completed.
statusreports the WAL. A leftover sidecar blocks the strictCurrentreadgate, so rather than querying rows it cannot corroborate,
statusdegrades to aread-only diagnostic: the typed refusal, the paths, and the DB and WAL sizes,
without counts or
journalMode. It never checkpoints or deletes anything.walSizeBytesappears only when a WAL exists, so a healthy index's JSON key setis unchanged.
Upstream also extended a liveness watchdog and stamped daemon.log lines; neither
applies here. Our only watchdog watches the parent pid and holds no database
handle, and every tracing sink already carries an RFC3339 timestamp.
Verification
make cigreen;cargo test --workspace128 suites / 3121 passed / 0 failed.configuration points fails all three threshold tests (
left: -1, right: 268435456/1048576, andthe first resetting write must clip the WAL to 1048576 bytes, got 3724512); removing the two pre-heal calls makes bothlease-barrier tests time out, because only one exclusive acquisition happens
without it; removing the status degradation arm fails with
Current index state has an unexpected SQLite sidecar.journal_size_limitdoes not clip atwal_checkpoint(RESTART)— RESTARTonly establishes the reset point and the next write performs the clip. With no
cap the sequence is
3518512 → 3518512 → 3518512; with a 1 MiB cap it is3518512 → 3518512 → 1048576, 401 rows readable either way.TRUNCATEisdeliberately not used there since it zeroes the file regardless of the cap.
status --jsonon a blockednamespace exits 0 with
walSizeBytesmatching the file exactly, all fouruncorroborable fields absent, and the WAL byte-identical afterward;
RUST_LOG=info synclogs the fold beforeScanning files…, ends sidecar-free,and a main-file-only copy then reads the previously WAL-only rows. A controlled
A/B separated "declined correctly" from "never attempted": with no contention
the fold count is 1, and with an external
flock -xon the lease it is 0 withno false healed claim. Two further interrupt rounds each produced a 3 530 872-byte
residue and each was cleared — the ratchet is gone.
environment of a multi-threaded test binary: making
Store::openread thevariable widened the read surface past what a test-local mutex can guard.
Verified both directions — with the child's
env_removein place a parentCODEGRAPH_WAL_VALVE_MB=7still yields268435456; removing it yields7340032.git status --porcelain reference/empty,CURRENT_EXTRACTION_VERSIONstill 2. Batch 4 is the round'sonly golden-moving item and lands separately.
Plan approved after four momus rounds; gates F1–F4 all APPROVE (F2 after two).