Skip to content

fix(store): bound the WAL a killed session leaves behind - #184

Merged
sunerpy merged 1 commit into
mainfrom
fix/wal-bounded-after-killed-session
Aug 1, 2026
Merged

fix(store): bound the WAL a killed session leaves behind#184
sunerpy merged 1 commit into
mainfrom
fix/wal-bounded-after-killed-session

Conversation

@sunerpy

@sunerpy sunerpy commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Summary

Batch 3 of the upstream v1.5.0 sync, porting 02c0e2c. A session killed
mid-write leaves an uncheckpointed -wal sidecar, and SQLite keeps that file at
its high-water mark forever — passive checkpoints fold the frames back but never
shrink the file.

Measured on released 0.42.1: a 9 204 112-byte residue stayed at exactly that
size across 8 samples during a sync touching one line, and six interrupted
rounds ratcheted 5 702 112 → 6 439 592 → 6 760 952 bytes with no path that ever
reclaimed it.

journal_size_limit on both write-connection configuration points, resolved
from the same wal_valve_threshold_bytes() the active valve already uses, so the
next WAL-resetting write clips the file. One concept, one env var:
CODEGRAPH_WAL_VALVE_MB drives the pragma, the valve, and the status warning.
The rebuild path re-validates its exclusive lease before the new pragma exactly
as it does for the others. Threshold resolution moved to a pure resolver with
checked_mul plus an i64 bound, so an oversized override falls back instead of
overflowing SQLite's signed integer.

sync and index fold a leftover WAL before their ordinary writer opens the
database.
This is deliberately not upstream's fire-and-forget from open():
our fold needs proof the previous daemon owner is dead AND a bounded exclusive
lease, so it stays synchronous, stays on the mutation entry points, and reports a
heal only when the fold actually completed.

status reports the WAL. A leftover sidecar blocks the strict Current read
gate, so rather than querying rows it cannot corroborate, status degrades to a
read-only diagnostic: the typed refusal, the paths, and the DB and WAL sizes,
without counts or journalMode. It never checkpoints or deletes anything.
walSizeBytes appears only when a WAL exists, so a healthy index's JSON key set
is unchanged.

Upstream also extended a liveness watchdog and stamped daemon.log lines; neither
applies here. Our only watchdog watches the parent pid and holds no database
handle, and every tracing sink already carries an RFC3339 timestamp.

Verification

  • make ci green; cargo test --workspace 128 suites / 3121 passed / 0 failed.
  • Each fix RED-proven by isolated revert: removing the pragma from both
    configuration points fails all three threshold tests (left: -1, right: 268435456 / 1048576, and the first resetting write must clip the WAL to 1048576 bytes, got 3724512); removing the two pre-heal calls makes both
    lease-barrier tests time out, because only one exclusive acquisition happens
    without it; removing the status degradation arm fails with Current index state has an unexpected SQLite sidecar.
  • The clip test is written around a measured SQLite subtlety:
    journal_size_limit does not clip at wal_checkpoint(RESTART) — RESTART
    only establishes the reset point and the next write performs the clip. With no
    cap the sequence is 3518512 → 3518512 → 3518512; with a 1 MiB cap it is
    3518512 → 3518512 → 1048576, 401 rows readable either way. TRUNCATE is
    deliberately not used there since it zeroes the file regardless of the cap.
  • Hands-on QA against a rebuilt release binary: status --json on a blocked
    namespace exits 0 with walSizeBytes matching the file exactly, all four
    uncorroborable fields absent, and the WAL byte-identical afterward;
    RUST_LOG=info sync logs the fold before Scanning files…, ends sidecar-free,
    and a main-file-only copy then reads the previously WAL-only rows. A controlled
    A/B separated "declined correctly" from "never attempted": with no contention
    the fold count is 1, and with an external flock -x on the lease it is 0 with
    no false healed claim. Two further interrupt rounds each produced a 3 530 872-byte
    residue and each was cleared — the ratchet is gone.
  • The three threshold tests run in child processes rather than mutating the
    environment of a multi-threaded test binary: making Store::open read the
    variable widened the read surface past what a test-local mutex can guard.
    Verified both directions — with the child's env_remove in place a parent
    CODEGRAPH_WAL_VALVE_MB=7 still yields 268435456; removing it yields
    7340032.
  • Golden-neutral by design: no extraction change, git status --porcelain reference/ empty, CURRENT_EXTRACTION_VERSION still 2. Batch 4 is the round's
    only golden-moving item and lands separately.

Plan approved after four momus rounds; gates F1–F4 all APPROVE (F2 after two).

A session killed mid-write leaves an uncheckpointed `-wal` sidecar, and SQLite
keeps that file at its high-water mark forever: passive checkpoints fold the
frames back but never shrink the file. Measured on 0.42.1, a 9 204 112-byte
residue stayed at exactly that size through a sync touching one line, and six
interrupted rounds ratcheted 5 702 112 -> 6 439 592 -> 6 760 952 bytes with no
path that ever reclaimed it.

- `journal_size_limit` is now set on both write-connection configuration points,
  from the same `wal_valve_threshold_bytes()` the active valve already uses, so
  the next WAL-resetting write clips the file instead of preserving the mark.
  One concept, one env var: `CODEGRAPH_WAL_VALVE_MB` drives the pragma, the
  valve, and the status warning. The rebuild path re-validates its exclusive
  lease before the new pragma exactly as it does for the others, and threshold
  resolution moved to a pure resolver with `checked_mul` plus an `i64` bound so
  an oversized override falls back rather than overflowing SQLite.
- `sync` and `index` now fold a leftover WAL before their ordinary writer opens
  the database. This is not upstream's fire-and-forget-from-`open()`: our fold
  needs proof the previous daemon owner is dead AND a bounded exclusive lease,
  so it stays synchronous, stays on the mutation entry points, and reports a
  heal only when the fold actually completed. A live or contended owner is
  never folded underneath.
- `status` reports the WAL. A leftover sidecar blocks the strict Current read
  gate, so instead of querying rows it cannot corroborate, `status` degrades to
  a read-only diagnostic: the typed refusal, the paths, and the DB and WAL
  sizes, without counts or `journalMode`. It never checkpoints or deletes
  anything. `walSizeBytes` appears only when a WAL exists, so a healthy index's
  JSON keys are unchanged, and a WAL larger than both the database and the
  configured limit prints how to recover safely.

Upstream also extended a liveness watchdog and stamped daemon.log lines; neither
applies here. Our only watchdog watches the parent pid and holds no database
handle, and every tracing sink already carries an RFC3339 timestamp.

The three threshold tests run in child processes rather than mutating the
environment of a multi-threaded test binary: making `Store::open` read the
variable widened the read surface past what a test-local mutex can guard.
@codecov

codecov Bot commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.59459% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/codegraph-cli/src/main.rs 93.54% 4 Missing ⚠️
crates/codegraph-store/src/queries.rs 91.66% 3 Missing ⚠️
crates/codegraph-store/src/connection.rs 97.95% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (94.59%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #184      +/-   ##
==========================================
+ Coverage   95.12%   95.13%   +0.01%     
==========================================
  Files         136      136              
  Lines       71294    71419     +125     
==========================================
+ Hits        67818    67947     +129     
+ Misses       3476     3472       -4     
Files with missing lines Coverage Δ
crates/codegraph-store/src/rebuild.rs 92.70% <100.00%> (ø)
crates/codegraph-store/src/connection.rs 87.14% <97.95%> (+1.14%) ⬆️
crates/codegraph-store/src/queries.rs 98.31% <91.66%> (+0.11%) ⬆️
crates/codegraph-cli/src/main.rs 90.13% <93.54%> (+0.12%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sunerpy
sunerpy merged commit 09a4b4e into main Aug 1, 2026
7 checks passed
@sunerpy
sunerpy deleted the fix/wal-bounded-after-killed-session branch August 1, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant