refactor(dash-spv): network manager refactor and sync pipelines optimized #902
Draft
ZocoLini wants to merge 22 commits into
Draft
refactor(dash-spv): network manager refactor and sync pipelines optimized #902ZocoLini wants to merge 22 commits into
ZocoLini wants to merge 22 commits into
Conversation
Resolved conflicts:
- client/mod.rs: restored dev's two client tests
(birth_height_anchors_chain_to_nearest_checkpoint, client_exposes_shared_wallet_manager),
adapted to the new DashSpvClient::new signature (network manager no longer passed in).
- sync/filters/manager.rs:
- start_download: kept our frontier-aware scan_start computation AND dev's
stored_covers_scan guard (#893, discard stored filters that don't reach scan_start);
they are sequential concerns, not alternatives.
- tests: dropped dev's 46-test `mod tests`. It is built on the pre-refactor
network flow (RequestSender/NetworkRequest, removed here) and the old 4-arg
FiltersManager::new, so it no longer compiles against this branch. Our
scan_state_tests are kept; porting the salvageable subset is left as follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… the sync checkpoint The merge of dev brought in the checkpoint floor (a birth-0 HD wallet now anchors mainnet sync at the 200000 checkpoint instead of genesis, since no HD wallet has transactions before it). That exposed a latent bug in the filters manager's stale-wallet check: it compared each wallet's raw `synced_height` against `committed_height`. A fresh wallet reports `synced_height = 0`, but a checkpoint sync sets `committed_height = checkpoint - 1` (199999), so the wallet read as perpetually "behind" — every tick reset and restarted the scan, an infinite loop that starved block/filter-header processing and stalled the whole sync at ~12%. No filters exist below the sync floor, so a wallet cannot be behind there: clamp its effective scanned baseline to `header_start_height` before the comparison. A genuinely late-added wallet (synced within the synced range) still triggers a rescan; a fresh below-floor wallet no longer does. Verified on a real mainnet bench: headers and filter headers now reach 100% (were stuck at ~12%), zero "restarting scan" loops. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…in-memory checkpoint sync Second checkpoint-sync stall (after the rescan loop): the filters phase spun without ever committing or matching — committed stuck at the checkpoint, downloaded/matched 0 — because verifying the first filter batch failed with "Missing filter header at height <checkpoint-1>". Filter-header sync anchors the chain differently per mode. The storage path stores the anchor (`cfheaders.previous_filter_header`) at `header_start - 1`, so filter bodies verify from `header_start`. The in-memory path (which runs on a fresh checkpoint sync) anchors it at `header_start` itself, so filter_header [header_start - 1] never exists — yet the filters manager still tried to download and verify a body at `header_start`, which needs exactly that missing header. Raise the filters download/scan floor to `filter_header_start + 1` when the anchor sits above genesis, so the first body downloaded is the lowest one whose predecessor filter header is actually known. No-op for genesis (anchor at 0, predecessor is the all-zeros sentinel) and for the storage/resume path (anchor at header_start - 1, floor already equals header_start). Verified on a real mainnet checkpoint sync: zero "Missing filter header" errors, filters commit and match (blocks download with relevant transactions), phases reach 100%. Integration: lib 184/184, dashd basic 3/3, restart 4/4. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev #902 +/- ##
==========================================
- Coverage 74.54% 72.61% -1.94%
==========================================
Files 327 319 -8
Lines 75032 69270 -5762
==========================================
- Hits 55936 50297 -5639
+ Misses 19096 18973 -123
|
Intermittently a mainnet sync stalled at the very start of header sync with only the first segment stored. Cause: at startup the reconnector races the probe — it connects peers from the discoverer and the sync sends them the initial per-segment GetHeaders, then `probe_and_select` settles the final peer set and replaced `connected` wholesale, dropping those peers mid-request. The stranded requests only recovered on the pipelines' own slow timeout, and occasionally a run never remounted (the surviving peers were slow/quiet), stalling all segments above the first. Retire displaced peers by draining instead of dropping: if a peer has requests in flight, keep its connection alive in the background (its reader keeps delivering responses and decrementing `in_flight`) and close it only once it has drained, capped at STALE_REQUEST so a peer that never drains is still reaped. A peer with nothing in flight is closed immediately as before. Bonus: previously these dropped peers leaked their reader tasks until global shutdown; now they close cleanly. Verified: 8/8 back-to-back mainnet syncs completed with zero stalls (a prior run of 5 stalled 1); the one run that hit the churn scenario recovered via the drain+retry instead of hanging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed number_prefix cargo-deny fails the branch: indicatif 0.17.11 pulls number_prefix 0.4.0, flagged unmaintained (RUSTSEC-2025-0119). indicatif 0.18 replaced number_prefix with unit-prefix, so the bump removes the advisory (and one duplicate windows-sys) from the tree. Drop-in — the bench only uses stable MultiProgress/ProgressBar/ ProgressStyle APIs. (Cargo.lock is gitignored, so CI resolves this fresh.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… doc
`cargo doc` with `-D warnings` failed: the public trait method
`SyncManager::handle_network_event` doc-linked `default_handle_network_event`,
which is `pub(super)` — a private-intra-doc-link, denied by
`rustdoc::private_intra_doc_links`.
The delegation advice ("override and defer the rest to the default body; Rust
can't call a trait's default body from an override") is guidance for the crate's
own sync-manager implementors, not the public API — an external reader can't call
the `pub(super)` helper anyway. So drop it from the public method doc, which now
just describes behavior, and move the full note onto `default_handle_network_event`
itself (a private item, only rendered under `--document-private-items`, where the
link resolves cleanly). `default_handle_network_event` stays private — exposing it
publicly just to satisfy a link would grow the API surface for no external benefit.
Verified: `RUSTDOCFLAGS="-D warnings" cargo doc -p dash-spv --all-features --no-deps` clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incredible big PR that does many things to get to a point where the sync time is as low as possible, I will be working on refactoring and cleaning the introduced code in the next PRs but since we need a fast sync for a good user experience at the iOS release I would like to get this merge as long as it can sync