bug: out-of-order block processing causes SPV wallet to miss UTXO spends

[lklimek]

Summary

When the SPV block processing pipeline processes blocks out of height order, UTXO spends are silently missed. If a UTXO is created in block H1 and spent in block H2 (H2 > H1), but H2 is processed before H1, the wallet never removes the spent UTXO — it appears as permanently spendable.

Impact

• Stale UTXOs: The wallet retains UTXOs that were already spent on-chain, showing an inflated balance.
• Failed broadcasts: Transactions built from stale UTXOs are silently rejected by peers as double-spends. The wallet reports "broadcast successfully" but the tx never enters mempools or gets mined.
• Cascading failures: Any flow that depends on spendable balance after a payment hangs indefinitely.

Root Cause

Compact block filter matching (check_compact_filters_for_addresses in key-wallet-manager/src/matching.rs) correctly identifies blocks containing wallet-relevant transactions. However, blocks are downloaded and processed in non-deterministic order — the download pipeline doesn't enforce ascending height ordering.

Observed in trace logs

Block 1458841 creates a UTXO for address yYTQPB4A... (received 2.39 DASH):

10:30:50.117596 New wallet transaction detected txid=706a9619... context=block 1458841
                net_change=-30000226 received=239274892 sent=269275118

Block 1458904 spends that UTXO (sends to external address yV8Bj8Td...):

10:30:50.069353 New wallet transaction detected txid=c8f671e5... context=block 1458904
                net_change=0 received=0 sent=0

Block 1458904 was processed 48ms BEFORE block 1458841. When the spending tx was checked against the wallet, the UTXO didn't exist yet (sent=0). The creating block was processed later and added the UTXO, but the spend was already missed.

Secondary concern: filter matching completeness

The compact filter matching function only queries GCS filters with output scriptPubKeys. Per BIP158, the basic filter also contains prevout scriptPubKeys for non-coinbase inputs. If Dash Core follows this (which it appears to, since the spending block was downloaded), the filter matching works. However, check_compact_filters_for_addresses should ideally also include watched_outpoints() serialized as bytes for defense-in-depth, similar to how the mempool bloom filter (dash-spv/src/sync/mempool/filter.rs) already does.

Steps to Reproduce

1. Create a wallet and receive funds (UTXO created at height H1)
2. Spend the UTXO from a different environment — confirmed at height H2 > H1
3. Restart the SPV wallet and let it sync past H2
4. Both H1 and H2 match the compact filter (our address appears in both blocks)
5. If H2 is downloaded/processed before H1, the spend is missed
6. The wallet still shows the UTXO as spendable

Verified on Dash testnet:

• UTXO 706a9619...22bd6a45:1 — created at block 1458841
• Spent by c8f671e5...1225c5bb — confirmed at block 1458904 (528 confirmations)
• SPV processed block 1458904 at 10:30:50.069 and block 1458841 at 10:30:50.117 — 48ms out of order

Proposed Fix

Option A (block pipeline): Enforce ascending height order in the block download/processing pipeline. Blocks should be processed strictly in height order so that UTXOs are always created before they can be spent.

Option B (wallet checker): After processing all matched blocks in a batch, do a second pass to reconcile: for any UTXO that was created, check if a later block in the same batch spent it. The spent_outpoints tracking in ManagedAccount already handles some out-of-order cases (it skips UTXO creation if the outpoint is in spent_outpoints), but this relies on the spending tx being processed first, which sets sent=0 when the UTXO doesn't exist yet.

Option C (retroactive detection): When a creating transaction is processed and adds a UTXO, check if any already-processed transaction in spent_outpoints references this outpoint. If so, immediately remove the UTXO.

Option A is the cleanest but may have performance implications. Option C is the most targeted fix.

_{🤖 Co-authored by Claudius the Magnificent AI Agent}

[xdustinface]

Actually block processing is enforced to be ordered so this shouldn't be possible. What branch was this? Do you have the full log?

rust-dashcore/dash-spv/src/sync/blocks/pipeline.rs

Lines 149 to 157 in 383b306

	pub(super) fn take_next_ordered_block(&mut self) -> Option<(Block, u32)> {
	let lowest_downloaded = *self.downloaded.keys().next()?;
	// Check if any pending blocks have lower heights
	if let Some(&min_pending) = self.pending_heights.first() {
	if min_pending < lowest_downloaded {
	return None; // Wait for lower block
	}
	}

rust-dashcore/dash-spv/src/sync/blocks/manager.rs

Lines 78 to 89 in 383b306

	pub(super) async fn process_buffered_blocks(&mut self) -> SyncResult<Vec<SyncEvent>> {
	let mut events = Vec::new();
	// Process blocks in height order using pipeline's ordering logic
	while let Some((block, height)) = self.pipeline.take_next_ordered_block() {
	let hash = block.block_hash();
	// Process block through wallet
	let mut wallet = self.wallet.write().await;
	let result = wallet.process_block(&block, height).await;
	drop(wallet);

[lklimek]

As discussed, this is not a bug, in some cases the same blocks are re-processed, what looks like out-of-order processing in the logs.

[lklimek]

This recurs — now with a deterministic, sub-second repro, and evidence the described mitigation doesn't actually hold

Traced a fresh recurrence of exactly this defect from a live testnet failure in dash-evo-tool (AssetLockFinalityTimeout, blocking identity registration entirely): a real UTXO's spend (block 1,474,746) was processed before its own funding transaction (block 1,474,688 — chronologically earlier on-chain — processed 0.87s later, same committed batch). Same H1/H2-reversed pattern this issue already describes.

New deterministic repro

Pushed to this repo: key-wallet-manager/tests/out_of_order_spend_repro_test.rs (906fd47d, branch repro/pr3549-rdc). No mnemonic, no network, ~0.05s. It feeds a synthetic funding tx and spend tx through WalletManager::process_block_for_wallets in the exact reversed order observed live — spend at height 200, funding at height 100 — then asserts the funding outpoint isn't left in the tracked UTXO set:

cargo test -p key-wallet-manager --test out_of_order_spend_repro_test

Result: FAILS. This directly falsifies the assumption in this issue's own "Root Cause" section that spent_outpoints "already handles some out-of-order cases... relies on the spending tx being processed first" — we processed the spend first, exactly as described, and the funding output still ends up tracked as spendable. update_utxos's is_outpoint_spent guard (managed_core_funds_account.rs) exists specifically for this case but does not prevent it.

Real-world confirmation of impact

Independently verified via a ground-truth balance oracle (derive addresses directly from the wallet's xpub, bypass key-wallet's own state entirely, query a block explorer for real balances): a real testnet wallet affected by this defect reports 11.29× its actual balance (229.99 DASH reported vs. 20.37 DASH real). Two additional live-network tests on the same branch (dash-spv/tests/asset_lock_stale_utxo_repro_test.rs) reproduce the real-world symptom against actual testnet history for further context — those didn't hit the exact trigger in isolation on their own, which is what motivated tracing down to the precise mechanism above.

Given the severity (identity registration completely broken downstream, funds effectively unspendable without hitting a 300s FinalityTimeout) and that we now have a deterministic, sub-second, non-network repro rather than only a live timing race — would it be worth reopening this?

_{🤖 Co-authored by Claudius the Magnificent AI Agent}