feat(p2p): slot-based soft deletion for TxPoolV2

yarn-project/p2p/src/mem_pools/tx_pool_v2/README.md

@@ -30,19 +30,20 @@ TxPoolV2 manages transactions through a state machine with clear transitions:
 └─────────────────────────────────────┘  (reorg)
         │                                     │
         │ handleFinalizedBlock()              │ eviction after reorg
+        │ / eviction / failed exec            │ (validation failure)
         ▼                                     ▼
 ┌─────────────────────────────────────┐  ┌─────────────────────────────────────┐
-│             DELETED                 │  │         SOFT-DELETED                │
-│   (hard-deleted or archived)        │  │   (kept in DB for debugging)        │
+│       SLOT-SOFT-DELETED             │  │       PRUNE-SOFT-DELETED            │
+│  (kept in DB until next slot)       │  │  (kept in DB until finalized)       │
+└─────────────────────────────────────┘  └─────────────────────────────────────┘
+        │                                        │
+        │ prepareForSlot()                       │ handleFinalizedBlock()
+        │ (slot advanced)                        │ (mined block finalized)
+        ▼                                        ▼
+┌─────────────────────────────────────┐  ┌─────────────────────────────────────┐
+│          HARD-DELETED               │  │          HARD-DELETED               │
+│   (permanently removed from DB)     │  │   (permanently removed from DB)     │
 └─────────────────────────────────────┘  └─────────────────────────────────────┘
-                                                  │
-                                                  │ handleFinalizedBlock()
-                                                  │ (mined block finalized)
-                                                  ▼
-                                         ┌─────────────────────────────────────┐
-                                         │          HARD-DELETED               │
-                                         │   (permanently removed from DB)     │
-                                         └─────────────────────────────────────┘
 ```
 
 ## Key Components
@@ -62,13 +63,11 @@ Core implementation containing:
 
 ### DeletedPool (`deleted_pool.ts`)
 
-Manages soft deletion of transactions from pruned blocks:
-- When a reorg (chain prune) occurs, transactions from pruned blocks are tracked with their original mined block number
-- When these transactions are later evicted (e.g., failed validation, nullifier conflict), they are "soft-deleted" instead of removed
-- Soft-deleted transactions remain in the database for debugging and potential resubmission
-- When the original mined block is finalized on the new chain, soft-deleted transactions are permanently hard-deleted
+Manages all transaction deletions in the pool with two soft-deletion mechanisms:
+- **Slot-based**: Non-pruned txs are kept in DB until the next slot, allowing other nodes to fetch them via reqresp
+- **Prune-based**: Txs from pruned blocks are kept in DB until their original mined block is finalized
 
-This ensures transactions from reorged blocks are kept around until we're certain they won't be needed.
+All deletions go through `DeletedPool.deleteTx()`, which routes to the appropriate path based on whether the tx is tracked as being from a pruned block.
 
 ### TxMetaData (`tx_metadata.ts`)
 
@@ -86,27 +85,40 @@ Lightweight metadata stored alongside each transaction:
 State is derived by TxPoolIndices:
 - `mined` if `minedL2BlockId` is set
 - `protected` if in protection map
-- `deleted` if soft-deleted (from a pruned block, evicted but kept in DB)
+- `deleted` if soft-deleted (slot-based or prune-based, evicted but kept in DB)
 - `pending` otherwise
 
 ## Soft Deletion
 
-When a chain reorganization occurs, transactions that were mined in pruned blocks are handled specially:
+Deleted transactions are kept in the database for a grace period before being permanently removed. There are two soft-deletion mechanisms:
+
+### Slot-Based Soft Deletion
+
+When a transaction is deleted from the pool (eviction, validation failure, failed execution) and is **not** from a pruned block, it is "slot-soft-deleted":
+
+1. **Soft Delete**: The tx is removed from indices but kept in the database, tagged with the current slot number
+2. **Retrieval**: Slot-soft-deleted txs can still be retrieved via `getTxByHash` and return status `'deleted'` from `getTxStatus`
+3. **Hard Delete**: When `prepareForSlot` advances to a new slot, txs deleted in earlier slots are permanently removed
+4. **Re-addition**: If a slot-soft-deleted tx is re-added to the pool, the slot-deleted tracking is cleared
+
+This allows other nodes to still fetch recently-deleted transactions via reqresp during the current slot.
+
+### Prune-Based Soft Deletion
+
+When a chain reorganization occurs, transactions that were mined in pruned blocks are handled with longer retention:
 
 1. **Tracking**: When `handlePrunedBlocks` is called, all un-mined transactions are tracked by their original mined block number
-2. **Soft Delete**: If these transactions are later evicted (failed validation, nullifier conflict, etc.), they are "soft-deleted" - removed from indices but kept in the database
-3. **Retrieval**: Soft-deleted transactions can still be retrieved via `getTxByHash` and `hasTxs`, and return status `'deleted'` from `getTxStatus`
+2. **Soft Delete**: If these transactions are later evicted (failed validation, nullifier conflict, etc.), they are "prune-soft-deleted" - removed from indices but kept in the database
+3. **Retrieval**: Prune-soft-deleted txs can still be retrieved via `getTxByHash` and return status `'deleted'` from `getTxStatus`
 4. **Hard Delete**: When `handleFinalizedBlock` is called and the finalized block number reaches or exceeds the transaction's original mined block, the transaction is permanently removed
+5. **Re-addition**: If a prune-soft-deleted tx is re-added, the `softDeleted` flag is reset to `false` but the prune tracking is preserved, so a subsequent deletion still uses the prune path
 
-This design allows:
-- Debugging reorg scenarios by keeping transaction data available
-- Potential resubmission of transactions that failed validation after a reorg
-- Clean eventual cleanup once we're certain the transaction won't be needed
+Prune-soft-deleted transactions are **not** affected by slot cleanup - they survive across slot boundaries until finalized.
 
-**Example scenario:**
+**Prune example:**
 1. Tx mined at block 10
 2. Chain prunes to block 5 (tx becomes un-mined, tracked as minedAtBlock=10)
-3. Tx fails validation and is soft-deleted
+3. Tx fails validation and is prune-soft-deleted
 4. Block 9 finalized → tx still in DB (minedAtBlock=10 > finalized=9)
 5. Block 10 finalized → tx hard-deleted (minedAtBlock=10 ≤ finalized=10)
 
@@ -117,6 +129,10 @@ If the tx is re-mined at a higher block before being soft-deleted:
 4. Block 10 finalized → tx still in DB
 5. Block 15 finalized → tx hard-deleted
 
+### Hydration
+
+On node restart, slot-soft-deleted transactions are immediately hard-deleted (they are stale by definition). Prune-soft-deleted transactions are loaded from the database and tracked normally.
+
 ## Architecture: Pre-add vs Post-event Rules
 
 **Pre-add rules** (run during `addPendingTxs`):

yarn-project/p2p/src/mem_pools/tx_pool_v2/deleted_pool.test.ts

@@ -1,4 +1,4 @@
-import { BlockNumber } from '@aztec/foundation/branded-types';
+import { BlockNumber, SlotNumber } from '@aztec/foundation/branded-types';
 import { createLogger } from '@aztec/foundation/log';
 import type { AztecAsyncMap } from '@aztec/kv-store';
 import { openTmpStore } from '@aztec/kv-store/lmdb-v2';
@@ -120,27 +120,25 @@ describe('DeletedPool', () => {
   });
 
   describe('deleteTx', () => {
-    it('soft-deletes tx from pruned block (keeps in DB)', async () => {
+    it('prune-soft-deletes tx from pruned block (keeps in DB)', async () => {
       await txsDB.set('tx1', Buffer.from('data1'));
       await pool.markFromPrunedBlock([{ txHash: 'tx1', minedAtBlock: BlockNumber(5) }]);
       expect(pool.isSoftDeleted('tx1')).toBe(false);
 
-      const result = await pool.deleteTx('tx1');
+      await pool.deleteTx('tx1');
 
-      expect(result).toBe('soft');
       expect(pool.isSoftDeleted('tx1')).toBe(true);
       expect(await txsDB.getAsync('tx1')).toBeDefined(); // Still in DB
     });
 
-    it('hard-deletes tx NOT from pruned block (removes from DB)', async () => {
+    it('slot-soft-deletes tx NOT from pruned block (keeps in DB)', async () => {
       await txsDB.set('tx1', Buffer.from('data1'));
       // tx1 is NOT marked as from pruned block
 
-      const result = await pool.deleteTx('tx1');
+      await pool.deleteTx('tx1');
 
-      expect(result).toBe('hard');
-      expect(pool.isSoftDeleted('tx1')).toBe(false);
-      expect(await txsDB.getAsync('tx1')).toBeUndefined(); // Removed from DB
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
     });
   });
 
@@ -208,8 +206,7 @@ describe('DeletedPool', () => {
       expect(pool.isFromPrunedBlock('tx1')).toBe(true);
 
       // 3. Soft-delete the tx (e.g., due to eviction)
-      const result = await pool.deleteTx('tx1');
-      expect(result).toBe('soft');
+      await pool.deleteTx('tx1');
       expect(pool.isSoftDeleted('tx1')).toBe(true);
       expect(await txsDB.getAsync('tx1')).toBeDefined(); // Still in DB
 
@@ -372,6 +369,158 @@ describe('DeletedPool', () => {
     });
   });
 
+  describe('slot-based soft deletion', () => {
+    it('slot-soft-deletes tx NOT from pruned block (stays in DB, isSoftDeleted true)', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+
+      await pool.deleteTx('tx1');
+
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
+    });
+
+    it('prune-soft-delete takes precedence over slot-soft-delete', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await pool.markFromPrunedBlock([{ txHash: 'tx1', minedAtBlock: BlockNumber(5) }]);
+
+      await pool.deleteTx('tx1');
+
+      // Should use prune path, not slot path
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(pool.isFromPrunedBlock('tx1')).toBe(true);
+    });
+
+    it('cleanupSlotDeleted hard-deletes txs from earlier slots', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await txsDB.set('tx2', Buffer.from('data2'));
+
+      // Delete both txs at slot 0 (default)
+      await pool.deleteTx('tx1');
+      await pool.deleteTx('tx2');
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(pool.isSoftDeleted('tx2')).toBe(true);
+
+      // Advance to slot 1 - should hard-delete both (deleted at slot 0 < 1)
+      await pool.cleanupSlotDeleted(SlotNumber(1));
+
+      expect(pool.isSoftDeleted('tx1')).toBe(false);
+      expect(pool.isSoftDeleted('tx2')).toBe(false);
+      expect(await txsDB.getAsync('tx1')).toBeUndefined();
+      expect(await txsDB.getAsync('tx2')).toBeUndefined();
+    });
+
+    it('cleanupSlotDeleted preserves txs from current slot', async () => {
+      // Advance to slot 5 first
+      await pool.cleanupSlotDeleted(SlotNumber(5));
+
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await pool.deleteTx('tx1'); // Deleted at slot 5
+
+      // Cleanup at slot 5 - should NOT hard-delete (deleted at 5, not < 5)
+      await pool.cleanupSlotDeleted(SlotNumber(5));
+
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
+
+      // Cleanup at slot 6 - NOW should hard-delete (deleted at 5 < 6)
+      await pool.cleanupSlotDeleted(SlotNumber(6));
+
+      expect(pool.isSoftDeleted('tx1')).toBe(false);
+      expect(await txsDB.getAsync('tx1')).toBeUndefined();
+    });
+
+    it('cleanupSlotDeleted does NOT affect prune-soft-deleted txs', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await pool.markFromPrunedBlock([{ txHash: 'tx1', minedAtBlock: BlockNumber(5) }]);
+      await pool.deleteTx('tx1'); // Prune-soft-deleted
+
+      // Advance to slot 10 - should NOT affect prune-soft-deleted tx
+      await pool.cleanupSlotDeleted(SlotNumber(10));
+
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
+      expect(pool.isFromPrunedBlock('tx1')).toBe(true);
+    });
+
+    it('clearSoftDeleted removes tx from slot-deleted tracking', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await pool.deleteTx('tx1');
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+
+      await pool.clearSoftDeleted('tx1');
+
+      expect(pool.isSoftDeleted('tx1')).toBe(false);
+      // Tx is still in DB (clearSoftDeleted doesn't delete the tx data)
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
+    });
+
+    it('clearSoftDeleted resets prune softDeleted flag but preserves prune tracking', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await pool.markFromPrunedBlock([{ txHash: 'tx1', minedAtBlock: BlockNumber(5) }]);
+      await pool.deleteTx('tx1');
+      expect(pool.isSoftDeleted('tx1')).toBe(true);
+      expect(pool.isFromPrunedBlock('tx1')).toBe(true);
+
+      await pool.clearSoftDeleted('tx1');
+
+      // No longer soft-deleted, but still tracked as from pruned block
+      expect(pool.isSoftDeleted('tx1')).toBe(false);
+      expect(pool.isFromPrunedBlock('tx1')).toBe(true);
+    });
+
+    it('clearSoftDeleted is a no-op for non-soft-deleted txs', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+
+      // Should not throw or error
+      await pool.clearSoftDeleted('tx1');
+      expect(pool.isSoftDeleted('tx1')).toBe(false);
+    });
+
+    it('hydration hard-deletes all slot-deleted txs', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await txsDB.set('tx2', Buffer.from('data2'));
+
+      // Slot-delete both
+      await pool.deleteTx('tx1');
+      await pool.deleteTx('tx2');
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
+      expect(await txsDB.getAsync('tx2')).toBeDefined();
+
+      // Simulate restart
+      const pool2 = new DeletedPool(store, txsDB, createLogger('test2'));
+      await pool2.hydrateFromDatabase();
+
+      // Both should be hard-deleted after hydration
+      expect(await txsDB.getAsync('tx1')).toBeUndefined();
+      expect(await txsDB.getAsync('tx2')).toBeUndefined();
+      expect(pool2.isSoftDeleted('tx1')).toBe(false);
+      expect(pool2.isSoftDeleted('tx2')).toBe(false);
+    });
+
+    it('hydration does not affect prune-soft-deleted txs', async () => {
+      await txsDB.set('tx1', Buffer.from('data1'));
+      await txsDB.set('tx2', Buffer.from('data2'));
+
+      // Prune-soft-delete tx1, slot-delete tx2
+      await pool.markFromPrunedBlock([{ txHash: 'tx1', minedAtBlock: BlockNumber(5) }]);
+      await pool.deleteTx('tx1');
+      await pool.deleteTx('tx2');
+
+      // Simulate restart
+      const pool2 = new DeletedPool(store, txsDB, createLogger('test2'));
+      await pool2.hydrateFromDatabase();
+
+      // tx1 (prune-soft-deleted) should still be in DB
+      expect(await txsDB.getAsync('tx1')).toBeDefined();
+      expect(pool2.isSoftDeleted('tx1')).toBe(true);
+      expect(pool2.isFromPrunedBlock('tx1')).toBe(true);
+
+      // tx2 (slot-deleted) should be hard-deleted
+      expect(await txsDB.getAsync('tx2')).toBeUndefined();
+      expect(pool2.isSoftDeleted('tx2')).toBe(false);
+    });
+  });
+
   describe('getCount and getPrunedTxHashes', () => {
     it('returns correct count', async () => {
       expect(pool.getCount()).toBe(0);