LSP panics on restart due to stale LSPS2 payment queue entries (duplicate intercept_id assertion)

[f3r10]

Description:

Summary

After a clean restart of an ldk-node LSPS2 service node, the node panics when the next inbound HTLC arrives for a JIT channel. The panic is caused by a stale payment queue entry from the previous session whose intercept SCID is still being used by the client node.

Panic

thread 'tokio-rt-worker' panicked at
lightning-liquidity/src/lsps2/payment_queue.rs:36:13:
assertion failed: entry.htlcs.iter().all(|htlc| htlc.intercept_id != new_htlc.intercept_id)

This poisons the internal mutex, cascading into:

thread 'tokio-rt-worker' panicked at lightning-liquidity/src/lsps2/service.rs:1510:53:
called Result::unwrap() on an Err value: PoisonError { .. }

thread 'tokio-rt-worker' panicked at lightning-net-tokio/src/lib.rs:359:5:
assertion failed: e.is_cancelled()

After this the LSP is completely non-functional until restarted again (and the cycle repeats).

Root Cause

Two problems combine to produce this:

1. The LSPS2 client (node-service) persists and reuses the intercept SCID after restart. receive_via_jit_channel stores its LSPS2 client state (including the assigned intercept SCID) in lightning_liquidity_state in SQLite. On restart, when a new invoice is requested, the client presents the same intercept SCID again via a new LSPS2 buy request.

2. The LSPS2 service (lsp-service) persists its payment queue but never cleans up settled entries. The lightning_liquidity_state|lsps2_service|<peer_id> row survives restart with the old queue entry for the completed payment still present. When the same intercept SCID arrives again, payment_queue.rs:36 asserts uniqueness and panics.

[f3r10]

@tnull @TheBlueMatt I could work on this issue since I have been checking this part, if you like

[tnull]

Thanks for reporting. That panic is really a debug_assert (i.e., only run on non-production builds) that we added as a consistency check that shouldn't usually be reachable. @f3r10 Could it be that you previously had a persistence issue and/or didn't shutdown cleanly?

We'll see to make this logic a bit more robust. Note though that part of the report (in particular Root Cause seem to be made up by AI: the client never persists anything in lightning-liquidity/lightning_liquidity_state.

[f3r10]

Thanks @tnull, for checking this. You are right, the root cause is related to my project and not rust-lightning.
The problem was with some small LSPS2 channels that were not successfully created. After that, the issue with persistence happens. I am going to try to create an integration test so I can reproduce the error.