renepay trying to send more than spendable_msat

[daywalker90]

so i merged the pr #6593 locally and i noticed something in the new output.

#   Flow 385: amount=23528000msat prob=0.000 fees=1023msat delay=140 path=-2471854x37x4/1(min=max=23528783msat)->-2414928x98x0/0->
#   Flow 386: amount=43625000msat prob=0.000 fees=1439msat delay=216 path=-2471854x37x13/1(min=max=43625433msat)->-2164740x20x0/0->-2136738x42x1/0->-2163293x89x1/1->-2254184x16x1/1->-2418354x44x1/1->-2418546x65x1/0->
#   Flow 387: amount=197460000msat prob=0.000 fees=14019msat delay=232 path=-2471854x37x5/1(min=max=197460000msat)->-2428790x34x2/0->-2420997x62x2/1->-2418354x44x1/1(43625043msat in 1 htlcs)->-2418546x65x1/0(43625000msat in 1 htlcs)->
#   Flow 388: amount=197460000msat prob=0.000 fees=14019msat delay=232 path=-2471854x37x15/1(min=max=197460000msat)->-2428790x34x4/0->-2420997x62x2/1(197462171msat in 1 htlcs)->-2418354x44x1/1(241085240msat in 2 htlcs)->-2418546x65x1/0(241085000msat in 2 htlcs)->
#   Flow 389: amount=197460000msat prob=0.000 fees=4343msat delay=158 path=-2471854x37x2/0(min=max=197460000msat)->-2467895x1x1/1->-2254184x16x1/1(43625479msat in 1 htlcs)->-2418354x44x1/1(438545437msat in 3 htlcs)->-2418546x65x1/0(438545000msat in 3 htlcs)->
#   Flow 390: amount=187457000msat prob=0.000 fees=5998msat delay=176 path=-2471854x37x7/1(min=max=187457878msat)->-2346806x7x0/1->-2191990x40x0/1->-2254194x23x0/0->-2418354x44x1/1(636005634msat in 4 htlcs)->-2418546x65x1/0(636005000msat in 4 htlcs)->
#   Flow 385: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 1620sat)
#   Flow 385: Failure of 23529023msat for 2471854x37x4/1 capacity [23528783msat,23528783msat] -> [23528783msat,23528783msat]
#   Flow 390: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 226sat)
#   Flow 390: Failure of 187462998msat for 2471854x37x7/1 capacity [187457878msat,187457878msat] -> [187457878msat,187457878msat]
#   Flow 388: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 226sat)
#   Flow 388: Failure of 197474019msat for 2471854x37x15/1 capacity [197460000msat,197460000msat] -> [197460000msat,197460000msat]
#   Flow 389: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 226sat)
#   Flow 389: Failure of 197464343msat for 2471854x37x2/0 capacity [197460000msat,197460000msat] -> [197460000msat,197460000msat]
#   Flow 386: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 1620sat)
#   Flow 386: Failure of 43626439msat for 2471854x37x13/1 capacity [43625433msat,43625433msat] -> [43625433msat,43625433msat]
#   Flow 387: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 226sat)
#   Flow 387: Failure of 197474019msat for 2471854x37x5/1 capacity [197460000msat,197460000msat] -> [197460000msat,197460000msat]
# we have computed a set of 6 flows with probability 0.000, fees 40841msat and delay 385
#   Flow 391: amount=23528000msat prob=0.000 fees=1023msat delay=140 path=-2471854x37x4/1(min=max=23528783msat)->-2414928x98x0/0->
#   Flow 392: amount=43625000msat prob=0.000 fees=1439msat delay=216 path=-2471854x37x13/1(min=max=43625433msat)->-2164740x20x0/0->-2136738x42x1/0->-2163293x89x1/1->-2254184x16x1/1->-2418354x44x1/1->-2418546x65x1/0->
#   Flow 393: amount=197460000msat prob=0.000 fees=14019msat delay=232 path=-2471854x37x5/1(min=max=197460000msat)->-2428790x34x2/0->-2420997x62x2/1->-2418354x44x1/1(43625043msat in 1 htlcs)->-2418546x65x1/0(43625000msat in 1 htlcs)->
#   Flow 394: amount=197460000msat prob=0.000 fees=14019msat delay=232 path=-2471854x37x15/1(min=max=197460000msat)->-2428790x34x4/0->-2420997x62x2/1(197462171msat in 1 htlcs)->-2418354x44x1/1(241085240msat in 2 htlcs)->-2418546x65x1/0(241085000msat in 2 htlcs)->
#   Flow 395: amount=197460000msat prob=0.000 fees=4343msat delay=158 path=-2471854x37x2/0(min=max=197460000msat)->-2467895x1x1/1->-2254184x16x1/1(43625479msat in 1 htlcs)->-2418354x44x1/1(438545437msat in 3 htlcs)->-2418546x65x1/0(438545000msat in 3 htlcs)->
#   Flow 396: amount=187457000msat prob=0.000 fees=5998msat delay=176 path=-2471854x37x7/1(min=max=187457878msat)->-2346806x7x0/1->-2191990x40x0/1->-2254194x23x0/0->-2418354x44x1/1(636005634msat in 4 htlcs)->-2418546x65x1/0(636005000msat in 4 htlcs)->
#   Flow 391: Failed at node #0 (WIRE_TEMPORARY_CHANNEL_FAILURE): failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Capacity exceeded - HTLC fee: 1620sat)
#   Flow 391: Failure of 23529023msat for 2471854x37x4/1 capacity [23528783msat,23528783msat] -> [23528783msat,23528783msat]

look at the lines where channel 2471854x37x4 is used. listpeerchannels says it has 23528783 msat spendable but there are lines above where it says
Failure of 23529023msat for 2471854x37x4/1 capacity [23528783msat,23528783msat] -> [23528783msat,23528783msat]

23529023 > 23528783

[rustyrussell]

Nice!
Indeed, this is fee confusion, in that it didn't seem to take into account the 1023 msat fee?

Flow 391: amount=23528000msat prob=0.000 fees=1023msat delay=140 path=-2471854x37x4/1(min=max=23528783msat)->-2414928x98x0/0->
Flow 391: Failure of 23529023msat for 2471854x37x4/1 capacity [23528783msat,23528783msat] -> [23528783msat,23528783msat]

This is a deeper issue (which may affect non-local channels), but as the release is imminent (and renepay experimental), I will write a test and take 1 sat + 1% off spendable_msat which should unstick this case.

[daywalker90]

workaround seems to work

[Lagrang3]

@renepickhardt @rustyrussell

The problem is that MCF algorithms assumes a flow conservation,
ie. it does not include fees paid to the forwarding nodes, this is for simplicity.
There must be some algorithms around to consider a network like ours in which
some flow (sats) are lost alonng the route, but for the moment I think it isn't
worth the extra trouble and possibly computational overhead.

In the current implementation fees are added after MCF computation has
been performed, then also the probability of success is computed.
So the flow to be forwarded by every channel is guaranteed to be smaller-equal
to the known maximum liquidity, but the flow+fees is not.

I recognize that if one channel is asked to forward an amount flow+fees that
exceeds the known maximum liquidity this channel will produce a 0 probability
of success and all the payment will have prob=0.
But I think removing flows with prob=0 will not be very efficient.
Maybe it would be better to:

1. artificially lower the flow capacity in the MCF arcs allowing some room
   for potential fees by some amount eg. fee_reserve for each channel and
2. after each MCF computation check if the fee_reserve is sufficient to allocate
   the actual fee, if not then fee_reserve is increased.

[Lagrang3]

Since #7125, this problem should be fixed. spendable_msat in our channels is used as the known_max that can be sent through them and the route builder would call flow_maximum_deliverable to adjust the amount in the flows to fit for liquidity (known_max) and htlc_max constraints.