errors after a restart after a successful splice

[hMsats]

Issue and Steps to Reproduce

I managed to add 100 ksats to a channel between 2 of my nodes (and made a reddit post about it!). It all went smooth and the nodes and channel were working fine (was able to pay invoices).

Before splicing:

100 ksats (main node) <=> 100 ksats (test node)

after

200 ksats (main node) <=> 100 ksats (test node)

I then tried to restart my test node (because the nodeid was no longer present in listnodes at my main node side of the channel (why?)) and got this:

023-08-08T06:27:17.156Z INFO    plugin-clnrest.py: Killing plugin: disabled itself: No module named 'flask'
2023-08-08T06:27:22.714Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2023-08-08T06:27:32.581Z INFO    lightningd: Restarting onchaind for channel 33
2023-08-08T06:27:32.581Z UNUSUAL 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Peer permanent failure in FUNDING_SPEND_SEEN: Funding transaction spent
2023-08-08T06:27:32.582Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from FUNDING_SPEND_SEEN to FUNDING_SPEND_SEEN
2023-08-08T06:27:32.615Z INFO    lightningd: --------------------------------------------------
2023-08-08T06:27:32.616Z INFO    lightningd: Server started with public key 02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392, alias myshtery (color #02aecd) and lightningd v23.08rc1-29-g54bcb10
lightning_onchaind: onchaind/onchaind.c:196: trim_maximum_feerate: Assertion `tal_count(commitment->inputs) == 1' failed.
lightning_onchaind: FATAL SIGNAL 6 (version v23.08rc1-29-g54bcb10)
0x55ad1ca95d26 send_backtrace
	common/daemon.c:33
0x55ad1ca95dc0 crashdump
	common/daemon.c:75
0x7f1d037ac08f ???
	/build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7f1d037ac00b __GI_raise
	../sysdeps/unix/sysv/linux/raise.c:51
0x7f1d0378b858 __GI_abort
	/build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79
0x7f1d0378b728 __assert_fail_base
	/build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92
0x7f1d0379cfd5 __GI___assert_fail
	/build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101
0x55ad1ca895c5 trim_maximum_feerate
	onchaind/onchaind.c:196
0x55ad1ca8f0af main
	onchaind/onchaind.c:3535
0x7f1d0378d082 __libc_start_main
	../csu/libc-start.c:308
0x55ad1ca88c0d ???
	???:0
0xffffffffffffffff ???
	???:0
lightning_onchaind: FATAL SIGNAL (version v23.08rc1-29-g54bcb10)
0x55ad1ca95d26 send_backtrace
	common/daemon.c:33
0x55ad1ca9c006 status_failed
	common/status.c:221
0x55ad1ca9c118 status_backtrace_exit
	common/subdaemon.c:18
0x55ad1ca95dc6 crashdump
	common/daemon.c:78
0x7f1d037ac08f ???
	/build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7f1d037ac00b __GI_raise
	../sysdeps/unix/sysv/linux/raise.c:51
0x7f1d0378b858 __GI_abort
	/build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79
0x7f1d0378b728 __assert_fail_base
	/build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92
0x7f1d0379cfd5 __GI___assert_fail
	/build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101
0x55ad1ca895c5 trim_maximum_feerate
	onchaind/onchaind.c:196
0x55ad1ca8f0af main
	onchaind/onchaind.c:3535
0x7f1d0378d082 __libc_start_main
	../csu/libc-start.c:308
0x55ad1ca88c0d ???
	???:0
0xffffffffffffffff ???
	???:0
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: FATAL SIGNAL 6 (version v23.08rc1-29-g54bcb10)
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:38 (send_backtrace) 0x55c25fd6ed6e
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:75 (crashdump) 0x55c25fd6edc0
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0x7fc39739508f
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../sysdeps/unix/sysv/linux/raise.c:51 (__GI_raise) 0x7fc39739500b
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79 (__GI_abort) 0x7fc397374858
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92 (__assert_fail_base) 0x7fc397374728
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101 (__GI___assert_fail) 0x7fc397385fd5
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:196 (trim_maximum_feerate) 0x55c25fd625c5
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:3535 (main) 0x55c25fd680af
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0x7fc397376082
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0x55c25fd61c0d
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0xffffffffffffffff
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: FATAL SIGNAL (version v23.08rc1-29-g54bcb10)
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:38 (send_backtrace) 0x55c25fd6ed6e
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/status.c:221 (status_failed) 0x55c25fd75006
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/subdaemon.c:18 (status_backtrace_exit) 0x55c25fd75118
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:78 (crashdump) 0x55c25fd6edc6
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0x7fc39739508f
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../sysdeps/unix/sysv/linux/raise.c:51 (__GI_raise) 0x7fc39739500b
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79 (__GI_abort) 0x7fc397374858
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92 (__assert_fail_base) 0x7fc397374728
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101 (__GI___assert_fail) 0x7fc397385fd5
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:196 (trim_maximum_feerate) 0x55c25fd625c5
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:3535 (main) 0x55c25fd680af
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0x7fc397376082
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0x55c25fd61c0d
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0xffffffffffffffff
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Owning subdaemon onchaind died (61952)                                                                                                                                                                              

EDIT 1: removed the gossip_store and restarted my test node. Now I see this:

2023-08-08T14:13:04.956Z INFO    plugin-clnrest.py: Killing plugin: disabled itself: No module named 'flask'
2023-08-08T14:13:12.642Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2023-08-08T14:13:14.864Z INFO    lightningd: Restarting onchaind for channel 33
2023-08-08T14:13:14.865Z UNUSUAL 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Peer permanent failure in ONCHAIN: Funding transaction spent
2023-08-08T14:13:14.930Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2023-08-08T14:13:14.947Z UNUSUAL 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Peer permanent failure in FUNDING_SPEND_SEEN: Funding transaction spent
2023-08-08T14:13:14.949Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from FUNDING_SPEND_SEEN to FUNDING_SPEND_SEEN
2023-08-08T14:13:15.011Z INFO    lightningd: --------------------------------------------------
2023-08-08T14:13:15.011Z INFO    lightningd: Server started with public key 02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392, alias myshtery (color #02aecd) and lightningd v23.08rc1-29-g54bcb10
2023-08-08T14:13:15.020Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from FUNDING_SPEND_SEEN to ONCHAIN

getinfo output

Both: are version 23.08rc1-29-g54bcb10 with

Config (test node)

bind-addr=192.168.178.20:6837
alias=myshtery
network=bitcoin
proxy=127.0.0.1:9050
fee-base=1
fee-per-satoshi=1
experimental-offers
allow-deprecated-apis=false
experimental-splicing

Config (main node)

bind-addr=192.168.178.20:9735
announce-addr=84.26.69.91:9735
announce-addr=2bh7psr4dinasuz3d6jm5o56ebkmlnotqqptoesikzvsyqxwmaneyiad.onion:9735
alias=bitcoinserver.nl
network=bitcoin
proxy=127.0.0.1:9050
fee-base=0
fee-per-satoshi=0
experimental-offers
allow-deprecated-apis=false
experimental-splicing

EDIT 2: This is what listpeerchannels shows on my test node:

     "state_changes": [
            {
               "timestamp": "2023-04-27T07:51:11.739Z",
               "old_state": "CHANNELD_AWAITING_LOCKIN",
               "new_state": "CHANNELD_NORMAL",
               "cause": "user",
               "message": "Lockin complete"
            },
            {
               "timestamp": "2023-08-07T17:45:52.608Z",
               "old_state": "CHANNELD_NORMAL",
               "new_state": "CHANNELD_AWAITING_SPLICE",
               "cause": "remote",
               "message": "Broadcasting splice"
            },
            {
               "timestamp": "2023-08-07T21:28:04.164Z",
               "old_state": "CHANNELD_AWAITING_SPLICE",
               "new_state": "CHANNELD_NORMAL",
               "cause": "remote", 
               "message": "Lockin complete"
            },
            {
               "timestamp": "2023-08-08T06:22:55.383Z",
               "old_state": "CHANNELD_NORMAL",
               "new_state": "AWAITING_UNILATERAL",
               "cause": "onchain", 
               "message": "Funding transaction spent"
            },
            {
               "timestamp": "2023-08-08T06:22:55.383Z",
               "old_state": "AWAITING_UNILATERAL",
               "new_state": "FUNDING_SPEND_SEEN",
               "cause": "onchain", 
               "message": "Onchain funding spend"
            },
            {
               "timestamp": "2023-08-08T06:33:48.279Z",
               "old_state": "FUNDING_SPEND_SEEN",
               "new_state": "ONCHAIN",
               "cause": "onchain",
               "message": "Onchain init reply"
            },
            {
               "timestamp": "2023-08-08T06:40:32.126Z",
               "old_state": "ONCHAIN",
               "new_state": "FUNDING_SPEND_SEEN",
               "cause": "onchain",
               "message": "Onchain funding spend"
            },
            {
               "timestamp": "2023-08-08T06:40:46.482Z",
               "old_state": "FUNDING_SPEND_SEEN",
               "new_state": "ONCHAIN",
               "cause": "onchain",
               "message": "Onchain init reply"
            },
            {
               "timestamp": "2023-08-08T14:13:14.932Z",
               "old_state": "ONCHAIN",
               "new_state": "FUNDING_SPEND_SEEN",
               "cause": "onchain",
               "message": "Onchain funding spend"
            },
            {
               "timestamp": "2023-08-08T14:13:15.021Z",
               "old_state": "FUNDING_SPEND_SEEN",
               "new_state": "ONCHAIN",
               "cause": "onchain",
               "message": "Onchain init reply"
            }
         ],
         "status": [
            "ONCHAIN:Tracking our own unilateral close",
            "ONCHAIN:3 outputs unresolved: in 107 blocks will spend DELAYED_OUTPUT_TO_US (a74ba0c6f9c3444eda58ad36e9dc1f7888644cf187d3778b06d3fefeb5551744:0) using OUR_DELAYED_RETURN_TO_WALLET"
         ],

and this on my main node:

         "state_changes": [
            {
               "timestamp": "2023-04-27T07:51:12.223Z",
               "old_state": "CHANNELD_AWAITING_LOCKIN",
               "new_state": "CHANNELD_NORMAL",
               "cause": "remote",
               "message": "Lockin complete"
            },
            {
               "timestamp": "2023-08-07T17:45:52.605Z",
               "old_state": "CHANNELD_NORMAL",
               "new_state": "CHANNELD_AWAITING_SPLICE",
               "cause": "user",
               "message": "Broadcasting splice"
            },
            {
               "timestamp": "2023-08-07T21:28:04.188Z",
               "old_state": "CHANNELD_AWAITING_SPLICE",
               "new_state": "CHANNELD_NORMAL",
               "cause": "user",
               "message": "Lockin complete"
            },
            {
               "timestamp": "2023-08-08T06:33:32.400Z",
               "old_state": "CHANNELD_NORMAL",
               "new_state": "AWAITING_UNILATERAL",
               "cause": "onchain",
               "message": "Funding transaction spent"
            },
            {
               "timestamp": "2023-08-08T06:33:32.401Z",
               "old_state": "AWAITING_UNILATERAL",
               "new_state": "FUNDING_SPEND_SEEN",
               "cause": "onchain",
               "message": "Onchain funding spend"
            },
            {
               "timestamp": "2023-08-08T06:33:32.756Z",
               "old_state": "FUNDING_SPEND_SEEN",
               "new_state": "ONCHAIN",
               "cause": "onchain",
               "message": "Onchain init reply"
            }
         ],
         "status": [
            "CHANNELD_NORMAL:Reconnected, and reestablished.",
            "ONCHAIN:Tracking their unilateral close",
            "ONCHAIN:All outputs resolved: waiting 63 more blocks before forgetting channel"
         ],

[hMsats]

@rustyrussell rc2 while this issue hasn't been fixed? Is that because splicing is still experimental or has it been fixed in some other commit? When fixed I would like to test splicing again.

[vincenzopalazzo]

I think just because no one put this in the roadmap sorry

[rustyrussell]

Sorry, I did miss this!

Tests for splicing are badly lacking:(

[ddustin]

Thanks for the report 🙏

Can you post the logs from when the unilateral close started, around timestamp 2023-08-08T06:22:55.383Z?

Specifically looking to see if "Restarting onchaind for channel..." appears beforehand, meaning it was caused by a lightningd restart flow or if it happened some other way.

[hMsats]

Currently in train but here is I think what you asked for:

2023-08-08T06:22:37.310Z INFO    plugin-clnrest.py: Killing plugin: disabled itself: No module named 'flask'                        2023-08-08T06:22:44.194Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.                                    2023-08-08T06:22:55.380Z INFO    lightningd: Restarting onchaind for channel 33         2023-08-08T06:22:55.380Z UNUSUAL 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Peer permanent failure in CHANNELD_NORMAL: Funding transaction spent                                           2023-08-08T06:22:55.382Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL          2023-08-08T06:22:55.383Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN       2023-08-08T06:22:55.420Z INFO    lightningd:

[hMsats]

Home now. Here is everything:

2023-08-08T06:22:37.310Z INFO    plugin-clnrest.py: Killing plugin: disabled itself: No module named 'flask'
2023-08-08T06:22:44.194Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2023-08-08T06:22:55.380Z INFO    lightningd: Restarting onchaind for channel 33
2023-08-08T06:22:55.380Z UNUSUAL 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Peer permanent failure in CHANNELD_NORMAL: Funding transaction spent
2023-08-08T06:22:55.382Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from CHANNELD_NORMAL to AWAITING_UNILATERAL
2023-08-08T06:22:55.383Z INFO    02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2023-08-08T06:22:55.420Z INFO    lightningd: --------------------------------------------------
2023-08-08T06:22:55.420Z INFO    lightningd: Server started with public key 02aecde3b3dde450889755d4dbce538e6b9e9dfa7702d14b5bc5c4379e1b330392, alias myshtery (color #02aecd) and lightningd v23.08rc1-29-g54bcb10
lightning_onchaind: onchaind/onchaind.c:196: trim_maximum_feerate: Assertion `tal_count(commitment->inputs) == 1' failed.
lightning_onchaind: FATAL SIGNAL 6 (version v23.08rc1-29-g54bcb10)
0x55c25fd6ed26 send_backtrace
        common/daemon.c:33
0x55c25fd6edc0 crashdump
        common/daemon.c:75
0x7fc39739508f ???
        /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7fc39739500b __GI_raise
        ../sysdeps/unix/sysv/linux/raise.c:51
0x7fc397374858 __GI_abort
        /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79
0x7fc397374728 __assert_fail_base
        /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92
0x7fc397385fd5 __GI___assert_fail
        /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101
0x55c25fd625c5 trim_maximum_feerate
        onchaind/onchaind.c:196
0x55c25fd680af main
        onchaind/onchaind.c:3535
0x7fc397376082 __libc_start_main
        ../csu/libc-start.c:308
0x55c25fd61c0d ???
        ???:0
0xffffffffffffffff ???
        ???:0
lightning_onchaind: FATAL SIGNAL (version v23.08rc1-29-g54bcb10)
0x55c25fd6ed26 send_backtrace
        common/daemon.c:33
0x55c25fd75006 status_failed
        common/status.c:221
0x55c25fd75118 status_backtrace_exit
        common/subdaemon.c:18
0x55c25fd6edc6 crashdump
        common/daemon.c:78
0x7fc39739508f ???
        /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
0x7fc39739500b __GI_raise
        ../sysdeps/unix/sysv/linux/raise.c:51
0x7fc397374858 __GI_abort
        /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79
0x7fc397374728 __assert_fail_base
        /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92
0x7fc397385fd5 __GI___assert_fail
        /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101
0x55c25fd625c5 trim_maximum_feerate
        onchaind/onchaind.c:196
0x55c25fd680af main
        onchaind/onchaind.c:3535
0x7fc397376082 __libc_start_main
        ../csu/libc-start.c:308
0x55c25fd61c0d ???
        ???:0
0xffffffffffffffff ???
        ???:0
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: FATAL SIGNAL 6 (version v23.08rc1-29-g54bcb10)
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:38 (send_backtrace) 0x55c25fd6ed6e
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:75 (crashdump) 0x55c25fd6edc0
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0x7fc39739508f
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../sysdeps/unix/sysv/linux/raise.c:51 (__GI_raise) 0x7fc39739500b
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79 (__GI_abort) 0x7fc397374858
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92 (__assert_fail_base) 0x7fc397374728
2023-08-08T06:22:56.781Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101 (__GI___assert_fail) 0x7fc397385fd5
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:196 (trim_maximum_feerate) 0x55c25fd625c5
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:3535 (main) 0x55c25fd680af
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0x7fc397376082
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0x55c25fd61c0d
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0xffffffffffffffff
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: FATAL SIGNAL (version v23.08rc1-29-g54bcb10)
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:38 (send_backtrace) 0x55c25fd6ed6e
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/status.c:221 (status_failed) 0x55c25fd75006
2023-08-08T06:22:56.782Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/subdaemon.c:18 (status_backtrace_exit) 0x55c25fd75118
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: common/daemon.c:78 (crashdump) 0x55c25fd6edc6
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0 ((null)) 0x7fc39739508f
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../sysdeps/unix/sysv/linux/raise.c:51 (__GI_raise) 0x7fc39739500b
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:79 (__GI_abort) 0x7fc397374858
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:92 (__assert_fail_base) 0x7fc397374728
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: /build/glibc-SzIz7B/glibc-2.31/assert/assert.c:101 (__GI___assert_fail) 0x7fc397385fd5
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:196 (trim_maximum_feerate) 0x55c25fd625c5
2023-08-08T06:22:56.783Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: onchaind/onchaind.c:3535 (main) 0x55c25fd680af
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: ../csu/libc-start.c:308 (__libc_start_main) 0x7fc397376082
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0x55c25fd61c0d
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: backtrace: (null):0 ((null)) 0xffffffffffffffff
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-onchaind-chan#33: STATUS_FAIL_INTERNAL_ERROR: FATAL SIGNAL
2023-08-08T06:22:56.784Z **BROKEN** 02888244029c5909593038ab19f269947c720de3423e491791b46c7c92f76279b6-chan#33: Owning subdaemon onchaind died (61952)

[ddustin]

PR #6564 should specifically address the crash on restart issues.

Still need to diagnose missing nodeid issue.

[ddustin]

PR #6564 will prevent the creation of unneeded justice transactions but it wont clear out ones that are already there.

You shouldn't make a habit of it but if you need to get into the node to recover your funds, you can remove the queued up justice transaction.

You can do that by going into the database and delete the (hopefully single) row in table channeltxs, after that you should be able to start up again -- though the funds may still require manually recovery.

[hMsats]

@ddustin Thanks a lot. I haven't lost any funds and opened a new channel between the two nodes and all is running as before I tried the splice. I will retry the splice after the fix has been merged and report back.