commando message hook blocking?

[jb55]

I noticed long pay commands can hang all commando requests, it looks like the custommsg hook is blocking? Is there a way to make it non blocking?

I found ElementsProject/lightning#2326, not sure if this will help

cc @cdecker

[cdecker]

I don't think that the custommsg hook is synchronous (otherwise pay would also not proceed), it could rather be the consuming plugin being single-threaded, which would mean that any commands that come in while the hook hasn't returned would be delayed. Which client lib are you using?

[jb55]

@cdecker wrote:

Which client lib are you using?

I'm using lnsocket. All it does is open a lightning connection and send commando messages. It's pretty easy to block commando by making a request with a large response:

To reproduce:

$ git clone git://jb55.com/lnsocket && cd lnsocket
$ make lnrpc
$ export LN_NODE=nodeid LN_HOST=host LN_RUNE=rune
$ ./lnrpc $LN_NODE $LN_HOST $LN_RUNE listchannels
error: rpc request timeout
$ ./lnrpc $LN_NODE $LN_HOST $LN_RUNE getinfo
error: rpc request timeout

# eventually once the listchannels is done...
$ ./lnrpc $LN_NODE $LN_HOST $LN_RUNE getInfo
{"result": {"id": "03f3c108ccd536b8526841f0a5c58212bb9e6584a1eb493080e7c1cc34f82dad71", "alias": "jb55.com", "color": "ff0000", "num_peers": 24, "num_pending_channels": 1, "num_active_channels": 22, "num_inactive_channels": 0, "address": [{"type": "ipv4", "address": "24.84.152.187", "port": 9735}], "binding": [{"type": "ipv4", "address": "0.0.0.0", "port": 9735}], "version": "v0.10.2-194-g52b1f5a-modded", "blockheight": 726321, "network": "bitcoin", "msatoshi_fees_collected": 13220067, "fees_collected_msat": "13220067msat", "lightning-dir": "/home/jb55/.lightning-bitcoin/bitcoin"}}

It looks like commando is probably buffering the large listchannels result into memory? making the request eventually timeout. It will block all subsequent RPC calls on other connections like getinfo until listchannels is finished. listchannels could be a long running command like pay, it's the same problem (I think)

[jb55]

I'm fine with commando rpc calls being syncronous per connection, it just seems to block all other connections as well. I'm guessing because a new plugin instance isn't spawned per connection? which is why I thought making things more asyncronous in the plugin would fix things, but I don't know enough about plugin architecture to speculate further.

[jb55]

cc @rustyrussell I dunno if this is a commando or plugin architecture issue, or if there's an easy fix for this

[fiatjaf]

It wouldn't need to support these blocking calls if it could instead allow us to subscribe to these notifications: https://lightning.readthedocs.io/PLUGINS.html#event-notifications

It isn't possible to do it through JSON-RPC directly, so they don't really fit lightning-cli, but for commando it would be great unless I'm missing something.

[rustyrussell]

Does that fix it for you? Thanks!

[jb55]

Hmm this doesn't seem to fix anything, I'll take a stab at debugging this.

[jb55]

another way to brick commando: just call waitanyinvoice 10000, then any future request won't work until the node+plugin is restarted