failed to verify certicate after lnd certificate renewal

[warioishere]

Title: AlbyHub fails to start after LND TLS certificate renewal - x509 certificate verification error

Environment:

• AlbyHub version: 1.21.0
• LND version: 0.20.0-beta
• OS: Ubuntu 24.04
• Installation method: Official Linux x86_64 script (https://github.com/getAlby/hub/tree/master/scripts/linux-x86_64)
• LND and AlbyHub running on the same server

Description:

AlbyHub fails to start after renewing the LND TLS certificate. The previous certificate had expired and was successfully renewed. Other applications (e.g., LNbits) are accepting and working with the new certificate without issues.

Error Message:

connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"

Steps to Reproduce:

1. LND TLS certificate expires
2. Renew LND TLS certificate
3. Attempt to start AlbyHub
4. AlbyHub fails to connect to LND with the above error

Expected Behavior:
AlbyHub should accept the new TLS certificate and connect to LND successfully, similar to how other Lightning applications (LNbits) handle the certificate renewal.

Additional Context:

• The certificate renewal was done correctly as verified by other applications
• No issues with LND itself - it's running normally
• The error suggests AlbyHub might be caching the old certificate or certificate authority information

Possible Solution:
AlbyHub may need to either:

1. Refresh/clear cached TLS certificate data
2. Re-read the certificate files from disk on startup
3. Provide a way to manually trigger certificate reload

[rolznz]

@warioishere Alby Hub should always read the newest version of the certificate but possibly only if the environment variable is set. Do you remember how you setup Alby Hub, was LND selected in the onboarding? maybe there you pasted the hex values and unfortunately they might not be able to be updated.

https://github.com/getAlby/hub/blob/master/config/config.go#L68-L75

[rolznz]

This is not such a good flow if hex values (rather than filenames) are passed as it doesn't allow for any way to update the certificate except by manually setting environment variables after. CC @im-adithya @bumi

[warioishere]

@warioishere Alby Hub should always read the newest version of the certificate but possibly only if the environment variable is set. Do you remember how you setup Alby Hub, was LND selected in the onboarding? maybe there you pasted the hex values and unfortunately they might not be able to be updated.

https://github.com/getAlby/hub/blob/master/config/config.go#L68-L75

yes I selected LND and entered the hex values for the certificate.

I suggest a button reset setup but keep data of the alby installation? There are no envs to change in the binary installation via script or am I missing something. At least I didnt see anything in the docs and I didint find any envs in the working dir. Maybe we can add a settings file in a future /etc/albyhub for binary / script setup?

[rolznz]

@warioishere we cannot expose such a button as it would be a security vulnerability.

Unfortunately the only way you can fix this right now is to add a .env file in your hub directory with this:
LND_CERT_FILE=/path/to/your/cert/file

[rolznz]

Maybe we can add a settings file in a future /etc/albyhub for binary / script setup?

Maybe this would be nice to add, an empty .env file with some comments on how to use it.

[warioishere]

It doesnt work adding an .env to the base repo and loading it with an adapted systemd file:

the .env file:

PORT=8029

LN_BACKEND_TYPE=LND
LND_ADDRESS=localhost:10009
LND_CERT_FILE=/home/admin/.lnd/tls.cert
LND_MACAROON_FILE=/home/admin/.lnd/data/chain/bitcoin/mainnet/admin.macaroon

systemd:

[Unit]
Description=Alby Hub
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=admin
WorkingDirectory=/home/admin/albyhub

EnvironmentFile=/home/admin/albyhub/.env
ExecStart=/home/admin/albyhub/start.sh

Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

Alby crashes with a panic, maybe it still expects something encrypted?

Dec 15 08:08:04 wallet-node start.sh[1490]: {"level":"info","msg":"Launching LN Backend: LND","time":"2025-12-15T08:08:04Z"}
Dec 15 08:08:04 wallet-node start.sh[1490]: panic: runtime error: index out of range [1] with length 1
Dec 15 08:08:04 wallet-node start.sh[1490]: goroutine 69 [running]:
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/config.AesGcmDecryptWithPassword({0xc00064cc40?, 0x17f6e40?}, {0xc0006af170, 0x18})
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/config/aesgcm.go:49 +0x116
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/config.(*config).get(0xc000256000?, {0x1b90573, 0xa}, {0xc0006af170, 0x18}, 0xc000236390)
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/config/config.go:192 +0x145
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/config.(*config).Get(0xc0001b8000?, {0x1b90573?, 0x1ba7648?}, {0xc0006af170?, 0xc0003dbd48?})
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/config/config.go:180 +0x25
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/service.(*service).launchLNBackend(0xc000650000, {0x22ee340, 0xc00038e640}, {0xc0006af170, 0x18})
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/service/start.go:331 +0x46a
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/service.(*service).StartApp(0xc000650000, {0xc0006af170, 0x18})
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/service/start.go:281 +0x1f8
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/api.(*api).StartInternal(0x8?, 0x0?)
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/api/api.go:1353 +0xa4
Dec 15 08:08:04 wallet-node start.sh[1490]: github.com/getAlby/hub/api.(*api).Start(0xc0002688f0, 0xc0009e0140?)
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/api/api.go:1339 +0x47
Dec 15 08:08:04 wallet-node start.sh[1490]: created by github.com/getAlby/hub/http.(*HttpService).startHandler in goroutine 29
Dec 15 08:08:04 wallet-node start.sh[1490]:         /home/runner/work/hub/hub/http/http_service.go:296 +0x150
Dec 15 08:08:04 wallet-node systemd[1]: albyhub.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 15 08:08:04 wallet-node systemd[1]: albyhub.service: Failed with result 'exit-code'.
Dec 15 08:08:06 wallet-node systemd[1]: albyhub.service: Scheduled restart job, restart counter is at 1.

[rolznz]

Hmm, that's not good, it's failing to decrypt. I think you've run into an issue we have already fixed in master, which is that there is a bug saving a config entry that was previously encrypted with your unlock password (which is the case with the cert hex) to decrypted (when you are pointing at a local certificate file). This will be fixed in version 1.21.1.

Unfortunately for now to fix this the simplest way would be to manually update the config entry from your sqlite database.

Assuming this is where your nwc sqlite db file is: /home/admin/albyhub/nwc.db

Note: make a backup of your nwc.db file before proceeding

The 2nd command below should return 1 but we want it to be 0.

sqlite3 /home/admin/albyhub/nwc.db
select encrypted from user_configs where key = "LNDCertHex";
update user_configs set encrypted = 0 where key = "LNDCertHex";

[warioishere]

still crashing with:

Dec 15 16:32:48 wallet-node start.sh[54415]: panic: runtime error: index out of range [1] with length 1
Dec 15 16:32:48 wallet-node start.sh[54415]: goroutine 82 [running]:
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/config.AesGcmDecryptWithPassword({0xc00064c7e0?, 0x17f6e40?}, {0xc0007a63f0, 0x18})
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/config/aesgcm.go:49 +0x116
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/config.(*config).get(0xc0002fc000?, {0x1b90573, 0xa}, {0xc0007a63f0, 0x18}, 0xc0000b0a50)
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/config/config.go:192 +0x145
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/config.(*config).Get(0xc000246180?, {0x1b90573?, 0x1ba7648?}, {0xc0007a63f0?, 0xc000221d48?})
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/config/config.go:180 +0x25
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/service.(*service).launchLNBackend(0xc000370000, {0x22ee340, 0xc0009b40a0}, {0xc0007a63f0, 0x18})
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/service/start.go:331 +0x46a
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/service.(*service).StartApp(0xc000370000, {0xc0007a63f0, 0x18})
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/service/start.go:281 +0x1f8
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/api.(*api).StartInternal(0x0?, 0x0?)
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/api/api.go:1353 +0xa4
Dec 15 16:32:48 wallet-node start.sh[54415]: github.com/getAlby/hub/api.(*api).Start(0xc0001698c0, 0xc00070c0b0?)
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/api/api.go:1339 +0x47
Dec 15 16:32:48 wallet-node start.sh[54415]: created by github.com/getAlby/hub/http.(*HttpService).startHandler in goroutine 56
Dec 15 16:32:48 wallet-node start.sh[54415]:         /home/runner/work/hub/hub/http/http_service.go:296 +0x150
Dec 15 16:32:48 wallet-node systemd[1]: albyhub.service: Main process exited, code=exit

I verified LNDCertHex ist set to 0

admin@wallet-node:~/albyhub/data$ sqlite3 nwc.db 
SQLite version 3.45.1 2024-01-30 16:01:20
Enter ".help" for usage hints.
sqlite> select encrypted from user_configs where key = "LNDCertHex";
0

Edit: it was showing 1 before as you said.

[rolznz]

@warioishere sorry, you will also need to do the same with LNDAddress and LNDMacaroonHex as well since you also added those related environment variables to your .env. I think that should fix it. I added the lines so you can check if the values are 1 there too.

sqlite3 /home/admin/albyhub/nwc.db
select encrypted from user_configs where key = "LNDMacaroonHex";
update user_configs set encrypted = 0 where key = "LNDMacaroonHex";
select encrypted from user_configs where key = "LNDAddress";
update user_configs set encrypted = 0 where key = "LNDAddress";

[warioishere]

@warioishere sorry, you will also need to do the same with LNDAddress and LNDMacaroonHex as well since you also added those related environment variables to your .env. I think that should fix it. I added the lines so you can check if the values are 1 there too.

sqlite3 /home/admin/albyhub/nwc.db
select encrypted from user_configs where key = "LNDMacaroonHex";
update user_configs set encrypted = 0 where key = "LNDMacaroonHex";
select encrypted from user_configs where key = "LNDAddress";
update user_configs set encrypted = 0 where key = "LNDAddress";

ah right could have investigated a bit more by myself, thanks for the hint.
Its working now and albyhub is up again! If you wanna keep this open for your reference and addressing this, if you have this already on your radar, you may of course close it

Thanks for your help and cheers

[rolznz]

@warioishere thanks for confirming!

I think we need to improve the LND flow somehow - I think it's fundamentally broken how you can paste a hex certificate that can expire and have no way to update it. I will leave this issue open. (also maybe about adding an example .env - I could move this to a separate issue if others on the team agree it's a good thing to add)