Esplora/ElectrumX inconsistency

[tcharding]

Hi,

I don't know if I am using bdk incorrectly, if this is a bug in bdk or if this is a bug in Esplora.

As far as I understand the software running the Esplora infrastructure is a fork of electrs with a new JSON RPC API bolted on top, it still serves the Electrum REST API.

I have a Xpriv hardcoded and am using esplora-reqwest backend and also electrum backend.

The issue I'm facing is that the Wallet<EsploraBlockchain, AnyDatabase> does not see transactions but Wallet<ElectrumBlockchain, AnyDatabase> does. (There is a single receive transaction).

So, as stated above, am I doing something wrong? Is there a bug in bdk? Or should I be reporting this on at https://github.com/Blockstream/electrs?

I have pushed the code showing this issue to https://github.com/tcharding/btc-wallet-rs/tree/esplora-no-find-tx. To verify I'm not mad, clone the repo and run RUST_LOG=info,bdk=debug,bdk::blockchain::esplora::reqwest=info cargo run -- debug.

Here is how I'm creating the wallets in case the error is obvious.

const NETWORK: Network = Network::Testnet;

/// Create a wallet that connects to bitcoind by way of esplora.
pub fn esplora_wallet(db_path: PathBuf) -> Result<Wallet<EsploraBlockchain, AnyDatabase>> {
    info!("Creating {} wallet using esplora as the indexer", NETWORK);
    debug!("Using database at: {}", db_path.display());
    let db = sled::open(db_path)?;
    let tree = db.open_tree(b"esplora wallet")?;

    let blockchain = EsploraBlockchain::new(ESPLORA_URL, 20);

    let wallet = Wallet::new(
        DESC,
        Some(CHANGE_DESC),
        NETWORK,
        AnyDatabase::from(tree),
        blockchain,
    )?;

    Ok(wallet)
}

/// Create a wallet that connects to bitcoind by way of the ElectrumX REST API.
pub fn electrs_wallet(db_path: PathBuf) -> Result<Wallet<ElectrumBlockchain, AnyDatabase>> {
    info!("Creating {} wallet using electrs as the indexer", NETWORK);
    debug!("Using database at: {}", db_path.display());
    let db = sled::open(db_path)?;
    let tree = db.open_tree(b"electrumx wallet")?;

    let client = Client::new(ELECTRS_URL)?;
    let blockchain = ElectrumBlockchain::from(client);

    let wallet = Wallet::new(
        DESC,
        Some(CHANGE_DESC),
        NETWORK,
        AnyDatabase::from(tree),
        blockchain,
    )?;

    Ok(wallet)
}

If this is the wrong place to bring this up please say and I'll move it. Thanks for your time reading this.

[afilini]

Looking into it...

[afilini]

This is the output I get running the command:

[2021-11-10T11:51:44Z INFO  btc_wallet::testnet] Creating testnet wallet using electrs as the indexer
[2021-11-10T11:51:46Z DEBUG bdk::wallet] Begin sync...
[2021-11-10T11:51:46Z DEBUG bdk::wallet] max_address 100
[2021-11-10T11:51:46Z DEBUG bdk::wallet] caching external addresses
[2021-11-10T11:51:46Z INFO  bdk::wallet] Derivation of 100 addresses from 0 took 163 ms
[2021-11-10T11:51:46Z DEBUG bdk::wallet] caching internal addresses
[2021-11-10T11:51:46Z INFO  bdk::wallet] Derivation of 100 addresses from 0 took 264 ms
[2021-11-10T11:51:46Z DEBUG bdk::wallet] run_setup: true
[2021-11-10T11:51:46Z DEBUG bdk::blockchain::utils] start setup
[2021-11-10T11:51:46Z DEBUG bdk::blockchain::utils] #0 of External results:1
[2021-11-10T11:51:46Z DEBUG bdk::blockchain::utils] #1 of External results:0
[2021-11-10T11:51:47Z DEBUG bdk::blockchain::utils] #0 of Internal results:0
[2021-11-10T11:51:47Z INFO  bdk::blockchain::utils] max indexes are: {External: 0}
[2021-11-10T11:51:47Z INFO  bdk::blockchain::utils] got 1 txs to download
[2021-11-10T11:51:47Z INFO  bdk::blockchain::utils] 1 previous txs to download
[2021-11-10T11:51:47Z INFO  bdk::blockchain::utils] 1 headers to download for timestamp
[2021-11-10T11:51:47Z DEBUG bdk::blockchain::utils] e8a7d8bc97f114255d99aaf20a71a7e9d80ed627c6217be8d00528b2c8daaa09 output #0 is mine, adding utxo
[2021-11-10T11:51:47Z INFO  bdk::blockchain::utils] finish setup, elapsed 896ms

used addr:   n3vcAZYZQzBSF4TnA8mRsdqG4Z5SFh4q7p
last unused: mvpQSFzKx6putMmEaumTf1CjEZ4DAyt7Vy
new addr:    n1ikD6GMJwizUk9aM9h1YMbqrFwwEYedgT

Balance: 0.00100000

Ok(Some(TransactionDetails { transaction: Some(Transaction { version: 2, lock_time: 2065449, input: [TxIn { previous_output: OutPoint { txid: 2f0870d1088072fbe6e6488e4f9a7aa0a6f5438f117db7b2dea3901401309ad4, vout: 0 }, script_sig: Script(OP_PUSHBYTES_22 00148100ca7111c81067e92e1470d9b65e28f40a083c), sequence: 4294967294, witness: [[48, 68, 2, 32, 73, 81, 124, 75, 156, 220, 65, 232, 73, 22, 152, 86, 85, 0, 157, 126, 71, 42, 69, 171, 151, 111, 75, 98, 69, 47, 164, 53, 97, 201, 102, 210, 2, 32, 105, 116, 77, 120, 44, 148, 5, 246, 141, 42, 223, 89, 185, 38, 228, 134, 10, 170, 107, 182, 199, 48, 153, 12, 168, 56, 90, 237, 205, 134, 153, 195, 1], [3, 146, 11, 208, 98, 157, 98, 156, 249, 24, 78, 54, 47, 17, 197, 201, 113, 163, 102, 202, 153, 189, 172, 7, 129, 85, 146, 126, 100, 160, 32, 137, 192]] }], output: [TxOut { value: 100000, script_pubkey: Script(OP_DUP OP_HASH160 OP_PUSHBYTES_20 f5cbf284b905d3b213c634663411aac7c5175a5f OP_EQUALVERIFY OP_CHECKSIG) }, TxOut { value: 8211826, script_pubkey: Script(OP_HASH160 OP_PUSHBYTES_20 48957985e03b2e9bd968b690b68d5c016e45257c OP_EQUAL) }] }), txid: e8a7d8bc97f114255d99aaf20a71a7e9d80ed627c6217be8d00528b2c8daaa09, received: 100000, sent: 0, fee: Some(168), confirmation_time: Some(ConfirmationTime { height: 2065450, timestamp: 1629421646 }), verified: true }))

so I guess it did find the transaction (?)

[tcharding]

Thanks a bunch for looking into this @afilini. My mistake, I was not totally clear. I left the branch in a running state using the electrs_wallet() call. This one uses the ElectrumBlockchain and it works. Using esplora_wallet() (uses EsploraBlockchain) is the one that fails. I pushed a commit to the branch that makes the change so it fails. Its just

diff --git a/src/main.rs b/src/main.rs
index 782071b..f0c1c55 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -29,7 +29,7 @@ fn main() -> Result<()> {

let db = database_path()?;

-    let wallet = testnet::electrs_wallet(db)?;
+    let wallet = testnet::esplora_wallet(db)?;
wallet.sync(blockchain::log_progress(), None)?;

match opt.cmd {

Thanks

[LLFourn]

This is fixed by: #461

The only thing I know that this PR fixes with esplora is that it will choose the best of conflicting transactions that esplora sends (and thereby show a lower balance).
I'm not sure what is being fixed here though. It'd be good to have a test for it whatever it is but it looks like @tcharding just sent coins to the first address and they just don't show up.

Looking into it now.

[LLFourn]

@tcharding your problem is that you are not using the testnet endpoint on esplora. It should be:

const ESPLORA_URL: &str = "https://blockstream.info/testnet/api";

This is fixed by: #461

Of course it wasn't. I clearly confused myself somehow.

[LLFourn]

Errr but how does electrum work then?

[tcharding]

Thanks bro!

Errr but how does electrum work then?

Perhaps the testnet endpoint is valid because of the port number, I can't find any online docs saying that 143 is testnet though.

[tcharding]

Resolved by @LLFourn, like a boss.

[tcharding]

Nope, I have no idea where I even got port 143 from?

[afilini]

Those are documented here: https://blog.blockstream.com/en-esplora-and-other-alternatives-to-electrumx/

I actually remember some Blockstream people telling me those were "internal" ports that they used for development, and that the public should use the more standard ports like 50001 for tcp mainnet, 50002 for ssl mainnet and 60001/60002 for testnet.

I guess they've changed their mind or messed something up since they ended up in the blog post..

[tcharding]

Legend! I remember that page now :)

[LLFourn]

isn't it a bad idea to connect to a remote electrum without tls?

[tcharding]

Fair point :)