Frequent errors displayed when trying to sync mainnet: "Peer not suitable for sync: Invalid genesis hash"

[halfalicious]

I'm attempting to sync mainnet with a locally built Aleth sync'd to dbbfd5b (this includes the sync fix from PR #5550) and I'm still seeing mainnet sync issues:

nilsf@berlin:~/Documents/Code/aleth/build/aleth$ ./aleth -v4 --log-channels sync
aleth, a C++ Ethereum client
aleth 1.7.0-alpha.0-1+commit.dbbfd5be
Node ID: enode://156f6db80161095aef879e5c5562db25b5e6e742bfa9e37cc4ae8623b2eb72b1829daa2c71101046b5d80210c68f549a3cf6e18a7f9e913d08ce5139d72a2ff0@0.0.0.0:0
JSONRPC Admin Session Key: Ilnf3dFvzUA=
DEBUG 04-09 20:49:02 p2p  sync   865a6325…|Parity-Ethereum/v2.2.9-stable-5d5b372-20190203/x86_64-linux-gnu/rustc1.31.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 20:49:03 p2p  sync   cdd653bc…|Parity-Ethereum/v2.1.11-stable-e9194b0-20190107/x86_64-linux-gnu/rustc1.31.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 20:49:03 p2p  sync   1016aadc…|Geth/v4.1.1/linux/go1.8 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 20:49:08 p2p  sync   865a6325…|Parity-Ethereum/v2.2.9-stable-5d5b372-20190203/x86_64-linux-gnu/rustc1.31.1 Peer not suitable for sync: Invalid genesis hash.

Note that I can sync Ropsten fine.

[halfalicious]

This isn't an issue with the DAO block header check since that's not being hit. The code which is failing is in the else clause here:

aleth/libethereum/BlockChainSync.cpp

Lines 190 to 195 in dbbfd5b

	std::string disconnectReason;
	if (peerSessionInfo->clientVersion.find("/v0.7.0/") != string::npos)
	disconnectReason = "Blacklisted client version.";
	else
	disconnectReason = _peer.validate(
	host().chain().genesisHash(), host().protocolVersion(), host().networkId());

[halfalicious]

This still repros on privately built Aleth with enlistment sync'd to e52571a (which is before the dao block header changes)

[halfalicious]

This was broken before release 1.6.0-alpha.1 (de53524)

[halfalicious]

Still hitting this error on 1.5.0 (bfd19bb):

aleth, a C++ Ethereum client
aleth 1.5.0
Mining Beneficiary: 84258fde-b0d9-747e-b70f-f55e14831192 - 00fd4aaf9713f5bb664c20a462acc4ebc363d1a6
Node ID: enode://b8a7d0274d0c9c5f310834f1097ff7be146504653f8f027420a965760542d3244a21d88d494e5b02792ddc2510ab348c2a5bfa3d2c897abb390642e30e5890eb@0.0.0.0:0
JSONRPC Admin Session Key: l397k9CiEsQ=
DEBUG 04-09 22:10:36 p2p  sync   865a6325…|Parity-Ethereum/v2.2.9-stable-5d5b372-20190203/x86_64-linux-gnu/rustc1.31.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:39 p2p  sync   6332792c…|Parity-Ethereum/v2.3.4-beta-0e95db1-20190220/x86_64-linux-gnu/rustc1.32.0 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:41 p2p  sync   d3cafe77…|GoChain/v3.1.1/linux-amd64/go1.12beta2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:41 p2p  sync   608b01a6…|Nodez/nodez.minerpool.net/v1.0.0/linux-amd64/go1.9.2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:45 p2p  sync   865a6325…|Parity-Ethereum/v2.2.9-stable-5d5b372-20190203/x86_64-linux-gnu/rustc1.31.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:46 p2p  sync   d3cafe77…|GoChain/v3.1.1/linux-amd64/go1.12beta2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:46 p2p  sync   608b01a6…|Nodez/nodez.minerpool.net/v1.0.0/linux-amd64/go1.9.2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:50 p2p  sync   865a6325…|Parity-Ethereum/v2.2.9-stable-5d5b372-20190203/x86_64-linux-gnu/rustc1.31.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   6f86387e…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   bed72917…|Pirl/v1.8.2-v5-masternode-premium-hulk-fd4b5d67/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   facdc4be…|Pirl/v1.8.2-v5-masternode-content-hulk-fd4b5d67/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   5e6341d3…|Pirl/v1.8.2-v5-masternode-premium-hulk-fd4b5d67/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   7e67b461…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   4d1ec344…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:52 p2p  sync   18e219e5…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   fe38f5e4…|Geth/v1.8.11-stable-dea1ce05/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   6064002a…|Pirl/v1.8.2-v5-masternode-premium-hulk-fd4b5d67/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   01886245…|Sipe/v1.0.1-stable-fe109f69/linux-amd64/go1.11.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   98c72023…|Geth/v0.0.9-Ether1_MN_SN-Security-Update/linux-amd64/go1.11.2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   1a926224…|Geth/testnet-miner-1/v1.8.10-stable-eae63c51/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   34052e64…|Geth/upool.in/v1.1.7.1-Ether1-Security-Update/linux-amd64/go1.12.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   a51d353c…|REOSC/v2.1.8-unstable-c04bcbf-20190125/x86_64-linux-gnu/rustc1.32.0 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:53 p2p  sync   4643eabb…|Geth/v1.9.0-unstable-2f5b6cb4/linux-amd64/go1.12.1 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:56 p2p  sync   d3cafe77…|GoChain/v3.1.1/linux-amd64/go1.12beta2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:56 p2p  sync   6332792c…|Parity-Ethereum/v2.3.4-beta-0e95db1-20190220/x86_64-linux-gnu/rustc1.32.0 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:56 p2p  sync   608b01a6…|Nodez/nodez.minerpool.net/v1.0.0/linux-amd64/go1.9.2 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:57 p2p  sync   6f86387e…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:57 p2p  sync   facdc4be…|Pirl/v1.8.2-v5-masternode-content-hulk-fd4b5d67/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:57 p2p  sync   5e6341d3…|Pirl/v1.8.2-v5-masternode-premium-hulk-fd4b5d67/linux-amd64/go1.10 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:57 p2p  sync   4d1ec344…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-09 22:10:57 p2p  sync   7e67b461…|egem/v1.1.0-valkryie-cd9024c2/linux-amd64/go1.12 Peer not suitable for sync: Invalid genesis hash.

[halfalicious]

It looks like mainnet syncing has been broken for quite a while... @gumb0 : When is the last time you were able to successfully sync mainnet?

[halfalicious]

I've investigated further and mainnet sync does appear to work, the "invalid genesis hash" errors are red herrings and are presumably due to Aleth attempting to sync with nodes which are running other networks.

[halfalicious]

I assumed that Aleth wouldn't attempt to sync with nodes on other networks since it can check the network id in the status capability packet that it receives from peers, but it looks like it isn't safe to depend on a particular network ID.

[gumb0]

Invalid genesis hash is not an error by itself, we check both genesis hash and networkID before we consider the peer suitable for sync.
So your log means discovery can't find main net peers. Deleting network.rlp to start discovery from scratch might help.

Mainnet sync works for me, though it can take 1-2 minutes to find the first suitable peer (when you start with empty peer list).

[gumb0]

It might make sense to check genesis hash after networkID, then we'll get fewer "Invalid genesis" messages.

Also maybe we should reformulate the message to something like "not equal to host's hash" or "not suitable hash" to make it appear less like an error. I remember people being confused about this before.

aleth/libethereum/EthereumPeer.cpp

Lines 67 to 81 in dbbfd5b

	std::string EthereumPeer::validate(
	h256 const& _hostGenesisHash, unsigned _hostProtocolVersion, u256 const& _hostNetworkId) const
	{
	std::string error;
	if (m_genesisHash != _hostGenesisHash)
	error = "Invalid genesis hash.";
	else if (m_protocolVersion != _hostProtocolVersion)
	error = "Invalid protocol version.";
	else if (m_networkId != _hostNetworkId)
	error = "Invalid network identifier.";
	else if (m_asking != Asking::State && m_asking != Asking::Nothing)
	error = "Peer banned for unexpected status message.";
	return error;
	}

[gumb0]

I see sometimes some other weird stuff in the sync log though

DEBUG 04-10 17:48:49 p2p  ethcap 742b99fe…|Geth/chain1_data_tokyo/v1.8.23-stable-c9427004/linux-amd64/go1.10.4 Status: 63 / 2030 / #c607cadf…, TD: 913141828821 = #25f768b8…
DEBUG 04-10 17:48:49 p2p  sync   742b99fe…|Geth/chain1_data_tokyo/v1.8.23-stable-c9427004/linux-amd64/go1.10.4 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-10 17:48:49 p2p  net    742b99fe…|Geth/chain1_data_tokyo/v1.8.23-stable-c9427004/linux-amd64/go1.10.4 Closing peer session :-(
DEBUG 04-10 17:48:49 p2p  peer   742b99fe…|Geth/chain1_data_tokyo/v1.8.23-stable-c9427004/linux-amd64/go1.10.4 Requesting 1 block headers starting from 790943

This looks like we try to send header request, after we detected mismatching hash and disconnected. This shouldn't happen, and I don't remember this happening before.

Could be also logging issue (maybe in reality we send request to another node, not the one that it says)

Sometimes it happens several times in a row:

EBUG 04-10 17:48:48 p2p  ethcap 84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Status: 63 / 17 / #303a4804…, TD: 7482247 = #1121b2bf…
DEBUG 04-10 17:48:48 p2p  sync   84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Peer not suitable for sync: Invalid genesis hash.
DEBUG 04-10 17:48:48 p2p  net    84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Closing peer session :-(
DEBUG 04-10 17:48:48 p2p  peer   84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Requesting 1 block headers starting from 790943
DEBUG 04-10 17:48:48 p2p  peer   84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Requesting 1 block headers starting from 790943
DEBUG 04-10 17:48:48 p2p  peer   84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Requesting 1 block headers starting from 790943
DEBUG 04-10 17:48:48 p2p  peer   84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 Requesting 1 block headers starting from 790943

I'm trying to investigate this.

[gumb0]

This looks like we try to send header request, after we detected mismatching hash and disconnected. This shouldn't happen, and I don't remember this happening before.

Could be also logging issue (maybe in reality we send request to another node, not the one that it says)

Ok, that's a logging issue, we send these requests to other nodes, but now I think it was like this always. It's just the first "context" part like 84d69e48…|Geth/v1.8.17-stable-1094c793/linux-amd64/go1.11.6 means that we're inside the handler of the message from this node. But we can send requests to other nodes from there.

I'll add request destination to these messages to make it less confusing.

But then I discovered another issue #5556