Avoid using ordered maps in LLMQ signing code

[codablock]

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.

[UdjinM6]

LGTM, will re-review after rebase

[codablock]

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

[UdjinM6]

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

[codablock]

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

[codablock]

Rebased and force-pushed

[UdjinM6]

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

[UdjinM6]

needs rebase

[codablock]

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

1. Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
2. Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
3. SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
4. Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
5. After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
6. Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

[UdjinM6]

re-utACK

UPDATE: and thanks for a very informative explanation!

[codablock]

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

---

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

1. If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
2. LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
3. After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

[UdjinM6]

Ohhh, nice :) 👍

[PastaPastaPasta]

post-utACK