Ensure that pending to-device events are sent over federation at startup

[richvdh]

Fixes #16680, as well as a related bug, where servers which we had never successfully sent an event to would not be retried.

In order to fix the case of pending to-device messages, we hook into the existing wake_destinations_needing_catchup process, by extending it to look for destinations that have pending to-device messages. The federation transmission loop then attempts to send the pending to-device messages as normal.

Suggest review commit-by-commit.

[erikjohnston]

otherwise lgtm

[erikjohnston]

I think we should add a now param to looping_call (as the underlying twisted thing supports it), that way if the initial run takes a long time a second run won't get started by the looping call.

[richvdh]

done

[richvdh]

Hum. Turns out this screws up the type checker.

In order to add the new parameter without breaking existing code, it needs to be a named parameter (because excess positional parameters are passed to the wrapped function). However, it turns out that ParamSpec is incompatible with additional keyword args (https://peps.python.org/pep-0612/#id2 actually mentions this: "Note that this also why we have to reject signatures of the form (*args: P.args, s: str, **kwargs: P.kwargs).)

So, three options:

• Add a positional parameter to looping_call, and update all existing calls to looping_call (55 of them, according to my IDE).
• Add a new method looping_call2 (or something) which takes a positional now parameter. Use it here, and deprecate looping_call.
• Add a new method looping_call_now, which is exactly the same as looping_call except for the obvious. (The implementation of this will probably involve a private equivalent to looping_call2, shared between looping_call_now and looping_call).

My instinct is number 3, but happy with whatever you think.

[richvdh]

Well, I took an executive decision.

[richvdh]

The tests are now failing due to db connection pool shenanigans. #17017 should fix it.