Support sending and receiving MSC4354 Sticky Event metadata.

[reivilibre]

Part of: MSC4354 whose experimental feature tracking issue is #19409

Follows: #19340 (a necessary bugfix for /event/ to set this metadata)

Partially supersedes: #18968

This PR implements the first batch of work to support MSC4354 Sticky Events.

Sticky events are events that have been configured with a finite 'stickiness' duration,
capped to 1 hour per current MSC draft.

Whilst an event is sticky, we provide stronger delivery guarantees for the event, both to
our clients and to remote homeservers, essentially making it reliable delivery as long as we
have a functional connection to the client/server and until the stickiness expires.

This PR merely supports creating sticky events and receiving the sticky TTL metadata in clients.
It is not suitable for trialling sticky events since none of the other semantics are implemented.

The current plan is to follow this PR up with more PRs, roughly parcelled up as follows:

• Implement the sliding sync extension specified in MSC4354, to proactively notify
• Implement the oldschool sync support, specified in MSC4354, to do the same but for clients still using oldschool sync.
• Notice new servers joining rooms and proactively tell them about sticky events.
• Add special federation catch-up support for sticky events, so that (as long as we have a connection) they don't get 'dropped' in the gaps between federation /send requests.
• Re-evaluate soft-failed sticky events when we think that might be possible

1. Add MSC4354 experimental feature flag

2. Expose MSC4354 enablement on /versions

3. Add constants for sticky events

4. Add sticky_events table

5. Add sticky events store and stream

6. EventBase: add the concept of sticky_duration

7. EventBuilder: allow building events with sticky event fields

8. store method: insert_sticky_events_txn

9. When persisting currently-sticky events, add to sticky event stream

10. Allow clients to send sticky events
    Including delayed events

11. Add test helper for sending sticky events

12. Expose the sticky event TTL to clients

13. Add a test for sticky TTL calculation and exposure to clients

[MadLittleMods]

The separation between StickyEventField and StickyEvent.FIELD_NAME is slightly mind bending.

[MadLittleMods]

Ugh! Future stuff but we should consider making soft_failed a dedicated column of the events table or some table. Is this performant enough?

[reivilibre]

I can't say I'm entirely happy with it, but it feels like it shouldn't be a very common case and the table is altogether small.

I don't think it is super expensive to fish out of the small internal_metadata JSON field in the event_json table (other than doing that heap lookup).
(It would be nicer if we didn't have a freeform internal_metadata and instead had nullable fields IMO, but I'm not sure that would greatly improve much.)

At this point I think it's better than risking stale denormalised data.

[MadLittleMods]

trial-olddeps -> sqlite3.OperationalError: near ">>": syntax error

trial-olddeps CI failing ❌, https://github.com/element-hq/synapse/actions/runs/21635340214/job/62359543300?pr=19365#step:10:7282

tests.storage.test_sticky_events.StickyEventsTestCase.test_get_updated_sticky_events
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/runner/work/synapse/synapse/tests/storage/test_sticky_events.py", line 153, in test_get_updated_sticky_events_with_limit
    updates = self.get_success(
  File "/home/runner/work/synapse/synapse/tests/unittest.py", line 707, in get_success
    return self.successResultOf(deferred)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/trial/_synctest.py", line 706, in successResultOf
    self.fail(
twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f36aca5efe0 current result: None>, found failure result instead:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 517, in errback
    self._startRunCallbacks(fail)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 580, in _startRunCallbacks
    self._runCallbacks()
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 1514, in gotResult
    current_context.run(_inlineCallbacks, r, g, status)
--- <exception caught here> ---
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 1443, in _inlineCallbacks
    result = current_context.run(result.throwExceptionIntoGenerator, g)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/databases/main/sticky_events.py", line 144, in get_updated_sticky_events
    return await self.db_pool.runInteraction(
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 1015, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 1443, in _inlineCallbacks
    result = current_context.run(result.throwExceptionIntoGenerator, g)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 981, in _runInteraction
    result: R = await self.runWithConnection(
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 1117, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/internet/defer.py", line 662, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/runner/work/synapse/synapse/tests/server.py", line 814, in <lambda>
    d.addCallback(lambda x: function(*args, **kwargs))
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/python/compat.py", line 403, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 1110, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 819, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/databases/main/sticky_events.py", line 160, in _get_updated_sticky_events_txn
    txn.execute(
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 458, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/synapse/storage/database.py", line 520, in _do_execute
    return func(sql, *args, **kwargs)
sqlite3.OperationalError: near ">>": syntax error

[MadLittleMods]

Related discussion in #synapse-dev:matrix.org

[MadLittleMods]

Probably fine as-is. We don't have any great options ⏩

[reivilibre]

Because it's not very trivial to unstick our SQLite supported version situation, for right now I'm going to merge this with a workaround for when the SQLite version is technically unsupported, such as on Ubuntu 22.04. I've opened a ticket to track this #19452 because I'd like to remove this workaround soon.

This does mean that the feature is not getting tested under olddeps, which currently uses Ubuntu 22.04. This is definitely not OK when the feature gets stabilised.

Another approach we could do, if SQLite 3.37 support is a requirement, is to mount a custom function into SQLite to deal with the JSON.