Refactor `backfilled` into specific behavior function arguments pt.1 by MadLittleMods · Pull Request #11396 · matrix-org/synapse

MadLittleMods · 2021-11-19T00:43:41Z

Refactor backfilled into specific behavior function arguments pt.1. This PR does not replace all backfilled instances. Just persist_events_and_notify and all downstream calls

Using keyword-only arguments (*) to reduce the mistakes from backfilled being left as a positional argument somewhere and being interpreted wrong by our new arguments.

Part of #11300

Dev notes

persist_events_and_notify (added inhibit_push_notifications)
- persist_events and persist_event
  - _event_persist_queue

_event_persist_queue ->

_persist_event_batch (added should_calculate_state_and_forward_extrems)
- _persist_events_and_state_updates (added use_negative_stream_ordering)
  - _persist_events_txn
    - _update_room_depths_txn (added update_room_forward_stream_ordering)
    - _update_metadata_tables_txn
      - _store_room_members_txn (added inhibit_local_membership_updates)

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
~~Pull request includes a sign off~~
Code style is correct
(run the linters)

Part of #11300

``` synapse/replication/http/federation.py:72: error: Signature of "_serialize_payload" incompatible with supertype "ReplicationEndpoint" [override] ```

MadLittleMods · 2021-11-19T07:54:05Z

synapse/replication/http/federation.py

+        should_calculate_state_and_forward_extrems,
+        use_negative_stream_ordering,
+        inhibit_local_membership_updates,
+        update_room_forward_stream_ordering,


Is there a way we can define the arguments here like the following? I can't get it to work even without the keyword-only marker.

async def _serialize_payload( store, room_id, event_and_contexts, *, inhibit_push_notifications: bool = False, should_calculate_state_and_forward_extrems: bool = True, use_negative_stream_ordering: bool = False, inhibit_local_membership_updates: bool = False, update_room_forward_stream_ordering: bool = True, ):

When I do this, I get the following error:

synapse/replication/http/federation.py:72: error: Signature of "_serialize_payload" incompatible with supertype "ReplicationEndpoint" [override]

And I can't figure out how to adjust the abstract method to be more flexible,

synapse/synapse/replication/http/_base.py

Lines 131 to 132 in 7ffddd8

@abc.abstractmethod

async def _serialize_payload(**kwargs):

I'm not entirely sure why, but it's the :bool type annotations that are upsetting it rather than the * or the default values. If I remove the :bools, it's fine.

MadLittleMods · 2021-11-19T08:25:07Z

synapse/storage/persist_events.py

 )

 import attr
+from mypy_extensions import DefaultNamedArg


Is there a better way to tackle this?

The build is failing because I used it builtins.ModuleNotFoundError: No module named 'mypy_extensions', https://github.com/matrix-org/synapse/runs/4261552290?check_suite_focus=true#step:6:739

MadLittleMods · 2021-11-19T08:27:57Z

synapse/handlers/federation_event.py

+                # Backfilled events have negative stream_ordering and happened
+                # in the past so we know that we don't need to update the
+                # stream_ordering tip for the room.
+                update_room_forward_stream_ordering=not backfilled,


In a future PR, I'll extend the behavior arguments further upstream in place of backfilled.

This PR does not replace all backfilled instances. Just persist_events_and_notify and all downstream calls

MadLittleMods · 2021-11-19T08:28:41Z

synapse/handlers/federation_event.py

                    self._message_handler.maybe_schedule_expiry(event)

-            if not backfilled:  # Never notify for backfilled events
+            if not inhibit_push_notifications:  # Never notify for backfilled events


Behavioral argument replacement: inhibit_push_notifications

MadLittleMods · 2021-11-19T08:29:12Z

synapse/storage/persist_events.py

            potentially_left_users: Set[str] = set()

-            if not backfilled:
+            if should_calculate_state_and_forward_extrems:


Behavioral argument replacement: should_calculate_state_and_forward_extrems

MadLittleMods · 2021-11-19T08:29:34Z

synapse/storage/databases/main/events.py

        # Note: Multiple instances of this function cannot be in flight at
        # the same time for the same room.
-        if backfilled:
+        if use_negative_stream_ordering:


Behavior argument replacement: use_negative_stream_ordering

Behavior argument replacement: use_negative_stream_ordering

sorry, what does this mean?

It's just the root place that I originally replaced and defines the behavior of the new argument.

MadLittleMods · 2021-11-19T08:30:20Z

synapse/storage/databases/main/events.py

            if (
                self.is_mine_id(event.state_key)
-                and not backfilled
+                and not inhibit_local_membership_updates


Behavioral argument replacement: inhibit_local_membership_updates

MadLittleMods · 2021-11-19T08:31:07Z

synapse/storage/databases/main/events.py


-            if not backfilled:
+            # TODO: test that this actuall works
+            if stream < 0:


TODO: Test that this actually works

MadLittleMods · 2021-11-19T08:31:29Z

synapse/storage/databases/main/events.py

            # Remove the any existing cache entries for the event_ids
            txn.call_after(self.store._invalidate_get_event_cache, event.event_id)
-            if not backfilled:
+            if update_room_forward_stream_ordering:


Behavioral argument replacement: update_room_forward_stream_ordering

MadLittleMods · 2021-11-19T08:31:38Z

synapse/handlers/federation_event.py

+        should_calculate_state_and_forward_extrems: bool = True,
+        use_negative_stream_ordering: bool = False,
+        inhibit_local_membership_updates: bool = False,
+        update_room_forward_stream_ordering: bool = True,


@richvdh How does this look? Is this on the right track?

richvdh

Thanks for putting this up, @MadLittleMods!

I haven't read all the way through the diff, because it's a bit hard to follow. Could I ask you to break it down a bit - let's start somewhere nearer the bottom of the stack: say _persist_events_and_state_updates. I think that will make it much easier to convince ourselves that the changes are correct.

Generally it seems like the right sort of thing, but I fear we might be taking it to extremes. If there are ways of grouping some of these flags together into coherent bits of functionality, I think that would be better. For example - does it ever make sense to set use_negative_stream_ordering=True and update_room_forward_stream_ordering=True together (or both False)? If not, let's get rid of update_room_forward_stream_ordering - otherwise it's just a nightmare to use the function.

richvdh · 2021-11-19T18:44:01Z

synapse/replication/http/federation.py

+        should_calculate_state_and_forward_extrems,
+        use_negative_stream_ordering,
+        inhibit_local_membership_updates,
+        update_room_forward_stream_ordering,


I'm not entirely sure why, but it's the :bool type annotations that are upsetting it rather than the * or the default values. If I remove the :bools, it's fine.

richvdh · 2021-11-19T18:47:04Z

synapse/handlers/federation_event.py

+                # Backfilled events have negative stream_ordering and happened
+                # in the past so we know that we don't need to update the
+                # stream_ordering tip for the room.
+                update_room_forward_stream_ordering=not backfilled,


shouldn't update_room_forward_stream_ordering be implied by use_negative_stream_ordering ?

richvdh · 2021-11-19T18:47:50Z

synapse/handlers/federation_event.py

+                # come down incremental `/sync`
+                use_negative_stream_ordering=backfilled,
+                # Backfilled events do not affect the current local state
+                inhibit_local_membership_updates=backfilled,


would it be sensible to have this implied by should_calculate_state_and_forward_extrems ?

I understand the mental leap where updating the state in the room also encompasses member state events.

My first impression was that it wasn't very obvious but it makes sense ⏩

richvdh · 2021-11-19T18:48:17Z

synapse/handlers/federation_event.py

+                # there is no need to update the forward extrems because we
+                # already know this event happened in the past if it was
+                # backfilled.
+                should_calculate_state_and_forward_extrems=not backfilled,


the should_ prefix seems a bit redundant here.

richvdh · 2021-11-19T18:51:48Z

synapse/handlers/federation_event.py

                    room_id=room_id,
                    event_and_contexts=batch,
-                    backfilled=backfilled,
+                    inhibit_push_notifications=inhibit_push_notifications,


doesn't this need to pass through all the keyword args?

Good catch 👍 I think so. Should match _serialize_payload.

It's a loose connection and I was having trouble typing it correctly, #11396 (comment), to have the linter yell at me for this.

richvdh · 2021-11-19T18:54:33Z

synapse/storage/databases/main/events.py

        # Note: Multiple instances of this function cannot be in flight at
        # the same time for the same room.
-        if backfilled:
+        if use_negative_stream_ordering:


Behavior argument replacement: use_negative_stream_ordering

sorry, what does this mean?

MadLittleMods · 2021-11-24T05:59:29Z

Thanks for the review @richvdh, great points! I've made a smaller PoC in #11417

MadLittleMods requested a review from a team as a code owner November 19, 2021 00:43

MadLittleMods removed the request for review from a team November 19, 2021 00:43

MadLittleMods marked this pull request as draft November 19, 2021 00:43

MadLittleMods force-pushed the madlittlemods/11300-refactor-backfilled-pt1 branch from 35e036a to 96d6b5c Compare November 19, 2021 00:47

MadLittleMods changed the title ~~Draft: Refactor backfilled behavior into specific behavior function parameters~~ Draft: Refactor backfilled behavior into specific behavior function arguments Nov 19, 2021

MadLittleMods force-pushed the madlittlemods/11300-refactor-backfilled-pt1 branch from 96d6b5c to f5e3731 Compare November 19, 2021 01:37

Refactor backfilled behavior into specific function parameters

d203d22

Part of #11300

MadLittleMods force-pushed the madlittlemods/11300-refactor-backfilled-pt1 branch from f5e3731 to d203d22 Compare November 19, 2021 01:37

MadLittleMods added 2 commits November 18, 2021 20:09

Pipe arguments across the function stack

b8c60b9

Fix event queue

d44fdcc

MadLittleMods changed the title ~~Draft: Refactor backfilled behavior into specific behavior function arguments~~ Draft: Refactor backfilled behavior into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods changed the title ~~Draft: Refactor backfilled behavior into specific behavior function arguments pt.1~~ Draft: Refactor backfilled arguments into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods changed the title ~~Draft: Refactor backfilled arguments into specific behavior function arguments pt.1~~ Draft: Refactor backfilled into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods added 2 commits November 19, 2021 00:43

Add changelog

ec82407

Fix incompatible type

56ea69f

``` synapse/replication/http/federation.py:72: error: Signature of "_serialize_payload" incompatible with supertype "ReplicationEndpoint" [override] ```

MadLittleMods commented Nov 19, 2021

View reviewed changes

Update comments docs to explain the why

f01d506

MadLittleMods commented Nov 19, 2021

View reviewed changes

MadLittleMods changed the title ~~Draft: Refactor backfilled into specific behavior function arguments pt.1~~ Refactor backfilled into specific behavior function arguments pt.1 Nov 19, 2021

MadLittleMods commented Nov 19, 2021

View reviewed changes

MadLittleMods requested a review from richvdh November 19, 2021 08:31

richvdh suggested changes Nov 19, 2021

View reviewed changes

MadLittleMods mentioned this pull request Nov 24, 2021

Refactor backfilled into specific behavior function arguments (_persist_events_and_state_updates) #11417

Merged

4 tasks

MadLittleMods added the T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. label Nov 29, 2021

MadLittleMods closed this Jun 14, 2022

Uh oh!

Conversation

MadLittleMods commented Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev notes

Pull Request Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Nov 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richvdh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods commented Nov 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MadLittleMods commented Nov 19, 2021 •

edited

Loading

MadLittleMods Nov 24, 2021 •

edited

Loading

MadLittleMods Nov 19, 2021 •

edited

Loading

MadLittleMods Nov 24, 2021 •

edited

Loading

MadLittleMods Nov 24, 2021 •

edited

Loading