Add in-repo Complement tests by MadLittleMods · Pull Request #19406 · element-hq/synapse

MadLittleMods · 2026-01-23T22:45:31Z

Add in-repo Complement tests. This is useful so we can test Synapse specific behaviors like our admin API.

(see docs in PR, complement/README.md)

COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo

Complement calls these "out-of-repo" tests but it's a bit of a misnomer once they're in your project. (just depends on the perspective)

There has been previous desire for this kind of thing but this is spawning from wanting to have some tests for our purge history admin API (https://github.com/element-hq/synapse-rust-apps/issues/430). There are some Sytest tests (matrix-org/sytest -> tests/48admin.pl#L91-L618) for this already but I'd much rather work in Complement instead of Sytest. I'm wanting these tests to ensure that our new event-cache rust app for Synapse Pro doesn't break these kind of erasure features (https://github.com/element-hq/synapse-rust-apps/issues/366 and https://github.com/element-hq/synapse-rust-apps/issues/153).

Interestingly, there is already matrix-org/complement -> tests/csapi/admin_test.go (added in matrix-org/complement#322) in the Complement repo iteslf that tests the /_synapse/admin/v1/send_server_notice endpoint but it's a bit of an interesting case as Dendrite also supports this endpoint. I don't think it's good practice to continually shove in more and more Synapse-specific behavior into the Complement repo itself.

We already have success with other out-of-repo tests for projects like the SBG, TI-Messenger Proxy, and our Synapse Pro for small hosts.

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

MadLittleMods · 2026-01-23T22:51:34Z

.github/workflows/latest_deps.yml

+        env:
+          POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
+          WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
+          TEST_ONLY_IGNORE_POETRY_LOCKFILE: 1


This is just aligning to what we already do in .github/workflows/tests.yml

The difference is the TEST_ONLY_IGNORE_POETRY_LOCKFILE here

MadLittleMods · 2026-01-23T22:52:28Z

.github/workflows/twisted_trunk.yml

+        env:
+          POSTGRES: ${{ (matrix.database == 'Postgres') && 1 || '' }}
+          WORKERS: ${{ (matrix.arrangement == 'workers') && 1 || '' }}
+          TEST_ONLY_SKIP_DEP_HASH_VERIFICATION: 1


This is just aligning to what we already do in .github/workflows/tests.yml

The difference is the TEST_ONLY_SKIP_DEP_HASH_VERIFICATION here

.github/workflows/tests.yml

MadLittleMods · 2026-01-23T22:57:36Z

complement/.golangci.yml

+  enable:
+    - gofmt
+    - goimports
+    - golines


Same config that we have in element-hq/synapse-small-hosts -> complement/.golangci.yml

https://github.com/element-hq/synapse/actions/runs/21303838770/job/61327615707?pr=19406 ``` +++ dirname synapse/scripts-dev/complement.sh ++ realpath synapse/scripts-dev/../complement realpath: synapse/scripts-dev/../complement: No such file or directory + in_repo_test_suite= ```

anoadragon453

Tried it out locally and it worked first try!

Couple suggestions below.

.github/workflows/tests.yml

scripts-dev/complement.sh

anoadragon453 · 2026-02-04T18:50:38Z

scripts-dev/complement.sh

+      "--in-repo")
+        use_in_repo_tests=1
+        ;;


We can also avoid fetching the latest Complement checkout when --in-repo is passed.

Probably easiest not to complicate things and just have it act as similarly as possible for either suite.

Similar reason, why I'd still recommend COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh --in-repo even though COMPLEMENT_DIR isn't relevant for the --in-repo tests. Just easier to run with the same command so that when you run COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh, it still just works.

Ideally, we'd also share the Complement version between both suites though.

I don't think it'd be too complicated to implement or too confusing for the user. Would it not just be adding a check for $use_in_repo_tests to:

synapse/scripts-dev/complement.sh

Line 135 in 5f21a30

if [[ -z "$COMPLEMENT_DIR" ]]; then

This would prevent the Complement codebase from being downloaded when CI runs only the in-repo tests.

I think it is that simple but not worth it 👍

You almost had be convinced with the CI argument but since both suites are run in the same job, the file system will be shared and so the second complement.sh will see the already checked out Complement. This actually even applies to both steps because .ci/scripts/setup_complement_prerequisites.sh checks out the repo for us.

Fair enough! In that case - since the Complement repo is fairly small in size - let's just KISS.
_{(...acronym. ACRONYM!!)}

See #19406 (comment)

anoadragon453

Exciting foundation step - thank you!

anoadragon453 · 2026-02-05T10:34:21Z

scripts-dev/complement.sh

+      "--in-repo")
+        use_in_repo_tests=1
+        ;;


Fair enough! In that case - since the Complement repo is fairly small in size - let's just KISS.
_{(...acronym. ACRONYM!!)}

MadLittleMods · 2026-02-05T23:12:43Z

Thanks for the review @anoadragon453 🐓

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [element-hq/synapse](https://github.com/element-hq/synapse) | minor | `v1.147.1` → `v1.148.0` | --- ### Release Notes <details> <summary>element-hq/synapse (element-hq/synapse)</summary> ### [`v1.148.0`](https://github.com/element-hq/synapse/releases/tag/v1.148.0) [Compare Source](element-hq/synapse@v1.147.1...v1.148.0) ### Synapse 1.148.0 (2026-02-24) No significant changes since 1.148.0rc1. ### Synapse 1.148.0rc1 (2026-02-17) #### Features - Support sending and receiving [MSC4354 Sticky Event](matrix-org/matrix-spec-proposals#4354) metadata. ([#19365](element-hq/synapse#19365)) #### Improved Documentation - Fix reference to the `experimental_features` section of the configuration manual documentation. ([#19435](element-hq/synapse#19435)) #### Deprecations and Removals - Remove support for [MSC3244: Room version capabilities](matrix-org/matrix-spec-proposals#3244) as the MSC was rejected. ([#19429](element-hq/synapse#19429)) #### Internal Changes - Add in-repo Complement tests so we can test Synapse specific behavior at an end-to-end level. ([#19406](element-hq/synapse#19406)) - Push Synapse docker images to Element OCI Registry. ([#19420](element-hq/synapse#19420)) - Allow configuring the Rust HTTP client to use HTTP/2 only. ([#19457](element-hq/synapse#19457)) - Correctly refuse to start if the Rust workspace config has changed and the Rust library has not been rebuilt. ([#19470](element-hq/synapse#19470)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/4203 Co-authored-by: Renovate Bot <renovate-bot@alexlebens.net> Co-committed-by: Renovate Bot <renovate-bot@alexlebens.net>

MadLittleMods added 2 commits January 23, 2026 15:59

Add in-repo Complement tests

b681dc8

Update and align CI to run in-repo Complement tests

3718517

MadLittleMods added the A-Testing label Jan 23, 2026

MadLittleMods added 2 commits January 23, 2026 16:48

Explain temporary test

ff3472c

Add changelog

fdfe621

MadLittleMods commented Jan 23, 2026

View reviewed changes

Better label second format step

616b07a

MadLittleMods commented Jan 23, 2026

View reviewed changes

.github/workflows/tests.yml Show resolved Hide resolved

MadLittleMods commented Jan 23, 2026

View reviewed changes

MadLittleMods added 3 commits January 23, 2026 17:21

Run in-repo quicker for feedback

abe6a65

Fix CI paths

5f21a30

MadLittleMods marked this pull request as ready for review January 24, 2026 00:29

MadLittleMods requested a review from a team as a code owner January 24, 2026 00:29

anoadragon453 reviewed Feb 4, 2026

View reviewed changes

MadLittleMods added 3 commits February 4, 2026 14:31

Merge branch 'develop' into madlittlemods/in-repo-complement-tests

8b4ce24

Fix extra "test" typo

9b4e237

See #19406 (comment)

Better docs

5aba74e

MadLittleMods requested a review from anoadragon453 February 4, 2026 21:08

anoadragon453 approved these changes Feb 5, 2026

View reviewed changes

Merge branch 'develop' into madlittlemods/in-repo-complement-tests

289890d

MadLittleMods merged commit 46d235c into develop Feb 5, 2026
45 checks passed

MadLittleMods deleted the madlittlemods/in-repo-complement-tests branch February 5, 2026 23:11

Conversation

MadLittleMods commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anoadragon453 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MadLittleMods commented Feb 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MadLittleMods commented Jan 23, 2026 •

edited

Loading

MadLittleMods Feb 4, 2026 •

edited

Loading