Run the providers against the real thing - #44
Merged
Conversation
Four things were shipped on argument rather than evidence, and this is the evidence. PostgreSQL and SQL Server had never executed their own SQL. Only SQLite ran, and SQLite serialises writers with a file lock, which hides the difference between a correct statement and a lucky one -- the exact difference the two dialects exist to express. Both now run the whole suite against a real engine in a container, and both prove their own concurrency claim: remove ON CONFLICT DO NOTHING from PostgreSQL and the create race fails with 23505, duplicate key; remove UPDLOCK, HOLDLOCK from SQL Server and it fails with a primary key violation. Restore either and forty rounds of eight writers pass. That race test needed strengthening first. One round of eight writers passed against the racy statement it was meant to catch -- measured, not assumed -- so it runs forty rounds over fresh keys now, and the racy form fails in the first few seconds. CosmosDB had a hand-written fake that modelled what the service is documented to do. It now runs against the emulator, which is the only thing that can say whether CosmosDB really answers 412 to a stale If-Match and 409 to a second create, and whether the ETag a read hands back is one a write accepts. The provider is built entirely on those three answers. The dashboard component's unsubscribe was the half that leaks and the half without a test. bunit renders it, disposes it, and checks the handler it gives back is the one it subscribed with; mounting twice and disposing once leaves exactly one subscription outstanding. Removing the unsubscribe fails two of the three. One design fault found on the way, in tests written earlier today. xUnit builds a fresh instance of a test class per test method, so a container started from IAsyncLifetime on the class starts once per test. The CosmosDB emulator was being started nine times for nine tests, took sixteen and a half minutes, and two of them timed out waiting and skipped. Class fixtures start one container and share it: Cosmos went to nine passing in seconds, SQL Server from seventy seconds to one. Tests scope their keys by test name, because a shared server means shared state. The suite is 501 tests with nothing skipped, where it was 473 with eighteen skipped for want of a container.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four things were shipped on argument rather than evidence. This is the evidence.
PostgreSQL and SQL Server had never executed their own SQL
Only SQLite ran. SQLite serialises writers with a file lock, which hides the exact difference the two dialects exist to express. Both now run the full provider suite against a real engine in a container — and each proves its own concurrency claim by failing without it:
ON CONFLICT (name) DO NOTHING23505: duplicate key value violates unique constraintUPDLOCK, HOLDLOCKViolation of PRIMARY KEY constraint 'PK__healthie__...'Restore either and 40 rounds of 8 concurrent writers pass.
The race test needed strengthening first. One round of eight writers passed against the racy statement it was written to catch — measured, not assumed. It runs 40 rounds over fresh keys now, and the racy form fails within seconds.
CosmosDB had a fake standing in for the service
The fake modelled what CosmosDB is documented to do. It now also runs against the real emulator, which is the only thing that can confirm the service actually answers
412to a staleIf-Matchand409to a second create, and that the ETag a read hands back is one a write accepts. The whole provider is built on those three answers.The dashboard component's unsubscribe
The service-level method was tested; the component calling it — the half that leaks — was not. bunit renders it, disposes it, and checks the handler handed back is the one it subscribed with. Mounting twice and disposing once leaves exactly one subscription outstanding. Removing the unsubscribe fails 2 of the 3.
A design fault in tests I wrote earlier today
xUnit builds a fresh instance of a test class per test method, so a container started from
IAsyncLifetimeon the class starts once per test. The CosmosDB emulator was being started nine times for nine tests, took 16m35s, and two timed out waiting and silently skipped.Class fixtures start one container and share it:
Tests scope their keys by test name, because a shared server means shared state.
Result
501 tests, nothing skipped, on net8.0 and net10.0 — up from 473 with 18 skipped for want of a container. 0 errors, 0 warnings. Locked restore still passes; the four new packages are in the lock file.
Still not verified, and I can't fix it alone
Slack, Teams and PagerDuty delivery. The payloads are asserted on the wire and checked against published schemas — that's how the Adaptive Card colour-casing bug was caught — but nothing has ever been accepted by those services. That needs a real webhook URL and a PagerDuty routing key. Give me a throwaway of each and I'll verify delivery properly.
One flaky E2E test. Two failures across roughly eight full runs today. Every attempt to capture it with a trx logger passes, so I still cannot name it. Likely browser timeouts under load — the failing runs took 2-3× longer — but that is a hypothesis, not a finding.