fix: interrupt publisher send-at-slot sleep on sequencer stop#23990
Merged
spypsy merged 2 commits intoJun 10, 2026
Conversation
spalladino
approved these changes
Jun 10, 2026
AztecBot
pushed a commit
that referenced
this pull request
Jun 10, 2026
## Summary - Propagate `CheckpointProposalJob.interrupt()` to its `SequencerPublisher` so the publisher's `sendRequestsAt` slot-deadline sleep is cancelled on sequencer stop. - Check `interrupted` *before* sleeping in `sendRequestsAt`, since `InterruptibleSleep.interrupt()` only resolves sleeps already in flight. Completes #23930, which made the job's own polling waits interruptible but not the publisher's. In [this `e2e_ha_full.test.ts` flake](http://ci.aztec-labs.com/1c46f3d4a226073d) a node became proposer 165ms before teardown and took the no-broadcast votes path, leaving `pendingL1Submission = publisher.sendRequestsAt(targetSlot)` sleeping until the target slot — ~22 wall-clock minutes away under the warped test clock. `Sequencer.stop()` blocked on it ("Awaiting pending L1 payload submission"), node stops were abandoned, and Jest hung on leaked handles until CI killed the run. Nothing in the shutdown path interrupts the per-job `SequencerPublisher`: `publisherFactory.stopAll()` only interrupts the underlying `L1TxUtils`. ## Tests - `checkpoint_proposal_job.test.ts`: pending L1 submission blocked in the publisher resolves promptly on `job.interrupt()` (red without the fix). - `sequencer-publisher.test.ts`: `sendRequestsAt` returns immediately when interrupted before the sleep starts (red without the fix).
Collaborator
|
✅ Successfully backported to backport-to-v5-next-staging #23994. |
spalladino
added a commit
that referenced
this pull request
Jun 10, 2026
Port of #23990 from merge-train/spartan. Propagates CheckpointProposalJob.interrupt() to its SequencerPublisher so the publisher's sendRequestsAt slot-deadline sleep is cancelled on sequencer stop, and checks interrupted before sleeping since InterruptibleSleep.interrupt() only resolves sleeps already in flight. The e2e_ha_full teardown changes from the original PR are superseded by the afterAll rework in this branch and are not ported.
spalladino
added a commit
that referenced
this pull request
Jun 10, 2026
Port of #23990 from merge-train/spartan. Propagates CheckpointProposalJob.interrupt() to its SequencerPublisher so the publisher's sendRequestsAt slot-deadline sleep is cancelled on sequencer stop, and checks interrupted before sleeping since InterruptibleSleep.interrupt() only resolves sleeps already in flight. The e2e_ha_full teardown changes from the original PR are superseded by the afterAll rework in this branch and are not ported.
PhilWindle
pushed a commit
that referenced
this pull request
Jun 11, 2026
…g anvil (#23979) Fixes the flaky HA full suite (`e2e_ha_full`) seen in http://ci.aztec-labs.com/8e1e980c4886df0d, where "should distribute work across multiple HA nodes" timed out awaiting a trigger tx. Also re-enables the suite, which #23976 had skipped. ## Root cause The HA compose suite was the only block-building suite running against an L1 with no self-advancing clock. Its anvil container ran in automine with no `--block-time`, and being external, it was excluded from the `TestDateProvider` sync that locally-spawned anvils get. L1 chain time only moved when something mined, while the shared sequencer clock free-ran. #23821 removed the `AnvilTestWatcher` that used to couple the two clocks in this mode and replaced it with per-iteration nudges in the test (clock warp + blind `mine(8)`). Two consequences, both visible in the failed run's logs: - The `mine(8)` overshoot put L1 ~1.5 slots ahead of the test clock, so each iteration's first propose raced its slot boundary and was silently dropped, followed by a prune that destroyed the pipelined builders' forks (`Fork not found` on all surviving nodes). This race was lost in passing runs too. - Recovery then required the proposers' archiver-sync gate to clear, but the gate's deadline runs on the free-running test clock while nothing mines L1 during the test's `waitForTx` — `Archiver did not sync L1 past slot 109 before slot 110 expired, discarding pipelined work`, repeated until the jest timeout. Whether a run passed or failed came down to seconds of margin on this gate. ## Fix Stop emulating L1 time in the test and run the suite in the same regime as every other block-building e2e (e.g. `e2e_epochs`): - Drop the anvil container and `ETHEREUM_HOSTS` from the HA compose file. With no external L1 configured, `setup()` spawns anvil in-proc with interval mining (`--block-time = ethereumSlotDuration`) and keeps the `TestDateProvider` snapped to L1 block timestamps via the existing stdout listener. The sibling web3signer compose suite already works this way. - Add `automineL1Setup: true` so L1 contract deployment runs under temporary automine before interval mining starts. - Delete all time scaffolding from the test (clock warps, cheat-mining heartbeats, archiver sync nudges). Tests submit a tx and wait, in real time. No assertions change. No production code changes: with a self-advancing L1, the sequencer and publisher behave exactly as on a real network. ## Parallelization The suite file is renamed to `e2e_ha_full.parallel.test.ts`, so CI runs each of its 8 tests as an isolated job in its own compose stack instead of one 15+ minute serial job: - `bootstrap.sh` expands the HA suite per test name (same mechanism as the existing `.parallel` simple tests). - `run_test.sh` forwards the test name into the compose stack and namespaces the docker compose project per test so concurrent jobs on one host don't collide. - `sendTriggerTx` now starts the HA sequencers idempotently, since under per-test isolation the governance/reload/distribute tests run without the first test (previously the only caller of `startHASequencers`). - Three clock-skew test titles contained parentheses, which jest's `--testNamePattern` interprets as regex groups (the filter would silently match nothing); they are retitled. ## Teardown fix (follow-up to the first CI round) The first CI round passed every test body but three jobs (produce-blocks, governance, reload) hung in `afterAll` until the job timeout. Two compounding causes, both fixed here: - `afterAll` reset the shared `TestDateProvider` *before* stopping nodes. The reset rewinds the clock from chain time to wall time — minutes apart after the automine deploy burst — so vote submissions armed against the rewound clock pushed sequencer stops out by that gap. The old 30s abandon-race then gave up, and the abandoned nodes outlived the jest environment, keeping the worker alive until the CI timeout (jest runs without `forceExit`). `afterAll` now stops sequencers first, awaits every node stop fully, and resets the clock last. These three jobs are the ones whose tests end with sequencers still running; the distribute test (which stops nodes in-test, before any reset) passed for the same reason. - Ports #23990 from `merge-train/spartan` (not previously on the v5 line): `CheckpointProposalJob.interrupt()` now propagates to the publisher, cancelling the `sendRequestsAt` slot-deadline sleep on sequencer stop, so a pending vote submission can never block shutdown. The original PR's `e2e_ha_full` teardown changes are superseded by the rework above and were not ported. ## Verification - Three full local runs of the suite via `run_test.sh ha` (all 8 tests each): green in 255s / 254s / 268s of jest time (the old warp-based suite ran 10+ minutes), with zero occurrences of the old failure signatures (`Fork not found`, `Archiver did not sync`, `discarding pipelined work`) — passing runs of the old code showed 12+ `Fork not found` errors even when green. - One per-test CI-style run (`run_test.sh ha <file> "should distribute work across multiple HA nodes"`): the originally flaky test passes standalone in its own compose stack (7 skipped, 1 passed), exercising the full `TEST_NAME` plumbing. - `yarn build`, `yarn format`, `yarn lint` clean; `sequencer-client` unit tests pass (back to the pre-change suite after the revert).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CheckpointProposalJob.interrupt()to itsSequencerPublisherso the publisher'ssendRequestsAtslot-deadline sleep is cancelled on sequencer stop.interruptedbefore sleeping insendRequestsAt, sinceInterruptibleSleep.interrupt()only resolves sleeps already in flight.e2e_ha_full.test.tsteardown so shared test clock skew is reset before shutdown and any HA node stop timeout fails explicitly after cleanup instead of surfacing later as a Jest open-handle flake.Completes #23930, which made the job's own polling waits interruptible but not the publisher's. In this
e2e_ha_full.test.tsflake a node became proposer 165ms before teardown and took the no-broadcast votes path, leavingpendingL1Submission = publisher.sendRequestsAt(targetSlot)sleeping until the target slot — ~22 wall-clock minutes away under the warped test clock.Sequencer.stop()blocked on it ("Awaiting pending L1 payload submission"), node stops were abandoned, and Jest hung on leaked handles until CI killed the run. Nothing in the shutdown path interrupts the per-jobSequencerPublisher:publisherFactory.stopAll()only interrupts the underlyingL1TxUtils.Tests
checkpoint_proposal_job.test.ts: pending L1 submission blocked in the publisher resolves promptly onjob.interrupt()(red without the fix).sequencer-publisher.test.ts:sendRequestsAtreturns immediately when interrupted before the sleep starts (red without the fix).sequencer.test.ts: related sequencer shutdown coverage still passes.yarn format end-to-endNODE_OPTIONS=--max-old-space-size=4096 yarn lint end-to-end