feat(sequencer): AutomineSequencer for single-sequencer e2e tests#23354
Merged
Conversation
spalladino
added
ci-draft
spalladino
force-pushed
the
palla/automine-sequencer
branch
from
5c63cb3 to
cc5f781
Compare
PhilWindle
approved these changes
May 20, 2026
A minimal, deterministic, queue-driven sequencer for e2e tests that don't exercise block-building or consensus. Reuses SequencerPublisher, FullNodeCheckpointsBuilder, and GlobalVariableBuilder; skips proposer-turn checks, validator orchestration, attestations, pipelining, P2P gossip, timetable enforcement, and event emission. Uses anvil setAutomine(true) with no interval mining; pre-sets next L1 block timestamp at slot boundaries when needed. Mempool-driven builds and explicit warp/buildEmptyBlock requests share a single serial queue and never interleave. Requires aztecTargetCommitteeSize=0 on the deployed rollup (the e2e default) so empty CommitteeAttestationsAndSigners is accepted by L1.
…chestration Adds the AUTOMINE_E2E_OPTS preset that opts a single-sequencer non-block-building test into the AutomineSequencer path. Adds a useAutomineSequencer flag to SetupOptions for the fixture to branch on. Adds four getters to AztecNodeService (getWorldStateSynchronizer, getL1ToL2MessageSource, getEpochCache, getGlobalVariableBuilder) so the test fixture can construct an AutomineSequencer alongside an otherwise-headless node (disableValidator + dontStartSequencer).
…ture Adds a useAutomineSequencer config flag. When set, AztecNodeService.createAndSync constructs an AutomineSequencer inside the existing validator-enabled branch, reusing the same L1 deps (l1TxUtils, publisher manager, publisher factory, checkpoints builder) instead of going through SequencerClient.new. AztecNodeService.mineBlock routes to AutomineSequencer.buildEmptyBlock when the automine path is wired. CheatCodes.warpL2TimeAtLeastTo delegates to the queue when an AutomineSequencer is wired, so existing test helpers work unchanged. Adds the AutomineSequencer + AutomineSequencerDeps + AutomineSequencerConstants exports from sequencer-client, exposes getPublisherConfigFromSequencerConfig, and adds a USE_AUTOMINE_SEQUENCER env var. Adds e2e_automine_smoke.test.ts exercising sequential txs, parallel txs, warp, and mineBlock under AUTOMINE_E2E_OPTS.
Without waiting, the next mempool-driven build picks up a stale tip and L1 rejects the propose with Rollup__InvalidArchive — the freshly-built header points at the pre-publish lastArchive, but L1's lastArchive has already advanced to the just-published checkpoint. Also cap the smoke-test warp at 24s (2 slots) so it doesn't cross the L1 proof-submission window and trigger a separate prune-related code path.
Mechanical swap from PIPELINING_SETUP_OPTS to AUTOMINE_E2E_OPTS for tests that don't exercise block-building or consensus: - e2e_authwit - e2e_keys - e2e_partial_notes - e2e_orderbook - e2e_double_spend Each tx now lands as soon as the AutomineSequencer picks it up from the mempool rather than waiting for the next 12s slot boundary.
- Drop duplicate re-export of getPublisherConfigFromSequencerConfig from publisher/index.ts; it's already available via sequencer-client's top-level re-export through ./config.js -> ./publisher/config.js. - Drop redundant async keyword from the buildIfPending queue callback (returns runBuild's promise directly, no await needed).
…OPTS Mechanical swap from PIPELINING_SETUP_OPTS to AUTOMINE_E2E_OPTS for 28 more single-sequencer non-block-building tests. Tests with hand-rolled warps (e2e_lending_contract), multi-PXE setups (e2e_2_pxes), or block-building semantics (e2e_block_building, e2e_pruned_blocks, e2e_sequencer_config, e2e_l1_with_wall_time, e2e_expiration_timestamp, e2e_genesis_timestamp) are intentionally held for follow-up.
Pulls the ~50 LOC of inline AutomineSequencer construction out of AztecNodeService.createAndSync into a dedicated factory under the sequencer-client package. Server-side code is back to a single function call; the factory owns publisher-manager wiring, attestor lookup, and EthCheatCodes construction. Pure refactor — no behavior change.
…encer compat
Test-side helper relied on setConfig({ minTxsPerBlock: 0 }) and a polling
wait, expecting the production Sequencer's polling loop to fire an empty
block when no txs were pending. AutomineSequencer never fires unprompted —
it builds on tx arrival or explicit request only, so the wait timed out.
aztecNode.mineBlock() already routes to AutomineSequencer.buildEmptyBlock
when wired, or temporarily drops minTxsPerBlock and triggers the production
sequencer otherwise — works under both presets.
The test does a full L1 reorg via rollbackTo + resumeSync + forceEmptyBlock. AutomineSequencer has no reorg awareness — after a rollback, its archiver- sync wait races against the archiver re-ingesting the original checkpoint from L1, causing either a Rollup__InvalidArchive on re-publish or a deadlock on the sync wait. Reorg semantics are out of scope for AUTOMINE_E2E_OPTS; this test stays on PIPELINING_SETUP_OPTS where it works cleanly.
Implements revertToCheckpoint(targetCheckpoint) on AutomineSequencer, running inside the serial queue so it never interleaves with builds. The sequence: 1. Fetch the target checkpoint's L1 block number from the archiver. 2. Call archiverRollback() to reset the archiver to the target checkpoint boundary (must happen before the L1 reorg so the archiver can still read the target checkpoint's L1 block hash). 3. Call worldState.syncImmediate() to propagate the archiver prune event to world-state before the next build runs. 4. Reorg L1 with reorg(depth) to drop blocks strictly after the target checkpoint publish block, keeping that block as the new tip. 5. Drop all pending L1 txs (anvil_rollback re-queues them) and reset the publisher's cached nonce so the next propose tx uses the correct post-reorg nonce. 6. Reset lastBuiltSlot and sync the date provider. Also adds resetNonce() to L1TxUtils to support step 5, and wires archiverRollback / resetPublisherNonces callbacks through the factory. Smoke test: adds "revertToCheckpoint rolls back L1+L2 state" to e2e_automine_smoke.test.ts, verifying the L2 tip drops and a new tx lands cleanly after the revert. All 5 scenarios pass.
… revertToCheckpoint
Replaces the PIPELINING_SETUP_OPTS + pauseSync/rollbackTo/resumeSync
pattern with AUTOMINE_E2E_OPTS + AutomineSequencer.revertToCheckpoint().
Changes:
- Switch setup preset from PIPELINING_SETUP_OPTS to AUTOMINE_E2E_OPTS.
- forceReorg now captures checkpointBeforeTx (the checkpointed tip
before the transfer tx lands) and calls revertToCheckpoint() with
that number. This atomically rolls back L1, the archiver, and
world-state in one call.
- forceEmptyBlock uses aztecNodeService.mineBlock() instead of
setConfig({ minTxsPerBlock: 0 }), which is the correct path under
AutomineSequencer.
- Remove pauseSync/resumeSync calls (no longer needed; the archiver
never pauses under AutomineSequencer).
- After the reorg, the p2p tx pool restores the rolled-back transfer tx
to pending, so the AutomineSequencer re-mines it automatically.
Three bugs caused the reorg test to fail: 1. After `archiverRollback`, the P2P block stream's `chain-pruned` event was only processed on the next poll cycle (~50ms later), so rolled-back txs weren't restored to pending before the next build ran. Fix: call `syncP2P()` in `runRevert` to force one immediate P2P block stream work cycle. 2. `runRevert` was calling `dateProvider.setTime(newL1Ts * 1000)` to roll the date provider back to the target checkpoint's L1 timestamp. Since the restored tx's `receivedAt` was recorded after that timestamp, `getEligiblePendingTxHashes` filtered it out as "too new". Fix: remove the `setTime` call from `runRevert` so the date provider keeps its current time. 3. `runBuild` called `worldState.fork(syncedToBlockNumber)` using the archiver's tip, but world state syncs from the archiver asynchronously. When the mempool-driven build for the re-queued tx completed and the follow-up empty block build ran immediately after, world state could still be one block behind, causing "Unable to initialize from future block". Fix: call `worldState.syncImmediate(syncedToBlockNumber)` before forking. Also adds a `retryUntil` loop in the test to wait for PXE to process the re-mined block's notes, since PXE syncs asynchronously from the archiver.
…k-hash race The test's far L1 time warp in an earlier it-block races the archiver's pruning of unpublished blocks; the PXE was mid-prove with an anchor on a now-pruned block. Pre-existing fragility under pipelining (verified: this PR's diff doesn't touch the affected code paths). Scoped to the specific error so it doesn't mask unrelated failures.
The test's far L1 time warp previously raced the archiver's pruning of unpublished blocks. Under AUTOMINE_E2E_OPTS the warp goes through the AutomineSequencer's serial queue, which guarantees no in-flight build when the warp runs — eliminating the race. Removes the corresponding flaky entry from .test_patterns.yml. Also adjusts the "lower than next block" expiration threshold: under pipelining the next block was 2 slots ahead (so the threshold was 2*slotDuration - 1n); under AutomineSequencer the next block is always 1 slot ahead, so the threshold is slotDuration - 1n.
…eps in AutomineSequencer archiverRollback -> Pick<Archiver, 'rollbackTo'>; resetPublisherNonces -> typed publisher/L1TxUtils dep; syncP2P -> method on p2pClient (with the dep type widened to the concrete P2PClient if the P2P interface doesn't expose sync). Pure refactor — no behavior change.
Pushes non-awaited tx sends interleaved with warpL2TimeAtLeastBy calls and verifies all txs land. Exercises the AutomineSequencer's SerialQueue guarantee that warps and builds never interleave.
… add README Pure reorganization. AutomineSequencer + factory now live under src/sequencer/automine/ with a dedicated README explaining the design and when to use AUTOMINE_E2E_OPTS. No behavior change.
Long chain of dependent deploy txs runs much faster under automine (each tx mines instantly); incidentally addresses the pre-existing CI flakiness this test was reported with.
Proven anchor stays at genesis automatically because AUTOMINE_E2E_OPTS disables the AnvilTestWatcher and the AutomineSequencer never marks checkpoints as proven on its own, removing the need for the prior isMarkingAsProven / minTxsPerBlock dance. The explicit empty-block advance now goes through aztecNode.mineBlock() instead of toggling minTxsPerBlock via aztecNodeAdmin.setConfig.
…MINE_E2E_OPTS Mechanical PIPELINING_SETUP_OPTS -> AUTOMINE_E2E_OPTS swap for tests that don't exercise block-building or consensus. Covers blacklist token, deploy contract harnesses plus assorted standalone tests.
…grate e2e_mempool_limit The new admin RPCs freeze block production without depending on minTxsPerBlock manipulation. Implemented on both production Sequencer and AutomineSequencer paths. e2e_mempool_limit now uses pauseSequencer to fill the pool deterministically, and runs under AUTOMINE_E2E_OPTS.
Replaces progressSlots' manual warp + mineBlock with the queue-aware warpL2TimeAtLeastTo (atomic warp + mineBlock under AutomineSequencer), and switches the suite from PIPELINING_SETUP_OPTS to AUTOMINE_E2E_OPTS. markAsProven is kept since the e2e fixture does not wire an EpochTestSettler and SLOT_JUMP=10 crosses unproven epoch boundaries that would otherwise trigger a reorg.
…oCheckpoint Uses the AutomineSequencer's queue-driven revertToCheckpoint to perform the prune scenarios instead of the legacy aztecNodeAdmin.rollbackTo path. Pruning becomes an atomic queue step that can't race with builds.
Switches the setup to AUTOMINE_E2E_OPTS and pins DEFAULT_TEST_UPDATE_DELAY to MINIMUM_UPDATE_DELAY (no longer derived from the AZTEC_SLOT_DURATION env var). Adds an explicit aztecNode.mineBlock() after each warp so the PXE's anchor header sits past the update's timestampOfChange (the warp advances L1 time but does not by itself produce an L2 block). Removes the pipelining-only TODO about propose_action_not_successful cascades — under AUTOMINE_E2E_OPTS enableProposerPipelining is false, so the failure mode the TODO described no longer applies.
…er caveat - fixtures.ts AUTOMINE_E2E_OPTS JSDoc pointed at the pre-move file path. - automine/README.md said the factory returns a started sequencer; only the PublisherManager is started by the factory (the node service starts the AutomineSequencer separately). - automine/README.md now warns that the e2e fixture does NOT wire EpochTestSettler; tests crossing epoch boundaries must call cheatCodes.rollup.markAsProven explicitly.
…sher manager - funderL1TxUtils was created and used by PublisherManager but excluded from the per-reorg resetNonce() list, leaving stale nonces if funding txs got rolled back. - AutomineSequencer.stop() now also stops the PublisherManager so node teardown drains its funding loop and L1TxUtils workers cleanly.
… and migrate 14 tests TokenContractTest.setup() and NestedContractTest.setup() now accept a Partial<SetupOptions>, mirroring BlacklistTokenContractTest.setup. All 14 sub-tests they back are migrated to AUTOMINE_E2E_OPTS.
…nly getters
- stop() now stops the AutomineSequencer alongside the production Sequencer
(the started-resource cleanup hook only fired on create failure, not normal
teardown).
- pauseSequencer + setConfig({ minTxsPerBlock }) now compose correctly:
setConfig while paused updates the saved restore value, keeping the freeze
applied until resume.
- Drop unused public getters (getWorldStateSynchronizer, getL1ToL2MessageSource,
getEpochCache, getGlobalVariableBuilder).
…Queued Mirrors checkpoint_proposal_job's failed-tx handling under AutomineSequencer: wraps buildBlock in try/catch for InsufficientValidTxsError, drops the error's failedTxs from the P2P mempool, and also drops failedTxs from successful buildResults. Without this, invalid txs would stay pending and the mempool poller would retry them forever. Also changes buildQueued from a boolean cleared at queue dequeue (before runBuild started) to a Promise that stays set until runBuild completes in the finally block. Coalesced callers now wait on the same promise instead of seeing a stale undefined. Addresses codex round-2 findings #1 and #2.
AutomineSequencer.warpTo only advances L1 time — cheat_codes.warpL2TimeAtLeastTo promises in its JSDoc to also mine an L2 block. The automine branch was returning before mining, leaving callers to add manual aztecNode.mineBlock() calls. Adds await node.mineBlock() after the queued automine.warpTo and removes the now-redundant manual calls from e2e_contract_updates. Keeps the explicit cheatCodes.eth.warp calls in e2e_crowdfunding_and_claim and e2e_expiration_timestamp: those warp multiple slots forward (a 7-day deadline; multi-slot in expiration) and mining after would cause the archiver to predict-reorg prior checkpoints.
… helper
- Read the L1 pending block's timestamp via eth_getBlock('pending') in
runBuild instead of assuming lastBlockTimestamp + 1. The previous +1
assumption breaks if setNextBlockTimestamp was called or blockInterval
differs from 1.
- Reshape runWarp to setNextBlockTimestamp + runBuild({ allowEmpty: true })
rather than ethCheatCodes.warp + caller-side node.mineBlock(). The old
flow mined two L1 blocks (one in warp, one in the subsequent mineBlock),
overshooting the target slot boundary by L2_SLOT_DURATION. Now we mine
exactly one L1 block via the empty checkpoint's propose tx. The mineBlock
call in cheat_codes.warpL2TimeAtLeastTo is removed because runWarp does
the mine itself.
- Extract the buildBlock try/catch in runBuild into tryBuildBlock.
… target slot
Adds `EthCheatCodes.nextBlockTimestamp()` mirroring `lastBlockTimestamp()` but
reading the pending block tag. Uses it in AutomineSequencer.runBuild to derive
the target slot, replacing the inline `publicClient.getBlock({blockTag:'pending'})`
call. Drops the now-redundant `lastBlockTimestamp()` call: the guard
`slotBoundaryTs > pendingBlockTs` is sufficient because pendingBlockTs is always
> lastBlockTimestamp.
…after AutomineSequencer was waiting (via retryUntil) for the archiver's L1 polling cycle to surface each newly-published checkpoint before the next build could proceed. Replace with two synchronous steps: - BEFORE sendRequests: push the block and proposed checkpoint into the archiver locally via addBlock + addProposedCheckpoint. Doing this before the L1 publish avoids racing the archiver's L1 polling, which would otherwise pick up the just-published checkpoint and reject our local push as a duplicate. - AFTER sendRequests: call archiver.syncImmediate() to force one full L1-sync cycle. This advances tips.checkpointed and the L1->L2 inbox tree state, neither of which is touched by the local push. Removes the retryUntil loop and one slotDuration*2 worst-case wait per checkpoint.
…uilds
Tests that bundle multiple txs into one block via aztecNodeAdmin.setConfig({
minTxsPerBlock: N }) failed under AUTOMINE_E2E_OPTS because the
AutomineSequencer ignored its own config: every mempool poll built a block
as soon as one tx was pending, so a second tx sent moments later landed in
a separate block.
Two changes:
- AutomineSequencer.runBuild gates on max(config.minTxsPerBlock, 1) pending
txs before building, matching production's waitForMinTxs. Explicit empty
builds (warp / mineBlock) still pass allowEmpty: true to skip the gate.
- AutomineSequencer gains updateConfig/getConfig, and AztecNodeService.setConfig
forwards updates to it alongside the production sequencer. Without this,
setConfig({ minTxsPerBlock }) had no effect on the automine path.
Fixes CI failure http://ci.aztec-labs.com/e628da8c9b888d11 in
e2e_deploy_contract/deploy_method.test.ts: "publicly deploys a contract in
one tx and calls a public function on it later in the same block".
BlacklistTokenContractTest.crossTimestampOfChange warps L1 time forward by
86400s (7200 slots, far past the proof-submission window of 2 epochs).
Under AUTOMINE_E2E_OPTS no proofs are submitted, so by the time the warp's
empty-checkpoint propose simulates, STFLib.canPruneAtTime is true and
getEffectivePendingCheckpointNumber returns the proven tip (genesis). The
new checkpoint header carries the previous checkpoint's archive root, which
no longer matches archives[0]=genesisArchiveRoot, so the simulation reverts
with Rollup__InvalidArchive and AutomineSequencer raises
"propose did not succeed for slot ${targetSlot}".
Calling cheatCodes.rollup.markAsProven() before the warp pins proven ==
pending so canPruneAtTime short-circuits and the warp's propose lands on the
current tip. Mirrors the existing pattern in lending_simulator.progressSlots
documented in sequencer-client/src/sequencer/automine/README.md.
Fixes CI failure http://ci.aztec-labs.com/dac9a2ae95a678c1 in
e2e_blacklist_token_contract/unshielding.test.ts; same fix applies to the
other 7 sibling tests (burn, minting, shielding, transfer_private,
transfer_public, access_control) since they all share the
BlacklistTokenContractTest base class.
spalladino
force-pushed
the
palla/automine-sequencer
branch
from
d87b39f to
973af65
Compare
Collaborator
Flakey Tests🤖 says: This CI run detected 1 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
danielntmd
pushed a commit
to danielntmd/aztec-packages
that referenced
this pull request
Jun 4, 2026
BEGIN_COMMIT_OVERRIDE refactor(p2p): merge FastTxCollection into TxCollection with sequential pipeline (AztecProtocol#23245) refactor(publisher): bundle-level simulate; drop per-action enqueue sims (AztecProtocol#23165) refactor(stdlib): remove deprecated RevertCode/TxExecutionResult aliases (AztecProtocol#23249) test(e2e): fix race in 'proposer invalidates multiple checkpoints' (AztecProtocol#23259) fix: clean up old jobs regardless of pending status (AztecProtocol#23260) refactor(p2p): remove unused sendBatchRequest (AztecProtocol#23273) chore(p2p): remove proposal_tx_collector leftovers (AztecProtocol#23276) feat: slash truncated checkpoint proposals (AztecProtocol#23250) refactor: remove unused map in attestation pool (AztecProtocol#23284) chore(p2p): assert last block in checkpoint proposal is correct (AztecProtocol#23274) refactor(l1-tx-utils): use DateProvider for fail-fast timeout check (AztecProtocol#23257) feat(sandbox): support proposer pipelining in local network (AztecProtocol#23277) test(e2e): fix race in broadcasted_invalid_block_proposal_slash under pipelining (AztecProtocol#23302) fix(archiver): atomic getter for L2 tips (AztecProtocol#23295) fix(sequencer): use targetSlot in tryVoteWhenEscapeHatchOpen under pipelining (AztecProtocol#23296) fix(world-state): make fork close idempotent for pruned forks (AztecProtocol#23298) test(e2e): migrate passing tests to proposer pipelining (AztecProtocol#23275) chore: update dashboard (AztecProtocol#23312) chore: Revert "feat(sandbox): support proposer pipelining in local network" (AztecProtocol#23313) test: slash on bad attestation (AztecProtocol#23184) feat(slasher): per-slot data-withholding watcher (A-523, A-525) (AztecProtocol#23116) test(e2e): enable pipelining on e2e debug trace (AztecProtocol#23301) test(e2e): enable pipelining on l1-to-l2 test (AztecProtocol#23300) test(e2e): switch fee_settings to organic fee bumps under pipelining (AztecProtocol#23303) fix(ci): retry sqlite3mc-wasm download on transient DNS/TLS failures (AztecProtocol#23333) test(e2e): wait for real oracle rotation in fee_settings inflate helper (AztecProtocol#23334) test(e2e): anchor e2e_amm PXE to checkpointed tip under pipelining (AztecProtocol#23336) fix(spartan-bench): tolerate older node images in SlasherConfig schema (AztecProtocol#23351) fix: interrupt prover jobs in stop (AztecProtocol#23358) test(e2e): enable pipelining on bot, fees, and avm simulator tests (AztecProtocol#23329) feat(sentinel): end-of-epoch evaluation with re-execution outcomes (AztecProtocol#23286) feat: slash for invalid checkpoint proposals (AztecProtocol#23270) fix: fork closure in epoch proving jobs (AztecProtocol#23390) fix(slasher): anchor watcher scans at archiver synced L2 slot (AztecProtocol#23394) fix: avoid npm uplink for aztec-up local publishes (AztecProtocol#23396) test(e2e): ignore benign 'Insufficient valid txs' block-build-failed in epochs tests (AztecProtocol#23424) chore: refactor weekly proving test wait (AztecProtocol#23395) refactor: add fifo set (AztecProtocol#23271) feat(sandbox): support proposer pipelining in local network (AztecProtocol#23327) fix(p2p): validate BLOCK_TXS in BatchTxRequester (AztecProtocol#23371) chore(p2p): simplify IBatchRequestTxValidator (AztecProtocol#23373) feat(sequencer): AutomineSequencer for single-sequencer e2e tests (AztecProtocol#23354) fix(prover): wait for previous epoch to be proven (AztecProtocol#23458) chore: collocate provers (AztecProtocol#23439) chore: rm staging-ignition (AztecProtocol#23440) chore: rm unused networks (AztecProtocol#23441) test(e2e): migrate block_building, multi_validator_node, publisher_funding, invalid_checkpoint_proposal to pipelining (AztecProtocol#23414) fix(archiver): reconcile local blocks with L1 checkpoints by block number (AztecProtocol#23461) feat: Updated slash conditions on block proposals (AztecProtocol#23466) test(e2e): migrate HA full test to pipelining (AztecProtocol#23463) chore: update resource profiles (AztecProtocol#23442) chore: update debug log levels (AztecProtocol#23456) test: fix flaky sentinel_status_slash by asserting the fault on the checkpoint slot (AztecProtocol#23483) feat(slasher): slash checkpoint equivocation between P2P and L1 (A-980) (AztecProtocol#23436) refactor(slasher): rename ATTESTED_DESCENDANT_OF_INVALID -> PROPOSED_DESCENDANT_OF_CHECKPOINT_WITH_INVALID_ATTESTATIONS (AztecProtocol#23468) fix: reject block proposals in poisoned slots (AztecProtocol#23411) fix: retry nargo dep + solc downloads to survive transient DNS drops (AztecProtocol#23490) fix: enrich json-rpc tracing (AztecProtocol#23412) feat: add trace export controls (AztecProtocol#23413) test(e2e): assert no equivocation offenses in HA full test (AztecProtocol#23496) test: cover invalid checkpoint proposal slashing (AztecProtocol#23503) test(e2e): migrate more e2e suites to proposer pipelining (AztecProtocol#23482) test: flag e2e_slashing_attested_invalid_proposal as flake under pipelining (AztecProtocol#23501) test: flag e2e_p2p_duplicate_proposal_slash as flake under pipelining (AztecProtocol#23515) test(e2e): require cross-observer agreement on sentinel fault slot (AztecProtocol#23513) test: flag e2e_ha_full afterAll hook timeout as flake under pipelining (AztecProtocol#23524) fix(e2e): propagate l1ContractsArgs into node config so archiver matches L1 (AztecProtocol#23514) test: flag e2e_multi_validator_node_key_store P2P tx-dropped failure as flake (AztecProtocol#23528) test(cheat-codes): retry warpL2TimeAtLeastTo in-current-slot test on L1 race (AztecProtocol#23533) test(e2e_ha_full): parallel HA peer node teardown with per-node deadline (AztecProtocol#23539) test: flag e2e_ha_full as flake under HA pipelining (AztecProtocol#23541) test(ci): skip e2e_ha_full entirely on merge-train/spartan (AztecProtocol#23542) test(ci): skip e2e_multi_validator_node_key_store entirely on merge-train/spartan (AztecProtocol#23544) END_COMMIT_OVERRIDE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
E2e tests outside
e2e_p2p,e2e_epochs,e2e_slashing, ande2e_block_buildingdon't exercise block-building or consensus — they just need their tx to land. Running them on the productionSequencerandCheckpointProposalJobwith 12s slot cadence is pure overhead: every test pays for proposer-turn checks, pipelining bookkeeping, validator attestations, and slot-aligned waits that aren't being tested. This PR adds a minimal, deterministic, queue-driven alternative that runs txs as fast as the block builder allows, and migrates 68 contract-logic tests onto it.Approach
Adds an
AutomineSequenceralongside the production one. It reusesSequencerPublisher,FullNodeCheckpointsBuilder, andGlobalVariableBuilder; skips proposer-turn checks, validator orchestration, attestations, pipelining, P2P gossip, and timetable enforcement. Anvil runs in automine mode with no interval mining; the sequencer derives the target slot from anvil's pending-block timestamp and pre-sets the next L1 block timestamp at slot boundaries when needed. All test time control (warps, empty blocks, reorgs) shares a single serial queue with mempool-driven builds — they never interleave. Published blocks and proposed checkpoints are pushed into the archiver locally before each L1 propose, thensyncImmediateis forced after, so tips advance synchronously without polling. RequiresaztecTargetCommitteeSize == 0(the e2e default), so an emptyCommitteeAttestationsAndSignersis accepted by L1.Changes
AutomineSequencerwith serial queue, mempool poller, and APIsbuildIfPending/buildEmptyBlock/warpTo/warpBy/revertToCheckpoint/pause/resume. HonorsminTxsPerBlock. Drops failed txs from the mempool on bothInsufficientValidTxsErrorand successful build paths. Pushes block + proposed checkpoint into the archiver before publishing to L1, then forces an immediate L1 sync.useAutomineSequencerconfig flag.AztecNodeService.createAndSyncconstructs the AutomineSequencer from the existing L1 deps instead of going throughSequencerClient.new.mineBlockroutes toAutomineSequencer.buildEmptyBlockwhen wired.setConfigforwards updates to the automine sequencer;pauseSequencer/resumeSequenceradmin APIs work for both implementations.EthCheatCodes.nextBlockTimestamp()mirroringlastBlockTimestamp()for the pending block tag.CheatCodes.warpL2TimeAtLeastTodelegates to the queue when an AutomineSequencer is wired, so existing test helpers (warpL2TimeAtLeastBy, etc.) work unchanged.AUTOMINE_E2E_OPTSpreset,e2e_automine_smoke.test.tsexercising sequential txs, parallel txs, warp, empty checkpoint, interleaved txs+warps, andrevertToCheckpoint. 68 tests migrated to the preset (see list below).Migrated tests (68)
Expand
Top-level (41):
e2e_blacklist_token_contract/(7): access_control, burn, minting, shielding, transfer_private, transfer_public, unshieldinge2e_deploy_contract/(4): contract_class_registration, deploy_method, legacy, private_initializatione2e_nested_contract/(4): importer, manual_private_call, manual_private_enqueue, manual_publice2e_storage_proof/(1): e2e_storage_proofe2e_token_contract/(10): access_control, burn, minting, private_transfer_recursion, reading_constants, transfer, transfer_in_private, transfer_in_public, transfer_to_private, transfer_to_publicguides/(1): writing_an_account_contractExcluded:
e2e_fees,e2e_cross_chain_messaging, and alle2e_p2p/e2e_epochs/e2e_slashing/e2e_block_buildingsuites — those need real block building, validators, or multi-sequencer setups.