feat(sandbox): support proposer pipelining in local network#23277
Conversation
Turns on SEQ_ENABLE_PROPOSER_PIPELINING=true in every docker-compose.yml that boots `aztec start --local-network` for testing (end-to-end compose suite, docs/examples runner, playground). This routes the compose-based sandbox tests through the pipelined block production path so we catch sandbox/pipelining regressions in CI.
Under proposer pipelining the publisher for slot N is built during wall-clock slot N-1 and parks waiting for L1 to advance past the build slot before its background submission task can run `enqueueCheckpointForSubmission` (the parent-L1-sync wait inside `waitForValidParentCheckpointOnL1`). The AnvilTestWatcher had no trigger that fired in this window: the "slot filled" branch needed the not-yet-published checkpoint, the pending-tx debounce required txs in the mempool (already drained after broadcast), and the wall-clock fallback / `syncDateProviderToL1IfBehind` paths only fired *after* wall clock had crossed the next slot — i.e. the slow path itself. The result was that each L2 slot under pipelining took a full real-time slot (~72 s) before the L1 multicall fired. This change hooks the sequencer's `block-proposed` event in the local-network setup and forwards the target slot to the watcher via a new `setProposedTargetSlot`. The watcher then warps L1 (and, via `cheatcodes.warp`, the injected date provider) to that slot's timestamp ahead of the publisher's `sendRequestsAt` sleep. The multicall lands inside the target slot, the existing "slot filled" branch advances to the next slot, and the cascade keeps going at the speed of the build path. Sandbox boot with pipelining drops from ~5 minutes back to ~30 s. `block-proposed` is used rather than the cleaner `state-changed → PUBLISHING_CHECKPOINT` because the latter only fires *after* `waitForValidParentCheckpointOnL1` unblocks (which itself is what we are trying to break); see comment in local-network.ts for the full chain. The watcher only ratchets the stored slot up, and `warpToTimestamp` already rejects warps that would move L1 backwards, so duplicate or stale `block-proposed` events are no-ops. The new code path is gated on `isLocalNetwork`, so non-pipelined sandbox behavior and tests that disable the watcher are unaffected.
Routes the three aztec-up smoke tests that boot `aztec start --local-network` (basic_install, amm_flow, bridge_and_claim) through the pipelined sequencer path. These tests run on PR CI via the aztec-up/bootstrap.sh test suite, which installs the working-tree's `aztec` package via a local Verdaccio registry — so the new AnvilTestWatcher pipelining hook from the previous commit is available to them. The acceptance test under aztec-cli-acceptance-test/ is not migrated because it runs only post-release against the published nightly, which doesn't carry the watcher fix yet.
Pipelining auditVerified the sandbox tests in this PR's scope ran on CI with
Both runs are SUCCESS. All compose-routed sandbox tests under Direct config-dump evidence (
|
| Test | Ran | Pipelining=true in node config | Log |
|---|---|---|---|
aztec-up/test/amm_flow.sh |
yes (325s, PASSED) | yes | http://ci.aztec-labs.com/af2b68e7a1d884e1 |
aztec-up/test/bridge_and_claim.sh |
yes (248s, PASSED) | yes | http://ci.aztec-labs.com/418776554c4dce4c |
aztec-up/test/basic_install.sh |
yes (283s, PASSED) | n/a — script sets LOG_LEVEL=silent, but exports SEQ_ENABLE_PROPOSER_PIPELINING=true (see basic_install.sh L11-L13) |
http://ci.aztec-labs.com/47cd2a28761b8cd6 |
playground chromium |
yes (1 passed, 7.9s) | yes (line 33: "l1ChainId":31337,"enableProposerPipelining":true) |
http://ci.aztec-labs.com/928d273ed3d0849f |
playground firefox |
yes (1 passed, 21.0s) | yes (same config dump) | http://ci.aztec-labs.com/fcb032ebb35eac96 |
docs/examples runner (all 7 examples) |
yes (all PASS: aztecjs_connection, aztecjs_getting_started, aztecjs_advanced, aztecjs_authwit, aztecjs_testing, example_swap, recursive_verification) |
local-network stdout is not multiplexed into the docs-examples container log, but the compose env sets SEQ_ENABLE_PROPOSER_PIPELINING: 'true' (docs/examples/ts/docker-compose.yml L31) |
http://ci.aztec-labs.com/8b68c8bc3be31e54 |
Sample config-line for amm_flow (line 89 of its log):
INFO: ethereum:deploy_aztec_l1_contracts Deploying L1 contracts with config: {... "l1ChainId":31337,"enableProposerPipelining":true,"archiverPollingIntervalMS":500, ...}
Compose-routed e2e/cli-wallet tests (yarn-project/end-to-end/scripts/docker-compose.yml)
run_compose_test only streams the end-to-end container stdout, so the local-network node's config dump isn't surfaced in these test logs. Pipelining is set via the compose env at docker-compose.yml L31 (SEQ_ENABLE_PROPOSER_PIPELINING: 'true'); all tests below are cached green from a run on commit 323a501a (the commit that carries the compose env change + watcher fix):
| Test | Ran | Result | Log |
|---|---|---|---|
src/composed/e2e_cheat_codes.test.ts |
yes | 6/6 PASS | http://ci.aztec-labs.com/baf7e7b8f90a29f0 |
src/composed/e2e_local_network_example.test.ts |
yes | PASS | http://ci.aztec-labs.com/5e79c18b50c60b38 |
src/composed/e2e_token_bridge_tutorial_test.test.ts |
yes | 1/1 PASS | http://ci.aztec-labs.com/abaa477cdb607759 |
src/guides/up_quick_start.test.ts |
yes | 1/1 PASS | http://ci.aztec-labs.com/e1125c678f3e5b1b |
cli-wallet/test/flows/basic.sh |
yes | exit 0 | http://ci.aztec-labs.com/fba0aab5fe57d92a |
cli-wallet/test/flows/capture_authwits.sh |
yes | exit 0 | http://ci.aztec-labs.com/c12d62acfe072d8d |
cli-wallet/test/flows/create_account_pay_native.sh |
yes | exit 0 | http://ci.aztec-labs.com/5cfacc59657f277e |
cli-wallet/test/flows/no_alias.sh |
yes | exit 0 | http://ci.aztec-labs.com/ff7767636172afda |
cli-wallet/test/flows/private_transfer.sh |
yes | exit 0 | http://ci.aztec-labs.com/5ce2ec286391d4ab |
cli-wallet/test/flows/profile.sh |
yes | exit 0 | http://ci.aztec-labs.com/596834026cea9df8 |
cli-wallet/test/flows/public_authwit_transfer.sh |
yes | exit 0 | http://ci.aztec-labs.com/9c05d97fda12051c |
cli-wallet/test/flows/shield_and_transfer.sh |
yes | exit 0 | http://ci.aztec-labs.com/c59e6a2cb8003643 |
cli-wallet/test/flows/sponsored_create_account_and_mint.sh |
yes | exit 0 | http://ci.aztec-labs.com/52313069797f12c9 |
cli-wallet/test/flows/tx_management.sh |
yes | exit 0 | http://ci.aztec-labs.com/bbf5788486697094 |
Watcher fix observed firing in CI
The new AnvilTestWatcher log line Warped L1 to target slot N for pipelined publish was observed in CI for every test whose log surfaces local-network stdout. A few examples:
aztec-up amm_flow.sh(15 occurrences, slots 3,6,9,12,15,18,21,24,27,30,33,36,39,42,45): http://ci.aztec-labs.com/af2b68e7a1d884e1[16:01:14.300] INFO: aztecjs:utils:watcher Warped L1 to target slot 30 for pipelined publish
aztec-up bridge_and_claim.sh(9 occurrences): http://ci.aztec-labs.com/418776554c4dce4c[16:00:54.144] INFO: aztecjs:utils:watcher Warped L1 to target slot 21 for pipelined publish
playgroundchromium (≥5 occurrences): http://ci.aztec-labs.com/928d273ed3d0849faztec-1 | INFO: aztecjs:utils:watcher Warped L1 to target slot 2 for pipelined publish
playgroundfirefox (≥4 occurrences): http://ci.aztec-labs.com/fcb032ebb35eac96
(basic_install.sh runs at LOG_LEVEL=silent, and the e2e run_compose_test flow doesn't multiplex local-network stdout, so the line is invisible in those logs but pipelining is still ON per the compose env.)
Anything flagged
Nothing to flag.
.test_patterns.ymlhasskip: trueonly forsrc/composed/uniswap_trade_on_l1_from_l2.test.ts(already out of scope per the SANDBOX.md exclusion list — "bricked in new version of anvil").- All other compose/aztec-up entries in
.test_patterns.ymlcarryerror_regex(flaky-on-match) withoutskip, so they ran normally. - The test-engine summary confirms
No flaked tests foundon bothci/x-fastandci/x-full. - The 3 aztec-up tests we deliberately skipped from this audit (
counter_contract,default_scaffold,no_shadow_user_bins) do not startaztec start --local-network, so they have no sandbox to pipeline.
… composes Both docs/examples/ts/docker-compose.yml and playground/docker-compose.yml ran with SEQ_ENABLE_PROPOSER_PIPELINING=true (added in #23277), but the sandbox is not yet configured to absorb pipelining's side effects: - example_swap stalls on `wait for proven block N` because the proven tip stops advancing in an idle pipelined sandbox (the original PR #23253 dequeue, http://ci.aztec-labs.com/b08ac48286302949). - aztecjs_advanced fails on `Cannot get L1 to L2 messages for checkpoint N: inbox tree in progress is N, messages not yet sealed` because under pipelining `AztecNodeService.simulatePublicCalls` reads L1->L2 messages from an in-progress checkpoint (http://ci.aztec-labs.com/419c4513023a1799). This is the same `simulator + inboxLag` mismatch already TODO'd in e2e_bot.test.ts and several e2e_fees tests. Disable the flag in the two sandbox composes to unblock the spartan merge train; aztec-up scripts (basic_install / bridge_and_claim / amm_flow) keep the flag and continue exercising pipelining in CI.
BEGIN_COMMIT_OVERRIDE refactor(p2p): merge FastTxCollection into TxCollection with sequential pipeline (AztecProtocol#23245) refactor(publisher): bundle-level simulate; drop per-action enqueue sims (AztecProtocol#23165) refactor(stdlib): remove deprecated RevertCode/TxExecutionResult aliases (AztecProtocol#23249) test(e2e): fix race in 'proposer invalidates multiple checkpoints' (AztecProtocol#23259) fix: clean up old jobs regardless of pending status (AztecProtocol#23260) refactor(p2p): remove unused sendBatchRequest (AztecProtocol#23273) chore(p2p): remove proposal_tx_collector leftovers (AztecProtocol#23276) feat: slash truncated checkpoint proposals (AztecProtocol#23250) refactor: remove unused map in attestation pool (AztecProtocol#23284) chore(p2p): assert last block in checkpoint proposal is correct (AztecProtocol#23274) refactor(l1-tx-utils): use DateProvider for fail-fast timeout check (AztecProtocol#23257) feat(sandbox): support proposer pipelining in local network (AztecProtocol#23277) test(e2e): fix race in broadcasted_invalid_block_proposal_slash under pipelining (AztecProtocol#23302) fix(archiver): atomic getter for L2 tips (AztecProtocol#23295) fix(sequencer): use targetSlot in tryVoteWhenEscapeHatchOpen under pipelining (AztecProtocol#23296) fix(world-state): make fork close idempotent for pruned forks (AztecProtocol#23298) test(e2e): migrate passing tests to proposer pipelining (AztecProtocol#23275) chore: update dashboard (AztecProtocol#23312) chore: Revert "feat(sandbox): support proposer pipelining in local network" (AztecProtocol#23313) test: slash on bad attestation (AztecProtocol#23184) feat(slasher): per-slot data-withholding watcher (A-523, A-525) (AztecProtocol#23116) test(e2e): enable pipelining on e2e debug trace (AztecProtocol#23301) test(e2e): enable pipelining on l1-to-l2 test (AztecProtocol#23300) test(e2e): switch fee_settings to organic fee bumps under pipelining (AztecProtocol#23303) fix(ci): retry sqlite3mc-wasm download on transient DNS/TLS failures (AztecProtocol#23333) test(e2e): wait for real oracle rotation in fee_settings inflate helper (AztecProtocol#23334) test(e2e): anchor e2e_amm PXE to checkpointed tip under pipelining (AztecProtocol#23336) fix(spartan-bench): tolerate older node images in SlasherConfig schema (AztecProtocol#23351) fix: interrupt prover jobs in stop (AztecProtocol#23358) test(e2e): enable pipelining on bot, fees, and avm simulator tests (AztecProtocol#23329) feat(sentinel): end-of-epoch evaluation with re-execution outcomes (AztecProtocol#23286) feat: slash for invalid checkpoint proposals (AztecProtocol#23270) fix: fork closure in epoch proving jobs (AztecProtocol#23390) fix(slasher): anchor watcher scans at archiver synced L2 slot (AztecProtocol#23394) fix: avoid npm uplink for aztec-up local publishes (AztecProtocol#23396) test(e2e): ignore benign 'Insufficient valid txs' block-build-failed in epochs tests (AztecProtocol#23424) chore: refactor weekly proving test wait (AztecProtocol#23395) refactor: add fifo set (AztecProtocol#23271) feat(sandbox): support proposer pipelining in local network (AztecProtocol#23327) fix(p2p): validate BLOCK_TXS in BatchTxRequester (AztecProtocol#23371) chore(p2p): simplify IBatchRequestTxValidator (AztecProtocol#23373) feat(sequencer): AutomineSequencer for single-sequencer e2e tests (AztecProtocol#23354) fix(prover): wait for previous epoch to be proven (AztecProtocol#23458) chore: collocate provers (AztecProtocol#23439) chore: rm staging-ignition (AztecProtocol#23440) chore: rm unused networks (AztecProtocol#23441) test(e2e): migrate block_building, multi_validator_node, publisher_funding, invalid_checkpoint_proposal to pipelining (AztecProtocol#23414) fix(archiver): reconcile local blocks with L1 checkpoints by block number (AztecProtocol#23461) feat: Updated slash conditions on block proposals (AztecProtocol#23466) test(e2e): migrate HA full test to pipelining (AztecProtocol#23463) chore: update resource profiles (AztecProtocol#23442) chore: update debug log levels (AztecProtocol#23456) test: fix flaky sentinel_status_slash by asserting the fault on the checkpoint slot (AztecProtocol#23483) feat(slasher): slash checkpoint equivocation between P2P and L1 (A-980) (AztecProtocol#23436) refactor(slasher): rename ATTESTED_DESCENDANT_OF_INVALID -> PROPOSED_DESCENDANT_OF_CHECKPOINT_WITH_INVALID_ATTESTATIONS (AztecProtocol#23468) fix: reject block proposals in poisoned slots (AztecProtocol#23411) fix: retry nargo dep + solc downloads to survive transient DNS drops (AztecProtocol#23490) fix: enrich json-rpc tracing (AztecProtocol#23412) feat: add trace export controls (AztecProtocol#23413) test(e2e): assert no equivocation offenses in HA full test (AztecProtocol#23496) test: cover invalid checkpoint proposal slashing (AztecProtocol#23503) test(e2e): migrate more e2e suites to proposer pipelining (AztecProtocol#23482) test: flag e2e_slashing_attested_invalid_proposal as flake under pipelining (AztecProtocol#23501) test: flag e2e_p2p_duplicate_proposal_slash as flake under pipelining (AztecProtocol#23515) test(e2e): require cross-observer agreement on sentinel fault slot (AztecProtocol#23513) test: flag e2e_ha_full afterAll hook timeout as flake under pipelining (AztecProtocol#23524) fix(e2e): propagate l1ContractsArgs into node config so archiver matches L1 (AztecProtocol#23514) test: flag e2e_multi_validator_node_key_store P2P tx-dropped failure as flake (AztecProtocol#23528) test(cheat-codes): retry warpL2TimeAtLeastTo in-current-slot test on L1 race (AztecProtocol#23533) test(e2e_ha_full): parallel HA peer node teardown with per-node deadline (AztecProtocol#23539) test: flag e2e_ha_full as flake under HA pipelining (AztecProtocol#23541) test(ci): skip e2e_ha_full entirely on merge-train/spartan (AztecProtocol#23542) test(ci): skip e2e_multi_validator_node_key_store entirely on merge-train/spartan (AztecProtocol#23544) END_COMMIT_OVERRIDE
Motivation
The local network sandbox (
aztec start --local-network) historically ran without proposer pipelining, so the compose-routed e2e suite (src/composed/*,src/guides/*, cli-wallet flows, docs examples, playground) never exercised the pipelined sequencer path. Turning pipelining on revealed that each L2 slot took a full real-time slot (~72 s) before the L1 multicall fired, blowing up sandbox boot from ~30 s to ~5 min, because the existingAnvilTestWatchertriggers don't fire in the pipelined-publish window.Approach
First commit flips
SEQ_ENABLE_PROPOSER_PIPELINING=trueon the three sandbox-test compose envs so every compose-routed test runs through the pipelined path. Second commit teachesAnvilTestWatcherabout the proposer's target slot by hooking the sequencer'sblock-proposedevent increateLocalNetwork; when the proposer has built a block destined for a slot beyond L1, the watcher warps L1 (and, viacheatcodes.warp, the injected date provider) forward, waking the pipelined publisher'ssendRequestsAtsleep and the upstreamwaitForValidParentCheckpointOnL1wait.block-proposedis used rather than the cleanerstate-changed → PUBLISHING_CHECKPOINTbecause the latter only fires afterwaitForValidParentCheckpointOnL1unblocks — which is what we are trying to break — so it would be circular.Changes
SEQ_ENABLE_PROPOSER_PIPELINING=trueto thelocal-network/aztecservice env so every compose-routed sandbox test runs pipelined.AnvilTestWatcher): newsetProposedTargetSlotsetter and awarpTimeIfNeededbranch (gated onisLocalNetwork) that warps L1 to the target slot's timestamp when it's ahead of L1.createLocalNetwork): subscribe to the sequencer'sblock-proposedevent and forwardslotto the watcher.Verified locally: sandbox boot drops from ~5 min back to ~27 s under pipelining, and
e2e_local_network_example.test.ts(both tests) passes in ~33 s.