Skip to content

fix(cheat-codes): wait for post-warp L2 block in warpL2TimeAtLeastTo#23213

Merged
benesjan merged 1 commit into
merge-train/spartanfrom
claudebox/fix-blacklist-expiration-ts
May 13, 2026
Merged

fix(cheat-codes): wait for post-warp L2 block in warpL2TimeAtLeastTo#23213
benesjan merged 1 commit into
merge-train/spartanfrom
claudebox/fix-blacklist-expiration-ts

Conversation

@AztecBot

@AztecBot AztecBot commented May 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the merge-queue failure in e2e_blacklist_token_contract/shielding (CI run) where every test fails in applyMint with Invalid tx: Invalid expiration timestamp.

Root cause

warpL2TimeAtLeastTo (introduced in #22084) calls eth.warp followed by node.mineBlock(). The sequencer's polling loop captures nowSeconds/slot at the top of each work() cycle. An in-flight cycle that started just before the warp will mine an L2 block at the pre-warp slot — L1 sync prunes that block from the canonical chain, but it lingers in local world state and the PXE anchors subsequent txs against it. With MAX_TX_LIFETIME == CHANGE_ROLES_DELAY == 86400s, the resulting expiration_timestamp lands exactly on the post-warp slot boundary and the validator rejects the tx as soon as the wall-clock crosses to the next slot.

Fix

After eth.warp, retry mineBlock until the latest L2 block's slot is at or past the slot corresponding to the warped timestamp. The first mineBlock may return a stale block produced by an in-flight cycle; the next triggers a fresh sequencer cycle that reads the post-warp time and builds a block at the post-warp slot. Subsequent txs then anchor against a fresh block whose expiration_timestamp is well in the future.

The signature of warpL2TimeAtLeastTo/warpL2TimeAtLeastBy widens from AztecNodeDebug to AztecNode & AztecNodeDebug so we can read the latest block via getBlockData('latest'). All current callers already type their node as the intersection.

This re-applies the diagnosis from the prior #22796 (which never merged), adapted to the current getBlockData('latest') API.

Full analysis: https://gist.github.com/AztecBot/67815cbe3c3f853d97ec3345dfb0c985

Test plan

  • e2e_blacklist_token_contract/shielding (originally failing)
  • e2e_blacklist_token_contract/{access_control,burn,minting,transfer_*,unshielding} — share applyBaseSetup → crossTimestampOfChange
  • e2e_contract_updates
  • composed/e2e_cheat_codes (verifies the type-widening change still resolves the methods correctly)

ClaudeBox log: https://claudebox.work/s/28594b4dc64f1cd0?run=1

@AztecBot AztecBot added ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR. labels May 12, 2026
@spalladino spalladino marked this pull request as ready for review May 12, 2026 18:25
@benesjan benesjan self-requested a review May 13, 2026 05:59
benesjan
benesjan approved these changes May 13, 2026
View reviewed changes

@benesjan benesjan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me

Thanks for (hopefully) fixing my stuff

@benesjan benesjan merged commit 6b7d279 into merge-train/spartan May 13, 2026
39 of 45 checks passed
@benesjan benesjan deleted the claudebox/fix-blacklist-expiration-ts branch May 13, 2026 06:06
@AztecBot AztecBot mentioned this pull request May 13, 2026
Merged
This was referenced May 13, 2026
Closed
Merged
PhilWindle pushed a commit that referenced this pull request May 15, 2026
@spalladino
test(e2e): migrate passing tests to proposer pipelining (#23275)
943ca27 
## Summary

> **Depends on PR #23296** -- this PR is rebased on top of
`palla/fix-b5-escape-hatch-slot-targeting`, which forward-ports the §6
B5 escape-hatch slot-targeting fix onto the modern
`buildCheckpointSimulationOverridesPlan` + flat `l1Contracts` API. With
B5 in, `e2e_sequencer/escape_hatch_vote_only` and
`e2e_sequencer/gov_proposal.parallel` "should vote even when unable to
build blocks" are now re-enabled under pipelining on this PR.

Extracts the tests known to pass under proposer pipelining from PR
#23150, without flipping the global default. Tests opt into pipelining
explicitly via a new `PIPELINING_SETUP_OPTS` helper. The global
`enableProposerPipelining` default stays `false` on
`merge-train/spartan`; this PR migrates tests file-by-file so each one
is opted in by name.

This PR is intentionally scoped: it only includes tests whose
pipelining-ready status is reasonably well understood. Tests that depend
on shared base-class fixtures (`FeesTest`, `BlacklistTokenContractTest`,
`CrossChainMessagingTest`, `DeployTest`, `FullProverTest`, etc.) keep
their branch changes but are not yet wired to pipelining via their base
class -- those base classes are used by tests outside this batch and a
blanket opt-in would over-migrate. They will be migrated in follow-up
PRs.

Two commits:

1. **`test(e2e): opt unchanged tests into proposer pipelining`** -- adds
`PIPELINING_SETUP_OPTS` to `fixtures.ts`, the small deploy-phase
`accountsDeployMinTxs` conditional to `setup.ts`, and the explicit
opt-in to every §1 test that calls `setup()` directly.
2. **`test(e2e): migrate tests that needed fixes into proposer
pipelining`** -- the §2 tests with their branch fixes plus the
infrastructure they depend on (sequencer.ts B5 fix, dummy_service.ts
loopback, sequencer-publisher.ts error logging, sequencer-client READMEs
rewrite, bootstrap.sh / test_simple.sh timeout bumps).

The global default flip and the migration of base-class-using tests are
intentionally deferred. They will land separately once each batch can be
verified independently.

---

## §1 -- Pipelining enabled and passing (no code changes)

Tests that pick up `enableProposerPipelining=true` from the explicit
opt-in and pass without any per-test fix. This is the majority of the
suite -- too many to enumerate. Examples include the unmodified
`e2e_authwit`, `e2e_nft`, `e2e_amm`, `e2e_partial_notes`,
`e2e_token_contract/*` (non-overflow), `e2e_offchain_*`,
`e2e_orderbook`, `e2e_event_*`, `e2e_keys`, `e2e_avm_simulator` (after
the suite-level timeout bump only), `e2e_pending_note_hashes_contract`,
etc. None of these required test-level pipelining adaptations.

Pre-existing `it.skip`s in this bucket are unrelated to pipelining (they
predate the branch) and were not touched:
- `e2e_token_contract/{transfer,transfer_in_private,transfer_in_public}`
"transfer into account to overflow"
- `e2e_blacklist_token_contract/{transfer_private,transfer_public}`
"transfer into account to overflow"
- `e2e_synching` "replay history and then do a fresh sync" / "a wild
prune appears"
- `e2e_p2p/reex` "validators re-execute transactions before attesting"

## §2 -- Pipelining enabled and needed fixes

Tests that needed test- or fixture-level changes to pass under
pipelining. All currently passing under PR #23150.

**Fixture-level (`src/fixtures/fixtures.ts` + `src/fixtures/setup.ts`)**
- New `PIPELINING_SETUP_OPTS` preset exporting `inboxLag=2`,
`minTxsPerBlock=0`, `aztecSlotDuration=12s`, `ethereumSlotDuration=4s`,
`walletMinFeePadding=PIPELINED_FEE_PADDING` (30x), and
`enableProposerPipelining=true`.
- `setup.ts` gains a small conditional so the deploy-phase
`minTxsPerBlock` override uses `0` instead of `1` under pipelining
(otherwise the chain stalls on alternating slots).

**Cheat-codes (`src/testing/cheat_codes.ts`)** -- already on
`merge-train/spartan` via cherry-pick of #23213.

**P2P (`src/services/dummy_service.ts`)**
- `notifyOwnCheckpointProposal` now invokes the all-nodes callback
synchronously, mirroring libp2p loopback. Without this the in-process
e2e sequencer never sees its own proposal and the pipelined parent
verification blocks indefinitely.

**Sequencer-client**
- `sequencer.ts::tryVoteWhenEscapeHatchOpen` -- §6 B5 fix: takes
`targetSlot`, signs the voter for `targetSlot`, and delays submission
via `sendRequestsAt(getTimestampForSlot(targetSlot))` when pipelining is
enabled. Mirrors the existing `tryVoteWhenSyncFails` and
`CheckpointProposalJob.execute` patterns. Plus a refactor of
`canProposeAt` simulation overrides via `SimulationOverridesBuilder`.
- `sequencer-publisher.ts` -- error log on publisher exhaustion now
includes the underlying viem error and tried-addresses context.

**Per-suite test fixes**
- `e2e_lending_contract` -- predictable-time stub, longer hook windows.
- `e2e_fees/private_payments` "pays fees for tx that dont run public app
logic".
- `e2e_blacklist_token_contract/{burn, minting, shielding,
transfer_private, transfer_public, unshielding}` -- 6/7 suites
re-enabled (`access_control` still skipped, see §5).
- `e2e_contract_updates` -- all 4 tests re-enabled (covered by §1 opt-in
in this PR).
- `e2e_expiration_timestamp` invalidates tests -- L1-only
`eth.warp(target, { resetBlockInterval: true })`, no publisher cascade.
- `e2e_ordering` -- switched from "latest block" to receipt-block reads;
helper renamed to `expectLogsFromBlockToBe(logMessages, fromBlock)`.
- `e2e_fees/failures` -- snapshot `provenCheckpointBefore/After`, use
`waitForProven` with extended timeout, account for newly-proven
checkpoint deltas in reward math, read committed fee headers via
`getCommittedProverFee` / `getCommittedBurn`.
- `e2e_fees/gas_estimation` -- pad `maxFeesPerGas` via
`getPaddedMaxFeesPerGas(aztecNode)` in `beforeEach` to absorb fee-asset
price evolution between snapshot and submission. 3/3 passing.
- `e2e_crowdfunding_and_claim` "cannot donate after a deadline" --
L1-only `cheatCodes.eth.warp(deadline+1, { resetBlockInterval: true })`.
- `e2e_deploy_contract/contract_class_registration` private-ctor
variants -- thread `receipt.blockNumber` through `deployFn`, read logs
from that specific block instead of "latest". 21/21 passing.
- `e2e_state_vars` DelayedPublicMutable -- root cause was slot-duration
mismatch (`delay(4)` assumed `aztecSlotDuration=72s` from
`DefaultL1ContractsConfig`; fixture forces `12s` under pipelining).
Replaced `delay(4)` with a loop that pumps no-op txs until `timestamp >=
timestamp_of_change`, and asserted exact equality against
`tx.data.constants.anchorBlockHeader.globalVariables.timestamp +
newDelay - 1n`. Tight `toEqual`, no widened bound.
- `e2e_pending_note_hashes_contract` -- squash helpers use the latest
*non-empty* block.
- `e2e_expiration_timestamp` -- include-by computation bumped by 2x
`aztecSlotDuration`.
- `e2e_p2p/*` and `e2e_epochs/*` -- explicit `enableProposerPipelining:
true` + `inboxLag: 2` on every test that builds its own config (so
behavior is intentional rather than implicit).
- `e2e_block_building` "processes txs until hitting timetable" --
replaced legacy `canStartNextBlock` mock + single-deadline timetable
with the pipelined sub-slot budget (`blockDurationMs=2000`,
`enforceTimeTable=true`, `fakeProcessingDelayPerTxMs=500`). 10
simultaneous txs must span at least 2 distinct blocks; would fail if the
proposer reverted to single-block-per-slot or stopped enforcing sub-slot
deadlines.
- `e2e_block_building` "assembles a block with multiple txs" (x2) --
pre-publish the contract class once and pass `skipClassPublication:
true` on each per-tx deploy so the deploys don't all share the same
`ContractClassRegistry.publish` nullifier and get RBF-rejected against
each other. Also reset `blockDurationMs` in `afterEach` so the
multi-block-per-slot state from the previous test doesn't leak.
- `e2e_block_building` "publishes two empty blocks" --
`buildCheckpointIfEmpty: true` so the proposer doesn't skip empty
sub-slots; retry budget bumped from 10s -> 60s because empty checkpoints
land every `aztecSlotDuration` (12s) rather than every legacy block.
- `e2e_epochs/epochs_mbps.parallel` "builds multiple blocks per slot
with L2 to L1 messages" -- pipelined timing loses one sub-slot to
attestation propagation; expectation dropped from
`EXPECTED_BLOCKS_PER_CHECKPOINT=3` to `>= 2`, mirroring the sibling MBPS
tests.
- `e2e_l1_with_wall_time` -- test was explicitly passing
`ethereumSlotDuration` from env (=12s), defeating the fixture's
pipelining override (=4s). With `aztec=eth=12s`, pipelined timing can't
fit propose+attest+publish in one Aztec slot. Removed the explicit
`ethereumSlotDuration`; also wrapped `teardown` in `afterEach` so setup
failures surface their real error.
- `e2e_p2p/add_rollup` re-enabled (entire describe; 1 test, passes in
~9:14 locally). AttestationTimeoutError still fires in some slots, but
the bundled-multicall governance-signal preCheck is independent of the
propose preCheck -- signals accumulate and reach quorum even when
checkpoint proposes fail to attest.
- `e2e_pruned_blocks` "can discover and use notes created in both pruned
and available blocks" -- restored the explicit `markAsProven` call (as
it had pre-#21156) + a 2-block buffer for Anvil's `finalized = latest -
2` heuristic; test re-enabled and passes.
- `e2e_sequencer/escape_hatch_vote_only` re-enabled. Source fix at
`sequencer.ts::tryVoteWhenEscapeHatchOpen` (see §B5 in PR #23150).
Test-side: attach event listeners *after* the warp, explicitly drain
trailing in-flight votes before counting.
- `e2e_sequencer/gov_proposal.parallel` re-enabled (both tests). Two
pipelining-aware adjustments: warp offset bumped to
`nextRoundBeginsAtTimestamp - AZTEC_SLOT_DURATION -
ETHEREUM_SLOT_DURATION`, and per-tx wait timeouts tuned for two slots of
catch-up (proposer + L1 mine).

**Bash-level timeout adjustments (`end-to-end/bootstrap.sh`)** --
pipelined sequential dependent txs run at ~2x legacy latency:
- simple e2e default: 10m -> 20m
- `e2e_block_building`: 25m
- `e2e_avm_simulator`: 30m
- compose/web3signer: 20m
- HA: 30m
- `scripts/test_simple.sh` Jest `--testTimeout` 5m -> 10m
- ~21 test files: per-file `const TIMEOUT` raised from 100/120/150/180s
-> 300s.

---

## Out of scope

- **Global default flip**: PR #23150 flipped
`enableProposerPipelining=true` everywhere. This PR keeps the default
`false` and migrates per-test. The global flip will land in a follow-up.
- **§3 opt-outs** (`e2e_l1_publisher` "with attestations" describe,
`epoch_cache.test.ts` non-pipelined branch coverage, demo
`docker-compose.yml`): no change required while the default is `false`.
- **§5 still-skipped tests**: the tests in §5 of PR #23150's
categorization (e.g. `e2e_blacklist_token_contract/access_control`,
`e2e_publisher_funding_multi`, `e2e_fees/fee_settings`, etc.) remain at
`merge-train/spartan` state.
- **Base-class fixtures** (`FeesTest`, `BlacklistTokenContractTest`,
`CrossChainMessagingTest`, `DeployTest`, `FullProverTest`,
`EpochesTest`, P2P fixtures): test files using these get their
branch-side changes preserved but are not wired to pipelining via the
base class -- those base classes are shared with tests not in this batch
and a blanket opt-in would over-migrate. Follow-up PRs will opt them in
selectively.

Reference: PR #23150 (`palla/kill-non-pipelined-flow`) for full context
on the categorization, source-level bugs surfaced (§6 B1-B6), and
per-suite investigation notes.
rangozd pushed a commit to rangozd/aztec-packages that referenced this pull request May 16, 2026
@PhilWindle
feat: merge-train/spartan (AztecProtocol#22980)
25fa2cc 
BEGIN_COMMIT_OVERRIDE
fix(test): warp L1 forward when proposer scan hits EpochNotStable
(AztecProtocol#22967)
test(e2e): fail epochs tests on proposer-rollup-check-failed (AztecProtocol#22965)
fix: grafana switch to aztec_status="proposed" (AztecProtocol#22978)
chore: update benchmark scraper (AztecProtocol#22984)
test(e2e): migrate simple epoch tests to pipelining (AztecProtocol#22973)
chore: remove top-level yarn.lock (AztecProtocol#22987)
refactor(archiver)!: unify L2BlockSource checkpoint lookups via query
objects (AztecProtocol#22933)
fix(sequencer): bounded sweep instead of event scan for governance
proposal check (AztecProtocol#22989)
fix(docs): allow webapp-tutorial yarn install to populate empty lockfile
in CI (AztecProtocol#23000)
test(e2e): enable pipelining in l1-reorgs and mbps redistribution tests
(AztecProtocol#23009)
fix(archiver): restore pending block height metric under pipelining
(AztecProtocol#22994)
chore(p2p): remove skipped validation result option (AztecProtocol#23034)
refactor(p2p)!: remove slow tx collection flow (AztecProtocol#22878)
chore(spartan): add next-net-clone environment config (AztecProtocol#22995)
chore(sequencer): add context to proposer-rollup-check-failed logs
(AztecProtocol#23071)
test(e2e): wait for archiver sync before asserting pipelining (AztecProtocol#22997)
refactor(node-rpc)!: remove deprecated AztecNode methods and
L2BlockSource tip helpers (AztecProtocol#22934)
feat(p2p): detect and track announce IP changes at runtime (AztecProtocol#22405)
test: mark tx_stats_bench 10 TPS as flake-retryable on
merge-train/spartan (AztecProtocol#23083)
fix(sequencer): bind vote-only multicalls to target slot under
pipelining (AztecProtocol#23090)
feat(sequencer): build optimistically across pruning epoch boundary
(AztecProtocol#23056)
fix(sequencer): use chainTipsOverride.pending for log context (AztecProtocol#23098)
test(e2e): relax post-boundary slot assertion in
epochs_proof_at_boundary (AztecProtocol#23108)
fix(bb-prover): pool long-lived bb verifier processes instead of
spawning per-call (AztecProtocol#23093)
fix(sequencer): anchor fee asset price modifier to predicted parent
(AztecProtocol#23113)
chore: error log when L1 head timestamp drifts (AztecProtocol#22947)
fix(sequencer): override full parent checkpoint cell in pipelined
simulation (AztecProtocol#23073)
test(e2e): enable pipelining on missed l1 slot test (AztecProtocol#23068)
fix: more robust metrics reporting in IRM monitor (AztecProtocol#23038)
fix: preserve LMDB slashing protection (AztecProtocol#23145)
test(e2e): enable pipelining on p2p tests (AztecProtocol#23070)
fix(archiver): move L2 tips cache refresh out of write transactions
(AztecProtocol#23110)
test(e2e): fix data_withholding_slash flake by freezing L1 across
restart (AztecProtocol#23162)
fix(validator): include proposed checkpoint out-hashes when validating
checkpoint proposals (AztecProtocol#23119)
refactor(config): drop nested config option, flatten l1Contracts
(AztecProtocol#23143)
test(e2e): bump bash TIMEOUT for e2e_p2p/add_rollup to match jest 20m
(AztecProtocol#23177)
fix(p2p): chunk archive of mined txs on block finalization (A-969)
(AztecProtocol#23085)
fix(p2p): stream tx pool hydration to bound startup memory (A-968)
(AztecProtocol#23086)
chore: remove orphan --archiver flag usages from start invocations
(AztecProtocol#23186)
feat(ci): daily merge-train/spartan stale-PR notifier (AztecProtocol#23189)
fix: preserve contract artifact permissions (AztecProtocol#23174)
fix(ci3): accept slashes in /list/<path:key> for merge-train
history (AztecProtocol#23160)
feat(ci): route merge-train/spartan flake notifications to
#team-alpha-ci (AztecProtocol#23219)
fix(cheat-codes): wait for post-warp L2 block in warpL2TimeAtLeastTo
(AztecProtocol#23213)
feat: slash attesters signing over bad checkpoints (AztecProtocol#23180)
refactor(prover-client): split orchestrator into sub-tree + top-tree
pair (AztecProtocol#22996)
fix(srs): retry transient CRS HTTP downloads with exponential backoff
(AztecProtocol#23244)
refactor(p2p): remove old reqresp mode (AztecProtocol#23158)
docs(sequencer-client): rewrite top-level and timing READMEs (AztecProtocol#23149)
fix(aztec-node): include upcoming checkpoint's L1 to L2 messages in
simulatePublicCalls (AztecProtocol#23163)
END_COMMIT_OVERRIDE
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@benesjan benesjan benesjan approved these changes

Assignees

No one assigned

Labels

ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AztecBot @benesjan