test(e2e): fix proposer invalidates multiple checkpoints timeout#23608
Merged
Conversation
**Test Summary** `proposer invalidates multiple checkpoints` verifies that two intended bad checkpoints land with insufficient attestations, a later good proposer invalidates the first bad checkpoint, and the chain then progresses. **Failed Run Error** CI run `8b1c0f4ec6031f2b` timed out at Jest’s 600s limit. The failure was not the shutdown L1 send error; that happened after the timeout while teardown was interrupting pending work. **Failed vs Successful Divergence** First meaningful divergence: checkpoint 4 at slot 23. Failed log: slot 23 published checkpoint 4 with only 1 attestation, then archivers reported `Insufficient attestations ... actualAttestations:1`. Successful log: slot 23 collected all 5 attestations before publishing checkpoint 4, so the first intentionally bad checkpoints were later. **Timeline** Failed: - `15:59:11` selected intended bad slots 25/26, applied bad config to proposer `0x15...` - `15:59:35` slot 23 job prepared by that same proposer - `16:00:15` checkpoint 4 at slot 23 landed with 1 attestation - repeated rollback/retry consumed enough time to hit Jest timeout Successful: - slot 23 checkpoint landed cleanly with 5 attestations - intended bad checkpoints at slots 24/25 landed with 1 attestation - checkpoint 5 was invalidated - test completed successfully **Hypothesis** High confidence: the test’s bad-slot selection only excluded `candidateSlot1 - 1` as a pre-bad pipelined target. In the failed run, `candidateSlot1 - 2` was still unsnapshotted and owned by a bad proposer, so applying malicious config leaked into slot 23. **Evidence** - Logs: failed run selected slots 25/26 but slot 23 later published with 1 attestation from the newly bad proposer. - Source: pipelined checkpoint jobs snapshot sequencer config when the target-slot job is created, so applying config while sequencers are running can affect any not-yet-created pre-bad job. - Skeptic check: no contradiction found; it also caught a broken local timeout race. **Proposed Fix** Implemented in [epochs_invalidate_block.parallel.test.ts](/home/santiago/Projects/aztec-1/yarn-project/end-to-end/src/e2e_epochs/epochs_invalidate_block.parallel.test.ts:393): the selector now excludes bad proposers from every pre-bad target slot from `currentSlot + 2` through `candidateSlot1 - 1`, not just the immediately prior slot. Also fixed the broken timeout race at [line 475](/home/santiago/Projects/aztec-1/yarn-project/end-to-end/src/e2e_epochs/epochs_invalidate_block.parallel.test.ts:475) by removing the accidental inner `await`.
spalladino
changed the title
proposer invalidates multiple checkpoints timeout
spalladino
added
the
ci-no-fail-fast
Collaborator
Flakey Tests🤖 says: This CI run detected 2 tests that failed, but were tolerated due to a .test_patterns.yml entry. |
PhilWindle
approved these changes
May 27, 2026
danielntmd
pushed a commit
to danielntmd/aztec-packages
that referenced
this pull request
Jun 4, 2026
BEGIN_COMMIT_OVERRIDE fix(archiver): skip descendants of invalid-attestations checkpoints (AztecProtocol#23502) chore: scale network validators (AztecProtocol#23579) fix(ci): nightly 10 TPS bench GCP auth and checkout (AztecProtocol#23586) chore: set eth node resource profile (AztecProtocol#23583) fix: wait for checkpoint before sentinel assertions (AztecProtocol#23573) fix: slash attestations for invalid checkpoint proposals (AztecProtocol#23506) test: fix web3signer pipelining `e2e_multi_validator_node_key_store.test.ts` (AztecProtocol#23568) fix: cap CI devbox hostname (AztecProtocol#23591) test: stabilize invalid checkpoint descendant e2e (AztecProtocol#23582) test(e2e): stabilize invalidation slots in `proposer invalidates multiple checkpoints` (AztecProtocol#23590) test(e2e): stabilize invalid proposal slashing target slot in `attested_invalid_proposal` (AztecProtocol#23589) chore(foundation): faster toBufferBE via zero fast-path (AztecProtocol#23592) fix: honour BB_BINARY_PATH (AztecProtocol#23570) chore: bump reth and lighthouse (AztecProtocol#23588) chore: add web3signer and postgres node selectors (AztecProtocol#23598) fix: do not symlink .codex folders (AztecProtocol#23593) chore: fix claude and codex symlinking tests (AztecProtocol#23599) test(e2e): narrow down sentinel check in `multiple_validators_sentinel` (AztecProtocol#23604) test(e2e): fix `proposer invalidates multiple checkpoints` timeout (AztecProtocol#23608) fix: record zero-amount slashing offenses (AztecProtocol#23556) fix: log slashing offense names (AztecProtocol#23565) feat(p2p): tx validation cache (AztecProtocol#23585) chore: add KEDA deployment module (AztecProtocol#23553) chore: add KEDA prover agent autoscaling (AztecProtocol#23554) chore: update destroy_bootnode.sh (AztecProtocol#23626) chore: skip failing chonk_pinned_inputs.test in CI (AztecProtocol#23643) chore(ci): tolerate public authwit P2P receipt flake (AztecProtocol#23648) END_COMMIT_OVERRIDE
spalladino
pushed a commit
that referenced
this pull request
Jun 11, 2026
…idates multiple checkpoints` (#24017) Fixes a flake in `proposer invalidates multiple checkpoints` (`e2e_epochs/epochs_invalidate_block.parallel.test.ts`) reported on `v5-next`: [failed run](http://ci.aztec-labs.com/e4076dd86c434c6f). Replaces #24016 (was based on `merge-train/spartan`; this one targets the v5 line where the flake fired and restructures the test instead of just resizing the timeout). ## Root cause of the flake `TimeoutError: Operation timed out after 256000ms` — the bare 8-slot `timeoutPromise` waiting for the two bad checkpoints. The bad-slot search from #23608 rejects any candidate pair whose proposer also owns an earlier un-snapshotted pipelined slot, and the rejection window grows with each attempt. In the failed run the current slot was 21 and the search rejected (24,25)…(29,30) before accepting slots **30/31** — 9–10 slots out. The fixed 256s wait expired at 22:48:55, before slot 30 even began (~22:49:00), while the chain healthily mined checkpoints at slots 22–28 underneath; the run was unwinnable at selection time. The race's `.then(() => [CheckpointNumber(0), …])` fallback was also dead code, since `timeoutPromise` rejects. ## Fix: search first, then warp Instead of starting the sequencers and waiting in real time for whatever slots the search lands on: - With sequencers stopped, search for a `warpSlot` such that the proposers of the three lead-in slots `warpSlot+1..warpSlot+3` are not the proposers of the bad slots `warpSlot+4`/`warpSlot+5`. A far-away candidate now costs a warp instead of a real-time wait, and `EpochNotStable` during the search is handled by warping forward one epoch (same pattern as the `archiver skips a descendant` test in this file). - Warp to one L1 block before `warpSlot`, so sequencers get a full L2 slot to boot before the first pipelined build window we rely on (end of `warpSlot`, targeting `warpSlot+1`). - Start the sequencers and wait for the first good checkpoint (lands at `warpSlot`, or up to `warpSlot+2` on a slow start). - Apply the malicious config to the bad-slot proposers. The three good lead-in slots guarantee no pipelined job before `badSlot1` can snapshot it, since jobs snapshot config during the last L1 slot of the previous L2 slot. - Fail fast with a clear assertion if config application was somehow late enough to reach `badSlot1`'s build window, rather than timing out opaquely. - The 8-slot wait for the bad checkpoints is now correctly sized by construction (`badSlot2` is at most ~6 slots from the wait start), and gets a descriptive timeout message. Worst case the wait phase is bounded at ~6 slots regardless of how many candidates the search rejects, where previously each rejected candidate pushed the bad checkpoints one slot further past the fixed timeout. --- *Created by [claudebox](https://claudebox.work/v2/sessions/d509a218614bf4ac) · group: `slackbot`*
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes flake in
proposer invalidates multiple checkpointse2e_epochs/epochs_invalidate_block.parallel.test.tstest that caused a timeout (see this run). See below for the Codex analysis and fix.Test Summary
proposer invalidates multiple checkpointsverifies that two intended bad checkpoints land with insufficient attestations, a later good proposer invalidates the first bad checkpoint, and the chain then progresses.Failed Run Error
CI run
8b1c0f4ec6031f2btimed out at Jest’s 600s limit. The failure was not the shutdown L1 send error; that happened after the timeout while teardown was interrupting pending work.Failed vs Successful Divergence
First meaningful divergence: checkpoint 4 at slot 23.
Failed log: slot 23 published checkpoint 4 with only 1 attestation, then archivers reported
Insufficient attestations ... actualAttestations:1.Successful log: slot 23 collected all 5 attestations before publishing checkpoint 4, so the first intentionally bad checkpoints were later.
Timeline
Failed:
15:59:11selected intended bad slots 25/26, applied bad config to proposer0x15...15:59:35slot 23 job prepared by that same proposer16:00:15checkpoint 4 at slot 23 landed with 1 attestationSuccessful:
Hypothesis
High confidence: the test’s bad-slot selection only excluded
candidateSlot1 - 1as a pre-bad pipelined target. In the failed run,candidateSlot1 - 2was still unsnapshotted and owned by a bad proposer, so applying malicious config leaked into slot 23.Evidence
Proposed Fix
Implemented in epochs_invalidate_block.parallel.test.ts: the selector now excludes bad proposers from every pre-bad target slot from
currentSlot + 2throughcandidateSlot1 - 1, not just the immediately prior slot.Also fixed the broken timeout race at line 475 by removing the accidental inner
await.