feat(prover-node): checkpoint-driven optimistic proving

[PhilWindle]

Drives the prover-node onto the split CheckpointSubTreeOrchestrator + TopTreeOrchestrator pair, with checkpoint-driven proving that pipelines sub-trees against tx-gathering and the top-tree against the in-flight sub-trees.

Stacked on #22996 (orchestrator split). Diff above is just the prover-node + e2e + cleanup delta.

What's new

prover-node — EpochProvingJob job-model rewrite

EpochProvingJob becomes an orchestrator over a Map<string, CheckpointJob> keyed by ${number}:${slot}. Each CheckpointJob owns a single CheckpointSubTreeOrchestrator with its own per-checkpoint context (txs, attestations, previous-block header, l1ToL2 messages, archive sibling path). A TopTreeJob drives the epoch root rollup once all checkpoint sub-trees have started block-level proving.

Public API:

• registerCheckpoint — synchronous; sets up sub-tree, kicks off chonk-verifier cache fill, attaches the blockProofs promise to the eventual top-tree job.
• provideTxs — supplies simulated txs, transitions the checkpoint job from registered → block-proving.
• removeCheckpointsAfter(threshold) — sole removal API. Suffix-only: cancels every job whose checkpoint number sits strictly above the threshold, atomically. Suffix-only by design — middle-removal would leave a non-contiguous survivor set that cannot be proved.
• getCheckpointCount, getCheckpointNumbers.

TopTreeJob constructor defensively asserts that its snapshot's checkpoint numbers form a contiguous run (prev + 1 === curr). With suffix-only removal this is structurally guaranteed; the assertion is a runtime tripwire against future regressions.

prover-node — L2BlockStream-driven checkpoint pipeline

The prover-node consumes chain-checkpointed / chain-pruned events from an L2BlockStream rooted at the first block of the first unproven epoch. On each chain-checkpointed:

1. Resolve the epoch via getEpochAtSlot.
2. Get-or-create the per-epoch EpochProvingJob.
3. Register the checkpoint synchronously and detach a tx-gathering task.
4. Call tryCompleteEpoch for both older epochs (tertiary signal) and the current epoch (covers the case where the EpochMonitor already fired and we were waiting on this last checkpoint to arrive).

On chain-pruned: call removeCheckpointsAfter(threshold) on every job whose first checkpoint sits at or above the threshold. Pending gather tasks are cancelled via AbortSignal.

Finalization is driven by the union of three signals — EpochMonitor (epoch closes on L1), tertiary (a checkpoint for a strictly later epoch arrives), or final-checkpoint registration on an already-monitored epoch. Each signal calls tryCompleteEpoch, which queries l2BlockSource.isEpochComplete directly: the archiver is the source of truth, so the prover-node carries no in-memory cache of "epochs complete on L1".

prover-node — reorg-after-finalization restart

When the L2BlockStream emits a prune that retroactively invalidates an epoch already in finalize, the prover-node aborts the in-flight publish, clears the job, and restarts proving from the new tip.

e2e

• New epochs_optimistic_proving.parallel.test.ts: full e2e covering the pipelining, replacement-checkpoint reuse, and reorg-during-proving paths.
• The two happy-path tests use a wall-clock-vs-job-epoch sampler to assert that the prover-node has registered checkpoints with the in-flight job for an epoch while still inside that epoch — proves checkpoints aren't deferred until epoch end. The multi-epoch test runs 4 epochs and asserts this for each.
• epochs_proof_fails, epochs_upload_failed_proof, epochs_long_proving_time, epochs_multi_proof updated to assert the new in-flight epoch behaviour.

Logging

Structured-context logs added across the CheckpointJob / TopTreeJob / EpochProvingJob lifecycles: register/provide-txs/sub-tree-created/per-block-progress/blockProofs-ready/cancel/teardown for checkpoints, and create/cancel/prove-start/prove-success for top-tree. Every log entry indexes on epochNumber / checkpointNumber per repo convention.

What's removed

EpochProver interface and ServerEpochProver are removed: the prover-node no longer drives a single-class epoch prover, so the legacy API has no production callers. ProvingOrchestrator survives only as a base class for CheckpointSubTreeOrchestrator and as the single-class driver used by prover-client's integration tests; it no longer implements EpochProver.

EpochProvingJob.removeCheckpoint(checkpointNumber) and cancelPendingCheckpoints() are also removed: middle-removal could leave a non-contiguous survivor set that the TopTreeJob constructor would reject. Suffix-only removeCheckpointsAfter is the only checkpoint-removal API now.

Test plan

• yarn workspace @aztec/prover-client test — 261 tests pass.
• yarn workspace @aztec/prover-node test — 116 tests pass (incl. new dedicated top-tree-job.test.ts and checkpoint-job.test.ts).
• e2e tests covering optimistic proving, reorgs during proving, failed proof publish, and multi-checkpoint flows are included in this PR.

[spalladino]

Got halfway through it, but looks good so far. Pasting here some observations by Claude that I didn't get to review myself:

• Same-epoch replacement checkpoints are silently dropped after completeEpoch(). prover-node.ts:213-220 skips any checkpoint for an epoch whose job has isEpochComplete(). Sequence: (1) all checkpoints registered, completeEpoch() fires; (2) prune removes the suffix; (3) the L1 reorg replacement for the same epoch arrives. The replacement is dropped because existingJob.isEpochComplete() is true. The finalize loop then restarts against an incomplete prefix of the epoch and eventually publishes a proof for a checkpoint range that does not match the canonical L1 chain. This contradicts the stated "orphan + replacement coexist briefly" model. The check should be conditioned on the surviving checkpoint set, not on the boolean flag.

• A prune that lands after topTreeJob.start() but during publishProof is not interceptable. epoch-proving-job.ts:478 clears this.topTreeJob immediately after the prove resolves; the publish at epoch-proving-job.ts:487 operates on the captured snapshot array. Meanwhile removeCheckpointsAfter (epoch-proving-job.ts:332-369) only acts on the live topTreeJob — which is now undefined — so the publish proceeds with stale data and the (out-of-date) attestations from snapshot.at(-1). Best case L1 rejects the proof (gas burn, retry confusion); worst case it lands against a window the canonical chain has since invalidated. Either re-validate the snapshot's checkpoint set (by id) against the live registry just before submitting, or hold the top-tree-job reference until publish completes and route prune through it.

• EpochMonitor permanently records "processed" on a no-op tryCompleteEpoch. prover-node.ts:145-153 returns true from handleEpochReadyToProve whenever tryCompleteEpoch doesn't throw — even if it bailed at prover-node.ts:394 (archiver says incomplete) or prover-node.ts:399-405 (not all checkpoints known to the job). epoch-monitor.ts:93 then sets latestEpochNumber = epochToProve, and epoch-monitor.ts:76-79 ensures the same epoch is never re-considered. In a healthy network the next checkpoint event for a later epoch reseeds via the tertiary signal, but if the epoch in question is the last produced (network stall, end-of-test) it stays stuck forever. tryCompleteEpoch should return whether completeEpoch() was actually invoked, and handleEpochReadyToProve should propagate that.

[spalladino]

Note that by updating the store first, any error in the handle methods below will not be retried in the next iteration of the L2BlockStream. Not sure if this is intentional.

[PhilWindle]

Yes it should go at the end I think.

[spalladino]

We should check in the archiver and blockstream what happens first: do checkpoints get emitted, or does the proven block number get updated? I'm worried that, when starting a long sync after offline for a while, all the checkpoints are emitted first and then the proven tip is updated, so the prover node would trigger a ton of jobs only to cancel them immediately afterwards.

[spalladino]

Ah just saw the compute starting block number for the blockstream. And as long as the prover node waits for the archiver initial sync to be completed before starting, we should be good I guess.

[spalladino]

I think we can now replace the entire epoch monitor with a running promise that just calls l2BlockSource.isEpochComplete periodically.

[spalladino]

Won't this create a hole in a given epoch? I don't see another code path where we patch an epoch job with checkpoint jobs we failed to add due to lack of capacity. I'm worried about a scenario where we don't add a checkpoint to an epoch, but then a prior epoch finishes, so we do add the next, and that epoch job ends up waiting for that missing checkpoint forever.

Speaking of which: is there anything that cleans up very old jobs when the epoch proof submission window closes?

[PhilWindle]

Yes, was going to ask about this. As it stands here there could be problems if this is set. Do you think this is needed or makes sense anymore? Proving is more of a continuous process rather than a set of large jobs.

[spalladino]

Maybe it makes sense to move the responsibility of collecting their data to the checkpoint jobs, to keep the prover node leaner?

[PhilWindle]

I think it does. As a further refactor, I want to break the checkpoint proving out of EpochProvingJob. Then we maintain a collection of checkpoint jobs being proven. EpochProvingJob then becomes a collection of references to a subset of those checkpoints and a top tree. This would enable us to create instances of EpochProvingJob with arbitrary checkpoints (of a valid range of course) enabling efficient partial epoch proving.

[spalladino]

Looks comprehensive. I'd add tests for cross-chain messages as well, those are usually a source of problems.

[PhilWindle]

Will do

[spalladino]

Heads up this should pass even without optimistic proving: prover would start as soon as the last checkpoint of the epoch was pushed to L1, which could happen a few L1 slots before the actual end of the epoch. Maybe we should offset this a bit so the check is more meaningful, or check for specific checkpoint jobs vs all jobs in the epoch?

[spalladino]

I'd also add an assertion that the prover actually started the job for the given checkpoint before reorging it out. Otherwise, this test could pass if the prover just takes a bit more time to start assembling the checkpoint job.

[spalladino]

This should also assert that both checkpoints landed on the same epoch, which could not be the case if this test started near the end of an epoch.

[spalladino]

We should also add txs. And it'd be interesting to see a case where one of the reorgs means that a tx that was mined in a checkpoint is now mined in a different one.

[spalladino]

I'm wondering if there's a cleaner way of doing this, perhaps reusing the EpochProvingJobHooks? Not sure if worth stressing over it though.

[spalladino]

Is there a race condition where a reorg removes all checkpoints from an epoch and then new ones are immediately added again, but we hit this case before the new ones get added? Seems like something that would never happen in prod, but could affect e2e tests that have short epochs.

[PhilWindle]

There shouldn't be. The prover node does this in it's prune handler.

      if (job.getCheckpointCount() === 0) {
        this.log.info(`Cancelling epoch ${epochNum} job — all checkpoints pruned`);
        await this.cancelAndCleanupJob(epochNum, job);
      }

So that epoch job instance is removed and replaced with another when a new checkpoint arrives for the same epoch number.

[PhilWindle]

I will add a test though

[spalladino]

How about we move the responsibility of uploading this data to the epoch-proving-job itself, so we don't need to collect this just for the prover-node to maybe read if it's configured for uploads?

[spalladino]

I don't think it makes a difference based on how reorgs are handled, but I'd throw in the checkpoint archive as part of the identifier, just in case

[spalladino]

I know this didn't change with this PR, but for private-only txs, who verifies the chonk proof?

[PhilWindle]

The private base rollup. Public txs have a different process. They require a dedicated chonk verifier circuit.

[spalladino]

I take it it's responsibility of the caller to check that all dependencies are ready, right?

[PhilWindle]

Which dependencies are you referring to? At the point this class is constructed, it's arguments contain all information required to start the proving process (e.g. compute blob challenges). If the checkpoint root input proofs aren't ready yet that is fine, the orchestrator fires the appropriate events as and when they are.

[alexghr]

Looks good!

[alexghr]

I think this is a great idea! I'd consider extending it to Base/Root parity circuits too because they could benefit from the same caching behaviour across reorged checkpoints