Skip to content

fix(ci): give e2e_multiple_blobs AVM check-circuit a larger timeout/CPU budget#23787

Closed
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-multiple-blobs-timeout-fix
Closed

fix(ci): give e2e_multiple_blobs AVM check-circuit a larger timeout/CPU budget#23787
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-multiple-blobs-timeout-fix

Conversation

@AztecBot

@AztecBot AztecBot commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Problem

The avm-check-circuit job failed on next (run 26784473487) with exit code 124 (timeout).

A single input timed out — e2e_multiple_blobs tx 0x17e546c9… — while every other input passed in ~4s:

FAILED ...: bb-avm avm_check_circuit ... e2e_multiple_blobs/...-0x17e546c9....bin (35s) (code: 124)
run_test_cmd '...:ISOLATE=1:TIMEOUT=30s:NAME=avm_cc_e2e_multiple_blobs_0x17e546c9 ...'

Root cause

Not a correctness bug or a hang. The e2e_multiple_blobs tx produces by far the largest AVM circuit (~700k rows). On the default isolate resources (2 CPUs, 8g), the per-input run breaks down as:

  • Simulating: ~2s
  • Generating trace: ~24s (CPU-bound, 2 CPUs)
  • Checking circuit over 700,560 rows (peak ~3.9 GiB) — started but killed before finishing

All AVM check-circuit inputs share a single TIMEOUT=30s budget in avm_check_circuit_cmds. This input simply doesn't fit in 30s on 2 CPUs — exactly the case the existing WARNING comment in that function anticipated ("transactions could need more CPU and MEM than we allocate by default … they might start timing out").

Fix

Add a per-input override (mirroring the per-test override idiom already used in test_cmds) so the large e2e_multiple_blobs circuit gets CPUS=8:TIMEOUT=180s. More CPUs cut the CPU-bound trace generation substantially, and the longer timeout gives generous headroom for the check itself. All other small inputs keep the default TIMEOUT=30s / 2 CPUs, so overall stage parallelism is unaffected. Peak memory (~3.9 GiB) is already under the 8g default, so MEM is left unchanged.

Notes

  • Targeting next as requested by the CI failure dispatch.
  • This is a CI resource/timeout adjustment only; no production or circuit logic changes.

Created by claudebox · group: slackbot

@AztecBot AztecBot added the claudebox Owned by claudebox. it can push to this PR. label Jun 1, 2026
@AztecBot

AztecBot commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator Author

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.

@AztecBot AztecBot closed this Jun 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant