Skip to content

fix(ci): bump AVM check-circuit per-test timeout to 120s for large circuits#23929

Closed
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-timeout
Closed

fix(ci): bump AVM check-circuit per-test timeout to 120s for large circuits#23929
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-cc-timeout

Conversation

@AztecBot

@AztecBot AztecBot commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

What

Raise the per-test budget for the avm_check_circuit runs in yarn-project/end-to-end/bootstrap.sh from TIMEOUT=30s to TIMEOUT=120s.

Why

The AVM check-circuit check failed on next (run 27132247511, commit d9b6e71). Every dumped input passed in 3–5s except one, which was killed with code: 124 (timeout):

FAILED: bb-avm avm_check_circuit -v --avm-inputs .../e2e_multiple_blobs/avm-circuit-inputs-tx-0x1772e8af....bin (34s) (code: 124)
run_test_cmd 'cf0b953d140d26e4:ISOLATE=1:TIMEOUT=30s:NAME=avm_cc_e2e_multiple_blobs_0x1772e8af ...'

The sub-log shows where the time went (container limited to CPUS=2, TIMEOUT=30s):

11:15:34 Simulating...
11:15:35 Generating trace...            (mem 823 MiB)
11:15:58 Checking circuit...            (mem 3885 MiB)   <- trace generation alone took ~23s
11:15:58 Running check (with skippable) circuit over 700560 rows.
11:16:04 timeout: sending signal TERM to command 'bash'  <- killed at the 30s limit

e2e_multiple_blobs produces the largest AVM circuit we dump (~700k rows). Trace generation takes ~23s on its own, leaving under 7s for the relation check — so the run exceeds the 30s budget that was originally sized for "relatively small" e2e txs (exactly the case the existing WARNING comment anticipated).

Root cause

This is a per-test time-budget issue, not a logic bug, and it is not caused by the barretenberg Pippenger/MSM refactor in the triggering merge-train commit: avm_check_circuit performs AVM trace generation and gate-relation checking, neither of which uses Pippenger MSM. The 30s timeout has been in place since the feature landed (#18747); the largest circuit simply tipped over the edge.

Fix

Bump the timeout to 120s (≈3× the observed ~40s worst case, leaving headroom for CI contention) and update the WARNING comment to point at the largest known case and the lever (TIMEOUT/CPUS) to pull if a bigger tx starts timing out. Parallelism and per-job resources are unchanged, so the small txs are unaffected and overall wall-clock is essentially unchanged (only the one large run now runs to completion instead of being killed).


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR. labels Jun 8, 2026
@AztecBot

Copy link
Copy Markdown
Collaborator Author

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.

@AztecBot AztecBot closed this Jun 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant