feat: SRS point compression - download 50% less CRS data by johnathan79717 · Pull Request #21112 · AztecProtocol/aztec-packages

johnathan79717 · 2026-03-04T15:56:51Z

Summary

Switch entirely to compressed BN254 G1 points (32 bytes/point vs 64 bytes/point) everywhere: C++ native, WASM/bb.js, CI scripts, and release images.

C++ decompresses via parallel from_compressed() using ThreadChunk
SHA-256 chunk hash verification of downloaded compressed CRS data
Prover-agent release image ships bn254_g1_compressed.dat, halving the CRS layer size (~3.2GB vs ~6.4GB)
Added skipSrsInit option to BackendOptions so hash-only WASM tests skip CRS initialization

Browser wallet startup benchmark

Tested on the playground embedded wallet. Download from local L2 node, measuring G1 download time, srsInitSrs (WASM deserialization/decompression), and total createPXE time.

Compressed (32 bytes/point, g1_compressed.dat):

Cores	G1 download	srsInitSrs	CRS total	createPXE
4	3059ms	1988ms	5840ms	10351ms
2	3097ms	3781ms	7677ms	12150ms
1	3111ms	7372ms	11249ms	15858ms

Uncompressed baseline (64 bytes/point, g1.dat, parallelized from_buffer):

Cores	G1 download	srsInitSrs	CRS total	createPXE
4	7565ms	2019ms	10381ms	15549ms
2	6078ms	3802ms	10742ms	15635ms
1	6065ms	7307ms	14173ms	18744ms

Key findings:

Decompression overhead is effectively zero in the browser. srsInitSrs is dominated by msgpack serialization and JS-to-WASM data transfer, not point parsing. The raw from_compressed cost is masked by this overhead.
The win is the 2x smaller download: ~3.1s vs ~6-7.5s for G1 data.
createPXE improvement: 33% at 4 cores, 22% at 2 cores, 15% at 1 core.

Note: earlier wasmtime benchmarks showed from_compressed at ~19s for 2^20 points (single-threaded), but V8's WASM JIT is roughly 5x faster on this field arithmetic workload. The browser from_compressed cost is estimated at ~3.9s (after subtracting serialization overhead), well within the existing overhead budget.

Native C++ decompression (full 100M point SRS, 3.2GB compressed)

Threads	Time
8	111s
32	28s

Changes

C++ (bb CLI + WASM)

bbapi_srs.cpp -- SrsInitSrs decompresses 32-byte compressed points via parallel from_compressed()
get_bn254_crs.cpp -- downloads compressed CRS with CDN fallback, parallel SHA-256 chunk verification, parallel decompression
chonk.bench.cpp -- added bn254_point_decompression benchmark

TypeScript (bb.js)

net_crs.ts -- downloads from g1_compressed.dat endpoint
crs/node/index.ts -- caches and loads bn254_g1_compressed.dat (32 bytes/point)
crs/browser/cached_net_crs.ts -- same for browser/IndexedDB
bb_backends/index.ts -- added skipSrsInit option to BackendOptions
barretenberg/index.ts -- respect skipSrsInit in Barretenberg.new()
blake2s.test.ts, poseidon.bench.test.ts -- use skipSrsInit: true (hash-only, no CRS needed)

Release image

release-image/bootstrap.sh -- prepare_crs ships bn254_g1_compressed.dat

CI scripts

download_bb_crs.sh -- downloads compressed with fallback/retries

Test plan

CI passes (bb tests, WASM tests, acir tests)
Prover-agent image builds with compressed CRS

ludamad · 2026-03-04T20:59:08Z

Damn, talk about a big decentralization win.

Remove references to uncompressed g1.dat; all tests now use g1_compressed.dat consistently.

- Add 3 retries with 5s delay to download_with_fallback in both crs/bootstrap.sh and scripts/download_bb_crs.sh - Call crs/bootstrap.sh from run_test.sh to ensure compressed CRS is available (CI AMI may only have old uncompressed g1.dat)

Include the file path and suggest running crs/bootstrap.sh.

…ect parity sign bit The compression script was using y > p/2 to set the sign bit, but bb::affine_element::from_compressed uses LSB of y (parity). This caused the second element verification to fail in ASAN builds. Regenerated all 763 chunk hashes from the correctly compressed 3.2GB file.

Switch prepare_crs to copy bn254_g1_compressed.dat (32 bytes/point) instead of bn254_g1.dat (64 bytes/point), halving the CRS layer size.

No longer needed since compressed CRS is the only format and download_bb_crs.sh already handles it.

johnathan79717 · 2026-03-18T14:52:56Z

/claudebox why is the CI still failing?

AztecBot · 2026-03-18T14:53:11Z

⏳ Run #1 — Session completed (6m)
Live status

CI fails because blake2s.test.js's beforeAll hook exceeds Jest's 5s default timeout — Barretenberg.new() now decompresses 2^20 compressed CRS points in WASM via srsInitSrs, which takes ~7s at 1 thread. Before this PR there was no decompression step. Full analysis + fix options: https://gist.github.com/AztecBot/3758b0b4e40380895108f54ab473065f

Gist: CI failure analysis for PR #21112 (SRS point compression) — blake2s.test.js timeout

… tests blake2s and poseidon tests only use hashing functions and don't need CRS/SRS data. With compressed CRS, the WASM decompression of 2^20 points exceeds Jest's 5s beforeAll timeout. Skip SRS init entirely for these tests.

…h/srs-point-compression # Conflicts: # barretenberg/cpp/src/barretenberg/benchmark/chonk_bench/chonk.bench.cpp

…avior) Only the C++ native path and bootstrap scripts change: - C++ downloads compressed, decompresses, stores uncompressed bn254_g1.dat - Bootstrap scripts download uncompressed g1.dat directly - JS/WASM path unchanged from PR #21112 (compressed throughout)

BEGIN_COMMIT_OVERRIDE fix: use llvm-objdump-20 for Mach-O re-signing in version injection (#21953) chore: Fix nightly debug vk check (#21957) feat: SRS point compression - download 50% less CRS data (#21112) END_COMMIT_OVERRIDE

CI machines may have bn254_g1_compressed.dat (from PR #21112) but not bn254_g1.dat. Check both filenames: uncompressed first, then compressed. New downloads still store as uncompressed bn254_g1.dat.

Two fixes for debug build: 1. Legacy CRS fallback: when compressed CRS (bn254_g1_compressed.dat) is not available but old uncompressed CRS (bn254_g1.dat) exists, read the uncompressed format directly. This ensures tests work on machines that haven't re-downloaded the CRS after the compression change in #21112. 2. Non-throwing debug pairing checks: the stdlib PairingPoints constructor and aggregate() method perform native pairing verification in debug builds (#ifndef NDEBUG). When mock/dummy data produces invalid points (e.g. during VK generation with cleared witnesses), this threw an uncatchable exception. Wrapped these debug-only checks in try-catch so they log failures without crashing.

## Summary - After PR #21112 switched CRS storage to compressed format (32 bytes/point), every CRS load decompresses from scratch. For 2^20 points in a browser that's ~4s on every page load; for 100M points on an 8-core native prover that's ~110s. - This PR makes compression a bandwidth-only optimization: after the first decompression, the uncompressed points (64 bytes/point) are cached and reused on subsequent loads. ### Changes **C++ native** (`get_bn254_crs.cpp`): Check for cached `bn254_g1.dat` (uncompressed) first. On cache miss, decompress from `bn254_g1_compressed.dat` and write the uncompressed cache. On download, cache both compressed and uncompressed. **C++ WASM** (`bbapi_srs.hpp/cpp`): `srsInitSrs` auto-detects input format (32B vs 64B per point). When decompressing compressed input, it returns the uncompressed bytes in the response so the caller can cache them. **Browser** (`cached_net_crs.ts`): Check IndexedDB for uncompressed `g1Data` first. Add `cacheUncompressed()` to persist after first decompression. **Node.js** (`crs/node/index.ts`): Check `bn254_g1.dat` first, fall back to `bn254_g1_compressed.dat`. Add `cacheUncompressed()` to write uncompressed file. **Wiring** (`barretenberg/index.ts`): `initSRSChonk` captures the `srsInitSrs` response and calls `crs.cacheUncompressed()` when uncompressed bytes are returned. ## Test plan - [ ] C++ native: run `bb` with only compressed CRS on disk, verify it creates `bn254_g1.dat` and uses it on second run - [ ] WASM: `./scripts/run_test.sh bbapi/exception_handling.test.js` passes (schema change) - [ ] WASM: `./scripts/run_test.sh barretenberg/blake2s.test.js` passes (skipSrsInit) - [ ] Browser: first load downloads compressed, caches uncompressed in IDB; second load skips decompression

The SRS compression PR (#21112) changed point format from 64 to 32 bytes but didn't add explicit buffer size validation. With the old uncompressed format, reading past the buffer would hit g1_identity validation and throw. With compressed format, WASM silently reads garbage memory without throwing. Add explicit bounds checks for both points_buf and g2_point before processing. Update TS test to expect the new error message and fix use of undefined fail().

Two fixes for debug build: 1. Legacy CRS fallback: when compressed CRS (bn254_g1_compressed.dat) is not available but old uncompressed CRS (bn254_g1.dat) exists, read the uncompressed format directly. This ensures tests work on machines that haven't re-downloaded the CRS after the compression change in #21112. 2. Non-throwing debug pairing checks: the stdlib PairingPoints constructor and aggregate() method perform native pairing verification in debug builds (#ifndef NDEBUG). When mock/dummy data produces invalid points (e.g. during VK generation with cleared witnesses), this threw an uncatchable exception. Wrapped these debug-only checks in try-catch so they log failures without crashing.

johnathan79717 force-pushed the jh/srs-point-compression branch from a7f4b6f to 6b57576 Compare March 4, 2026 16:59

johnathan79717 requested review from a team, IlyasRidhuan, LHerskind, LeilaWang, Maddiaa0, charlielye, fcarreiro, jeanmon, just-mitch and nventuro as code owners March 4, 2026 16:59

johnathan79717 changed the base branch from merge-train/barretenberg to claudebox/hybrid-crs-hash-verification March 4, 2026 16:59

johnathan79717 force-pushed the jh/srs-point-compression branch from 6b57576 to 18a1ab8 Compare March 4, 2026 17:05

johnathan79717 removed request for a team, IlyasRidhuan, LHerskind, LeilaWang, Maddiaa0, charlielye, fcarreiro, jeanmon, just-mitch and nventuro March 4, 2026 17:06

Base automatically changed from claudebox/hybrid-crs-hash-verification to merge-train/barretenberg March 4, 2026 17:26

johnathan79717 force-pushed the jh/srs-point-compression branch from 1bdb1da to a78b80a Compare March 4, 2026 17:44

johnathan79717 requested a review from ledwards2225 March 4, 2026 18:08

ludamad reviewed Mar 4, 2026

View reviewed changes

Comment thread barretenberg/ts/src/crs/net_crs.ts Outdated

ludamad reviewed Mar 4, 2026

View reviewed changes

Comment thread barretenberg/cpp/src/barretenberg/benchmark/chonk_bench/chonk.bench.cpp

johnathan79717 requested a review from ludamad March 5, 2026 12:25

johnathan79717 added 6 commits March 17, 2026 17:10

Update TS fetchWithFallback tests to use compressed CRS URLs

de94aae

Remove references to uncompressed g1.dat; all tests now use g1_compressed.dat consistently.

Improve error message when compressed CRS is missing

ac59021

Include the file path and suggest running crs/bootstrap.sh.

Ship compressed CRS in prover-agent release image

958a94f

Switch prepare_crs to copy bn254_g1_compressed.dat (32 bytes/point) instead of bn254_g1.dat (64 bytes/point), halving the CRS layer size.

Remove redundant crs/bootstrap.sh call from run_test.sh

9b6a7e8

No longer needed since compressed CRS is the only format and download_bb_crs.sh already handles it.

johnathan79717 force-pushed the jh/srs-point-compression branch from 47723ab to 9b6a7e8 Compare March 17, 2026 17:10

Base automatically changed from merge-train/barretenberg to next March 24, 2026 00:19

Merge remote-tracking branch 'origin/merge-train/barretenberg' into j…

b27e29e

…h/srs-point-compression # Conflicts: # barretenberg/cpp/src/barretenberg/benchmark/chonk_bench/chonk.bench.cpp

johnathan79717 changed the base branch from next to merge-train/barretenberg March 24, 2026 15:56

ludamad approved these changes Mar 24, 2026

View reviewed changes

johnathan79717 enabled auto-merge (squash) March 24, 2026 16:28

johnathan79717 merged commit 1683d61 into merge-train/barretenberg Mar 24, 2026
20 of 21 checks passed

johnathan79717 deleted the jh/srs-point-compression branch March 24, 2026 16:39

This was referenced Mar 24, 2026

feat: merge-train/barretenberg #21962

Merged

fix: always store uncompressed CRS on disk in C++ native path #21972

Open

AztecBot mentioned this pull request Mar 25, 2026

fix: add bounds checking to SrsInitSrs to fix debug build test #21985

Merged

johnathan79717 mentioned this pull request Mar 25, 2026

fix: cache uncompressed CRS after first decompression #22002

Merged

4 tasks

AztecBot mentioned this pull request Mar 26, 2026

fix: add bounds checking to SrsInitSrs for compressed point format #22054

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: SRS point compression - download 50% less CRS data#21112

feat: SRS point compression - download 50% less CRS data#21112
johnathan79717 merged 21 commits into
merge-train/barretenbergfrom
jh/srs-point-compression

johnathan79717 commented Mar 4, 2026 •

edited

Loading

Uh oh!

ludamad commented Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

johnathan79717 commented Mar 18, 2026

Uh oh!

AztecBot commented Mar 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

johnathan79717 commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Browser wallet startup benchmark

Native C++ decompression (full 100M point SRS, 3.2GB compressed)

Changes

Test plan

Uh oh!

ludamad commented Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

johnathan79717 commented Mar 18, 2026

Uh oh!

AztecBot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

johnathan79717 commented Mar 4, 2026 •

edited

Loading

AztecBot commented Mar 18, 2026 •

edited

Loading