feat: merge-train/barretenberg#21962
Merged
Merged
Conversation
…21953) ## Summary - macOS release binaries were broken (SIGKILL/exit 137) because `inject_version` in `bootstrap.sh` used `llvm-objdump` (unversioned) to detect Mach-O binaries for re-signing, but the CI environment only has `llvm-objdump-20` - The detection silently failed (`&>/dev/null`), skipping `ldid -S` re-signing after version injection, leaving binaries with an invalidated linker-signed code signature - Fix: `llvm-objdump` → `llvm-objdump-20`
The Chonk standalone vk check was failing because: - The native preset was not used to resolve `bb` (so we were resolving `build/bin/bb` instead of `build-debug/bin/bb`) - The VK check did not disable asserts (so when checking VKs we would detect mismatches due to the dummy data supplied to build VKs) This PR fixes the above issues: - Resolve `bb` using the script `preset-build-dir` with argument `NATIVE_PRESET` - Add a flag to the `check` command in the bb api that disables asserts when checking vks
## Summary Switch entirely to compressed BN254 G1 points (32 bytes/point vs 64 bytes/point) everywhere: C++ native, WASM/bb.js, CI scripts, and release images. - C++ decompresses via parallel `from_compressed()` using `ThreadChunk` - SHA-256 chunk hash verification of downloaded compressed CRS data - Prover-agent release image ships `bn254_g1_compressed.dat`, halving the CRS layer size (~3.2GB vs ~6.4GB) - Added `skipSrsInit` option to `BackendOptions` so hash-only WASM tests skip CRS initialization ### Browser wallet startup benchmark Tested on the playground embedded wallet. Download from local L2 node, measuring G1 download time, `srsInitSrs` (WASM deserialization/decompression), and total `createPXE` time. **Compressed (32 bytes/point, `g1_compressed.dat`):** | Cores | G1 download | srsInitSrs | CRS total | createPXE | |-------|------------|------------|-----------|-----------| | 4 | 3059ms | 1988ms | 5840ms | 10351ms | | 2 | 3097ms | 3781ms | 7677ms | 12150ms | | 1 | 3111ms | 7372ms | 11249ms | 15858ms | **Uncompressed baseline (64 bytes/point, `g1.dat`, parallelized `from_buffer`):** | Cores | G1 download | srsInitSrs | CRS total | createPXE | |-------|------------|------------|-----------|-----------| | 4 | 7565ms | 2019ms | 10381ms | 15549ms | | 2 | 6078ms | 3802ms | 10742ms | 15635ms | | 1 | 6065ms | 7307ms | 14173ms | 18744ms | **Key findings:** - **Decompression overhead is effectively zero in the browser.** `srsInitSrs` is dominated by msgpack serialization and JS-to-WASM data transfer, not point parsing. The raw `from_compressed` cost is masked by this overhead. - **The win is the 2x smaller download**: ~3.1s vs ~6-7.5s for G1 data. - **createPXE improvement**: 33% at 4 cores, 22% at 2 cores, 15% at 1 core. Note: earlier wasmtime benchmarks showed `from_compressed` at ~19s for 2^20 points (single-threaded), but V8's WASM JIT is roughly 5x faster on this field arithmetic workload. The browser `from_compressed` cost is estimated at ~3.9s (after subtracting serialization overhead), well within the existing overhead budget. ### Native C++ decompression (full 100M point SRS, 3.2GB compressed) | Threads | Time | |---------|------| | 8 | 111s | | 32 | 28s | ### Changes **C++ (bb CLI + WASM)** - `bbapi_srs.cpp` -- `SrsInitSrs` decompresses 32-byte compressed points via parallel `from_compressed()` - `get_bn254_crs.cpp` -- downloads compressed CRS with CDN fallback, parallel SHA-256 chunk verification, parallel decompression - `chonk.bench.cpp` -- added `bn254_point_decompression` benchmark **TypeScript (bb.js)** - `net_crs.ts` -- downloads from `g1_compressed.dat` endpoint - `crs/node/index.ts` -- caches and loads `bn254_g1_compressed.dat` (32 bytes/point) - `crs/browser/cached_net_crs.ts` -- same for browser/IndexedDB - `bb_backends/index.ts` -- added `skipSrsInit` option to `BackendOptions` - `barretenberg/index.ts` -- respect `skipSrsInit` in `Barretenberg.new()` - `blake2s.test.ts`, `poseidon.bench.test.ts` -- use `skipSrsInit: true` (hash-only, no CRS needed) **Release image** - `release-image/bootstrap.sh` -- `prepare_crs` ships `bn254_g1_compressed.dat` **CI scripts** - `download_bb_crs.sh` -- downloads compressed with fallback/retries ## Test plan - [ ] CI passes (bb tests, WASM tests, acir tests) - [ ] Prover-agent image builds with compressed CRS
Collaborator
Author
|
🤖 Auto-merge enabled after 4 hours of inactivity. This PR will be merged automatically once all checks pass. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BEGIN_COMMIT_OVERRIDE
fix: use llvm-objdump-20 for Mach-O re-signing in version injection (#21953)
chore: Fix nightly debug vk check (#21957)
feat: SRS point compression - download 50% less CRS data (#21112)
END_COMMIT_OVERRIDE