feat: migrate bb-prover from direct binary execution to bb.js API by charlielye · Pull Request #21564 · AztecProtocol/aztec-packages

charlielye · 2026-03-14T00:36:20Z

Summary

Migrates all bb-prover server-side proving and verification from spawning bb/bb-avm binaries via child_process.spawn() to using the bb.js TypeScript API (Barretenberg class via NativeUnixSocket backend). This eliminates temporary file I/O (writing/reading bytecode, witnesses, proofs, VKs to disk) and stdout-based result parsing. A fresh bb process is still spawned per proof to maintain parallelism, but communication happens via Unix socket instead of files.

What was migrated

Operation	Before	After
UltraHonk proving	`bb prove` via spawn	`api.circuitProve()` via bb.js
UltraHonk verification	`bb verify` via spawn	`api.circuitVerify()` via bb.js
Chonk (IVC) verification	`bb verify --scheme chonk` via spawn + temp files	`api.chonkVerify()` via bb.js
AVM proving	`bb-avm avm_prove` via spawn + temp files	`api.avmProve()` via bb.js
AVM verification	`bb-avm avm_verify` via spawn + temp files	`api.avmVerify()` via bb.js
AVM circuit check	`bb-avm avm_check_circuit` via spawn + temp files	`api.avmCheckCircuit()` via bb.js
Gate counting	`bb gates` via spawn	`api.circuitStats()` / `api.chonkStats()` via bb.js
Solidity contract gen	`bb contract` via spawn	`api.circuitWriteSolidityVerifier()` via bb.js

Key changes

New BBJsInstance / BBJsProverFactory (bb_js_backend.ts): Adapter layer wrapping the Barretenberg class with typed methods for all proof operations.
New DebugBBJsInstance (bb_js_debug.ts): Debug wrapper that writes inputs/outputs to disk and logs equivalent CLI commands (see debug section below).
C++ AVM msgpack commands (bbapi_avm.hpp/cpp): Three bbapi command structs (AvmProve, AvmVerify, AvmCheckCircuit) with in-memory function implementations in vm2/api_avm.cpp and stub implementations in vm2_stub/api_avm.cpp.
AVM per-stage timings over the API: New AvmStat serialized struct, plus a stats: AvmStat[] field on AvmProve::Response and AvmCheckCircuit::Response. Stats::snapshot() in barretenberg/vm2/tooling returns the per-stage map; avm_prove_from_bytes resets the singleton at entry and snapshots on exit. This replaces the legacy stdout-scraping used by avm_proving_tester.ts for per-stage timings, which now reads the stats array directly.
Chonk proof splitting (splitChonkProofToStructured): Mirrors C++ ChonkProof::from_field_elements — derives the hidingOinkProof length as the remainder after subtracting the four fixed-size sub-proofs (MERGE_PROOF_SIZE, ECCVM_PROOF_LENGTH, IPA_PROOF_LENGTH, JOINT_PROOF_LENGTH). Removes coupling to HIDING_OINK_LENGTH / HIDING_KERNEL_IO_PUBLIC_INPUTS_SIZE / numCustomPublicInputs, so changes in Mega witness entities can't silently misalign the TS split.
Removed bb-prover/src/bb/execute.ts: replaced by file_names.ts containing the three bb-CLI output filename constants (VK_FILENAME, PROOF_FILENAME, PUBLIC_INPUTS_FILENAME) that a few readers still need. The binary spawner (executeBB) is inlined into ivc-integration/src/bb_js_debug.test.ts, the only remaining consumer — a CLI-parity test that must spawn bb to compare its output against bb.js's.
Test harnesses migrated to bb.js: avm_proving_tester.ts (reads timings from the stats response field instead of scraping stdout) and ivc-integration/src/prove_native.ts::proveAvm (in-memory proof, no tempdir, no file I/O).

Debug workflow with `BB_DEBUG_OUTPUT_DIR`

Set the BB_DEBUG_OUTPUT_DIR environment variable to enable file-based debugging. When set, DebugBBJsInstance wraps all bb.js calls and:

Writes all inputs (bytecode, witness, VK, proof, etc.) to numbered subdirectories under the debug dir
Writes a command.sh script with the equivalent bb CLI command that can be re-run manually
Writes outputs (proof files, verification results) after each operation

Example:

BB_DEBUG_OUTPUT_DIR=/tmp/bb-debug yarn workspace @aztec/prover-client test
# Then inspect /tmp/bb-debug/0001_prove_MyCircuit/, /tmp/bb-debug/0002_verify_MyCircuit/, etc.

Each subdirectory contains the input files and a command.sh that reproduces the operation using the bb CLI binary, useful for isolating failures outside of the TypeScript stack.

fcarreiro · 2026-03-14T11:55:00Z

Hey Charlie, this looks nice, just one comment for the AVM. We regularly use the following flow for all kinds of debugging, testing, benchmarking and iterating: we run some test (e.g., the avm bulk test). This test writes the avm inputs file and outputs the exact bb-avm command that you'd need to run. After that, we can just use that command (and that file) and operate 100% on the CLI/C++ side. We can then easily change the passed environment variables, concurrency, etc. IIUC, these changes would break that because the files wouldn't be written?

charlielye · 2026-03-14T14:11:25Z

Hey Charlie, this looks nice, just one comment for the AVM. We regularly use the following flow for all kinds of debugging, testing, benchmarking and iterating.

This should now be covered by setting a BB_DEBUG_OUTPUT_DIR before running.

fcarreiro

LGTM in general except for some changes in vm2/api. I can review again after that is solved!

fcarreiro · 2026-05-07T09:03:39Z

I think the additions to this file should instead be in BB's avm_api. In particular the vm2/api should not use or know of the result types defined in api_avm (all that is one level higher).

Pulled the stats reset/snapshot up into bbapi_avm.cpp (linked it against vm2_sim so it can hit Stats::get() directly).
The byte-based functions (avm_prove_from_bytes etc.) are still in vm2/api_avm.cpp, moveing those needs bbapi to be AVM-conditional since AvmAPI::prove only exists in AVM-on builds.

Ok I see. I just don't like that vm2/api_avm depends on types and files from a level up (and doesn't even define part of the implementation functions in its own hpp). It's not great but given all that this PR needs to achieve, it's probably ok and can be refactored later if needed.

If avm folks are ok with it I'm ok with it :)

Agree not ideal. Refactor in follow up PR will be best.

fcarreiro · 2026-05-07T09:08:27Z

-      this.logger.error(`Proof generation failed: ${proofRes.reason}`);
+    const { proof, verified, stats } = await this.bbJsFactory.withFreshInstance(i => i.generateAvmProof(inputsBuffer));
+    this.recordProverMetrics(stats, txLabel);
+    // bb.js also verifies right after proving; treat a failed inner verification as a prove failure.


Interesting. Is this a bb.js thing or is it because our vm2/api actually does prove and verify? We added that some time ago but we thought we might remove it at some point.

It was just mimicking the avm_prove call which does a verify. But it's redundant because all call sites do their own verify checks. I guess cli did it to provide non zero exit code. Have removed the redundent verify from the api call.

Yes I don't remember now why we added that verify. The thing is that prove never fails (unless you crash), so you always generate a proof, however it may not verify later. Right now I don't remember which code path needed it. Maybe git blame helps. I'll let @IlyasRidhuan decide

fcarreiro · 2026-05-07T09:12:30Z

+ * The CLI always uses `--scheme ultra_honk`; flavors are expressed via
+ * `--oracle_hash` and `--ipa_accumulation`.
+ */
+function getCliFlags(flavor: UltraHonkFlavor): string {


Do we have something that incentivizes people to keep this (and other CLI code) in sync if they change the CLI itself?

Ah nice, the test for this file executes the cli so that should be more than enough.

fcarreiro · 2026-05-07T12:44:32Z

Ok I see. I just don't like that vm2/api_avm depends on types and files from a level up (and doesn't even define part of the implementation functions in its own hpp). It's not great but given all that this PR needs to achieve, it's probably ok and can be refactored later if needed.

If avm folks are ok with it I'm ok with it :)

## Summary Migrates all `bb-prover` server-side proving and verification from spawning `bb`/`bb-avm` binaries via `child_process.spawn()` to using the bb.js TypeScript API (`Barretenberg` class via `NativeUnixSocket` backend). This eliminates temporary file I/O (writing/reading bytecode, witnesses, proofs, VKs to disk) and stdout-based result parsing. A fresh bb process is still spawned per proof to maintain parallelism, but communication happens via Unix socket instead of files. ### What was migrated | Operation | Before | After | |---|---|---| | UltraHonk proving | `bb prove` via spawn | `api.circuitProve()` via bb.js | | UltraHonk verification | `bb verify` via spawn | `api.circuitVerify()` via bb.js | | Chonk (IVC) verification | `bb verify --scheme chonk` via spawn + temp files | `api.chonkVerify()` via bb.js | | AVM proving | `bb-avm avm_prove` via spawn + temp files | `api.avmProve()` via bb.js | | AVM verification | `bb-avm avm_verify` via spawn + temp files | `api.avmVerify()` via bb.js | | AVM circuit check | `bb-avm avm_check_circuit` via spawn + temp files | `api.avmCheckCircuit()` via bb.js | | Gate counting | `bb gates` via spawn | `api.circuitStats()` / `api.chonkStats()` via bb.js | | Solidity contract gen | `bb contract` via spawn | `api.circuitWriteSolidityVerifier()` via bb.js | ### Key changes - **New `BBJsInstance` / `BBJsProverFactory`** (`bb_js_backend.ts`): Adapter layer wrapping the `Barretenberg` class with typed methods for all proof operations. - **New `DebugBBJsInstance`** (`bb_js_debug.ts`): Debug wrapper that writes inputs/outputs to disk and logs equivalent CLI commands (see debug section below). - **C++ AVM msgpack commands** (`bbapi_avm.hpp/cpp`): Three bbapi command structs (`AvmProve`, `AvmVerify`, `AvmCheckCircuit`) with in-memory function implementations in `vm2/api_avm.cpp` and stub implementations in `vm2_stub/api_avm.cpp`. - **AVM per-stage timings over the API**: New `AvmStat` serialized struct, plus a `stats: AvmStat[]` field on `AvmProve::Response` and `AvmCheckCircuit::Response`. `Stats::snapshot()` in `barretenberg/vm2/tooling` returns the per-stage map; `avm_prove_from_bytes` resets the singleton at entry and snapshots on exit. This replaces the legacy stdout-scraping used by `avm_proving_tester.ts` for per-stage timings, which now reads the `stats` array directly. - **Chonk proof splitting** (`splitChonkProofToStructured`): Mirrors C++ `ChonkProof::from_field_elements` — derives the `hidingOinkProof` length as the remainder after subtracting the four fixed-size sub-proofs (`MERGE_PROOF_SIZE`, `ECCVM_PROOF_LENGTH`, `IPA_PROOF_LENGTH`, `JOINT_PROOF_LENGTH`). Removes coupling to `HIDING_OINK_LENGTH` / `HIDING_KERNEL_IO_PUBLIC_INPUTS_SIZE` / `numCustomPublicInputs`, so changes in Mega witness entities can't silently misalign the TS split. - **Removed `bb-prover/src/bb/execute.ts`**: replaced by `file_names.ts` containing the three bb-CLI output filename constants (`VK_FILENAME`, `PROOF_FILENAME`, `PUBLIC_INPUTS_FILENAME`) that a few readers still need. The binary spawner (`executeBB`) is inlined into `ivc-integration/src/bb_js_debug.test.ts`, the only remaining consumer — a CLI-parity test that must spawn `bb` to compare its output against bb.js's. - **Test harnesses migrated to bb.js**: `avm_proving_tester.ts` (reads timings from the `stats` response field instead of scraping stdout) and `ivc-integration/src/prove_native.ts::proveAvm` (in-memory proof, no tempdir, no file I/O). ### Debug workflow with `BB_DEBUG_OUTPUT_DIR` Set the `BB_DEBUG_OUTPUT_DIR` environment variable to enable file-based debugging. When set, `DebugBBJsInstance` wraps all bb.js calls and: 1. Writes all inputs (bytecode, witness, VK, proof, etc.) to numbered subdirectories under the debug dir 2. Writes a `command.sh` script with the equivalent `bb` CLI command that can be re-run manually 3. Writes outputs (proof files, verification results) after each operation Example: ```bash BB_DEBUG_OUTPUT_DIR=/tmp/bb-debug yarn workspace @aztec/prover-client test # Then inspect /tmp/bb-debug/0001_prove_MyCircuit/, /tmp/bb-debug/0002_verify_MyCircuit/, etc. ``` Each subdirectory contains the input files and a `command.sh` that reproduces the operation using the `bb` CLI binary, useful for isolating failures outside of the TypeScript stack.

…artan (#23083) ## Motivation The `verifies transactions at 10 TPS` sub-test of [`yarn-project/end-to-end/src/bench/tx_stats_bench.test.ts`](https://github.com/AztecProtocol/aztec-packages/blob/merge-train/spartan/yarn-project/end-to-end/src/bench/tx_stats_bench.test.ts) is now reliably flaking on the `bench all` step of `merge-train/spartan`. It has fired on at least two different merge-train commits hours apart, with no relation to either commit's diff: | Run | Triggering merge-train commit | CI log | |---|---|---| | [25546251580](https://github.com/AztecProtocol/aztec-packages/actions/runs/25546251580) | #22934 (refactor(node-rpc)! removing deprecated AztecNode methods) | http://ci.aztec-labs.com/1778227975844707 | | [25552992890](https://github.com/AztecProtocol/aztec-packages/actions/runs/25552992890) | #22405 (feat(p2p): detect and track announce IP changes at runtime) | http://ci.aztec-labs.com/1778237470322975 | Both runs hit the same assertion: ``` ● transaction benchmarks › verifies transactions at 10 TPS expect(received).toBe(expected) // Object.is equality Expected: true Received: false at bench/tx_stats_bench.test.ts:268 ``` Sub-test failing log on the latest run: http://ci.aztec-labs.com/ca459ca73d02002c (`bench all` parent: http://ci.aztec-labs.com/90616bad7bf7ebaa). The other three sub-tests in the suite (compression; single private verify x20 serial; single public verify x20 serial) pass cleanly against the same proven txs in both runs. The failure is in the stress sub-test that fires 600 IVC verifications at 10/s with 8 concurrent IVC verifiers (`BB_NUM_IVC_VERIFIERS=8`, `BB_IVC_CONCURRENCY=1`). At least one verification returns `valid: false` under load. ## Cause Neither triggering commit touches the IVC verifier path: - #22934 is a pure node-rpc surface refactor. - #22405 is p2p / discv5 ENR plumbing. The two failures sharing this signature across unrelated diffs is strong evidence that the flake is independent of the merge-train commit and stems from the bench infrastructure itself. The likely culprit is the recent bb-prover migration to the bb.js `NativeUnixSocket` backend (#21564), which spawns a fresh bb subprocess per Chonk verification via `withVerifierInstance`. Under 8x parallel verifications on the CPU-isolated bench host (each verifier requesting 16 threads, 8 × 16 = 128 threads on 56 isolated cores), transient verifier failures appear. The bench-output log shows continuous `bb.js - Received signal 15, shutting down gracefully...` traffic during the 10 TPS phase — verifier instances are being torn down rapidly, and at least one verification slips through with a stale/incomplete response. Because the serial sub-tests (`numIterations = 20` sequential) pass cleanly in both runs, this is a stress-only interaction, not a correctness regression. ## Approach Add `tx_stats_bench` to `.test_patterns.yml` with an `error_regex` anchored to the test file's stack-trace line (`tx_stats_bench.test.ts:<line>:<col>`), and assign `*charlie` as owner (author of the bb.js migration). With this entry, `ci3/run_test_cmd` retries the test once on failure and treats a single retry-pass as a flake instead of a hard fail, unblocking the merge train for unrelated commits while Charlie investigates the underlying concurrency interaction with the bb.js backend. The `error_regex` is intentionally narrow (file + line + column from the stack trace) so other ways tx_stats_bench could fail (timeout, OOM, infra) are still surfaced as hard fails. ## Changes - `.test_patterns.yml`: add a `tx_stats_bench` entry with an error_regex anchored to the test file's stack-trace line and `*charlie` as owner. ClaudeBox logs: - https://claudebox.work/s/6e7853d3a073145f?run=1 (initial diagnosis on #22934 failure) - https://claudebox.work/s/c12a360275f05ad3?run=1 (this update on #22405 recurrence)

…ning per-call (#23093) ## Why Follow-up to #21564 (bb-prover bb.js migration) addressing the IVC verification perf regression that surfaced in `tx_stats_bench`. The migration kept the legacy spawn-per-verification model: every chonk/ultra-honk verification through `BBCircuitVerifier` spawned a fresh `bb` process and SIGTERMed it after one proof. `BB_NUM_IVC_VERIFIERS=8` only capped concurrency at the queue layer (`QueuedIVCVerifier`), not the number of bb processes. That made the bench spawn ~600 bb processes over its 60s 10 TPS phase inside an 8-CPU isolate. Two compounding problems: 1. ~50–100 ms of `bb` startup tax on every verification's hot path. 2. The bind→listen race in `NativeUnixSocket`: bb's socket file appears after `bind()` but before `listen()`. A TS `connect()` landing in that window gets `ECONNREFUSED`. Vanishingly rare under low load; reliable flake under contention. Diagnosis at http://ci.aztec-labs.com/735256f13a268733. ## What ### Make `BB_NUM_IVC_VERIFIERS` mean what its name says (commits aa99817, 0f4cb77) Pool of long-lived bb verifier processes instead of fresh-per-call. The factory class is renamed `BBJsProverFactory` → `BBJsFactory` (it's used for both proving and verifying) and given a single `getInstance(): Promise<BBJsApi & AsyncDisposable>` method: - `new BBJsFactory(path)` → no pool. Every `getInstance()` spawns a fresh bb that is destroyed on dispose. Same as the previous `withFreshInstance` behaviour — used by `BBNativeRollupProver`, the AVM proving tester, and ivc-integration helpers, so their semantics are unchanged. - `new BBJsFactory(path, { poolSize: N })` → pool of N long-lived bb processes, lazily spawned on first acquire. Used by `BBCircuitVerifier` with `poolSize: numConcurrentIVCVerifiers`. Callers use `await using inst = await factory.getInstance()` for RAII-style release, matching the codebase's preference for `AsyncDisposable`. `BBCircuitVerifier.stop` (already wired through to aztec-node shutdown) tears the pool down. ### Close the bind→listen race in bb.js (commit 8e519b0) `barretenberg/ts/src/bb_backends/node/native_socket.ts`: retry `connect()` on `ECONNREFUSED` with exponential backoff (capped at 50 ms) up to the existing 5 s budget. Other socket errors fail fast as before. Pool startup still spawns N bb processes in parallel, so the race surface is reduced from ~600 to N — the retry handles the residual. ### Server-side Chonk proof split (commit 97577cf) `splitChonkProofToStructured` in TS had three hand-maintained constants (`MERGE_PROOF_SIZE`, `ECCVM_PROOF_LENGTH`, `JOINT_PROOF_LENGTH`) duplicating C++ values. When C++ shifted Chonk layout (e.g. databus relation changes shrinking the oink portion in the previous round of regressions), these went stale and verification failed deep in the verifier with an opaque "OinkVerifier: num_public_inputs mismatch with VK". Add a new `ChonkVerifyFromFields` bbapi command that takes a flat `Vec<bb::fr>` and calls `ChonkProof::from_field_elements` server-side, then runs the verifier. The TS layer now passes flat fields straight through — no layout knowledge, no hand-maintained constants. - `bbapi_chonk.{hpp,cpp}`: new struct + `execute()`. - `bbapi_execute.hpp`: register the variant. - `bb_js_backend.ts`: `verifyChonkProof` calls the new API; `splitChonkProofToStructured` and the 3 constants are deleted. ### Disposal robustness (commit 5cde220) The first cut of `BBJsFactory` had three `.catch(() => {})` clauses that silently swallowed bb `destroy()` errors, and an `initPool()` that dropped already-spawned bb children if a sibling creation failed (`Promise.all` short-circuit). Both would manifest as the Jest "worker failed to exit gracefully" warning we hit on one test run. Now: destroy errors propagate (`AggregateError` for the pool path); `initPool` uses `allSettled` and tears down anything it spawned if any sibling rejects. ### Playground bundle size (commit 1681d33) The new `ChonkVerifyFromFields` bbapi variant tipped the playground main entrypoint over the 1750 KB hard limit. Bumped to 1800 with a bump-log entry. ## Effect - `tx_stats_bench`: 600 bb spawns → 8 bb spawns at boot, then 8 long-lived processes serve every verification. The bind→listen race surface drops 75×, *and* the residual is handled by the connect retry. Per-call ~50–100 ms `bb` startup cost disappears from the verifier hot path. - Brittle TS Chonk constants are gone — Chonk layout changes in C++ can no longer manifest as opaque verifier errors in TS. - Disposal failures surface instead of leaking bb children. - Behaviour for proving paths (`BBNativeRollupProver`, AVM tests, ivc-integration) is unchanged — they still spawn fresh per call. ClaudeBox log: https://claudebox.work/s/2d65052b0deaeab2?run=3 --------- Co-authored-by: Charlie <5764343+charlielye@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

charlielye requested review from IlyasRidhuan, LeilaWang, Maddiaa0, fcarreiro and jeanmon as code owners March 14, 2026 00:36

charlielye added the ci-full-no-test-cache label Mar 14, 2026

charlielye removed request for IlyasRidhuan, LeilaWang, Maddiaa0, fcarreiro and jeanmon March 16, 2026 12:54

charlielye force-pushed the cl/bb_js_api_migrate branch 3 times, most recently from 613e654 to 970f34c Compare March 23, 2026 04:37

charlielye mentioned this pull request Apr 24, 2026

fix(aztec-cli): release TXE port and kill bb children on shutdown #22730

Merged

4 tasks

charlielye requested a review from MirandaWood as a code owner April 29, 2026 11:08

charlielye requested review from fcarreiro and removed request for MirandaWood May 5, 2026 13:34

fcarreiro requested review from fcarreiro and removed request for fcarreiro May 6, 2026 08:00

fcarreiro reviewed May 7, 2026

View reviewed changes

fcarreiro approved these changes May 7, 2026

View reviewed changes

charlielye added ci-squash-and-merge and removed ci-full-no-test-cache labels May 7, 2026

charlielye enabled auto-merge May 7, 2026 16:40

charlielye disabled auto-merge May 7, 2026 16:40

charlielye enabled auto-merge May 7, 2026 16:40

AztecBot removed the ci-squash-and-merge label May 7, 2026

AztecBot force-pushed the cl/bb_js_api_migrate branch from 0407d1a to ea1c9b6 Compare May 7, 2026 17:00

charlielye added this pull request to the merge queue May 7, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 7, 2026

charlielye added this pull request to the merge queue May 7, 2026

Merged via the queue into next with commit 480c552 May 7, 2026
23 checks passed

charlielye deleted the cl/bb_js_api_migrate branch May 7, 2026 18:44

AztecBot mentioned this pull request May 8, 2026

test: mark tx_stats_bench 10 TPS as flake-retryable on merge-train/spartan #23083

Merged

This was referenced May 8, 2026

test: skip tx_stats_bench 10 TPS sub-test on merge-train/spartan #23092

Closed

fix(bb-prover): pool long-lived bb verifier processes instead of spawning per-call #23093

Merged

vezenovm mentioned this pull request May 8, 2026

test: skip tx_stats_bench 10 TPS sub-test on merge-train/fairies #23101

Closed

Conversation

charlielye commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What was migrated

Key changes

Debug workflow with BB_DEBUG_OUTPUT_DIR

Uh oh!

fcarreiro commented Mar 14, 2026

Uh oh!

charlielye commented Mar 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fcarreiro left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charlielye commented Mar 14, 2026 •

edited

Loading

Debug workflow with `BB_DEBUG_OUTPUT_DIR`

charlielye commented Mar 14, 2026 •

edited

Loading