Skip to content

README-dev.md

Latest commit

 

History

History
540 lines (401 loc) · 21.3 KB

README-dev.md

File metadata and controls

540 lines (401 loc) · 21.3 KB

o1js README-dev

o1js is a TypeScript framework designed for zk-SNARKs and zkApps on the Mina blockchain.

For more information on our development process and how to contribute, see CONTRIBUTING.md. This document is meant to guide you through building o1js from source and understanding the development workflow.

It is a manual step to build the o1js Reference docs. See o1js Reference in the docs wiki style guide.

Prerequisites

Before starting, ensure you have the following tools installed:

After cloning the repository, you need to fetch the submodules:

git submodule update --init --recursive

Building o1js

For most users, building o1js is as simple as running:

npm install
gh auth login #gh is used to download the compiled artifacts
npm run build

This command downloads the artifacts from github if they are missing and compiles the TypeScript source files, making them ready for use. The compiled OCaml and WebAssembly artifacts are cached for each commit where CI is run. These artifacts are stored under src/bindings/compiled and src/bindings/mina-transaction/gen and contain the artifacts needed for both nodejs and web builds. These files only have to be regenerated if there are changes to the OCaml or Rust source files.

External contributors

Unfortunately you generally won't be able to run npm run build:bindings-download on your own commits because the artifacts won't have been built for them, so make sure to run it on main before you start making changes. In a fresh git repo npm run build also works.

Keep in mind that merging a newer version of o1js may include OCaml and Rust changes so you may need to redownload the artifacts. When this happens, as long as you aren't making changes to the OCaml and Rust yourself, you can run REV=<commit you just merged> npm run build:bindings-download, this will download the bindings for the commit you merged which should be the same as the ones you need.

Internal contributors

If you have an open PR npm run build:bindings-download should work on any commit where CI has run or is running. If CI is still running it uses gh run watch to show you the progress.

If your PR has a merge conflict, in which case CI will not run on each new commit, or you just don't have a PR you may can run npm run build:bindings-remote. This will trigger our self-hosted runner to build the bindings for your commit and download them once it finishes.

Building with nix

Much like the Mina repo, we use the nix registry to conveniently handle git submodules. You can enter the devshell with ./pin.sh and nix develop o1js#default or by using direnv with the .envrc provided. This devshell provides all the dependencies required for npm scripts including npm run build:bindings-all.

Building Bindings

To regenerate the OCaml and WebAssembly artifacts, you can do so within the o1js repo. The Mina repo is a submodule of o1js, and proof-systems is a submodule of Mina, so you can build them recursively from within the o1js repo.

o1js depends on OCaml code that is transpiled to JavaScript using Js_of_ocaml and Rust code that is transpiled to WebAssembly using wasm-pack. These artifacts allow o1js to call into Pickles, snarky, and Kimchi to write zk-SNARKs and zkApps.

The compiled artifacts are stored under src/bindings/compiled and are version-controlled to simplify the build process for end-users.

If you want to rebuild the OCaml and Rust artifacts, you must be able to build the mina repo before building the bindings. See the Mina Dev Readme for more information. After you have configured your environment to build mina, you can build the bindings:

npm run build:bindings-all

This command builds the OCaml and Rust Node & Web artifacts and copies them to the src/bindings/compiled directory.

You can also build only the OCaml and Rust Node bindings for local development by running:

npm run build:bindings-node

Build Scripts

The root build script which kicks off the build process is under src/bindings/scripts/update-o1js-bindings.sh. This script is responsible for building the Node.js and Web artifacts for o1js, and places them under src/bindings/compiled, to be used by o1js.

However, when first setting up your o1js development environment, simply run npm run:

npm run build

This command generates the Node bindings and compiles the project to JavaScript, allowing you to run both the existing examples and any new files as described here.

OCaml Bindings

o1js depends on Pickles, snarky, and parts of the Mina transaction logic, all of which are compiled to JavaScript and stored as artifacts to be used by o1js natively. The OCaml bindings are located under src/bindings. See the OCaml Bindings README for more information.

To compile the OCaml code, a build tool called Dune is used. Dune is a build system for OCaml projects, and is used in addition with Js_of_ocaml (JSOO) to compile the OCaml code to JavaScript. The dune file that is responsible for compiling the OCaml code is located under src/bindings/ocaml/dune. There are two build targets: o1js_node and o1js_web, which compile the Mina dependencies as well as link the wasm artifacts to build the Node.js and Web artifacts, respectively. The output file is o1js_node.bc.js, which is used by o1js.

For internal development and debugging, you can manually build the JSOO bindings for Node and Web using npm run build:jsoo:node and npm run build:jsoo:web, respectively. However, for typical local development, running the standard build commands automatically generates these bindings as part of the overall build process.

WebAssembly Bindings

o1js additionally depends on Kimchi, which is compiled to WebAssembly. Kimchi is located in the Mina repo under src/mina. See the Kimchi README for more information.

To compile the Wasm code, a combination of Cargo and Dune is used. Both build files are located under src/mina/src/lib/crypto/kimchi, where the wasm folder contains the Rust code that is compiled to Wasm, and the js folder that contains a wrapper around the Wasm code which allows Js_of_ocaml to compile against the Wasm backend.

For the Wasm build, the output files are:

  • plonk_wasm_bg.wasm: The compiled WebAssembly binary.
  • plonk_wasm_bg.wasm.d.ts: TypeScript definition files describing the types of .wasm or .js files.
  • plonk_wasm.js: JavaScript file that wraps the Wasm code for use in Node.js.
  • plonk_wasm.d.ts: TypeScript definition file for plonk_wasm.js.

Similarly, for internal development and debugging, you can manually build the WASM bindings for Node and Web using npm run build:wasm:node and npm run build:wasm:web, respectively. For typical local development, however, running the standard build commands automatically generates these bindings as part of the overall build process.

Generated Constant Types

In addition to building the OCaml and Rust code, the build script also generates TypeScript types for constants used in the Mina protocol. These types are generated from the OCaml source files, and are located under src/bindings/crypto/constants.ts and src/bindings/mina-transaction/gen. When building the bindings, these constants are auto-generated by Dune. If you wish to add a new constant, you can edit the src/bindings/ocaml/o1js_constants file, and then run npm run build:bindings-node to regenerate the TypeScript files.

o1js uses these types to ensure that the constants used in the protocol are consistent with the OCaml source files.

This process is already performed as part of the bindings build, but you can regenerate the constant types separately by running:

npm run build:bindings-transaction-layout

Bindings check in CI

If the bindings check fails in CI it will upload a patch you can use to update the bindings without having to rebuild locally. This can also be helpful when the bindings don't build identically, as unfortunately often happens.

To use this patch:

  • Click details on the Build o1js bindings / build-bindings-ubuntu job
  • Go to the patch-upload job and expand the logs for Upload patch
  • Download the file linked in the last line of the logs ie. Artifact download URL: https://github.com/o1-labs/o1js/actions/runs/12401083741/artifacts/2339952965
  • Unzip it
  • Navigate to src/bindings
  • Run git apply path/to/bindings.patch

Development

Branching Policy

o1js base branches Is default?
main Yes
develop No

When you start your work on o1js, please create the feature branch off of one of the above base branches. It's encouraged to submit your work-in-progress as a draft PR to raise visibility! When working with submodules and various interconnected parts of the stack, ensure you are on the correct branches that are compatible with each other.

Default to main as the base branch.

Other base branches (currently develop only) are used in specific scenarios where you want to adapt o1js to changes in the sibling repos on those other branches. Even then, consider whether it is feasible to land your changes to main and merge to develop afterwards. Only changes in main will ever be released, so anything in other branches has to be backported and reconciled with the main branch eventually.

Relationship Between Repositories and Branches

Repository o1js → o1js-bindings → mina
Branches main main compatible
develop develop develop

Where:

  • compatible: This is the Mina repository branch. It corresponds to the main branch in both o1js and o1js-bindings repositories. This branch is where stable releases and soft-fork features are maintained.

  • develop: This branch is maintained across all three repositories. It is used for ongoing (next hard-fork) development, testing new features and integration work.

Style Guide

This repo uses minimal oxlint and prettier configs to stay tidy. Here are some tips to comply with the style:

  1. Check for style violations by running the npm commands npm run lint path/to/file and npm run format:check path/to/file
  • To attempt to fix all style violations in all changed files, you can run:
    • git diff --cached --name-only --diff-filter=d | grep -E '\.(ts|js)$' | xargs npm run format
    • and git diff --cached --name-only --diff-filter=d | grep -E '\.(ts|js)$' | xargs npm run lint:fix
  1. Integrate prettier into your dev environment
  • For instance the VS Code plugin allows for format on save
  1. Enable pre-commit hooks
  • There is an opt-in pre-commit hook available that will attempt to fix styling for all diffed files. Enable it by running git config husky.optin true

[!NOTE] You can opt-out of linting in a PR by tagging it with skip-lint, in case the linting script is legitimately blocking an important PR.

Build Guide

To build the o1js project, simply run:

npm run build

This command builds the Node bindings and compiles the project’s TypeScript sources into JavaScript.

If you want to build o1js for both Node and Web, run:

npm run build:bindings-all

This script builds the bindings for both environments and compiles the project to JavaScript for Node and Web as well.

These are the two main build scripts.
For more fine-grained local development and debugging, you can use the following targeted scripts.

TypeScript Build Scripts

Command Description
npm run build:dev Builds the TypeScript source files and outputs the compiled JavaScript to the dist/ directory.
npm run build Builds the full project for Node.js, ensuring all bindings are present and compiling the code to JavaScript.
npm run build:web Builds the project for Web (browser) usage, only generating a browser-compatible bundle.
npm run build:examples Compiles the example files located under src/examples/.

Bindings Build Scripts

Command Description
npm run build:bindings-node Builds the OCaml and Rust Node bindings used by o1js.
npm run build:bindings-transaction-layout Regenerates the TypeScript constant and transaction layout definitions from OCaml sources.
npm run build:wasm Builds all WebAssembly (WASM) bindings for both Node and Web.
npm run build:wasm:node Builds the Kimchi WASM bindings for Node.js.
npm run build:wasm:web Builds and optimizes the Kimchi WASM bindings for Web browsers.
npm run build:jsoo Builds all Js_of_OCaml (JSOO) bindings for both Node and Web.
npm run build:jsoo:node Builds the Js_of_OCaml Node bindings.
npm run build:jsoo:web Builds the Js_of_OCaml Web bindings.

Additional Utilities:

Command Description
npm run check:bindings Checks for existing prebuilt bindings (downloads them if available).
npm run build:bindings-download Downloads prebuilt bindings directly from the CI artifacts.
npm run build:bindings-remote Builds bindings remotely or in a containerized CI environment.

Clean Scripts

You can also clean generated artifacts using the following scripts:

Command Description
npm run clean Removes build outputs such as dist/ and Node bindings artifacts.
npm run clean:all Removes Node build artifacts generated by the o1js build process as well as test-related output files.
npm run clean:artifacts Removes compiled bindings and intermediate artifacts from OCaml, Kimchi, and proof-systems builds.

Running Tests

To ensure your changes don't break existing functionality, run the test suite:

npm run test
npm run test:unit

In order for the mina-signer tests to run you must also build from inside its subdirectory:

cd src/mina-signer
npm run build
cd ../..

This runs all the unit tests and provides you with a summary of the test results.

Note that you can run individual jest tests via the command:

./jest <path/to/test.ts>

You can also run integration tests by running:

npm run test:integration

Finally, a set of end-to-end tests are run against the browser. These tests are not run by default, but you can run them by running:

npm install
npm run e2e:install
npm run build:web

npm run e2e:prepare-server
npm run test:e2e
npm run e2e:show-report

Run the GitHub actions locally

You can execute the CI locally by using act. First, generate a GitHub token and use:

act -j Build-And-Test-Server --matrix test_type:"Simple integration tests" -s $GITHUB_TOKEN

Releasing

To release a new version of o1js, you must first update the version number in package.json. Then, you can create a new pull request to merge your changes into the main branch. After the pull request is merged, a CI job automatically publishes the new version to npm.

Testing and Debugging

Test zkApps against Lightnet network

Use the lightweight Mina blockchain network (Lightnet) to test on a local blockchain before you test with a live network. To test zkApps against the local blockchain, first spin up Lightnet.

The easiest way is to use zkApp CLI sub-commands:

zk lightnet start # start the local network
# Do your tests and other interactions with the network
zk lightnet logs # manage the logs of the local network
zk lightnet explorer # visualize the local network state
zk lightnet stop # stop the local network

Use zk lightnet --help for more information.

You can also use the corresponding Docker image manually:

docker run --rm --pull=missing -it \
  --env NETWORK_TYPE="single-node" \
  --env PROOF_LEVEL="none" \
  --env LOG_LEVEL="Trace" \
  -p 3085:3085 \
  -p 5432:5432 \
  -p 8080:8080 \
  -p 8181:8181 \
  -p 8282:8282 \
  o1labs/mina-local-network:compatible-latest-lightnet

See the Docker Hub repository for more information.

Next up, get the Mina blockchain accounts information to be used in your zkApp. After the local network is up and running, you can use the Lightnet o1js API namespace to get the accounts information. See the corresponding example in src/examples/zkapps/hello-world/run-live.ts.

Profiling o1js

To enhance the development experience and optimize the performance of o1js, use the Chrome Debugger alongside Node.js. This setup is particularly useful when you want to profile the performance of your zkApp or o1js.

Using the run-debug script

To facilitate this process, use the provided script named run-debug. To use this script, run:

./run-debug <path-to-your-zkapp> --bundle

This script initializes a Node.js process with the --inspect-brk flag that starts the Node.js inspector and breaks before the user script starts (i.e., it pauses execution until a debugger is attached). The --enable-source-maps flag ensures that source maps are used to allow easy debugging of o1js code directly.

After the Node.js process is running, open the Chrome browser and navigate to chrome://inspect to attach the Chrome Debugger to the Node.js process. You can set breakpoints, inspect variables, and profile the performance of your zkApp or o1js. For more information on using the Chrome Debugger, see the DevTools documentation.

Debugging within the SDK

To debug a call into the SDK, you can link your local copy of the SDK with npm link. After that, you'll be able to add log statements, set breakpoints, and make code changes. Within the SDK, run:

npm run link

Then in your zkApp codebase, run:

npm link o1js

Logging from OCaml

If you need to debug a call into the OCaml code, the process is a little more complicated. The OCaml is compiled into JavaScript with js_of_ocaml during npm run build:bindings-all, so you'll need to add your logs into the OCaml code and rebuild the bindings to see them. Logging from OCaml in a way that will reflect as JS console.logs in the compiled code can be done like this:

let () = print_endline "This is a log" in