don't control concurrency from INSIDE_DUNE by Alizter · Pull Request #12800 · ocaml/dune

Alizter · 2025-11-27T11:18:03Z

INSIDE_DUNE sets the concurrency of dune to 1. We use this behaviour when running our tests. However, running dune inside of dune, as is common in package management, means that the child process also observes INSIDE_DUNE and therefore affects the concurrency when we haven't explicitly passed -j, as is the case for OxCaml:

Nesting of INSIDE_DUNE and build concurrency #12737

This PR introduces a new environment variable DUNE_JOBS whose sole job is to configure the concurrency from the environment.

Our second change is to decouple the concurrency control from INSIDE_DUNE. This means that INSIDE_DUNE no longer sets the concurrency to 1. This means that we have to explicitly set DUNE_JOBS in our cram tests. We don't however set the value for the expect tests.

Setting DUNE_JOBS for our cram tests has an unfortunate side-effect, and is was the cause of most of the noise in this PR. It triggers a warning message given by an RPC client when a dune's config differs from the default values. I've therefore disabled this warning when INSIDE_DUNE since it doesn't have any utility for us. Also, depending on the pid contents of the build lock in a racy way is not sound.

update docs
- mention new env var in cli docs and man pages
- document concurrency properly
changelog
Find a way to silence or perhaps get rid of the warning. (It's not a very good warning to begin with)

Alizter · 2025-11-28T09:14:58Z

FTR oxcaml still won't build with the correct concurrency with this patch because they are specifically vendoring Dune 3.20 and using that to build. We therefore have to patch their opam build to not do that. If we are going to do that anyway, then we might as well unset INSIDE_DUNE. @Sudha247 may have some updates on this.

Alizter · 2025-11-28T09:28:04Z

Another issue with this PR is that comparing against Common.Builder.default value is used as a heuristic for displaying the RPC forwarding message. This means that INSIDE_DUNE no longer sets this as 1 and every test differs from the default of Auto. Therefore triggering the message in each RPC forwarding situation.

Leonidas-from-XIV · 2025-11-28T10:47:25Z

I do wonder why the CI doesn't want to run the jobs that run the test-suite…

Alizter · 2025-11-28T10:57:33Z

@Leonidas-from-XIV The nix tests are a dependency of the rest of the tests. That way we can catch errors early and not bother using up all the allocated machines. The fact that the environment is Nix doesn't mean much here, its just the regular test suite.

Sudha247 · 2025-12-02T12:38:40Z

Yes, I can confirm that this doesn't help the OxCaml build by itself. I believe we will need to update the Dune version used to bootstrap the compiler to actually see the difference.

test/blackbox-tests/test-cases/watching/watching-eager-concurrent-runtest-command.t

test/blackbox-tests/test-cases/watching/watching-eager-concurrent-exec-command.t

Sudha247 · 2025-12-09T17:11:30Z

This looks reasonable to me. I'll run tests with this branch on the build of the OxCaml compiler and report back.

Sudha247 · 2025-12-10T10:49:39Z

I've run some tests for this with OxCaml compiler build and it does appear to enable using concurrency with nested Dune calls which it previously wasn't.

To reproduce:

Install Dune from this branch.

opam pin add dune git+https://github.com/Alizter/dune.git#push-ysnonzkmwzym

Use this branch Patched OPAM repository in dune-workspace: https://github.com/Sudha247/oxcaml-opam-repository/blob/inside-dune-test/packages/ocaml-variants/ocaml-variants.5.2.0%2Box/opam

(repository
 (name oxcaml)
 (url git+file:///home/user/path/to/oxcaml-opam-repository))

(lock_dir
 (repositories :standard oxcaml))

Build the project with dune build, I built hello-oxcaml. This will trigger a compiler build.

To compare some numbers on time taken to build hello-oxcaml:

Build	Time
Upstream `dune pkg`	32:00.89
`opam` switch creation	14:21.62
`dune pkg` with this patch	16:57.90
Manually unsetting INSIDE_DUNE	14:04.00

Given the performance of the patch, I believe it does enable concurrent builds in the case of nested Dune. I think we can go ahead and merge this PR, provided everything else is ok.

doc/changes/added/12800.md

Signed-off-by: Ali Caglayan <alizter@gmail.com>

@Alizter

CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the sandbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)

@Alizter

CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Place the default trace file inside the build directory at the workspace root, rather than relative to the current directory. (ocaml/dune#13735, @vouillon) - Fixed interpreting relative paths in `%{bin:..}` and `%{bin-available:..}`. These are now interpreted correctly, relative to the dune file they're in. (ocaml/dune#13712, fixes ocaml/dune#9564, @anmonteiro) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix the Dune cache on Windows by correctly handling renames onto read-only files. Before this change, the Dune cache would be filled but the stored artifacts would not generally be usable by Dune. (ocaml/dune#13713, @Nevor) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Fix package extraction on systems with tar implementations that don't auto-detect compression (e.g., OpenBSD). Dune now passes explicit decompression flags (-z for gzip, -j for bzip2) when needed, and provides clear error messages for unsupported formats like XZ and LZMA. (ocaml/dune#13688, fixes ocaml/dune#10123, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Dune cache: use of hard links under Windows. (ocaml/dune#13714, @Nevor) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the asndbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)

@Alizter

CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Place the default trace file inside the build directory at the workspace root, rather than relative to the current directory. (ocaml/dune#13735, @vouillon) - Fixed interpreting relative paths in `%{bin:..}` and `%{bin-available:..}`. These are now interpreted correctly, relative to the dune file they're in. (ocaml/dune#13712, fixes ocaml/dune#9564, @anmonteiro) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix the Dune cache on Windows by correctly handling renames onto read-only files. Before this change, the Dune cache would be filled but the stored artifacts would not generally be usable by Dune. (ocaml/dune#13713, @Nevor) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Fix package extraction on systems with tar implementations that don't auto-detect compression (e.g., OpenBSD). Dune now passes explicit decompression flags (-z for gzip, -j for bzip2) when needed, and provides clear error messages for unsupported formats like XZ and LZMA. (ocaml/dune#13688, fixes ocaml/dune#10123, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Dune cache: use of hard links under Windows. (ocaml/dune#13714, @Nevor) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the asndbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)

@Alizter

CHANGES: ### Fixed - `Dyn.to_string` now uses a smarter way to convert floats. This ensures that floats are printed with enough precision to round-trip and are valid OCaml lexemes. (ocaml/dune#12982, fixes ocaml/dune#12980, @Alizter) - Fix `dune install --prefix` failing with relative paths outside the workspace like `../foo` (ocaml/dune#12993, fixes ocaml/dune#12241, @benodiwal) - Place the default trace file inside the build directory at the workspace root, rather than relative to the current directory. (ocaml/dune#13735, @vouillon) - Fixed interpreting relative paths in `%{bin:..}` and `%{bin-available:..}`. These are now interpreted correctly, relative to the dune file they're in. (ocaml/dune#13712, fixes ocaml/dune#9564, @anmonteiro) - Delete sandboxes with broken permissions (ocaml/dune#13511, @rgrinberg) - Fix compiling Menhir parsers that refer to sibling modules within a subdirectory of `(include_subdirs qualified)`. (ocaml/dune#13118, fixes ocaml/dune#11119, @anmonteiro) - Fixed the dependency specification of C stubs, which could result in C stubs not getting rebuilt when needed (which could in turn lead to segmentation faults and other hard-to-track bugs). (ocaml/dune#13652, fixes ocaml/dune#13651, @nojb) - Fix the Dune cache on Windows by correctly handling renames onto read-only files. Before this change, the Dune cache would be filled but the stored artifacts would not generally be usable by Dune. (ocaml/dune#13713, @Nevor) - Fix rpc not transferring promotion warnings to the client (ocaml/dune#12604, fixes ocaml/dune#12578, @ElectreAAS) - Fix issue where `dune exec -w` was unable to kill running programs on rebuild. (ocaml/dune#12360, fixes ocaml/dune#12323, @Alizter) - Resolve context and workspace binaries introduced by the respective `(env (binaries ..))` stanzas. (ocaml/dune#12952, fixes ocaml/dune#6220, @anmonteiro) - Fix `diff` promotions originating from sandboxed rules. Previously, they would be completely ignored as the sandbox with the promoted file would be destroyed if the promotion fired (ocaml/dune#13520, @rgrinberg) - Fix failure to digest installed directory targets, allowing them to be used as dependencies to other rules. (ocaml/dune#13045, @anmonteiro) - Fix handling of `(select ..)` field when used with `(include_subdirs ..)`. `(select <path> from ..)` modules now parse `path` as a relative path starting from the module group root (ocaml/dune#13175, fixes ocaml/dune#4383, ocaml/dune#12450, @anmonteiro) - Fix dune trying to kill processes that were already reaped due to race conditions (ocaml/dune#13245, @rgrinberg) - Add `O_CLOEXEC` to all files used for stdin/stdout/stderr (ocaml/dune#13385, @rgrinberg) - Fix `$ dune promote dir/foo` when `dir` does not exist (ocaml/dune#13493, @rgrinberg) - Fix `(select ..)` field evaluation when a transitive library has optional dependencies (fixes ocaml/dune#13299, ocaml/dune#13389, @anmonteiro) - Fix sandboxed builds of `library` stanzas that set `(stdlib (modules_before_stdlib ..))` (ocaml/dune#13624, @anmonteiro) - Dune cache: use of hard links under Windows. (ocaml/dune#13714, @Nevor) - Fixed non-build caches not following `$DUNE_CACHE_ROOT` and instead only relying on `$XDG_CACHE_HOME`. This means the normal build cache moves: `$DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db` (no changes if that variable was unset). Affected users can prevent a full cache invalidation by moving previous contents: `cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db`. (ocaml/dune#11612, fixes ocaml/dune#11584, @ElectreAAS) - `$ dune promotion list` writes output to stdout rather than stderr (ocaml/dune#13462) - Improve handling of empty files in the `diff` action. These are now correctly distinguished from *empty* files. (ocaml/dune#13696, @rgrinberg) - Pass `/dev/null` to `--diff-command` instead of non-existent files (ocaml/dune#13696, @rgrinberg) - Fix failure when multiple `rocq.extraction` stanzas existing in a directory (ocaml/dune#13531, fixes ocaml/dune#8042, @rlepigre-skylabs-ai) - Print `$ dune promotion show` output to stdout rather than stderr (ocaml/dune#13481, @rgrinberg) - Fix deadlock in the `memo` library in the presence of dependency cycles (ocaml/dune#13625, @anmonteiro) - Fix promotions that modify a directory into a file (ocaml/dune#13516, fixes ocaml/dune#4067, @rgrinberg) - Fix installation of implementations of virtual libraries. This failed when the implementation had no private modules, but the virtual library did (ocaml/dune#10635, @rgrinberg) - Respect the `(dir ..)` field on packages when setting up cram tests (ocaml/dune#13581, @rgrinberg) ### Added - Add support for generating `.cms` files using oxcaml and adding `.cms` or `.cmt` files as compilation dependencies (ocaml/dune#13397, @spiessimon) - Add trace events for custom actions (ocaml/dune#13265, @rgrinberg) - Allow enabling extensions with `(using ..)` in `dune-workspace` files (ocaml/dune#13395, @spiessimon) - Add sandbox extraction trace event (ocaml/dune#13544, @rgrinberg) - Add the initial cwd to the first config event (ocaml/dune#13026, @rgrinberg) - Dune dune produces trace events in `DUNE_ACTION_TRACE_DIR` if this variable is set. (ocaml/dune#13302, @rgrinberg) - Add file watching events to the trace file (ocaml/dune#13038, @rgrinberg) - Introduce the `$ dune trace cat` subcommand to view the trace file. (ocaml/dune#13055, @rgrinberg) - Add diagnostic events to the trace. (ocaml/dune#13041, @rgrinberg) - Add `DUNE_JOBS` environment variable for controlling concurrency of Dune from environment. The `INSIDE_DUNE` variable also now no longer controls concurrency (ocaml/dune#12800, @Alizter) - Support for Rocq expected output tests (ocaml/dune#13632, @rlepigre-skylabs-ai) - Add `rusage` information to completed processes in the trace (@rgrinberg, ocaml/dune#13241) - Add process start events to the trace (ocaml/dune#13261, rgrinberg) - Generate odoc documentation in markdown using the `@doc-markdown` alias (ocaml/dune#12581, @davesnx) - Add timing information for every command executed by cram (ocaml/dune#13092, @rgrinberg) - Add the workspace root to the config trace event (ocaml/dune#12922, @rgrinberg) - Introduce the `dune-action-trace` library. This public library is to be used by custom actions to emit trace events while executed as part of a dune build. The trace events emitted through this library will be incorporated into dune's own trace (ocaml/dune#13348, @rgrinberg) - Add `dune-find-dominating` to `dune.el`, a command to find the dominating dune file. (ocaml/dune#12696, @arvidj) - Add a `--no-recursive` flag to `$ dune describe workspace` (ocaml/dune#13590, @rgrinberg) - Trace events for files written directly by dune (ocaml/dune#13618, @rgrinberg) - Allow expansion of special forms like `(:include ..)` and `%{read-lines:..}` in the `modules` specification for the `ocamllex`, `ocamlyacc` and `menhir` stanzas. (ocaml/dune#13105, ocaml/dune#13135, ocaml/dune#13157, @anmonteiro) - Add a trace event for snapshotting the asndbox (ocaml/dune#13541, @rgrinberg) - Add signal send and receive events to the trace (ocaml/dune#13193, @rgrinberg) - Emit final trace event before exiting. (ocaml/dune#13018, @rgrinberg) - `dune runtest` can now run individual test executables from `(tests)` stanzas and inline tests from `(library (inline_tests))` stanzas by providing their source files as arguments. (ocaml/dune#13064, fixes ocaml/dune#870, @Alizter) - Add a `shell` field to the cram stanza. This field allows customizing the shell to be `bash` rather than `sh` (ocaml/dune#13083, @haochenx) ### Changed - Start sandboxing the execution of tests defined with the `test` and `tests` stanzas (ocaml/dune#13510, ocaml/dune#13617, @rgrinberg) - Disabled cram tests can now be run explicitly with `dune runtest disabled.t`. The `enabled_if` field now only controls whether a test is included in the `@runtest` alias. (ocaml/dune#13081, @Alizter) - Process categories in trace events are moved to their own field in `args` (ocaml/dune#13024, @rgrinberg) - Sandbox running `ocamllex` and `ocamlyacc` actions. (ocaml/dune#13098, @anmonteiro) - Sandboxing mdx test actions is now the default starting from `0.5` (ocaml/dune#13504, @rgrinberg) - Start sandboxing Melange rules by default in the `(library ..)` and `(melange.emit ..)` stanzas (ocaml/dune#13619, @anmonteiro) - Introduce a promotion trace event and remove the corresponding verbose log message. (ocaml/dune#12949, ocaml/dune#13444, @rgrinberg) - Change dune's trace format to emit canonical s-expressions. This improves performance and is better aligned with dune's usage of the format elsewhere. `$ dune trace cat` can also emit the trace in `--chrome-trace` for perfetto, or `--sexp` for regular s-expressions for interactive usage. (ocaml/dune#13059, @rgrinberg) - Move all logging statements to the trace file. All log statements now contain structured payloads (ocaml/dune#13015, fixes ocaml/dune#12904, @rgrinberg) - Add a target resolution event to replace the equivalent log message (ocaml/dune#12955, @rgrinberg)

Leonidas-from-XIV force-pushed the push-ysnonzkmwzym branch from 0cb608f to 8822feb Compare November 28, 2025 09:11

Sudha247 mentioned this pull request Dec 2, 2025

Make compiler build with Dune package management faster oxcaml/opam-repository#25

Merged

Sudha247 mentioned this pull request Dec 2, 2025

Nesting of INSIDE_DUNE and build concurrency #12737

Closed

Alizter force-pushed the push-ysnonzkmwzym branch from 8822feb to e403ac6 Compare December 9, 2025 15:20

Alizter changed the title ~~WIP don't control concurrency from INSIDE_DUNE~~ don't control concurrency from INSIDE_DUNE Dec 9, 2025

Alizter mentioned this pull request Dec 9, 2025

3.21.0 release tracker #12619

Closed

43 tasks

Alizter force-pushed the push-ysnonzkmwzym branch 2 times, most recently from cf28308 to 20d5d73 Compare December 9, 2025 16:37

Alizter marked this pull request as ready for review December 9, 2025 16:37

Alizter requested review from Sudha247 and rgrinberg December 9, 2025 16:38

Alizter commented Dec 9, 2025

View reviewed changes

test/blackbox-tests/test-cases/watching/watching-eager-concurrent-runtest-command.t Outdated Show resolved Hide resolved

Alizter commented Dec 9, 2025

View reviewed changes

test/blackbox-tests/test-cases/watching/watching-eager-concurrent-exec-command.t Outdated Show resolved Hide resolved

Alizter force-pushed the push-ysnonzkmwzym branch from 20d5d73 to ed3cc8e Compare December 10, 2025 11:14

rgrinberg reviewed Dec 10, 2025

View reviewed changes

doc/changes/added/12800.md Outdated Show resolved Hide resolved

Alizter force-pushed the push-ysnonzkmwzym branch from ed3cc8e to 36e02a4 Compare December 10, 2025 11:23

feat: introduce DUNE_JOBS environment variable

335c387

Signed-off-by: Ali Caglayan <alizter@gmail.com>

Alizter marked this pull request as draft December 10, 2025 11:29

Alizter marked this pull request as ready for review December 10, 2025 11:31

Alizter force-pushed the push-ysnonzkmwzym branch from 36e02a4 to 610ba7f Compare December 10, 2025 11:32

don't control concurrency from INSIDE_DUNE

07a66be

Signed-off-by: Ali Caglayan <alizter@gmail.com>

Alizter force-pushed the push-ysnonzkmwzym branch from 610ba7f to 07a66be Compare December 10, 2025 11:33

Alizter requested a review from rgrinberg December 10, 2025 11:52

rgrinberg approved these changes Dec 11, 2025

View reviewed changes

Alizter merged commit ea10cbe into ocaml:main Dec 12, 2025
29 checks passed

Alizter deleted the push-ysnonzkmwzym branch December 12, 2025 08:10

shonfeder mentioned this pull request Mar 2, 2026

[new release] dune (18 packages) (3.22.0~alpha0) ocaml/opam-repository#29497

Closed

shonfeder mentioned this pull request Mar 11, 2026

[new release] dune (18 packages) (3.22.0~alpha1) ocaml/opam-repository#29535

Closed

shonfeder mentioned this pull request Mar 16, 2026

[new release] dune (18 packages) (3.22.0~alpha2) ocaml/opam-repository#29547

Closed

shonfeder mentioned this pull request Mar 19, 2026

[new release] dune (18 packages) (3.22.0) ocaml/opam-repository#29562

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

don't control concurrency from INSIDE_DUNE #12800

don't control concurrency from INSIDE_DUNE #12800
Alizter merged 2 commits intoocaml:mainfrom
Alizter:push-ysnonzkmwzym

Alizter commented Nov 27, 2025 •

edited

Loading

Uh oh!

Alizter commented Nov 28, 2025

Uh oh!

Alizter commented Nov 28, 2025

Uh oh!

Leonidas-from-XIV commented Nov 28, 2025

Uh oh!

Alizter commented Nov 28, 2025 •

edited

Loading

Uh oh!

Sudha247 commented Dec 2, 2025

Uh oh!

Uh oh!

Uh oh!

Sudha247 commented Dec 9, 2025

Uh oh!

Sudha247 commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Alizter commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Alizter commented Nov 28, 2025

Uh oh!

Alizter commented Nov 28, 2025

Uh oh!

Leonidas-from-XIV commented Nov 28, 2025

Uh oh!

Alizter commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Sudha247 commented Dec 2, 2025

Uh oh!

Uh oh!

Uh oh!

Sudha247 commented Dec 9, 2025

Uh oh!

Sudha247 commented Dec 10, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Alizter commented Nov 27, 2025 •

edited

Loading

Alizter commented Nov 28, 2025 •

edited

Loading