[new release] dune (18 packages) (3.22.0~alpha2)

[shonfeder]

Fast, portable, and opinionated build system

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

Fixed

• Dyn.to_string now uses a smarter way to convert floats. This ensures that
  floats are printed with enough precision to round-trip and are valid OCaml
  lexemes. (feat(dyn): use smarter printing of floats dune#12982, fixes Request to make Dyn.float more precise dune#12980, @Alizter)

• Fix dune install --prefix failing with relative paths outside the workspace
  like ../foo (fix: support relative paths in dune install --prefix dune#12993, fixes dune install doesn't process relative paths correctly dune#12241, @benodiwal)

• Place the default trace file inside the build directory at the
  workspace root, rather than relative to the current directory.
  (Place default trace file in the build directory dune#13735, @vouillon)

• Fixed interpreting relative paths in %{bin:..} and %{bin-available:..}.
  These are now interpreted correctly, relative to the dune file they're in.
  (fix: interpret relative paths in %{bin:..} and %{bin-available:..} dune#13712, fixes Relative paths and %{bin:..} and %{bin-available:..} forms dune#9564, @anmonteiro)

• Delete sandboxes with broken permissions (fix: try harder to delete sandboxes with modified permissions dune#13511, @rgrinberg)

• Fix compiling Menhir parsers that refer to sibling modules within a
  subdirectory of (include_subdirs qualified). (fix(menhir): compile inferred interface in its corresponding module group dune#13118, fixes Sibling modules are unbound in qualified Menhir parsers dune#11119,
  @anmonteiro)

• Fixed the dependency specification of C stubs, which could result in C
  stubs not getting rebuilt when needed (which could in turn lead to
  segmentation faults and other hard-to-track bugs).
  (C stubs: depend on Caml headers dune#13652, fixes [BUG] C stubs dependencies are underspecified dune#13651, @nojb)

• Fix the Dune cache on Windows by correctly handling renames onto read-only
  files. Before this change, the Dune cache would be filled but the stored
  artifacts would not generally be usable by Dune. (fix: allow rename to overwrite read-only destination on Windows when deduplicating Dune cache dune#13713, @Nevor)

• Fix rpc not transferring promotion warnings to the client
  (Fix rpc not transferring promotion warnings to the client dune#12604, fixes dune promote doesn't fail when not available in the promotion database dune#12578, @ElectreAAS)

• Fix issue where dune exec -w was unable to kill running programs on
  rebuild. ( fix: avoid killing process group when not applicable  dune#12360, fixes dune exec --watch broken for long-running processes dune#12323, @Alizter)

• Fix package extraction on systems with tar implementations that don't
  auto-detect compression (e.g., OpenBSD). Dune now passes explicit
  decompression flags (-z for gzip, -j for bzip2) when needed, and provides
  clear error messages for unsupported formats like XZ and LZMA. (fix(pkg): pass explicit -z/-j flags for non-auto-detecting tar dune#13688,
  fixes tar on OpenBSD dune#10123, @Alizter)

• Resolve context and workspace binaries introduced by the respective (env (binaries ..)) stanzas. (fix: account for workspace binaries when generating rules for ./.bin dune#12952, fixes "No rule found" when using "binaries" defined in "env" stanza in dune-workspace dune#6220, @anmonteiro)

• Fix diff promotions originating from sandboxed rules. Previously, they
  would be completely ignored as the sandbox with the promoted file would be
  destroyed if the promotion fired (fix: register promotions from sandboxed actions dune#13520, @rgrinberg)

• Fix failure to digest installed directory targets, allowing them to be used
  as dependencies to other rules. (test: demonstrate failure to digest installed dir target dependency dune#13045, @anmonteiro)

• Fix handling of (select ..) field when used with (include_subdirs ..).
  (select <path> from ..) modules now parse path as a relative path
  starting from the module group root (fix: (select ..) field + (include_subdirs ..) dune#13175, fixes Alternative dependencies do not work with include_subdirs unqualified dune#4383, Can include_subdirs qualified and select work together? dune#12450,
  @anmonteiro)

• Fix dune trying to kill processes that were already reaped due to race
  conditions (Scheduler: fix pid races dune#13245, @rgrinberg)

• Add O_CLOEXEC to all files used for stdin/stdout/stderr (fix: add missing O_CLOEXEC to redirect files dune#13385, @rgrinberg)

• Fix $ dune promote dir/foo when dir does not exist (fix: dune promote should promote into non existent dirs dune#13493, @rgrinberg)

• Fix (select ..) field evaluation when a transitive library has optional
  dependencies (fixes All MirageOS unikernels fail to build: dune build fails to succeed (with select in a monorepo) dune#13299, fix(select-field): eval closure when resolving select dune#13389, @anmonteiro)

• Fix sandboxed builds of library stanzas that set
  (stdlib (modules_before_stdlib ..)) (fix(stdlib): depend on modules_before_stdlib dune#13624, @anmonteiro)

• Dune cache: use of hard links under Windows. (Dune cache: use hard links on Windows dune#13714, @Nevor)

• Fixed non-build caches not following $DUNE_CACHE_ROOT and instead only
  relying on $XDG_CACHE_HOME.
  This means the normal build cache moves:
  $DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db (no changes if that variable was
  unset). Affected users can prevent a full cache invalidation by moving
  previous contents:
  cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db.
  (use DUNE_CACHE_HOME instead of XDG_CACHE_HOME dune#11612, fixes Toolchains cache does not follow DUNE_CACHE_ROOT dune#11584, @ElectreAAS)

• $ dune promotion list writes output to stdout rather than stderr (fix: dune promotion list should write its output to stdout dune#13462)

• Improve handling of empty files in the diff action. These are now correctly
  distinguished from empty files. (fix: diff action missing files dune#13696, @rgrinberg)

• Pass /dev/null to --diff-command instead of non-existent files (fix: diff action missing files dune#13696,
  @rgrinberg)

• Fix failure when multiple rocq.extraction stanzas existing in a directory
  (Fix #8042, multiple rocq.extraction in directory. dune#13531, fixes Having several Coq extraction stanzas leads to multiple rule error from Dune 3.7 dune#8042, @rlepigre-skylabs-ai)

• Print $ dune promotion show output to stdout rather than stderr (fix: dune promotion show should write to stdout dune#13481,
  @rgrinberg)

• Fix deadlock in the memo library in the presence of dependency cycles
  (fix(memo): fill ivar cells for dependency cycle nodes to avoid deadlock dune#13625, @anmonteiro)

• Fix promotions that modify a directory into a file (fix: promoting a file into a directory dune#13516, fixes cram tests: spurious error when transforming a single-file test into a directory test dune#4067,
  @rgrinberg)

• Fix installation of implementations of virtual libraries. This failed when
  the implementation had no private modules, but the virtual library did
  (Uncaught exception when a public library has a virtual module dune#10635, @rgrinberg)

• Respect the (dir ..) field on packages when setting up cram tests (fix: respect the dir stanza for assigning cram tests to packages dune#13581,
  @rgrinberg)

Added

• Add support for generating .cms files using oxcaml and adding .cms or
  .cmt files as compilation dependencies (Add support for OxCaml's .cms files dune#13397, @spiessimon)

• Add trace events for custom actions (feature: trace some custom actions dune#13265, @rgrinberg)

• Allow enabling extensions with (using ..) in dune-workspace files
  (Support for using oxcaml 0.1 in workspace files dune#13395, @spiessimon)

• Add sandbox extraction trace event (feature: sandbox extraction event dune#13544, @rgrinberg)

• Add the initial cwd to the first config event (Dune_trace: add pid and initial_cwd to initial event dune#13026, @rgrinberg)

• Dune dune produces trace events in DUNE_ACTION_TRACE_DIR if this variable
  is set. (feature: dune produces action traces dune#13302, @rgrinberg)

• Add file watching events to the trace file (dune_trace: add file watching events dune#13038, @rgrinberg)

• Introduce the $ dune trace cat subcommand to view the trace file. (Dune trace subcommand dune#13055,
  @rgrinberg)

• Add diagnostic events to the trace. (Add diagnostic events to the trace dune#13041, @rgrinberg)

• Add DUNE_JOBS environment variable for controlling concurrency of Dune from
  environment. The INSIDE_DUNE variable also now no longer controls
  concurrency (don't control concurrency from INSIDE_DUNE  dune#12800, @Alizter)

• Support for Rocq expected output tests (Add expected output tests for Rocq. dune#13632, @rlepigre-skylabs-ai)

• Add rusage information to completed processes in the trace (@rgrinberg,
  dune_trace: add resource usage to process events dune#13241)

• Add process start events to the trace (feature: process start events dune#13261, rgrinberg)

• Generate odoc documentation in markdown using the @doc-markdown alias
  (Markdown for odoc without system dune#12581, @davesnx)

• Add timing information for every command executed by cram (Cram: Add command level timing information dune#13092,
  @rgrinberg)

• Add the workspace root to the config trace event (Add dune root to config event dune#12922, @rgrinberg)

• Introduce the dune-action-trace library. This public library is to be used
  by custom actions to emit trace events while executed as part of a dune
  build. The trace events emitted through this library will be incorporated
  into dune's own trace (feature: introduce a public action tracing library dune#13348, @rgrinberg)

• Add dune-find-dominating to dune.el, a command to find the
  dominating dune file. (Add dune-find-dominating command to dune.el dune#12696, @arvidj)

• Add a --no-recursive flag to $ dune describe workspace (feature: add --no-recursive flag to $ dune describe workspace dune#13590, @rgrinberg)

• Trace events for files written directly by dune (feature: trace file writing events dune#13618, @rgrinberg)

• Allow expansion of special forms like (:include ..) and %{read-lines:..}
  in the modules specification for the ocamllex, ocamlyacc and menhir
  stanzas. (feat(ocamllex): support dynamic (modules ..) evaluation  dune#13105, feat(ocamlyacc): support expanding (modules ..) dune#13135, feat(menhir): allow expanding (modules ..) field dune#13157, @anmonteiro)

• Add a trace event for snapshotting the asndbox (feature: sandbox snapshot event dune#13541, @rgrinberg)

• Add signal send and receive events to the trace (dune_trace: trace all signals dune#13193, @rgrinberg)

• Emit final trace event before exiting. (Dune trace: emit final event dune#13018, @rgrinberg)

• dune runtest can now run individual test executables from (tests) stanzas
  and inline tests from (library (inline_tests)) stanzas by providing their
  source files as arguments. (feat(runtest): dune runtest for (tests) and (inline_tests) dune#13064, fixes It should be possible to run tests from a specific module with runtest dune#870, @Alizter)

• Add a shell field to the cram stanza. This field allows customizing the
  shell to be bash rather than sh (cram: allow customizing the shell dune#13083, @haochenx)

Changed

• Start sandboxing the execution of tests defined with the test and tests
  stanzas (feature: start sandboxing test execution starting from 3.22 dune#13510, fix: sandbox all test stanza variants starting from 3.22 dune#13617, @rgrinberg)

• Disabled cram tests can now be run explicitly with dune runtest disabled.t.
  The enabled_if field now only controls whether a test is included in
  the @runtest alias. (feat(cram): explicitly requested disabled tests now execute dune#13081, @Alizter)

• Process categories in trace events are moved to their own field in args
  (dune trace: move process categories to field in args dune#13024, @rgrinberg)

• Sandbox running ocamllex and ocamlyacc actions. (ocamllex / ocamlyacc: sandbox rules dune#13098, @anmonteiro)

• Sandboxing mdx test actions is now the default starting from 0.5 (feature(mdx): start sandboxing in version 0.5 dune#13504,
  @rgrinberg)

• Start sandboxing Melange rules by default in the (library ..) and
  (melange.emit ..) stanzas (feat(melange): sandbox rules by default dune#13619, @anmonteiro)

• Introduce a promotion trace event and remove the corresponding verbose log
  message. (Add a promotion trace event dune#12949, feature: trace events for promotion registration dune#13444, @rgrinberg)

• Change dune's trace format to emit canonical s-expressions. This improves
  performance and is better aligned with dune's usage of the format
  elsewhere. $ dune trace cat can also emit the trace in --chrome-trace
  for perfetto, or --sexp for regular s-expressions for interactive usage.
  (dune_trace: csexp dune#13059, @rgrinberg)

• Move all logging statements to the trace file. All log statements now contain
  structured payloads (Structured Logging dune#13015, fixes Rationalize Logging/Tracing dune#12904, @rgrinberg)

• Add a target resolution event to replace the equivalent log message (Dune_trace: target resolution event dune#12955,
  @rgrinberg)

[shonfeder]

The errors that did not appear in the last release, minus these classes of known noise:

    [ (* gcc incompatibility issue *)
      "-Wimplicit-function-declaration"
    ; "-Wincompatible-pointer-types"
    ; (* CI workers timeout or failure *)
      "(failed: Cancelled)"
    ; "(failed: Failed: Internal error)"
    ; (* Unable to get package source *)
      "Failed to get sources of"
    ; (* Package source has been changed *)
      "Bad checksum" ]

Fixed in #29555

• chrome-trace:
  • (revdeps/compilers/5.4): ocaml-lsp-server.1.24.0 (failed: This application of the functor Drpc.Client.Make is ill-typed.)
• xdg:
  • (revdeps/compilers/5.4): ocaml-lsp-server.1.24.0 (failed: This application of the functor Drpc.Client.Make is ill-typed.)

Unrelated test failures

• dune-build-info:
  • (revdeps/compilers/5.4): server-reason-react.0.4.1 (failed: The compilation of server-reason-react.0.4.1 failed at "dune build -p server-reason-react -j 255 @install @runtest".)
    • A test failure due to a trivial discrepency in expect tests (one missing line), likely due to a brittle test logic, and no signs of being due to breakage from dune, AFAICT.

Unrelated build failures

• dune-private-libs:
  • (revdeps/compilers/4.14): pilat.1.6 (failed: frama-c.22.0 failed to build)
    • frama-c builds are often flaky
• top-closure:
  • (revdeps/compilers/4.14): pilat.1.6 (failed: frama-c.22.0 failed to build)
  • (revdeps/compilers/4.14): sail_output.0.20.1 (failed: System package install failed with exit code 100 at command:)

Updates

Unbound value String.is_prefix in ocaml-lsp-server

• chrome-trace:
  • (revdeps/compilers/4.14): ocaml-lsp-server.1.20.1-4.14 (failed: Unbound value String.is_prefix)
• xdg:
  • (revdeps/compilers/4.14): ocaml-lsp-server.1.20.1-4.14 (failed: Unbound value String.is_prefix)

Fixed by upper bounds in #29561

FreeBSD failures

• dune-rpc:
  • (freebsd (experimental)/freebsd-14.3-ocaml-5.4-amd64 (experimental)): dune-rpc.3.22.0~alpha2 (failed: Failed: Build failed)
• dune-rpc-lwt:
  • (freebsd (experimental)/freebsd-14.3-ocaml-5.4-amd64 (experimental)): dune-rpc-lwt.3.22.0alpha2 (failed: dune-rpc.3.22.0alpha2 failed to build)

These appear to be flaky runners. See #29547 (comment)

[mseri]

Decompress and ocaml-lsp-server failures stand out. The rest does not seem strange but there is a surprising amount of cstubs failures, that seems more than usual (without checking carefully)

[Alizter]

The ocaml-lsp-server and xdg breakage is due to the renaming Stdune.String.is_prefix -> Stdune.String.starts_with. A simple fix would be to patch ocaml-lsp-server to have the old starts_with inside ocaml-lsp-server/src/import.ml.

[shonfeder]

After reviewing the FreeBSD errors and trying to rerun the jobs several times, I think the freebsd errors are just problems with the runners. Initially it looked like they were having trouble with the file system, and now they are failing non-deterministically with failed: Cancelled. I'm seeing the same sorts of failures on other jobs (e.g., https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/710097a1b048881d8a70c503fd3a92b6b2b0f887).

[shonfeder]

Decompress and ocaml-lsp-server failures stand out. The rest does not seem strange but there is a surprising amount of cstubs failures, that seems more than usual (without checking carefully)

Thanks for looking, @mseri !

• Decompress is not a new failure, and was discussed previously [new release] dune (17 packages) (3.21.0) #29216 (comment) . This problem has been reported upstream at decompress 1.3.0 tests fail to build due to bigarry pulling in dune  mirage/decompress#113 .
• ocaml-lsp-server should be addressed by Add upper bound on stdune 3.22 #29561
• Regarding the "surprising amount of cstubs failures", AFAICT, this is just a symptom of updated GCC rollouts on some of the workers. I raised this issue in the opam-repo-maintainers discord channel during the last release. At least, every case I've investigated has just been due to changing defaults of the c toolchain.

[shonfeder]

#29561 is allowing the ocaml-lsp-server revdeps to build, as expected. See the newer results at

• https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/19c70fd6a788b154ec5e9fe26bca1d12fb2519be/variant/compilers,4.14,chrome-trace.3.22.0~alpha2,revdeps,ocaml-lsp-server.1.20.1-4.14
• https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/19c70fd6a788b154ec5e9fe26bca1d12fb2519be/variant/compilers,4.14,chrome-trace.3.22.0~alpha2,revdeps,ocaml-lsp-server.1.18.0

I think we have whacked all the moles, and I am going to cut the full release now. Thanks for the reviews and support!