[new release] dune (18 packages) (3.22.0)

[shonfeder]

Fast, portable, and opinionated build system

• Project page: https://github.com/ocaml/dune
• Documentation: https://dune.readthedocs.io/

CHANGES:

Fixed

• Dyn.to_string now uses a smarter way to convert floats. This ensures that
  floats are printed with enough precision to round-trip and are valid OCaml
  lexemes. (feat(dyn): use smarter printing of floats dune#12982, fixes Request to make Dyn.float more precise dune#12980, @Alizter)

• Fix dune install --prefix failing with relative paths outside the workspace
  like ../foo (fix: support relative paths in dune install --prefix dune#12993, fixes dune install doesn't process relative paths correctly dune#12241, @benodiwal)

• Place the default trace file inside the build directory at the
  workspace root, rather than relative to the current directory.
  (Place default trace file in the build directory dune#13735, @vouillon)

• Fixed interpreting relative paths in %{bin:..} and %{bin-available:..}.
  These are now interpreted correctly, relative to the dune file they're in.
  (fix: interpret relative paths in %{bin:..} and %{bin-available:..} dune#13712, fixes Relative paths and %{bin:..} and %{bin-available:..} forms dune#9564, @anmonteiro)

• Delete sandboxes with broken permissions (fix: try harder to delete sandboxes with modified permissions dune#13511, @rgrinberg)

• Fix compiling Menhir parsers that refer to sibling modules within a
  subdirectory of (include_subdirs qualified). (fix(menhir): compile inferred interface in its corresponding module group dune#13118, fixes Sibling modules are unbound in qualified Menhir parsers dune#11119,
  @anmonteiro)

• Fixed the dependency specification of C stubs, which could result in C
  stubs not getting rebuilt when needed (which could in turn lead to
  segmentation faults and other hard-to-track bugs).
  (C stubs: depend on Caml headers dune#13652, fixes [BUG] C stubs dependencies are underspecified dune#13651, @nojb)

• Fix the Dune cache on Windows by correctly handling renames onto read-only
  files. Before this change, the Dune cache would be filled but the stored
  artifacts would not generally be usable by Dune. (fix: allow rename to overwrite read-only destination on Windows when deduplicating Dune cache dune#13713, @Nevor)

• Fix rpc not transferring promotion warnings to the client
  (Fix rpc not transferring promotion warnings to the client dune#12604, fixes dune promote doesn't fail when not available in the promotion database dune#12578, @ElectreAAS)

• Fix issue where dune exec -w was unable to kill running programs on
  rebuild. ( fix: avoid killing process group when not applicable  dune#12360, fixes dune exec --watch broken for long-running processes dune#12323, @Alizter)

• Resolve context and workspace binaries introduced by the respective (env (binaries ..)) stanzas. (fix: account for workspace binaries when generating rules for ./.bin dune#12952, fixes "No rule found" when using "binaries" defined in "env" stanza in dune-workspace dune#6220, @anmonteiro)

• Fix diff promotions originating from sandboxed rules. Previously, they
  would be completely ignored as the sandbox with the promoted file would be
  destroyed if the promotion fired (fix: register promotions from sandboxed actions dune#13520, @rgrinberg)

• Fix failure to digest installed directory targets, allowing them to be used
  as dependencies to other rules. (test: demonstrate failure to digest installed dir target dependency dune#13045, @anmonteiro)

• Fix handling of (select ..) field when used with (include_subdirs ..).
  (select <path> from ..) modules now parse path as a relative path
  starting from the module group root (fix: (select ..) field + (include_subdirs ..) dune#13175, fixes Alternative dependencies do not work with include_subdirs unqualified dune#4383, Can include_subdirs qualified and select work together? dune#12450,
  @anmonteiro)

• Fix dune trying to kill processes that were already reaped due to race
  conditions (Scheduler: fix pid races dune#13245, @rgrinberg)

• Add O_CLOEXEC to all files used for stdin/stdout/stderr (fix: add missing O_CLOEXEC to redirect files dune#13385, @rgrinberg)

• Fix $ dune promote dir/foo when dir does not exist (fix: dune promote should promote into non existent dirs dune#13493, @rgrinberg)

• Fix (select ..) field evaluation when a transitive library has optional
  dependencies (fixes All MirageOS unikernels fail to build: dune build fails to succeed (with select in a monorepo) dune#13299, fix(select-field): eval closure when resolving select dune#13389, @anmonteiro)

• Fix sandboxed builds of library stanzas that set
  (stdlib (modules_before_stdlib ..)) (fix(stdlib): depend on modules_before_stdlib dune#13624, @anmonteiro)

• Dune cache: use of hard links under Windows. (Dune cache: use hard links on Windows dune#13714, @Nevor)

• Fixed non-build caches not following $DUNE_CACHE_ROOT and instead only
  relying on $XDG_CACHE_HOME.
  This means the normal build cache moves:
  $DUNE_CACHE_ROOT -> $DUNE_CACHE_ROOT/db (no changes if that variable was
  unset). Affected users can prevent a full cache invalidation by moving
  previous contents:
  cd $DUNE_CACHE_ROOT; mkdir db; mv <contents of directory> db.
  (use DUNE_CACHE_HOME instead of XDG_CACHE_HOME dune#11612, fixes Toolchains cache does not follow DUNE_CACHE_ROOT dune#11584, @ElectreAAS)

• $ dune promotion list writes output to stdout rather than stderr (fix: dune promotion list should write its output to stdout dune#13462)

• Improve handling of empty files in the diff action. These are now correctly
  distinguished from empty files. (fix: diff action missing files dune#13696, @rgrinberg)

• Pass /dev/null to --diff-command instead of non-existent files (fix: diff action missing files dune#13696,
  @rgrinberg)

• Fix failure when multiple rocq.extraction stanzas existing in a directory
  (Fix #8042, multiple rocq.extraction in directory. dune#13531, fixes Having several Coq extraction stanzas leads to multiple rule error from Dune 3.7 dune#8042, @rlepigre-skylabs-ai)

• Print $ dune promotion show output to stdout rather than stderr (fix: dune promotion show should write to stdout dune#13481,
  @rgrinberg)

• Fix deadlock in the memo library in the presence of dependency cycles
  (fix(memo): fill ivar cells for dependency cycle nodes to avoid deadlock dune#13625, @anmonteiro)

• Fix promotions that modify a directory into a file (fix: promoting a file into a directory dune#13516, fixes cram tests: spurious error when transforming a single-file test into a directory test dune#4067,
  @rgrinberg)

• Fix installation of implementations of virtual libraries. This failed when
  the implementation had no private modules, but the virtual library did
  (Uncaught exception when a public library has a virtual module dune#10635, @rgrinberg)

• Respect the (dir ..) field on packages when setting up cram tests (fix: respect the dir stanza for assigning cram tests to packages dune#13581,
  @rgrinberg)

Added

• Add support for generating .cms files using oxcaml and adding .cms or
  .cmt files as compilation dependencies (Add support for OxCaml's .cms files dune#13397, @spiessimon)

• Add trace events for custom actions (feature: trace some custom actions dune#13265, @rgrinberg)

• Allow enabling extensions with (using ..) in dune-workspace files
  (Support for using oxcaml 0.1 in workspace files dune#13395, @spiessimon)

• Add sandbox extraction trace event (feature: sandbox extraction event dune#13544, @rgrinberg)

• Add the initial cwd to the first config event (Dune_trace: add pid and initial_cwd to initial event dune#13026, @rgrinberg)

• Dune dune produces trace events in DUNE_ACTION_TRACE_DIR if this variable
  is set. (feature: dune produces action traces dune#13302, @rgrinberg)

• Add file watching events to the trace file (dune_trace: add file watching events dune#13038, @rgrinberg)

• Introduce the $ dune trace cat subcommand to view the trace file. (Dune trace subcommand dune#13055,
  @rgrinberg)

• Add diagnostic events to the trace. (Add diagnostic events to the trace dune#13041, @rgrinberg)

• Add DUNE_JOBS environment variable for controlling concurrency of Dune from
  environment. The INSIDE_DUNE variable also now no longer controls
  concurrency (don't control concurrency from INSIDE_DUNE  dune#12800, @Alizter)

• Support for Rocq expected output tests (Add expected output tests for Rocq. dune#13632, @rlepigre-skylabs-ai)

• Add rusage information to completed processes in the trace (@rgrinberg,
  dune_trace: add resource usage to process events dune#13241)

• Add process start events to the trace (feature: process start events dune#13261, rgrinberg)

• Generate odoc documentation in markdown using the @doc-markdown alias
  (Markdown for odoc without system dune#12581, @davesnx)

• Add timing information for every command executed by cram (Cram: Add command level timing information dune#13092,
  @rgrinberg)

• Add the workspace root to the config trace event (Add dune root to config event dune#12922, @rgrinberg)

• Introduce the dune-action-trace library. This public library is to be used
  by custom actions to emit trace events while executed as part of a dune
  build. The trace events emitted through this library will be incorporated
  into dune's own trace (feature: introduce a public action tracing library dune#13348, @rgrinberg)

• Add dune-find-dominating to dune.el, a command to find the
  dominating dune file. (Add dune-find-dominating command to dune.el dune#12696, @arvidj)

• Add a --no-recursive flag to $ dune describe workspace (feature: add --no-recursive flag to $ dune describe workspace dune#13590, @rgrinberg)

• Trace events for files written directly by dune (feature: trace file writing events dune#13618, @rgrinberg)

• Allow expansion of special forms like (:include ..) and %{read-lines:..}
  in the modules specification for the ocamllex, ocamlyacc and menhir
  stanzas. (feat(ocamllex): support dynamic (modules ..) evaluation  dune#13105, feat(ocamlyacc): support expanding (modules ..) dune#13135, feat(menhir): allow expanding (modules ..) field dune#13157, @anmonteiro)

• Add a trace event for snapshotting the asndbox (feature: sandbox snapshot event dune#13541, @rgrinberg)

• Add signal send and receive events to the trace (dune_trace: trace all signals dune#13193, @rgrinberg)

• Emit final trace event before exiting. (Dune trace: emit final event dune#13018, @rgrinberg)

• dune runtest can now run individual test executables from (tests) stanzas
  and inline tests from (library (inline_tests)) stanzas by providing their
  source files as arguments. (feat(runtest): dune runtest for (tests) and (inline_tests) dune#13064, fixes It should be possible to run tests from a specific module with runtest dune#870, @Alizter)

• Add a shell field to the cram stanza. This field allows customizing the
  shell to be bash rather than sh (cram: allow customizing the shell dune#13083, @haochenx)

Changed

• Start sandboxing the execution of tests defined with the test and tests
  stanzas (feature: start sandboxing test execution starting from 3.22 dune#13510, fix: sandbox all test stanza variants starting from 3.22 dune#13617, @rgrinberg)

• Disabled cram tests can now be run explicitly with dune runtest disabled.t.
  The enabled_if field now only controls whether a test is included in
  the @runtest alias. (feat(cram): explicitly requested disabled tests now execute dune#13081, @Alizter)

• Process categories in trace events are moved to their own field in args
  (dune trace: move process categories to field in args dune#13024, @rgrinberg)

• Sandbox running ocamllex and ocamlyacc actions. (ocamllex / ocamlyacc: sandbox rules dune#13098, @anmonteiro)

• Sandboxing mdx test actions is now the default starting from 0.5 (feature(mdx): start sandboxing in version 0.5 dune#13504,
  @rgrinberg)

• Start sandboxing Melange rules by default in the (library ..) and
  (melange.emit ..) stanzas (feat(melange): sandbox rules by default dune#13619, @anmonteiro)

• Introduce a promotion trace event and remove the corresponding verbose log
  message. (Add a promotion trace event dune#12949, feature: trace events for promotion registration dune#13444, @rgrinberg)

• Change dune's trace format to emit canonical s-expressions. This improves
  performance and is better aligned with dune's usage of the format
  elsewhere. $ dune trace cat can also emit the trace in --chrome-trace
  for perfetto, or --sexp for regular s-expressions for interactive usage.
  (dune_trace: csexp dune#13059, @rgrinberg)

• Move all logging statements to the trace file. All log statements now contain
  structured payloads (Structured Logging dune#13015, fixes Rationalize Logging/Tracing dune#12904, @rgrinberg)

• Add a target resolution event to replace the equivalent log message (Dune_trace: target resolution event dune#12955,
  @rgrinberg)

[shonfeder]

This should be ready to go ASAP, without needing to wait for CI to run, assuming that maintainers are satisfied with the results of the last run, and agree with the diagnosis of the few new failures diagnosed and analyzed at #29547 (comment) .

For the fullest record of our triage, fixes, and analysis, see ocaml/dune#13674

[raphael-proust]

thanks a lot!

looking forward to using all the new tracing capabilities <3