Skip to content

bundle: serialize installs sharing an implicit dependency - #23342

Merged
MikeMcQuaid merged 1 commit into
mainfrom
bundle-implicit-dep-lock-conflicts
Jul 29, 2026
Merged

bundle: serialize installs sharing an implicit dependency#23342
MikeMcQuaid merged 1 commit into
mainfrom
bundle-implicit-dep-lock-conflicts

Conversation

@dduugg

@dduugg dduugg commented Jul 28, 2026

Copy link
Copy Markdown
Member

What does this change do, and why?

Addresses one of two independent causes behind #23328 (per @MikeMcQuaid's request for a separate PR per cause).

brew bundle's parallel scheduler (bundle/parallel_installer.rb's build_dependency_map) avoids running two entries in the same batch when their recursive declared dependencies overlap, matching the locking FormulaInstaller#lock does. On Linux, though, formulae can silently gain an implicit dependency at instantiation time (bubblewrap, and gcc/glibc when build formulae are needed) whenever the relevant executable isn't installed yet, added via Formula#add_global_deps_to_spec rather than declared on any formula. Two entries with no declared dependency in common (e.g. bat and xz) can therefore both silently need to fetch the same bottle at once. Since DownloadLock's flock is non-blocking, the loser used to die immediately with OperationInProgressError instead of waiting.

This adds DependencyCollector#implicit_dependency_names, empty by default and overridden on Linux to report whichever of those names would currently be added, and folds it into every entry's recursive dependency set before computing overlaps, so entries that would both need the same implicit dependency get serialized like any other shared dependency.

I wasn't able to fully verify the Linux-specific code path end to end on my own machine (macOS doesn't load extend/os/linux/* at all, confirmed while investigating), so the Linux-specific unit tests here will get their first real execution on Linux CI. Opening as a draft for that reason, please let me know if you'd like anything adjusted before it's ready for a full review.

Step-by-step reproduction

Given in #23328: brew bundle install with a Brewfile containing two formulae that share no declared dependency but would both pull in bubblewrap on a fresh Linux machine without it installed (e.g. bat and xz).


  • Have you followed our Contributing guidelines?
  • Have you checked for other open Pull Requests for the same change?
  • Have you explained what your changes do? Performance claims (e.g. "this is faster") must include Hyperfine benchmarks.
  • Have you explained why you'd like these changes included, not just what they do?
  • For bug fixes, have you given step-by-step brew commands to reproduce the bug?
  • Have you written new tests (excluding integration tests)? Here's an example.
  • Have you successfully run brew lgtm (style, typechecking and tests) locally?

  • AI was used to generate or assist with generating this PR.

Used Claude Code to investigate #23328, trace the scheduler's overlap-detection logic and the implicit-dependency mechanism it misses, implement the fix, and write the accompanying tests. Verified by running brew typecheck and brew style --changed (clean), running the full spec files touched (dependency_collector_spec.rb, bundle/installer_spec.rb) and the full bundle directory (all passing), and manually confirming via a scratch script that Formula#recursive_dependencies (no block) preserves :implicit-tagged dependencies rather than pruning them, which the scheduler's overlap check relies on. Reviewed the full diff by hand before opening this PR.


@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Approach makes sense, just think it's possible to get this information another way instead.

Comment thread Library/Homebrew/extend/os/linux/dependency_collector.rb Outdated
Comment thread Library/Homebrew/bundle/parallel_installer.rb Outdated
Comment thread Library/Homebrew/dependency_collector.rb Outdated
@dduugg
dduugg force-pushed the bundle-implicit-dep-lock-conflicts branch from b4f7f8d to f053ab1 Compare July 28, 2026 17:11

@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread Library/Homebrew/bundle/parallel_installer.rb Outdated
@dduugg
dduugg force-pushed the bundle-implicit-dep-lock-conflicts branch from f053ab1 to 6adb00a Compare July 28, 2026 23:12
@dduugg
dduugg marked this pull request as ready for review July 28, 2026 23:16
Copilot AI review requested due to automatic review settings July 28, 2026 23:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates brew bundle鈥檚 parallel scheduler to account for Linux-only implicit formula dependencies (e.g. bubblewrap, gcc, glibc) that can be injected at formula instantiation time, so entries that would contend on the same implicit dependency are scheduled in separate batches to avoid lock/download races.

Changes:

  • Add DependencyCollector#implicit_dependency_names to surface currently-applicable implicit dependency formula names.
  • Incorporate implicit dependency names into brew bundle鈥檚 recursive dependency sets used for overlap/lock-conflict scheduling.
  • Add unit tests covering implicit_dependency_names (generic + Linux) and the bundle scheduler鈥檚 serialization behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
Library/Homebrew/dependency_collector.rb Introduces implicit_dependency_names helper to report currently-applicable implicit deps.
Library/Homebrew/bundle/parallel_installer.rb Adds implicit deps into recursive dependency sets for lock-conflict scheduling in brew bundle.
Library/Homebrew/test/dependency_collector_spec.rb Verifies default (non-Linux) implicit_dependency_names is empty.
Library/Homebrew/test/os/linux/dependency_collector_spec.rb Adds Linux-specific tests for bubblewrap/gcc/glibc inclusion in implicit_dependency_names.
Library/Homebrew/test/bundle/installer_spec.rb Adds a scheduler test ensuring entries are serialized when sharing an implicit dep.

馃挕 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread Library/Homebrew/bundle/parallel_installer.rb Outdated
brew bundle's parallel scheduler avoids running two entries in the same
batch when their recursive declared dependencies overlap, but implicit
dependencies added at formula-instantiation time (e.g. bubblewrap and
its dependents on Linux, when the sandbox executable isn't installed
yet) aren't declared on any formula, so two otherwise-unrelated
formulae can both silently need to fetch the same bottle at once. Since
DownloadLock's flock is non-blocking, the loser dies instead of
waiting.

Add DependencyCollector#implicit_dependency_names (empty by default,
overridden on Linux to report bubblewrap/gcc/glibc when they'd
currently be added), and fold it into every entry's recursive
dependency set before computing overlaps, so entries needing the same
implicit dependency are serialized like any other shared dependency.

See #23328
@dduugg
dduugg force-pushed the bundle-implicit-dep-lock-conflicts branch from 6adb00a to f959477 Compare July 28, 2026 23:43

@MikeMcQuaid MikeMcQuaid left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeh, this approach makes more sense I think! Thanks!

@MikeMcQuaid
MikeMcQuaid enabled auto-merge July 29, 2026 07:32
@MikeMcQuaid
MikeMcQuaid added this pull request to the merge queue Jul 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 29, 2026
@MikeMcQuaid
MikeMcQuaid added this pull request to the merge queue Jul 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 29, 2026
@MikeMcQuaid
MikeMcQuaid added this pull request to the merge queue Jul 29, 2026
Merged via the queue into main with commit 248797d Jul 29, 2026
64 of 65 checks passed
@MikeMcQuaid
MikeMcQuaid deleted the bundle-implicit-dep-lock-conflicts branch July 29, 2026 17:31
dduugg added a commit that referenced this pull request Jul 30, 2026
Two `brew` processes downloading the same file raced on the download
lock and one failed outright with `OperationInProgressError`, which is
half of #23328 (the scheduling half was fixed in #23342). Waiting is
almost always what the user wants, since the holder is about to produce
exactly the file this process needs.

`DownloadLock#lock_or_wait` polls the existing non-blocking `flock`
every 0.1s instead of giving up on the first failure. At 0.1s the wait
costs about 0.15% of one core per waiting download, dominated by syscall
overhead rather than real work, so the interval buys responsiveness for
no meaningful CPU. The wait is capped at 3 minutes: an hour is always
going to be too long, and giving up beats waiting much longer because
`RetryableDownload` preserves the `.incomplete` file, so a retry resumes
the holder's partial download via `--continue-at`.

The warning is suppressed when the caller is already rendering progress.
`HOMEBREW_DOWNLOAD_CONCURRENCY` defaults to `cores * 2`, and above 1
`DownloadQueue#fetch` drives a cursor-addressed redraw whose arithmetic
assumes one line per download, so an unscheduled write from a pool
worker desyncs it. `OperationInProgressError` also takes an optional
`waited:` now, because telling someone to "wait for it to finish or
terminate it to continue" after three minutes of waiting is not useful.
The message is unchanged for every existing caller.

Mutation testing found `lock_file_spec.rb` passed with the inode/unlink
recheck neutered and with `ignore_interrupts` removed from `#lock`, so
cover both. Deferring the interrupt itself can't be asserted in-process,
since RSpec owns the `INT` handler that `ignore_interrupts` traps and
`Thread#raise` bypasses `trap`, so the wrapper's presence is asserted
instead.
dduugg added a commit that referenced this pull request Jul 30, 2026
Two `brew` processes downloading the same file raced on the download
lock and one failed outright with `OperationInProgressError`, which is
half of #23328 (the scheduling half was fixed in #23342). Waiting is
almost always what the user wants, since the holder is about to produce
exactly the file this process needs.

`DownloadLock#lock_or_wait` polls the existing non-blocking `flock`
every 0.1s instead of giving up on the first failure. At 0.1s the wait
costs about 0.15% of one core per waiting download, dominated by syscall
overhead rather than real work, so the interval buys responsiveness for
no meaningful CPU. The wait is capped at 3 minutes: an hour is always
going to be too long, and giving up beats waiting much longer because
`RetryableDownload` preserves the `.incomplete` file, so a retry resumes
the holder's partial download via `--continue-at`.

The warning is suppressed when the caller is already rendering progress.
`HOMEBREW_DOWNLOAD_CONCURRENCY` defaults to `cores * 2`, and above 1
`DownloadQueue#fetch` drives a cursor-addressed redraw whose arithmetic
assumes one line per download, so an unscheduled write from a pool
worker desyncs it. `OperationInProgressError` also takes an optional
`waited:` now, because telling someone to "wait for it to finish or
terminate it to continue" after three minutes of waiting is not useful.
The message is unchanged for every existing caller.

Mutation testing found `lock_file_spec.rb` passed with the inode/unlink
recheck neutered and with `ignore_interrupts` removed from `#lock`, so
cover both. Deferring the interrupt itself can't be asserted in-process,
since RSpec owns the `INT` handler that `ignore_interrupts` traps and
`Thread#raise` bypasses `trap`, so the wrapper's presence is asserted
instead.
dduugg added a commit that referenced this pull request Jul 30, 2026
Two `brew` processes downloading the same file raced on the download
lock and one failed outright with `OperationInProgressError`, which is
half of #23328 (the scheduling half was fixed in #23342). Waiting is
almost always what the user wants, since the holder is about to produce
exactly the file this process needs.

`DownloadLock#lock_or_wait` polls the existing non-blocking `flock`
every 0.1s instead of giving up on the first failure. At 0.1s the wait
costs about 0.15% of one core per waiting download, dominated by syscall
overhead rather than real work, so the interval buys responsiveness for
no meaningful CPU. The wait is capped at 3 minutes, or at the caller's
remaining `timeout:` budget when that is shorter, so waiting on the lock
can't blow a deadline the caller asked for. An hour is always going to
be too long, and giving up beats waiting much longer because
`RetryableDownload` preserves the `.incomplete` file, so a retry resumes
the holder's partial download via `--continue-at`.

The warning is suppressed when the caller is already rendering progress.
`HOMEBREW_DOWNLOAD_CONCURRENCY` defaults to `cores * 2`, and above 1
`DownloadQueue#fetch` drives a cursor-addressed redraw whose arithmetic
assumes one line per download, so an unscheduled write from a pool
worker desyncs it. `OperationInProgressError` also takes an optional
`waited:` now, because telling someone to "wait for it to finish or
terminate it to continue" after three minutes of waiting is not useful.
The message is unchanged for every existing caller.

Mutation testing found `lock_file_spec.rb` passed with the inode/unlink
recheck neutered and with `ignore_interrupts` removed from `#lock`, so
cover both. Deferring the interrupt itself can't be asserted in-process,
since RSpec owns the `INT` handler that `ignore_interrupts` traps and
`Thread#raise` bypasses `trap`, so the wrapper's presence is asserted
instead.
dduugg added a commit that referenced this pull request Jul 30, 2026
Two `brew` processes downloading the same file raced on the download
lock and one failed outright with `OperationInProgressError`, which is
half of #23328 (the scheduling half was fixed in #23342). Waiting is
almost always what the user wants, since the holder is about to produce
exactly the file this process needs.

`DownloadLock#lock_or_wait` polls the existing non-blocking `flock`
every 0.1s instead of giving up on the first failure. At 0.1s the wait
costs about 0.15% of one core per waiting download, dominated by syscall
overhead rather than real work, so the interval buys responsiveness for
no meaningful CPU. The wait is capped at 3 minutes, or at the caller's
remaining `timeout:` budget when that is shorter, so waiting on the lock
can't blow a deadline the caller asked for. An hour is always going to
be too long, and giving up beats waiting much longer because
`RetryableDownload` preserves the `.incomplete` file, so a retry resumes
the holder's partial download via `--continue-at`.

The warning is suppressed when the caller is already rendering progress.
`HOMEBREW_DOWNLOAD_CONCURRENCY` defaults to `cores * 2`, and above 1
`DownloadQueue#fetch` drives a cursor-addressed redraw whose arithmetic
assumes one line per download, so an unscheduled write from a pool
worker desyncs it. `OperationInProgressError` also takes an optional
`waited:` now, because telling someone to "wait for it to finish or
terminate it to continue" after three minutes of waiting is not useful.
The message is unchanged for every existing caller.

Mutation testing found `lock_file_spec.rb` passed with the inode/unlink
recheck neutered and with `ignore_interrupts` removed from `#lock`, so
cover both. Deferring the interrupt itself can't be asserted in-process,
since RSpec owns the `INT` handler that `ignore_interrupts` traps and
`Thread#raise` bypasses `trap`, so the wrapper's presence is asserted
instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants