Rebuilding and running an executable installs every executable configured

[jeenuv]

General summary/comments

When only a single executable is required to be (re)built, building and running that executable installs all other binaries configured.

Steps to reproduce

1. Start a new stack project;
2. Configure more than one (preferably many) executable;
3. Build all executables;
4. Make trivial modification to the Main.hs for any executable;
5. Run stack run that_executable;
6. Stack installs all exes, even though only a single executable was rebuilt.

Expected

Since the source changes (step 4 above) affected only a single executable, the reasonable expectation from Stack is that it rebuilds and run that executable alone—do nothing more.

Actual

Instead, as mentioned, Stack installs every executable whenever there is any changes made. This is wasteful and unnecessary.

The back story: was trying my hand at AOC 2022. I configured each day's puzzle as a separate binary, for example, under app/day12_1 and app/day12_2. Due to the incremental nature of development, every time I make changes to day12_2, for example, every other executable is installed before day12_2 is run. That is 12x2, 24 binary installs when a single executable source was changed! Needless to say, this increases the turn around time. And it gets worse with every passing day, when I keep adding more and more executables.

My current work around is to configure one executable at a time, and comment out the rest. I feel shouldn't have to do this. Stack is acting suboptimally. Executables are supposed to be independent of each other; changing one shouldn't cause any action to be taken upon the rest (unless their common dependency changes, which in this case, it didn't).

I believe I've tried all command line options to get Stack to break out of it, but without any success.

Stack version

Version 2.9.1.

Method of installation

Official binary, downloaded via haskellstack.org or from Stack's repository

[jeenuv]

I noticed this line from the stack run ... output:

Building all executables for `aoc2022' once. After a successful build of all of them, only specified executables will be rebuilt.

Other binaries are not built, though, because none of them had any relevant source changes. Yet, Stack proceeds to install the binaries as if they were rebuilt.

In any case, I don't get the need for rebuilding all binaries in this case. Perhaps someone can explain?

[mpilgrem]

@jeenuv, thanks for reporting. I will investigate. As I understand it, the origin of Stack's Building all executables ... once. was a feature/bug of the underlying Cabal library on which Stack is built - see #3229. I don't know if the Cabal library has moved on in this regard since 2017.

[mpilgrem]

@jeenuv, I can't reproduce this, on Windows. I have a project that looks a lot like a plain stack new, except that I have directory appA with one executable (named multiExeA) and a directory appB with another executable (multiExeB).

First time out, stack run multiExeA builds, and installs within Stack's working directory, all executables, as expected given the solution to #3229. After that, stack run multiExeA just runs the executable. If I edit the source code of multiExeB, stack run multiExeA still just runs the executable; modifying the B executable source code does not prompt a rebuild, if I stick to running the A executable.

[jeenuv]

@mpilgrem, thanks for checking. I forgot to mention that I'm on Linux, but I didn't think that'd matter.

However, this is my reproducing this afresh:

Ran stack new foobar, and setup the app directory with more than one exe. FWIW, this is the snippet from package.yaml:

executables:
  foobar1:
    main:                foobar1/Main.hs
    source-dirs:         app
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - foobar

  foobar2:
    main:                foobar1/Main.hs
    source-dirs:         app
    ghc-options:
    - -threaded
    - -rtsopts
    - -with-rtsopts=-N
    dependencies:
    - foobar

First run:

$ stack run foobar1
Building all executables for `foobar' once. After a successful build of all of them, only specified executables will be rebuilt.
foobar> configure (lib + exe)
Configuring foobar-0.1.0.0...
foobar> build (lib + exe)
Preprocessing library for foobar-0.1.0.0..
Building library for foobar-0.1.0.0..
[1 of 2] Compiling Lib
[2 of 2] Compiling Paths_foobar
Preprocessing executable 'foobar1' for foobar-0.1.0.0..
Building executable 'foobar1' for foobar-0.1.0.0..
[1 of 2] Compiling Main
[2 of 2] Compiling Paths_foobar
Linking .stack-work/dist/x86_64-linux-tinfo6/Cabal-3.6.3.0/build/foobar1/foobar1 ...
Preprocessing executable 'foobar2' for foobar-0.1.0.0..
Building executable 'foobar2' for foobar-0.1.0.0..
[1 of 2] Compiling Main
[2 of 2] Compiling Paths_foobar
Linking .stack-work/dist/x86_64-linux-tinfo6/Cabal-3.6.3.0/build/foobar2/foobar2 ...
foobar> copy/register
Installing library in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/lib/x86_64-linux-ghc-9.2.5/foobar-0.1.0.0-A8bmbQuRQBKIj644PEGzbW
Installing executable foobar1 in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/bin
Installing executable foobar2 in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/bin
Registering library for foobar-0.1.0.0..
someFunc

Nothing surprising here, but then:

$ stack run foobar2
foobar-0.1.0.0: unregistering (components added: exe:foobar2)
foobar> build (lib + exe)
Preprocessing library for foobar-0.1.0.0..
Building library for foobar-0.1.0.0..
Preprocessing executable 'foobar2' for foobar-0.1.0.0..
Building executable 'foobar2' for foobar-0.1.0.0..
foobar> copy/register
Installing library in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/lib/x86_64-linux-ghc-9.2.5/foobar-0.1.0.0-A8bmbQuRQBKIj644PEGzbW
Installing executable foobar1 in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/bin
Installing executable foobar2 in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/bin
Registering library for foobar-0.1.0.0..
someFunc

Notice the installations for both foobar1 and foobar2. To confirm, there were no source changes. If I run foobar1 again:

$ stack run foobar1
foobar-0.1.0.0: unregistering (components added: exe:foobar1)
foobar> build (lib + exe)
Preprocessing library for foobar-0.1.0.0..
Building library for foobar-0.1.0.0..
Preprocessing executable 'foobar1' for foobar-0.1.0.0..
Building executable 'foobar1' for foobar-0.1.0.0..
foobar> copy/register
Installing library in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/lib/x86_64-linux-ghc-9.2.5/foobar-0.1.0.0-A8bmbQuRQBKIj644PEGzbW
Installing executable foobar1 in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/bin
Installing executable foobar2 in /home/jeenu/tmp/foobar/.stack-work/install/x86_64-linux-tinfo6/bdb8424f31a4b32f29f6c8cfe7b7a748e5d0364100b14e0dc727f95ea0f0ba4c/9.2.5/bin
Registering library for foobar-0.1.0.0..
someFunc

I get rebuilding and wasteful installations if I switch between running foobar1 and foobar2 without any source changes.

FWIW, there's a similar reproducing from a Reddit thread.

I hope you'd be able to reproduce as above.

[mpilgrem]

@jeenuv, on the two aspects - (1) 'building' and (2) 'installing' (actually, copy/register) - with stack run foobar2, you can see that nothing is actually being built (because it has already been built). If something was actually being built you would have Cabal library output like:

[2 of 2] Compiling Main

I think the Cabal library output Building executable 'foobar2' for foobar-0.1.0.0.. should be read as: Beginning to turn to building executable 'foobar2' for foobar-0.1.0.0, if necessary..

On the copy/register step, if you use the --verbose flag, you will see that step involves Stack executing a single Cabal Setup.hs copy-type command, something like (on Windows; Linux may differ because the Windows version of Stack uses hashes to keep paths short):

Cabal-simple_sDt42OhJ_3.6.3.0_ghc-9.2.5.exe --verbose=1 --builddir=.stack-work\dist\8a54c84f copy

I understand from the documentation for the Cabal library that Setup.hs copy simply copies all files from the build directory into the installation directory. I think that is why files are copied for all executables that have been previously built. I assume it is 'cheaper' to simply copy everything than to check executable-by-executable if something already exists in the installation directory and only move a copy there if it does not already exist.

[mpilgrem]

Investigating this has, however, revealed something else that is odd on Windows. I've opened a separate issue for that.

[jeenuv]

nothing is actually being built (because it has already been built).

I'd have thought so, but having used --verbose, as you suggested, I see (with no source changes):

2022-12-18 11:29:07.758634: [info] foobar> build (lib + exe)
2022-12-18 11:29:07.759005: [debug] Run process within /tmp/foobar/: /home/jeenu/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_mPHDZzAJ_3.6.3.0_ghc-9.2.5 --verbose=1 --builddir=.stack-work/dist/x86_64-linux-tinfo6/Cabal-3.6.3.0 build lib:foobar exe:foobar1 --ghc-options " -fdiagnostics-color=always"

If I'm reading this right, there's an attempt to build it; perhaps no compilation occurs thereafter, because there were no source changes.

This is different to how make would have done it: compare the time stamps, and only then decide to invoke the build command. Here, it seems like the build command is always invoked, and the rest is left at the mercy of the build command as to whether or not to produce the binary.

Then I see copy command is being invoked, which, as the documentation suggests, installs all binaries.

I get the feeling that Stack and the underlying cabal system is operating at different granularity: I, as a Stack user, am asking to only run a single binary (and build if necessary). Stack's translation of that into Cabal tuns out to be "build and install everything".

Bringing this to back to my original problem: the install everything approach is workable as long as there are only a couple of binaries involved. If there happens to be more (as with my use case), the situation proves unworkable without resorting to commenting stuff out from package.yaml. As uncommon as this case may be, to my mind, the exhibited behaviour is very suboptimal and wasteful. Even worse, the installations can be triggered just by choosing a different binary to run—no source changes required anywhere!

But then, if Stack is reliant on cabal for these matters, I don't know if there will be a fix.

What would have been a relief is if Stack would tell me where the binary location is, by some variant of stack path. If so, I can run stack build foobar1 and then run the binary directly, without involving stack run, which is the gatekeeper here.

[mpilgrem]

I don't think stack build will help. It does the same thing (in terms of copying from the build directory to the installation directory) as stack run when it comes to the copy/register step. For example (using stack ide targets to remind myself of the format for specific target components):

❯ stack ide targets
multiExe:lib
multiExe:exe:multiExeA
multiExe:exe:multiExeB
multiExe:test:multiExe-test

❯ stack build multiExe:exe:multiExeB
multiExe-0.1.0.0: unregistering (components added: exe:multiExeB)
multiExe> build (lib + exe)
Preprocessing library for multiExe-0.1.0.0..
Building library for multiExe-0.1.0.0..
Preprocessing executable 'multiExeB' for multiExe-0.1.0.0..
Building executable 'multiExeB' for multiExe-0.1.0.0..
multiExe> copy/register
Installing library in D:\Users\mike\Code\Haskell\multiExe\.stack-work\install\275f83a7\lib\x86_64-windows-ghc-9.2.5\multiExe-0.1.0.0-4bO8x8ZwRvUFOzjMHcfUPb
Installing executable multiExeB in D:\Users\mike\Code\Haskell\multiExe\.stack-work\install\275f83a7\bin
Installing executable multiExeA in D:\Users\mike\Code\Haskell\multiExe\.stack-work\install\275f83a7\bin

With stack build you can get the full picture of what is going on at both the Stack end and the Cabal library end by asking for complete verbosity: stack --verbose build multiExe:exe:multiExeB --cabal-verbosity 3.

I am not across where the boundary between Stack and the Cabal library falls exactly, but I suspect that Stack hands over to the Cabal library to decide what, if anything, needs to be done to 'build'. I am still trying to work out where in Stack's code base the copy/register step is set up. Cabal's Setup.hs copy command does allow components to be specified:

❯ stack exec -- runghc setup.hs copy --help
Copy the files of all/specific components to install locations.

Usage: setup.hs copy [FLAGS]
   or: setup.hs copy COMPONENTS [FLAGS]

Components encompass executables and libraries. Does not call register, and
allows a prefix at install time. Without the --destdir flag, configure
determines location.

Flags for copy:
 -h, --help                     Show this help text
 -v, --verbose[=n]              Control verbosity (n is 0--3, default
                                verbosity level is 1)
 --builddir=DIR                 The directory where Cabal puts generated build
                                files (default dist)
 --destdir=DIR                  directory to copy files to, prepended to
                                installation directories

Examples:
  setup.hs copy               All the components in the package
  setup.hs copy foo           A component (i.e. lib, exe, test suite)

[jeenuv]

I don't think stack build will help. It does the same thing (in terms of copying from the build directory to the installation directory) as stack run when it comes to the copy/register step

You're right - what I may have tried is building the same binary over and over, which of course doesn't do anything. If I switch over to building another binary, it does the full shebang indeed.

Cabal's Setup.hs copy command does allow components to be specified:

That sounds like good news?

[mpilgrem]

Some notes for my own information:

Stack.Build.Execute.singleBuild has (as part of realBuild):

let shouldCopy = not isFinalBuild && (hasLibrary || hasInternalLibrary || hasExecutables)
when shouldCopy $ withMVar eeInstallLock $ \() -> do
  announce "copy/register"
  eres <- try $ cabal KeepTHLoading ["copy"]

isFinalBuid is an argument of singleBuild (-- ^ Is this a final build?) and distinguishes 'build' tasks from 'final' tasks (see Stack.Build.Execute.toActions). The Haddock documentation for singleBuild explains:

-- * Registers the library and copies the built executables into the
--   local install directory. Note that this is literally invoking Cabal
--   with @copy@, and not the copying done by @stack install@ - that is
--   handled by 'copyExecutables'.

[isovector]

I've just run into this, in a big (proprietary) project with ~15 executables. The bug therefore has a significant impact on our CI times. Is there any hope of a fix?

[mpilgrem]

I think progress on this issue will be related to #6377. That is because (extract):

stack --resolver ghc-8.0.2 exec --no-ghc-package-path -- runghc Setup.hs copy --help

Warning: Stack's support of Cabal versions below 2.2.0.0 is deprecated and may be
         removed from the next version of Stack. Cabal version 1.24.2.0 was found.
         Consider using a resolver that is lts-12.0 or later or nightly-2018-03-13
         or later.
Copy the files into the install locations.

Usage: Setup.hs copy [FLAGS]
...

but by GHC 8.4.1 and Cabal-2.2.0.0 the Cabal (library) copy command can specify components (extract):

❯ stack --resolver ghc-8.4.1 exec --no-ghc-package-path -- runghc Setup.hs copy --help
Copy the files of all/specific components to install locations.

Usage: Setup.hs copy [FLAGS]
   or: Setup.hs copy COMPONENTS [FLAGS]

Components encompass executables and libraries. Does not call register, and
allows a prefix at install time. Without the --destdir flag, configure
determines location.