/nix will not be writable on macOS Catalina

[mroi]

This is not a short term bug, but it will become an issue when macOS Catalina is released this fall. macOS is now split across two volumes (system and data) with a read-only system volume. This means that /nix will no longer be writable.

Some more information can be found in the related WWDC talk and some session notes people took from a Q&A.

Summary: the system volume, which is mounted at / will become non-writable. Some directories that need to be writable are connected via firmlinks (an Apple invention) to the data volume. /nix is not among these locations, so with the release of macOS Catalina, this location is no longer an option.

I see two possible solutions:

1. I could try and file a bug to convince Apple to pre-install /nix as a firmlink to a writable location. I think this has limited success and Nix would then depend on Apple to not drop this link in a future release.
2. We could move Nix on macOS to a different default location. Possible locations are those that Apple chooses to pre-install as links to writable locations. Two examples are /usr/local and /opt, so we could move to /usr/local/nix or /opt/nix. I would hope that these locations are common enough so that Apple would not drop them in the future.

I wanted to raise this issue early, before it becomes a problem for users. If this issue tracker is not the right place, please feel free to move this discussion elsewhere. I would also be available for testing any potential solution, since I have access to a macOS Catalina beta.

[edolstra]

Realistically, this may force us to drop support for macOS. Using a different store location for macOS would require a separate binary cache and a separate Hydra instance, and there would be no guarantee that the new location wouldn't break in the future.

I did notice in the talk that it's still possible to make the system volume writable, though not in a persistent way.

[mroi]

Speaking for myself: I would be quite sad if macOS support would go away.
I will file a bug with Apple and see what happens. Cannot hurt.

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

[jwiegley]

@edolstra Before canceling support, we can ask the community if it's willing to host its own binary cache and Hydra instance, just as we've enlisted a host of volunteers to keep nixpkgs at near parity on Darwin. Having the answer be, "This doesn't easily fit our model so we won't support Mac" would be an unfortunate response.

[mroi]

I think no such decision should be made lightly, but it is definitely a drastic change on Apple’s part.

[edolstra]

Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.

Maybe you can use this to create a /nix symlink to another volume, which would hopefully persist across reboots?

[mroi]

That was my first try to get it temporarily working again. But nix-env complains then:

error: cannot open connection to remote store 'daemon': the path '/nix' is a symlink; this is not allowed for the Nix store and its parent directories

If that error can be safely ignored, I think this would be a workaround worth exploring. The way I understand things, anything you place in the root folder would persist until the next system update.

[LnL7]

There's no real reason for the nix store to be on the system volume, with apfs it's easy to create a separate volume and mount it to /nix. But perhaps there's something I'm missing because I don't understand the need for these firmlinks, the equivalent is possible with mounts similar to how the store is mounted readonly on NixOS.

[edolstra]

@mroi You can set NIX_IGNORE_SYMLINK_STORE=1 to disable that check. (This should probably be turned into a nix.conf option.)

[jwiegley]

@edolstra Great idea, added #2926.

[mroi]

@edolstra Thanks for the tip, this is a useful workaround.

But future Nix-users on macOS still need to find a reliable way to inject this symlink into the read-only root directory. This probably involves some advanced steps like rebooting into recovery mode after every system upgrade.

@LnL7 The same would be true for /nix as a mount point, because we would need to create this directory on the read-only volume.

[LnL7]

Sure but if Apple doesn't want to provide some sort of solution for one of these cases there will be no alternative to changing the default prefix. Which is arguably is not worth it for a secondary platform that's continuously moving further away from UNIX.

Is there an overview of the full volume layout somewhere? /usr/local and /Users are used as examples but I can't imagine that's everything.

[kevingriffin]

@mroi How have you been testing this? In my testing, I was able to install nix, but I assumed it was due to the comment in the presentation that "system volume is writeable in the developer preview". I'd also like to help test, but was able to get nix installed:

kevingriffin@KevinnoiMac /nix % uname -a
Darwin KevinnoiMac.local 19.0.0 Darwin Kernel Version 19.0.0: Fri May 24 17:36:10 PDT 2019; root:xnu-6041.0.0.111.5~1/RELEASE_X86_64 x86_64
kevingriffin@KevinnoiMac /nix % ls /nix
store	var
kevingriffin@KevinnoiMac /nix % nix-env -iA nixpkgs.hello
installing 'hello-2.10'
these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked):
  /nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10
copying path '/nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10' from 'https://cache.nixos.org'...
building '/nix/store/iwcsfsh5hxyzpymnkp1r54nb3zzpd7mg-user-environment.drv'...
created 2 symlinks in user environment```

[matthewbauer]

It looks like you can still access the /nix directory after the update, it's just in /System/Volumes/Data/nix. Too bad there's no chroot equivalent on macOS.

[matthewbauer]

sudo mount -uw / && sudo ln -s /System/Volumes/Data/nix /nix works for me! This is on SIP even! Having the installer handle this might work.