From 40e5e3f07fb3ace5195a19a26b07db40c452abca Mon Sep 17 00:00:00 2001 From: Jacek Galowicz Date: Tue, 27 Aug 2024 11:10:57 +0200 Subject: [PATCH 1/2] Add tutorial about binary cache setup --- source/guides/recipes/post-build-hook.md | 1 + source/tutorials/nixos/binary-cache-setup.md | 238 +++++++++++++++++++ source/tutorials/nixos/index.md | 5 + 3 files changed, 244 insertions(+) create mode 100644 source/tutorials/nixos/binary-cache-setup.md diff --git a/source/guides/recipes/post-build-hook.md b/source/guides/recipes/post-build-hook.md index 700240ffbc..8b1525f590 100644 --- a/source/guides/recipes/post-build-hook.md +++ b/source/guides/recipes/post-build-hook.md @@ -1,3 +1,4 @@ +(post-build-hooks)= # Setting up post-build hooks This guide shows how to use the Nix [`post-build-hook`](https://nix.dev/manual/nix/2.22/command-ref/conf-file#conf-post-build-hook) configuration option to automatically upload build results to an [S3-compatible binary cache](https://nix.dev/manual/nix/2.22/store/types/s3-binary-cache-store). diff --git a/source/tutorials/nixos/binary-cache-setup.md b/source/tutorials/nixos/binary-cache-setup.md new file mode 100644 index 0000000000..f0c84b1ac9 --- /dev/null +++ b/source/tutorials/nixos/binary-cache-setup.md @@ -0,0 +1,238 @@ +--- +myst: + html_meta: + "description lang=en": "Setting up a binary cache to reuse builds" + "keywords": "Nix, caching" +--- + +# Setting up a binary cache + +A binary cache stores prebuilt [Nix store objects](https://nix.dev/manual/nix/latest/store/store-object) and provides them to other machines over the network. +This way, one machine can download prebuilt packages from another instead of rebuilding them. +Any machine with a Nix store can be a binary cache for other machines. + +## Introduction + +In this tutorial you will set up a Nix binary cache that will serve store objects from a NixOS machine over HTTP or HTTPS. + +### What will you learn? + +You'll learn how to +- Set up signing keys for your cache +- Enable the right services on the NixOS machine serving the cache +- Check that the setup works as intended + +### Prerequisites + +- A machine that runs NixOS + + If you're new to NixOS, learn about the [](module-system-deep-dive) and [](nixos-vms) to configure your first system. + +- (optional) A public IP and DNS domain + + If you don't host yourself, check [NixOS friendly hosters](https://wiki.nixos.org/wiki/NixOS_friendly_hosters) on the NixOS Wiki. + Follow the tutorial on [](provisioning-remote-machines) to deploy your NixOS configuration. + +For a cache on a local network, we assume: +- The hostname is `cache` (replace it with yours, or an IP address) +- The host serves store objects via HTTP on port 80 (this is the default) + +For a publicly accessible cache, we assume: +- The domain name is `cache.example.com` (replace it with yours) +- The host serves store objects via HTTPS on port 443 (this is the default) + +### How long will it take? + +~10 minutes + +## Set up services + +Create a new NixOS configuration module file `binary-cache.nix` in the same folder where your `configuration.nix` is: + +```{code-block} nix +{ config, ... }: + +{ + services.nix-serve = { + enable = true; + secretKeyFile = "/var/secrets/cache-private-key.pem"; + }; + + services.nginx = { + enable = true; + recommendedProxySettings = true; + virtualHosts."cache.example.com" = { + locations."/".proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}"; + }; + }; + + networking.firewall.allowedTCPPorts = [ + config.services.nginx.defaultHTTPListenPort + ]; +} +``` + +The attributes under `services.nix-serve.*` install and enable the binary cache service. +See also [the `services.nix-serve` options reference on search.nixos.org][nix-serve-options]. + +`nix-serve` does not serve via IPv6 and does not support SSL/HTTPS. +For this reason, this tutorial configures `services.nginx.*`. +Nginx listens on the HTTP port and forwards all connections to `nix-serve`. +There is an optional HTTPS section in the end of this tutorial. + +Add the new NixOS module to your existing `configuration.nix`: + +```{code-block} nix +{ config, ... }: + +{ + imports = [ + ./binary-cache.nix + ]; + + # ... +} +``` + +Activate the new configuration as root: + +```shell-session +nixos-rebuild switch +``` + +The binary cache daemon will report errors because there is no secret key file, yet. + +## Generate key pair + +A pair of private and public keys is important to ensure that the store objects in the cache can be trusted. +The private key is either used to sign store objects right after are built, or the binary cache signs the store objects while serving them. + +To generate a key pair for the binary cache, replace the example hostname `cache.example.com` with your hostname: + +```shell-session +mkdir /var/secrets +cd /var/secrets +nix-store --generate-binary-cache-key cache.example.com cache-private-key.pem cache-public-key.pem +``` + +`cache-private-key.pem` will be used by the binary cache daemon to sign the binaries as they are served. + +Up until now, the binary cache daemon was in a restart loop due to the missing secret key file. +It should now work correctly, which can be checked with the command `systemctl status nix-serve.service`. + +Distribute `cache-public-key.pem` to all machines that should be able to access the binary cache. + +:::{note} +The location `/var/secrets/` for keeping the key pair is not a requirement and can be chosen differently. +::: + +## Test availability + +The setup is complete. +The following steps check if everything is set up correctly and may help identifying problems. + +### 1. Check general availability + +Test if the binary cache, HTTP(s) reverse proxy, and firewall rules work correctly by running this command on a client machine: + +```shell-session +$ curl http://cache.example.com/nix-cache-info +StoreDir: /nix/store +WantMassQuery: 1 +Priority: 30 +``` + +### 2. Check binary signing + +Test if binaries are signed correctly with the following two steps. + +On the binary cache host, run this command: + +```shell-session +$ curl "http://cache.example.com/$(nix-build '' -A pkgs.hello | awk -F '/' '{print $4}' | awk -F '-' '{print $1}').narinfo" | grep "Sig: " +... +Sig: build01.nix-consulting.de:GyBFzocLAeLEFd0hr2noK84VzPUw0ArCNYEnrm1YXakdsC5FkO2Bkj2JH8Xjou+wxeXMjFKa0YP2AML7nBWsAg== +``` + +It is important that the output contains this line prefixed with `Sig:` with the previously generated public key. + +:::{note} +This one-liner builds a package and extracts its hash to calculate a valid URL in the cache. +(From `/nix/store/--`, it derives `http://cache.example.com/.narinfo`.) +Querying this URL exposes information about the cached nix store path. +::: + +## Outlook + +You can now distribute the hostname and public key to anyone who wants access to your new binary cache. +Configure clients with [this guide about binary cache client configuration](custom-binary-cache). + +If your binary cache is already a [remote build machine][remote-build-machine], it will serve all binaries in its nix store. + +Other hosts can be configured to automatically push binaries to the binary cache using [the `post-build-hook` feature (Guide)](post-build-hooks). + +## Next steps + +- [](custom-binary-cache) +- [](post-build-hooks) + +To save space, please refer to the following NixOS configuration attributes: + +- [`nix.gc.*`][nix-gc]: Automatic periodic garbage collection Settings +- [`nix.optimise.*`][nix-optimise]: Automatic periodic nix store optimisation + +### Serving via HTTPS from a public address + +If the binary cache is publicly accessible, it is possible to enforce HTTPS with [Let's encrypt](https://letsencrypt.org/) SSL certificates. +Edit your `binary-cache.nix` like this and make sure to replace the example URL and mail address with yours: + +```{code-block} nix +{ config, ... }: +{ + # ... + + services.nginx = { + # ... + virtualHosts."cache.example.com" = { + enableACME = true; + forceSSL = true; + # ... + }; + }; + + networking.firewall.allowedTCPPorts = [ + config.services.nginx.defaultHTTPListenPort + config.services.nginx.defaultSSLListenPort + ]; + + security.acme = { + acceptTerms = true; + certs = { + "cache.example.com".email = "you@example.com"; + }; + }; +} +``` + +Rebuild the system to activate these changes. + + +## Alternatives + +- [Cachix](https://www.cachix.org): Nix Binary Cache as a Service +- Amazon S3: Nix supports pushing to and pulling from S3 buckets (see [Nix manual about S3][nix-s3]) +- Tigris: An alternative to S3 +- Cloudflare R2: Another alternative to S3 +- [attic](https://github.com/zhaofengli/attic): Alternative to `nix-serve` (open source) + +## References + +- [Nix Manual on HTTP Binary Cache Store](https://nix.dev/manual/nix/latest/store/types/http-binary-cache-store) +- [`services.nix-serve` module definition](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/networking/nix-serve.nix) + +[nix-serve-options]: https://search.nixos.org/options?query=services.nix-serve +[nginx-ssl]: https://nixos.org/manual/nixos/stable/#module-security-acme +[nixos-ipv6]: https://nixos.org/manual/nixos/stable/#sec-ipv6 +[nix-gc]: https://search.nixos.org/options?query=nix.gc. +[nix-optimise]: https://search.nixos.org/options?query=nix.optimise. +[remote-build-machine]: https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds diff --git a/source/tutorials/nixos/index.md b/source/tutorials/nixos/index.md index be0d3dc60a..0e044acd20 100644 --- a/source/tutorials/nixos/index.md +++ b/source/tutorials/nixos/index.md @@ -8,6 +8,10 @@ Learn how to configure, test, and install or deploy NixOS. - [](building-bootable-iso-image) - [](building-and-running-docker-images) +## Nix related services + +- [](binary-cache-setup) + ## Integration testing - [](integration-testing-using-virtual-machines) @@ -29,4 +33,5 @@ integration-testing-using-virtual-machines.md provisioning-remote-machines.md installing-nixos-on-a-raspberry-pi.md deploying-nixos-using-terraform.md +binary-cache-setup.md ``` From efbbb127a274bff176a86f9e2a11f07c4f733c75 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Sat, 31 Aug 2024 21:40:12 +0200 Subject: [PATCH 2/2] review pass Co-authored-by: wamirez --- source/guides/recipes/add-binary-cache.md | 4 + source/tutorials/nixos/binary-cache-setup.md | 207 +++++++++--------- .../nixos/provisioning-remote-machines.md | 6 + 3 files changed, 113 insertions(+), 104 deletions(-) diff --git a/source/guides/recipes/add-binary-cache.md b/source/guides/recipes/add-binary-cache.md index e454c201d9..e360b63b91 100755 --- a/source/guides/recipes/add-binary-cache.md +++ b/source/guides/recipes/add-binary-cache.md @@ -3,6 +3,10 @@ Nix can be configured to use a binary cache with the [`substituters`](https://nix.dev/manual/nix/2.21/command-ref/conf-file.html#conf-substituters) and [`trusted-public-keys`](https://nix.dev/manual/nix/2.21/command-ref/conf-file.html#conf-trusted-public-keys) settings, either exclusively or in addition to cache.nixos.org. +:::{tip} +Follow the tutorial to [set up an HTTP binary cache](setup-http-binary-cache) and create a key pair for signing store objects. +::: + For example, given a binary cache at `https://example.org` with public key `My56...Q==%`, and some derivation in `default.nix`, make Nix exclusively use that cache once by passing [settings as command line flags](https://nix.dev/manual/nix/2.21/command-ref/conf-file#command-line-flags): ```shell-session diff --git a/source/tutorials/nixos/binary-cache-setup.md b/source/tutorials/nixos/binary-cache-setup.md index f0c84b1ac9..3372ce5d8c 100644 --- a/source/tutorials/nixos/binary-cache-setup.md +++ b/source/tutorials/nixos/binary-cache-setup.md @@ -1,15 +1,15 @@ --- myst: html_meta: - "description lang=en": "Setting up a binary cache to reuse builds" + "description lang=en": "Setting up a binary cache for store objects" "keywords": "Nix, caching" --- -# Setting up a binary cache +(setup-http-binary-cache)= +# Setting up an HTTP binary cache -A binary cache stores prebuilt [Nix store objects](https://nix.dev/manual/nix/latest/store/store-object) and provides them to other machines over the network. -This way, one machine can download prebuilt packages from another instead of rebuilding them. -Any machine with a Nix store can be a binary cache for other machines. +A binary cache stores pre-built [Nix store objects](https://nix.dev/manual/nix/latest/store/store-object) and provides them to other machines over the network. +Any machine with a Nix store can be an binary cache for other machines. ## Introduction @@ -17,16 +17,18 @@ In this tutorial you will set up a Nix binary cache that will serve store object ### What will you learn? -You'll learn how to +You'll learn how to: - Set up signing keys for your cache - Enable the right services on the NixOS machine serving the cache - Check that the setup works as intended -### Prerequisites +### What do you need? -- A machine that runs NixOS +- A working [Nix installation]() on your local machine - If you're new to NixOS, learn about the [](module-system-deep-dive) and [](nixos-vms) to configure your first system. +- SSH access to a NixOS machine to use as a cache + + If you're new to NixOS, learn about the [module system](module-system-deep-dive) and configure your first system with [](nixos-vms). - (optional) A public IP and DNS domain @@ -43,11 +45,11 @@ For a publicly accessible cache, we assume: ### How long will it take? -~10 minutes +- 25 minutes ## Set up services -Create a new NixOS configuration module file `binary-cache.nix` in the same folder where your `configuration.nix` is: +For the NixOS machine hosting the cache, create a new configuration module in `binary-cache.nix`: ```{code-block} nix { config, ... }: @@ -61,7 +63,7 @@ Create a new NixOS configuration module file `binary-cache.nix` in the same fold services.nginx = { enable = true; recommendedProxySettings = true; - virtualHosts."cache.example.com" = { + virtualHosts.cache = { locations."/".proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}"; }; }; @@ -72,15 +74,19 @@ Create a new NixOS configuration module file `binary-cache.nix` in the same fold } ``` -The attributes under `services.nix-serve.*` install and enable the binary cache service. -See also [the `services.nix-serve` options reference on search.nixos.org][nix-serve-options]. +The options under [`services.nix-serve`] configure the binary cache service. + +`nix-serve` doesn't support IPv6 or SSL/HTTPS. +The [`services.nginx`] options are used to set up a proxy, which does support IPv6, to handle requests to the hostname `cache`. + +[`services.nix-serve`]: https://search.nixos.org/options?query=services.nix-serve +[`services.nginx`]: https://search.nixos.org/options?query=services.nginx -`nix-serve` does not serve via IPv6 and does not support SSL/HTTPS. -For this reason, this tutorial configures `services.nginx.*`. -Nginx listens on the HTTP port and forwards all connections to `nix-serve`. -There is an optional HTTPS section in the end of this tutorial. +:::{important} +There is an [optional HTTPS section](https-binary-cache) at the end of this tutorial. +::: -Add the new NixOS module to your existing `configuration.nix`: +Add the new NixOS module to the existing machine configuration: ```{code-block} nix { config, ... }: @@ -94,145 +100,138 @@ Add the new NixOS module to your existing `configuration.nix`: } ``` -Activate the new configuration as root: +From your local machine, deploy the new configuration: ```shell-session -nixos-rebuild switch +nixos-rebuild switch --no-flake --target-host root@cache ``` +:::{note} The binary cache daemon will report errors because there is no secret key file, yet. +::: -## Generate key pair +## Generate a signing key pair -A pair of private and public keys is important to ensure that the store objects in the cache can be trusted. -The private key is either used to sign store objects right after are built, or the binary cache signs the store objects while serving them. +A pair of private and public keys is required to ensure that the store objects in the cache are authentic. + + To generate a key pair for the binary cache, replace the example hostname `cache.example.com` with your hostname: ```shell-session -mkdir /var/secrets -cd /var/secrets nix-store --generate-binary-cache-key cache.example.com cache-private-key.pem cache-public-key.pem ``` `cache-private-key.pem` will be used by the binary cache daemon to sign the binaries as they are served. +Copy it to the location configured in `services.nix-serve.secretKeyFile` on the machine hosting the cache: + +```shell-session +scp cache-private-key.pem root@cache:/var/secrets/cache-private-key.pem +``` Up until now, the binary cache daemon was in a restart loop due to the missing secret key file. -It should now work correctly, which can be checked with the command `systemctl status nix-serve.service`. +Check that it now works correctly: -Distribute `cache-public-key.pem` to all machines that should be able to access the binary cache. +```shell-session +ssh root@cache systemctl status nix-serve.service +``` -:::{note} -The location `/var/secrets/` for keeping the key pair is not a requirement and can be chosen differently. +:::{important} +[](custom-binary-cache) using `cache-public-key.pem` on your local machine. ::: ## Test availability -The setup is complete. -The following steps check if everything is set up correctly and may help identifying problems. +The following steps check if everything is set up correctly and may help with identifying problems. -### 1. Check general availability +### Check general availability -Test if the binary cache, HTTP(s) reverse proxy, and firewall rules work correctly by running this command on a client machine: +Test if the binary cache, reverse proxy, and firewall rules work as intended by querying the cache: ```shell-session -$ curl http://cache.example.com/nix-cache-info +$ curl http://cache/nix-cache-info StoreDir: /nix/store WantMassQuery: 1 Priority: 30 ``` -### 2. Check binary signing - -Test if binaries are signed correctly with the following two steps. +### Check store object signing -On the binary cache host, run this command: +To test if store objects are signed correctly, inspect the metadata of a sample derivation. +On the binary cache host, build the `hello` package and get the `.narinfo` file from the cache: ```shell-session -$ curl "http://cache.example.com/$(nix-build '' -A pkgs.hello | awk -F '/' '{print $4}' | awk -F '-' '{print $1}').narinfo" | grep "Sig: " +$ hash=$(nix-build '' -A pkgs.hello | awk -F '/' '{print $4}' | awk -F '-' '{print $1}') +$ curl "http://cache/$hash.narinfo" | grep "Sig: " ... -Sig: build01.nix-consulting.de:GyBFzocLAeLEFd0hr2noK84VzPUw0ArCNYEnrm1YXakdsC5FkO2Bkj2JH8Xjou+wxeXMjFKa0YP2AML7nBWsAg== +Sig: cache.example.org:GyBFzocLAeLEFd0hr2noK84VzPUw0ArCNYEnrm1YXakdsC5FkO2Bkj2JH8Xjou+wxeXMjFKa0YP2AML7nBWsAg== ``` -It is important that the output contains this line prefixed with `Sig:` with the previously generated public key. +Make sure that the output contains this line prefixed with `Sig:` and shows the public key you generated. -:::{note} -This one-liner builds a package and extracts its hash to calculate a valid URL in the cache. -(From `/nix/store/--`, it derives `http://cache.example.com/.narinfo`.) -Querying this URL exposes information about the cached nix store path. -::: +(https-binary-cache)= +### Serving the binary cache via HTTPS -## Outlook +If the binary cache is publicly accessible, it is possible to enforce HTTPS with [Let's Encrypt](https://letsencrypt.org/) SSL certificates. +Edit your `binary-cache.nix` like this and make sure to replace the example URL and mail address with yours: -You can now distribute the hostname and public key to anyone who wants access to your new binary cache. -Configure clients with [this guide about binary cache client configuration](custom-binary-cache). +```{code-block} diff + services.nginx = { + enable = true; + recommendedProxySettings = true; +- virtualHosts.cache = { ++ virtualHosts."cache.example.com" = { ++ enableACME = true; ++ forceSSL = true; + locations."/".proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}"; + }; + }; + ++ security.acme = { ++ acceptTerms = true; ++ certs = { ++ "cache.example.com".email = "you@example.com"; ++ }; ++ }; + + networking.firewall.allowedTCPPorts = [ + config.services.nginx.defaultHTTPListenPort ++ config.services.nginx.defaultSSLListenPort + ]; +``` -If your binary cache is already a [remote build machine][remote-build-machine], it will serve all binaries in its nix store. +Rebuild the system to deploy these changes: -Other hosts can be configured to automatically push binaries to the binary cache using [the `post-build-hook` feature (Guide)](post-build-hooks). +```shell-session +nixos-rebuild switch --no-flake --target-host root@cache.example.com +``` ## Next steps -- [](custom-binary-cache) -- [](post-build-hooks) - -To save space, please refer to the following NixOS configuration attributes: +If your binary cache is already a [remote build machine](https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds), it will serve all store objects in its Nix store. -- [`nix.gc.*`][nix-gc]: Automatic periodic garbage collection Settings -- [`nix.optimise.*`][nix-optimise]: Automatic periodic nix store optimisation +- [](custom-binary-cache) using the binary cache's hostname and the generated public key +- [](post-build-hooks) to upload store objects to the binary cache -### Serving via HTTPS from a public address +To save storage space, please refer to the following NixOS configuration attributes: -If the binary cache is publicly accessible, it is possible to enforce HTTPS with [Let's encrypt](https://letsencrypt.org/) SSL certificates. -Edit your `binary-cache.nix` like this and make sure to replace the example URL and mail address with yours: - -```{code-block} nix -{ config, ... }: -{ - # ... +- [`nix.gc`](https://search.nixos.org/options?query=nix.gc): Options for automatic garbage collection +- [`nix.optimise`](https://search.nixos.org/options?query=nix.optimise): Options for periodic Nix store optimisation - services.nginx = { - # ... - virtualHosts."cache.example.com" = { - enableACME = true; - forceSSL = true; - # ... - }; - }; - - networking.firewall.allowedTCPPorts = [ - config.services.nginx.defaultHTTPListenPort - config.services.nginx.defaultSSLListenPort - ]; - - security.acme = { - acceptTerms = true; - certs = { - "cache.example.com".email = "you@example.com"; - }; - }; -} -``` - -Rebuild the system to activate these changes. +## Alternatives +- The [SSH Store](https://nix.dev/manual/nix/latest/store/types/ssh-store), [Experimental SSH Store](https://nix.dev/manual/nix/latest/store/types/experimental-ssh-store), and the [S3 Binary Cache Store](https://nix.dev/manual/nix/latest/store/types/s3-binary-cache-store) can also be used to serve a cache. + There are many commercial providers for S3-compatible storage, for example: + - Amazon S3 + - Tigris + - Cloudflare R2 -## Alternatives +- [attic](https://github.com/zhaofengli/attic): Nix binary cache server backed by an S3-compatible storage provider -- [Cachix](https://www.cachix.org): Nix Binary Cache as a Service -- Amazon S3: Nix supports pushing to and pulling from S3 buckets (see [Nix manual about S3][nix-s3]) -- Tigris: An alternative to S3 -- Cloudflare R2: Another alternative to S3 -- [attic](https://github.com/zhaofengli/attic): Alternative to `nix-serve` (open source) +- [Cachix](https://www.cachix.org): Nix binary cache as a service ## References - [Nix Manual on HTTP Binary Cache Store](https://nix.dev/manual/nix/latest/store/types/http-binary-cache-store) -- [`services.nix-serve` module definition](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/services/networking/nix-serve.nix) - -[nix-serve-options]: https://search.nixos.org/options?query=services.nix-serve -[nginx-ssl]: https://nixos.org/manual/nixos/stable/#module-security-acme -[nixos-ipv6]: https://nixos.org/manual/nixos/stable/#sec-ipv6 -[nix-gc]: https://search.nixos.org/options?query=nix.gc. -[nix-optimise]: https://search.nixos.org/options?query=nix.optimise. -[remote-build-machine]: https://nix.dev/manual/nix/latest/advanced-topics/distributed-builds +- [`services.nix-serve` module options][`services.nix-serve`] +- [`services.nginx` module options][`services.nginx`] diff --git a/source/tutorials/nixos/provisioning-remote-machines.md b/source/tutorials/nixos/provisioning-remote-machines.md index 241aa7645a..ecd51f5212 100644 --- a/source/tutorials/nixos/provisioning-remote-machines.md +++ b/source/tutorials/nixos/provisioning-remote-machines.md @@ -285,6 +285,12 @@ nixos-rebuild switch --no-flake --target-host root@target-host `nixos-anywhere` is not needed any more, unless you want to change the disk layout. + +# Next steps + +- [](binary-cache-setup) +- [](post-build-hooks) + ## References - [`nixos-anywhere` project page][`nixos-anywhere`]