Aarch64 SD images missing pi-zero-2.dtb file require to boot.

[deftdawg]

[Apologies in advance if this is not the correct place to report this issue; there is basically no current documentation that I could find on how to boot and run nix on Raspberry Pi Zero 2]

Attempting to boot the Aarch64 SD card images built by Hydra on a Raspberry Pi Zero 2 fails because uboot cannot find gd19f48w8fwcbvb3g0pmfrlyj74zcnb0-linux-5.15.94-dtbs/broadcom/bcm2837-rpi-zero-2.dtb

Based on this line in master:

nixpkgs/pkgs/os-specific/linux/kernel/linux-rpi.nix

Line 78 in 270de46

copyDTB bcm2710-rpi-zero-2.dtb bcm2837-rpi-zero-2.dtb

It appears the file and a few others should be included, but for whatever reason (maybe this linux-rpi is not used or Hydra building off an different branch?) the file is not present in the image.

Here are the files actually present on the flashed SD card:

ls -1 /run/media/${USER}/NIXOS_SD/boot/nixos/gd19f48w8fwcbvb3g0pmfrlyj74zcnb0-linux-5.15.94-dtbs/broadcom
bcm2711-rpi-4-b.dtb
bcm2711-rpi-400.dtb
bcm2837-rpi-3-a-plus.dtb
bcm2837-rpi-3-b-plus.dtb
bcm2837-rpi-3-b.dtb
bcm2837-rpi-cm3-io3.dtb
bcm4908
northstar2
stingray

Steps To Reproduce

Steps to reproduce the behavior:

1. Download an AArch64 image from Hydra
2. Use etcher or dd to flash a micro sd card
3. Insert micro sd and boot a Pi Zero 2

Expected behavior

Expect it to boot to a nixOS shell

Workaround

It is possible to workaround the issue by mounting the micro SD on a operable Linux system by copying the RPi-3-b dtb file to the expected Pi Zero 2 file name:

cd /run/media/${USER}/NIXOS_SD/boot/nixos/gd19f48w8fwcbvb3g0pmfrlyj74zcnb0-linux-5.15.94-dtbs/broadcom/
cp bcm2837-rpi-3-b.dtb bcm2837-rpi-zero-2.dtb

Then unmount the SD and boot with your RPi Zero 2

Notify maintainers

Notify the 2 folks listed as having touched the RPi Zero 2 dtb line in linux-rpi.nix
@leo60228
@zhaofengli

[zhaofengli]

Could you try the new_kernel variant of the SD image? It should have the DTB which was added in 5.18. The Hydra images use the mainline kernel, and linux-rpi.nix carries the RPi Foundation kernel. Alternatively, you can try this expression if you want to use the Foundation kernel.

[deftdawg]

Thanks @zhaofengli!

Tried the new_kernel variant... still panics because of a typo.

The file expected by uboot is

/boot/nixos/dm53ipsnxinwkx2vmgmaxfh5rcl05hv9-linux-6.1.12-dtbs/broadcom/bcm2837-rpi-zero-2.dtb

The file provided by the image has -w in it's name:

/boot/nixos/dm53ipsnxinwkx2vmgmaxfh5rcl05hv9-linux-6.1.12-dtbs/broadcom/bcm2837-rpi-zero-2-w.dtb

Copying the file:

cd /run/media/${USER}/NIXOS_SD/boot/nixos/dm53ipsnxinwkx2vmgmaxfh5rcl05hv9-linux-6.1.12-dtbs/broadcom
sudo cp bcm2837-rpi-zero-2-w.dtb bcm2837-rpi-zero-2.dtb

Allows it to boot 👍 and wlan0 shows up (unlike with the RPi3 workaround above)... sweet!

[deftdawg]

Looks like I can get things to run, though the Pi Zero 2 needs help to keep from dying when running nixos-rebuild switch as it runs out of memory.

I've made a couple of tweaks:

1. Disable RPi Camera support and reduce VRAM to 16MB from the default of 128MB.

   edit /run/media/${USER}/FIRMWARE/config.txt:

   [pi02]
   kernel=u-boot-rpi3.bin
   # Give up VRAM for more Free System Memory
   # - Disable camera which automatically reserves 128MB VRAM
   start_x=0 
   # - Reduce allocation of VRAM to 16MB minimum for non-rotated (32MB for rotated)
   gpu_mem=16

2. Create a 2 GB swap file in root

   export SWAP_FILE=/swap
   # create a 2GB swap file
   if [ ! -f ${SWAP_FILE} ]; then
     sudo dd if=/dev/zero of=${SWAP_FILE} bs=100M count=20
     # format file as swap fs
     sudo mkswap -f ${SWAP_FILE}
   fi
   # activate swap file
   sudo swapon ${SWAP_FILE}
   echo

Still need to figure out permanent configuration, but at least with these I can rebuild without killing the Pi.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/raspberry-pi-compute-module-3-compatibility/16210/11

[deftdawg]

Looks like the problem is stale references in the uboot bins

[deftdawg@nixos:~/sda1]$ grep  bcm2837-rpi-zero-2.dtb *                                   
grep: u-boot-rpi3.bin: binary file matches
grep: u-boot-rpi4.bin: binary file matches

[deftdawg@nixos:~/sda1]$ strings u-boot-rpi3.bin | grep broadcom
... 
broadcom/bcm2837-rpi-zero-2.dtb
... 

This is based on the last new kernel sd image on hydra from Mar 20, 2023.

[deftdawg]

Looks like the problem was fixed in uboot 17 days ago... So we'd need a uboot update in nixos and a working hydra build

u-boot/u-boot@0e8c940

[plmercereau]

The PR with a newer version of uboot has been merged to nixpkgs, but it unfortunately does not solve the problem for me.

I created a small module that copies the right uboot file, creates a swap partition and allows to add options to the firmware configuration. Here's the gist.

Eventually my config looks like this:

{ config, lib, pkgs, ... }:
{
  imports = [
    <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64-installer.nix>
    ./sd-image.nix
  ];
  system.stateVersion = "23.11";

  # Pi Zero 2 struggles to work without swap
  sdImage.swap.enable = true;
  sdImage.extraFirmwareConfig = {
    # Give up VRAM for more Free System Memory
    # - Disable camera which automatically reserves 128MB VRAM
    start_x = 0;
    # - Reduce allocation of VRAM to 16MB minimum for non-rotated (32MB for rotated)
    gpu_mem = 16;
  };
  # bzip2 compression takes loads of time with emulation, skip it. Enable this if you're low on space.
  sdImage.compressImage = false;

  networking = {
    interfaces."wlan0".useDHCP = true;
    wireless = {
      enable = true;
      interfaces = [ "wlan0" ];
      networks = {
        "<ssid>" = {
          psk = "<ssid-key>";
        };
      };
    };
  };

  # Enable OpenSSH out of the box.
  services.sshd.enable = true;

  # NTP time sync.
  services.timesyncd.enable = true;

}

[alex-nt]

@plmercereau many thanks, I used your config and added it to my fork of nixpkgs here: https://github.com/NixOS/nixpkgs/compare/release-23.05...alex-nt:nixpkgs:feature-23.05-rpi?expand=1

Using sd-image-aarch64-new-kernel-no-zfs-installer.nix everything works! But whenever I try the rpi kernel via

boot = {
    kernelPackages = pkgs.linuxPackages_rpi3;
}

I loose wifi (wlan0 fails to init). I've been using a fork of nixpkgs since 2022-12 till now on RPI02w, latest kernel always provided usable results compared with the rpi provided one. I wonder what I keep messing up.

[deftdawg]

For wifi I observed everytime I got a message in journalctl -b -u dhcpcd relating to expiring ipv6 my wifi would go down... not sure if what you've got is related, but this workaround did the trick for me:

#87802 (comment)

[plmercereau]

my wifi works fine with the above configuration (connecting to a router with DHCP enabled), but sometimes IPv6 somehow takes precedence over IPv4, for instance:

[nixos@nixos:~]$ ping www.github.com
PING www.github.com (140.82.121.4) 56(84) bytes of data.
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=1 ttl=49 time=38.9 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=2 ttl=49 time=31.6 ms
64 bytes from lb-140-82-121-4-fra.github.com (140.82.121.4): icmp_seq=3 ttl=49 time=28.8 ms
^C
--- www.github.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 28.768/33.080/38.882/4.261 ms


[nixos@nixos:~]$ ping www.google.com
PING www.google.com(ams16s32-in-x04.1e100.net (2a00:1450:400e:80c::2004)) 56 data bytes
^C
--- www.google.com ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8170ms


[nixos@nixos:~]$ ping -4 www.google.com
PING  (142.250.179.164) 56(84) bytes of data.
64 bytes from ams15s41-in-f4.1e100.net (142.250.179.164): icmp_seq=1 ttl=56 time=28.3 ms
64 bytes from ams15s41-in-f4.1e100.net (142.250.179.164): icmp_seq=2 ttl=56 time=27.6 ms
64 bytes from ams15s41-in-f4.1e100.net (142.250.179.164): icmp_seq=3 ttl=56 time=23.2 ms
^C
---  ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 23.231/26.375/28.291/2.240 ms

I don't know if it is related to your problem though. I will try your suggestion @deftdawg

[alex-nt]

@plmercereau @deftdawg

With the following?

boot = {
    kernelPackages = pkgs.linuxPackages_rpi3;
}

Guys, you are giving me hope here, I might burn a few hours tomorrow and check to see what happens. All in all I am still very happy with your config @plmercereau. I have everything working (even i2c, but I didn't get to test spi yet, last time I failed to get spi working). Thanks yet again!

[rjpcasalino]

I am to understand that the core issue pi-zero-2.dtb file missing isn't fixed with latest u-boot? Many thanks to @deftdawg for helping me boot my Zero 2 with that hint. I think we also need to persevere @plmercereau's config someplace to help others that might not be looking at GitHub issues and considering the u-boot update didn't fix this yet. Perhaps updating the NixOS ARM wiki? In any case, thanks all!