NixOS test disk timeout, kernel panic

[l0b0]

nixpkgs commit 5620d24 causes the dev-disk-by-label-nixos.device service on the NixOS test VM to not start, causing a kernel panic. Bisecting:

❯ git bisect start 0726a0ecb6d4e08f6adced58726b95db924cef57 4c1018dae018162ec878d42fec712642d214fdfa
Bisecting: 4073 revisions left to test after this (roughly 12 steps)
[4a9d25e66358a773d2b0a5af6c953cf7633cfeb7] {,fcitx5-}openbangla-keyboard: migrate to by-name, preserve overrides (#508973)
❯ git bisect run nix build --override-input nixpkgs-unstable . ~/my\ projects/root/modules/nixos/tests#checks.x86_64-linux.sshServerAudit
[omitted start of log]
5620d245adf03c58b74f803031ba2a85d204db31 is the first bad commit
commit 5620d245adf03c58b74f803031ba2a85d204db31
Author: Will Fancher <elvishjerricco@gmail.com>
Date:   2025-08-15 15:43:01 -0400

    nixos/systemd-stage-1: Enable by default

 .../manual/administration/boot-problems.section.md | 84 +++++++++++++---------
 .../installing-from-other-distro.section.md        |  9 ++-
 nixos/doc/manual/redirects.json                    |  3 +
 nixos/doc/manual/release-notes/rl-2605.section.md  | 10 +++
 nixos/modules/system/boot/stage-1.nix              |  4 ++
 nixos/modules/system/boot/systemd/initrd.nix       | 84 +++++++++++++---------
 nixos/release-combined.nix                         | 20 ++++++
 7 files changed, 143 insertions(+), 71 deletions(-)
bisect found first bad commit

@ElvishJerricco @tfc

System info

• system: "x86_64-linux"
• host os: Linux 6.18.22, NixOS, 26.05 (Yarara), 26.05.20260414.4bd9165
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.34.6
• channels(root): "home-manager, nixos, nixos-hardware, nixos-stable-24.05"
• nixpkgs: /nix/store/0d1kf07ph2ply64i1y247f9q2an9cw00-source

[dtomvan]

Yeah I think it's because of services.udev.enable = false;. Tested with: https://gist.github.com/dtomvan/e90f8597ec85e8be98d2878fb6882136 . This reproducer is extracted from your repo https://gitlab.com/engmark/root. If I comment out udev.enable = false; both should-fail and should-succeed build.

I don't know the technical details, but seems logical how disabling udev leaves you with systemd not finding the virtual disk...

[ElvishJerricco]

Interesting. What is the use case for services.udev.enable = false;? This option seems to have a number of problems, and should probably not be user-facing.

• In stage 2, services.udev.enable = false; only prevents NixOS from configuring udev rules. It's still in the boot chain and will be started and will process the udev rules that are shipped with systemd. That's not really what I would call "disabled", so the option seems to not do what it says there.
• In scripted stage 1, services.udev.enable = false; does absolutely nothing. It still ends up using udev just as it would with services.udev.enable = true;. Again, not what I would call "disabled".
• In systemd stage 1, services.udev.enable = false; does actually remove udev from the initrd essentially completely (the binary and the rules with systemd are still copied in, but they don't run). This is the most true to what the option suggests, but as we see in this issue, disabling udev in initrd is rather nonsense.

The option was added so that container configs could avoid needlessly configuring udev, because udev doesn't run in containers. My inclination is to hide this option as internal and not support its use outside of nixpkgs, probably even asserting it be enabled in systemd stage 1. systemd and NixOS both depend on udevd; it's one of the few things about systemd that is not optional. So we can't really support disabling it except in environments like containers where it actually doesn't do anything.

[l0b0]

The use case is that the NixOS tests don't run reliably on GitLab's CI runners. I tried removing everything not absolutely required for the tests to pass, and it worked for a while. But they were flaky, and got worse over time. With this recent nixpkgs update they stopped working entirely. After re-enabling udev they seem to only ever pass locally, and not in CI.

If there are any tricks to getting NixOS tests to pass reliably in container-based CI runners like on GitLab, I'd love to know. I've converted a couple of the simpler tests to use NixOS test containers rather than VMs, but unfortunately that doesn't work for the more complex tests.

[dtomvan]

Huh, if I use cachix/install-nix-action on GitHub actions (which AFAIK is also container-based), I can perform nixos tests just fine: https://github.com/dtomvan/puntbestanden/actions/runs/24625538344/job/72003762790#step:6:28594

And I don't think it's doing anything fancy, except maybe this part where it adds a (host) udev rule for kvm. Is KVM actually supported on gitlab.com's CI runners?

Slightly offtopic

Now that you mentioned it, my workflow only applying to GitHub and not GitLab is sadly yet another way I'm locked into github...

[ElvishJerricco]

The use case is that the NixOS tests don't run reliably on GitLab's CI runners.

That sounds like unrelated stuff though, right? This particular issue is about the disk timeout problem, which is resolved by not disabling udev (and NixOS should change that option so that it isn't so broken and inaccurate). If you're having other buggy behavior with NixOS then of course opening additional issues with details about those problems would be very welcome.