[0.2.3-rpm] libxenlight failed to create new domain `sd-log`

[eloquence]

In my prod install, I have now seen this error message (displayed as a sticky Qubes notification) twice towards the end of updates. sd-log does end up running just fine.

/var/log/qubes/qubes.log shows this output:

Click to expand

2020-03-18 18:00:23,561 Setting Qubes DB info for the VM
2020-03-18 18:00:23,563 Starting Qubes DB
2020-03-18 18:00:23,633 Activating the sys-whonix VM
2020-03-18 18:00:28,747 Starting sd-proxy
2020-03-18 18:00:28,754 Starting sd-whonix
2020-03-18 18:00:32,529 Setting Qubes DB info for the VM
2020-03-18 18:00:32,529 Starting Qubes DB
2020-03-18 18:00:32,595 Activating the sd-whonix VM
2020-03-18 18:00:46,144 Setting Qubes DB info for the VM
2020-03-18 18:00:46,145 Starting Qubes DB
2020-03-18 18:00:46,179 Starting sd-log
2020-03-18 18:00:46,216 Activating the sd-proxy VM
2020-03-18 18:00:49,778 Start failed: internal error: libxenlight failed to create new domain 'sd-log'
2020-03-18 18:00:54,378 Starting sd-log
2020-03-18 18:00:58,560 Setting Qubes DB info for the VM
2020-03-18 18:00:58,560 Starting Qubes DB
2020-03-18 18:00:58,640 Activating the sd-log VM
2020-03-18 18:01:18,881 Starting sd-app
2020-03-18 18:01:26,525 Setting Qubes DB info for the VM
2020-03-18 18:01:26,526 Starting Qubes DB
2020-03-18 18:01:26,687 Activating the sd-app VM
2020-03-18 18:01:55,294 Starting sd-gpg
2020-03-18 18:02:01,086 Setting Qubes DB info for the VM
2020-03-18 18:02:01,088 Starting Qubes DB
2020-03-18 18:02:01,378 Activating the sd-gpg VM

[eloquence]

See #495 for another sd-log error during a prod install reported by Kev last week, which may or may not be related.

[eloquence]

Again seeing this during an update run today. Am blowing away this prod install momentarily, will report if I see it again on reinstall.

[eloquence]

I'm seeing this on a brand new install from scratch, but this time with sd-proxy has failed to start; again, during the update process. @ninavizz also saw it earlier today. Note that all those laptops only have 16GB RAM.

[rocodes]

Yup, repro'd. Installed prod yesterday, applied updates today, saw the error when I attempted to shut down sd-related vms.

Edit: see QubesOS/qubes-issues#3125
edit2: my logs

libxl-driver.log.20200325.txt

[eloquence]

@rocodes How much memory are you using?

[emkll]

@conorsch has provided a script to monitor the health of the qmemman service in https://gist.github.com/conorsch/bb8b573a6a7a98af70db2a20b4866122

I ran this script (watch ./check-qmemman.sh) as I was updating my workstation. At the reboot step, observed the qmememan service failed at the same moment as the sd-log vm failed to start (the error described in this ticket).

[eloquence]

Same but with sd-proxy. Always tends to happen at the 95% stage for me. Here's a smoking gun screenshot. Service does not appear to recover on its own after this happens.

[eloquence]

Last qmemman.log lines before this happened: https://gist.github.com/eloquence/043db1727d4c7cca215eb587abba64c2

[eloquence]

(For the record, we have reports of this happening on systems w/ 16 and 32 GB, the latter of which should be more than sufficient.)

[marmarek]

"EOF" message from qmemman isn't any kind of its failure. It is normal(*) message during VM startup, failed or not. It means qmemman resumes balancing normally, after allocating memory for the new VM.

(*) "normal" as in "expected", but indeed very cryptic.

[eloquence]

@marmarek The state change Conor's script checks for (EOF being the most recent journal entry) is only reached exactly when I see the VM startup error, and after that, qmemman does not seem to leave that state until it's bounced. I'll test it again today to see if that behavior is consistently reproducible.

I do not see any suspicious line in the libxl-driver.log, indeed I don't see any events in that log at all that directly correlate with the startup failure, but I don't know if the timestamps between the logs are 100% consistent. The failure occurred around 00:00 and these are the logs before and after. To be sure, I don't see anything like the "Stubdom" log entry or "startup timed out" anywhere in the logs, across all of /var/log.

Click to expand

2020-03-25 23:02:05.140+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2020-03-25 23:02:06.694+0000: libxl: libxl_device.c:1093:device_backend_callback: unable to remove device with path /local/domain/11/backend/vif/15/0
2020-03-25 23:02:06.698+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 15
2020-03-25 23:02:06.808+0000: libxl: libxl_device.c:1093:device_backend_callback: unable to remove device with path /local/domain/11/backend/vif/14/0
2020-03-25 23:02:06.862+0000: libxl: libxl.c:1669:devices_destroy_cb: libxl__devices_destroy failed for 14
2020-03-25 23:02:35.482+0000: libxl: libxl_linux.c:155:libxl__loopdev_cleanup: unable to release device /dev/loop0: No such device or address
2020-03-25 23:02:35.542+0000: libxl: libxl_linux.c:155:libxl__loopdev_cleanup: unable to release device /dev/loop0: No such device or address
2020-03-26 00:41:04.117+0000: libxl: libxl_pci.c:1199:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
2020-03-26 03:41:35.288+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/21/console/tty': Resource temporarily unavailable
2020-03-26 03:47:06.648+0000: libxl: libxl.c:1853:libxl_console_get_tty: unable to read console tty path `/local/domain/33/console/tty': Resource temporarily unavailable