Investigate inconsistent ordering of PCI bus IDs for USB devices

[eloquence]

During review of freedomofpress/securedrop-export#5, @zenmonkeykstop reported that the PCI bus ID used by the same USB port will sometimes be different after attach/detach events, causing subsequent exports to fail. We want to better understand the root cause of this issue, and investigate if we can mitigate against it, or of this will potentially require upstream changes.

[eloquence]

We've allocated a 4 hour timebox for this investigation during the 8/7-8/21 sprint. I can help with upstream coordination if required.

[zenmonkeykstop]

It looks like this is due to how XHCI support in Linux handles USB 2.0. Check out the output from lsusb:

[user@sys-usb ~]$ lsusb -t
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
   |__ Port 3: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
   |__ Port 8: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
   |__ Port 8: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
   |__ Port 9: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 12M
   |__ Port 10: Dev 5, If 0, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
   |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 480M

There are two buses, usb2 and usb3, using the xhci_hcd driver, both of which are associated with the computer's single XHCI controller:

lrwxrwxrwx 1 root root 0 Aug  9 14:21 usb2 -> ../../../devices/pci0000:00/0000:00:06.0/usb2
lrwxrwxrwx 1 root root 0 Aug  9 14:21 usb3 -> ../../../devices/pci0000:00/0000:00:06.0/usb3

Note the PCI id and see:

[user@sys-usb ~]$ lspci 
...
00:04.0 USB controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 10)
00:06.0 USB controller: Intel Corporation Sunrise Point-LP USB 3.0 xHCI Controller (rev 21)

When I connect my (Brother HL2390DW) printer, which uses USB 2.0, it gets connected to usb2 and has the sys-usb id 2-2. When I connect a USB 3.0 stick, it gets assigned to usb3 and has an id of 3-2.

(The other EHCI (USB 2.0) controller and the usb1 bus is just for HID and I don't think we can use it.)

This would suggest a few workarounds/solutions:

1. Require all devices to be USB 3.0 (though this would restrict printer choices)
2. Force the workstation to only use USB 2.0 on the XHCI controller somehow (no option in BIOS that I can see, might be possible with kernel args, would make file transfers slower)
3. Link more than one sys-usb ID to sd-export-usb, one for USB 3.0 devices, one for USB 2.0, both corresponding to the same physical port, and have the send-to-usb script probe to see which one is active.

One complication that I saw a couple of times was a USB 3.0 stick getting attached as a USB 2.0 device. I'm not sure why this was happening and I can't replicate it, but I can replicate the USB 2.0 printer being attached to 2-2 followed by the USB 3.0 stick on the same port being attached to 3-2. Option (3) above would cover that case anyway.

[zenmonkeykstop]

Next step here is to test if (3) is even possible, by setting up persistent attachments for both the 2.0 and 3.0 versions of a port and seeing if the export VM will boot with only one (or none) attached.

[eloquence]

Thanks! The agreed upon time-boxed investigation here was completed; moving follow-up work back to the near-term backlog for now.

[eloquence]

In testing the export feature, I experienced additional inconsistency of the ordering of the bus ID in sd-export-usb as compared with the bus ID reported by Qubes, and by lsusb inside of sys-usb. For example:

• device attached, qvm-usb ls in dom0 reports it as sys-usb:4:1
• sd-export-usb configured accordingly; it can mount and read the drive
• Qubes device manager also reports it as sys-usb:4:1
• lsusb on sys-usb reports it as Bus 004 Device 002
• lsusb on sd-export-usb reports it as Bus 003 Device 002

Note that the string identifiers may also differ between VMs (one drive was reported as "Kanguru" in sd-export-usb, which is correct, and "Lumension" in sys-usb, which is incorrect), I am guessing because of differences in the USB ID lookup database between Fedora and Debian.

Because the export code parses lsusb output in sd-export-usb and filters it by the bus ID, it fails to discover the device, and all export attempts fail.

In discussions about this today, @creviera suggested lsblk as a possible alternative that does not have the numbering issue.

[sssoleileraaa]

I was able to repro the issue reported by @eloquence:

• qvm-usb ls in dom0 reports BACKEND:DEVID as sys-usb:3:2 and DESCRIPTION as Generic_Mass_Storage_CC9....
• Qubes device manager also reports it as sys-usb:3:2 Generic_Mass_Storage_CC9....
• lsusb on sys-usb reports it as Bus 003 Device 005: ID 058f:6387 Alcor Micro Corp. Flash Drive (I unplugged it and plugged back in to verify that is indeed the identifier and name it sees for my flash drive)

After updating usb device to sys-usb:3-2 in config.json and running make sd-export:

• qvm-usb ls in dom0 reports BACKEND:DEVID as sys-usb:3:2 and DESCRIPTION as Generic_Mass_Storage_CC9.... and device is USED BY sd-export-usb
• Qubes device manager also reports it as sys-usb:3:2 Generic_Mass_Storage_CC9....
• lsusb on sys-usb reports it as Bus 003 Device 006: ID 058f:6387 Alcor Micro Corp. Flash Drive
• lsusb on sd-export-usb reports it as Bus 002 Device 003: ID 058f:6387 Alcor Micro Corp. Flash Drive
• sd-export-config.json on sd-export-usb VM has bus id 3 and device id 2 and I'm able to see my flash drive attached to the VM in the Files app and access files
• qvm-open-in-vm sd-export-usb archvie.sd-export fails with USB_NOT_CONNECTED even though it's attached to the Export VM

[emkll]

3. Link more than one sys-usb ID to sd-export-usb, one for USB 3.0 devices, one for USB 2.0, both corresponding to the same physical port, and have the send-to-usb script probe to see which one is active.

Unfortunately this doesn't work, I just tested this scenario: qvm-usb attach operation requires the device to be available at the time of the attachment. We cannot have sys-usb:2-1 and sys-usb:3-1 to be both connected at the same time, so we cannot attach in one operation to sd-export-usb. Even if that was possible, that still would not work: I attempted to attach both a sys-usb:2-1 device and a sys-usb:3-1 device, but both need to be plugged in for sd-export-usb to properly start (based on the theory described above this would be impossible since they refer to the same slot_

[eloquence]

@rmol will prototype the approach outlined in freedomofpress/securedrop-export#18 (comment): automatically attaching newly plugged in USB mass storage devices and printers to sd-export-usb, obviating the need for a persistent attachment using a specific device ID.

In this implementation, we want to make sure that users can still manually detach the device and attach it to another VM, to ensure advanced users aren't too fenced in by this approach.

Even if this approach doesn't prove viable after prototyping, we still agree that the persistent attachment to the sd-export-usb VM will have to be removed (our fallback is the Qubes device manager). This is tracked in #320.

[eloquence]

^ Doing the above prototyping work is our commitment for the 11/6-11/20 sprint, time-boxed to 16 hours. After that @rmol will make a recommendation on how to proceed.