Freenect(2) update thread (Linux/macOS support) #9
Description
Hi,
With the upcoming 0.3 release, I split obs-kinect into multiple parts: common stuff like obs plugin registration, depth/infrared/faux green-screen processing, source and general device handling is part of the core but it doesn't know about Kinect devices (or even windows). Kinect v1/v2 handling is done in separate dll which are loaded by obs-kinect at startup.
This was made to add support for Kinect v1 without requiring the install of both v1 and v2 runtimes. And also to support Linux/macOS.
Obviously, Microsoft Kinect SDKs don't work on Linux/macOS, hopefully projects like libfreenect and libfreenect2 allow the use of Kinect on these systems.
Since obs-kinect device handling is done in separate libraries, it's possible to add libfreenect support kinda easily.
I tried both of these libraries on Windows (since they do have Windows support as well) for testing and it's working well!
freenect (v1):
freenect2 (v2):
However since libfreenect(2) needs raw USB commands, it requires to install a custom USB driver (like UsbDk). Hopefully this doesn't conflict with Kinect v2 (I can use libfreenect2 and Kinect SDK v2.0 without problem), but well, for Kinect v1 it's a bit more like hell. You have to replace the Kinect USB driver (using Zadig) for libfreenect to work which prevents Kinect SDK v1.8 to work, and it took me one hour to reinstall the official Kinect drivers, so yeah, I don't think I will release an obs-kinect libfreenect backend on Windows because of that (since it doesn't seem to do things you cannot do with official Kinect v1.8 runtime, except maybe for led control?).
As for Kinect v2, libfreenect2 does add more control, especially with the RGB camera color settings (things like exposure mode, white balance, gain, and such), so I think I'll support it on Windows too (even though I hope to be able to do that as well with the official SDK backend, using this).
Important note
Both libfreenect libs are really cool, but they are lacking some higher-level stuff the Kinect SDK does, like body/skeletal detection, or dedicated background removal (like Kinect SDK v1.8 does). It doesn't mean you won't be able to have background removal on Linux/macOS but it means you won't be able to use body informations for that and there's not much I can do about that. (except implementing body detecting but it requires some skill I don't have).
I will be able to compile the plugin on Linux, or even test it without problems, but I won't be able to do that with macOS. So if you have a mac and are willing to help, please let me know!
Also, as the 0.3 release is already a big one, Linux/macOS support won't be part of it (but I'll try to do it asap).
TL;DR: Linux/macOS support requires an open-source backend since official Microsoft SDKs are for Windows. There are backend for Kinect v1 and v2 but they don't do higher-level processing like body information. Things like body detection/body-based greenscreen filtering won't work on Linux/macOS because of that.