Raspberry Pi Zero W support with Buster Lite Aug 2020 release

[jasonmnemonic]

Hi Kieran,

sorry for getting back to you a bit later than I would but I am happy that I have someone to talk to in figuring this out and I appreciate your time and community support. Just to let you know that I am OK with C++ (and C) but I am not a Linux expert so I do have my oddities where I think I may have done something incorrect but do let me know and point the way; I am more than happy to learn :-)

Also some context in why I am here. I have been trying to have the 4K Pi Camera V2 to stream from the Pi to a Windows10 PC like a webcam and I realised that there is this USB OTG where I can configure the Pi to become gadget(s). To start off with, I learned the various configurations from https://github.com/wlhe/uvc-gadget and https://github.com/climberhunt/uvc-gadget but could only tinker to get climberhunt's fork to work for me. That's where I started to get confused and also understand a bit more of what is going on... Anyway, I have been trying to configure a UVC webcam and a RNDIS gadget which somehow do not want to work with each other -- UVC fails but RNDIS works; more of this in raspberrypi/linux#3965 and that's when I thought it would be nice to understand the work of ideas-on-board and appreciate whats is going on so I can also help figure some issues and hopefully get somewhere in contributing to this; see https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=148361#p1650763 which I see you have also participated in.

But for now, I am interested in getting a RPI0W with Buster Lite Aug 2020 to run this uvc-gadget from you (or Ideas-on-board or Laurent's repository git://git.ideasonboard.org/uvc-gadget.git) which I see to have issues building and running. I got somewhere at the end with David's (climberhunt's) fork but this original fork has more updates. Hopefully, with this conversation/discussion, I can learn more about how stuff works which I appreciate! :-)

OK, from climberhunts, I have the uvc-gadget built and then modified the multi-gadget.sh to be loaded by systemd so that a UVC Camera and RNDIS gadget can be seen in Windows Device Manager but it does not work; so when I setup as a UVC Camera standalone, it works. Based on the experience there, I understand that UVC is a video standard that allows e.g. Windows to see it as a USB streaming device and this uvc-gadget code takes a capture video from Pi Camera via v4l2 and redirects it to UVC output to Windows so Skype can see the camera video feed.

With this same configuration as a standalone uvc-gadget with multi-gadget.sh that I have so far (here is where I have a feeling it is the wrong configuration); the Laurent's uvc-gadget does not work for me. Here is what I have done.

1. Download uvc-gadget git repository from here (which mirrors Laurent's?). Follow the cmake and make instructions so I have the uvc-gadget ELF binary.

2. Copy uvc-gadget binary to where my boot script's multi-gadget.sh (climberhunts) refers to this binary. I have a look at your ./scripts/uvc-gadget.sh which I thought of initially using but not sure if I can use it as it seems that it is not for the Pi (I think).

3. At this point, my Pi is configured to have UVC Camera and it is configured using ConfigFS similar to climberhunt's example. My RPI0W has /dev/video0, /dev/video1 and (/dev/video10, 11, 12, 13, 14, 15, 16; cannot remember why) but only /dev/video0 is the one that does v4l2 video capture and /dev/video1 is the UVC output.

So in climberhunt's script, it would be the binary is started as sudo /home/pi/uvc-gadget/uvc-gadget -f1 -s2 -r1 -u /dev/video1 -v /dev/video0 where -u device is UVC Video Output device and -v device is V4L2 Video Capture device and his script captures at 1920x1080.

My gadget's ConfigFS paths are /sys/kernel/config/usb_gadget/pi4 so I have my ConfigFS paths as

/sys/kernel/config/usb_gadget/pi4/functions/uvc.usb0
/sys/kernel/config/usb_gadget/pi4/configs/c.1
/sys/kernel/config/usb_gadget/pi4/functions/uvc.usb0/control/header/h
/sys/kernel/config/usb_gadget/pi4/functions/uvc.usb0/streaming/mjpeg/
and so on...

4. At this point, if I do not have uvc-gadget running but I intend to run uvc-gadget, this is what I try to do on the terminal.

This uvc-gadget has the following options:

Available options are
-c device V4L2 source device
-h Print this help screen and exit
-i image MJPEG image

UVC device instance specifier

and I have been trying different things on my terminal and I get:

`~/uvc-gadget $ ./uvc-gadget -c /dev/video0 pi4/functions/uvc.usb0
Error: driver returned invalid frame ival type 2
Error opening device /dev/video0: unable to enumerate formats.

~/uvc-gadget $ ./uvc-gadget -c /dev/video0 functions/uvc.usb0
Failed to identify function configuration

~/uvc-gadget $ ./uvc-gadget -c /dev/video0 uvc.usb0
Error: driver returned invalid frame ival type 2
Error opening device /dev/video0: unable to enumerate formats.

~/uvc-gadget $ ./uvc-gadget -c /dev/video0 g1/functions/uvc.usb0
Failed to identify function configuration
`

what have I done wrong?

PS: at the moment, I cannot seem to get the segmentation fault as I initially mentioned wlhe/uvc-gadget#1. I know over there I have done something wrong as well but here it would be nice if I can get it working for the RPI0W and then ask some questions and pointers on the code since it is quite an interesting piece of work to look at.

So, I guess the above is enough to help us have a conversation and hopefully guide me figure out what I am doing wrongly... Thanks Kieran! :-)

[kbingham]

`~/uvc-gadget $ ./uvc-gadget -c /dev/video0 pi4/functions/uvc.usb0
Error: driver returned invalid frame ival type 2
Error opening device /dev/video0: unable to enumerate formats.

So - it looks like the issue is at the /dev/video0. If it's failing to enumerate formats, that's why it's stopped.

I'm curious as to why that happened, if /dev/video0 worked with the other applications.

Could you try running:

v4l2-ctl -d /dev/video0 -D --list-formats

and paste the results here please?

By the way, because there is only one UDC on your platform the updated uvc-gadget should automatically find it. So you should be able to do just

./uvc-gadget -c /dev/video0

[jasonmnemonic]

Thanks Keiran, here we go.

v4l2-ctl -d /dev/video0 -D --list-formats

Driver Info:
Driver name : bm2835 mmal
Card type : mmal service 16.1
Bus info : platform:bcm2835-v4l2
Driver version : 5.4.51
Capabilities : 0x85200005
Video Capture
Video Overlay
Read/Write
Streaming
Extended Pix Format
Device Capabilities
Device Caps : 0x05200005
Video Capture
Video Overlay
Read/Write
Streaming
Extended Pix Format
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

[0]: 'YU12' (Planar YUV 4:2:0)
[1]: 'YUYV' (YUYV 4:2:2)
[2]: 'RGB3' (24-bit RGB 8-8-8)
[3]: 'JPEG' (JFIF JPEG, compressed)
[4]: 'H264' (H.264, compressed)
[5]: 'MJPG' (Motion-JPEG, compressed)
[6]: 'YVYU' (YVYU 4:2:2)
[7]: 'VYUY' (VYUY 4:2:2)
[8]: 'UYVY' (UYVY 4:2:2)
[9]: 'NV12' (Y/CbCr 4:2:0)
[10]: 'BGR3' (24-bit BGR 8-8-8)
[11]: 'YV12' (Planar YVU 4:2:0)
[12]: 'NV21' (Y/CrCb 4:2:0)
[13]: 'BGR4' (32-bit BGRA/X 8-8-8-8)

[kbingham]

ioctl: VIDIOC_ENUM_FMT
Type: Video Capture

that looks like formats were enumerated...

Ok - so tracing that back through the code:

v4l2_enum_frame_intervals() is where the error "Error: driver returned invalid frame ival type 2" is reported.
That refers to :
V4L2_FRMIVAL_TYPE_CONTINUOUS at https://elixir.bootlin.com/linux/latest/source/include/uapi/linux/videodev2.h#L840

You said you can code C.
Would you be able to add support for V4L2_FRMIVAL_TYPE_CONTINUOUS to the v4l2_enum_frame_intervals() function?

We can see the camera driver setting this at : https://elixir.bootlin.com/linux/latest/source/drivers/staging/vc04_services/bcm2835-camera/bcm2835-camera.c#L1357

The uvc-gadget is expecting a list of supported frame sizes, but because the RPi camera has a scaler when using the GPU - it can produce 'any' size - so UVC gadget doesn't expect that.

[jasonmnemonic]

Hi Kieran, I would love to have a go but I need some pointers and guidance. That is because I am not very well versed with Linux subsystems and stuff on v4l2... so coding in the blind. I am having baptism by fire :-)

I mean whereabouts in your C file (uvc-gadget project) that I will need to add this v4l2_enum_frame_intervals() in and is this a callback? Basically, I will try to do this work and contribute but maybe will need slightly more guidance than most lurkers here.

Thank you!

[kbingham]

v4l2_enum_frame_intervals() is already in uvc-gadget.c

Take a look in lib/v4l2.c. That's where this issue is.

uvc-gadget/lib/v4l2.c

Line 122 in b1d622d

[kbingham]

Of course it's frame intervals, not frame sizes that needs updating - sorry for confusing terms above.

[jasonmnemonic]

Hi Kieran, thanks for the tips. I initially thought of a making post when I have done some work but it seems that I need some days to look at this because of other things at work and also at home but nonetheless, I am interested in looking at this because others can benefit from this too.

So this is me just making the courtesy post that I am on it and will get back to you. All things appreciated! :-)

[kbingham]

No worries. I expect this is the reason that most people haven't been able to use upstream uvc-gadget with RPi.
The Broadcom(raspberrypi) drivers do this 'slightly' differently to every other platform so RPi sort of caught a nuance.

The thing is - no one ever reported it to us for the upstream properly, so until I saw your log messages we weren't aware of it.
I hope with this fixed, we'll be able to clear up the 'mess' that is 'how to use a RPi' as a UVC gadget.

I suspect there are more issues for RPi4 though, as I've tried testing on there - but hit different issues with the whole board/kernel locking up ... but if the zero gadget is working first that will be a big help to know the platform works.

[jasonmnemonic]

Hi Kieran,

in the function v4l2_enum_frame_intervals() in v4l2.c, I have added this block of code here.

case V4L2_FRMIVAL_TYPE_CONTINUOUS:
	ival->min = (struct v4l2_fract) {1, 90};	// FPS_MAX 90
	ival->max = (struct v4l2_fract) {1, 1};		// FPS_MIN 1
	ival->step.numerator = 1;
	ival->step.denominator = 1;
	break;

I built the code and with my current RPI0W setup where I have configured a ConfigFS that works with my Win10 and if I execute the uvc-gadget correctly then I will hear the bloop bloop sound on Win10 and UVC Camera becomes non-grey.

This does not work for me...

$ ./uvc-gadget -c /dev/video0 
Failed to open attribute bInterfaceNumber: No such file or directory
Failed to identify function configuration

But this works for me ...
$ ./uvc-gadget -c /dev/video0 uvc.usb0

Then the RPI terminal seems to be doing something (no previous errors) and cannot see any errors on the terminal but Win10 Camera App shows me error. The message is as such.

Something went wrong

If you need it, here's the error code:
0xa00f4271<MediaCaptureFailedEvent>(0x8007001f)

I am guessing it cannot receive a stream.

So here are the timeline of events from running the binary until I see the error. Maybe you could guide me to where I have done wrong? Ta.

$ ./uvc-gadget -c /dev/video0 uvc.usb0
Device /dev/video0 opened: mmal service 16.1 (platform:bcm2835-v4l2).
Device /dev/video1 opened: 20980000.usb (gadget).
bRequestType a1 bRequest 86 wValue 0400 wIndex 0100 wLength 0001
control request (req 86 cs 04)
bRequestType a1 bRequest 81 wValue 0200 wIndex 0000 wLength 0001
control request (req 81 cs 02)
bRequestType a1 bRequest 86 wValue 0900 wIndex 0100 wLength 0001
control request (req 86 cs 09)
bRequestType a1 bRequest 81 wValue 0200 wIndex 0000 wLength 0001
control request (req 81 cs 02)
bRequestType a1 bRequest 86 wValue 0200 wIndex 0200 wLength 0001
control request (req 86 cs 02)
bRequestType a1 bRequest 81 wValue 0200 wIndex 0000 wLength 0001
control request (req 81 cs 02)

Then when I run the CameraApp, there is no termination or errors but the terminal gets additional messages:

bRequestType a1 bRequest 81 wValue 0100 wIndex 0001 wLength 001a
streaming request (req 81 cs 01)
bRequestType 21 bRequest 01 wValue 0100 wIndex 0001 wLength 001a
streaming request (req 01 cs 01)
setting probe control, length = 26
bRequestType a1 bRequest 81 wValue 0100 wIndex 0001 wLength 001a
streaming request (req 81 cs 01)
bRequestType a1 bRequest 83 wValue 0100 wIndex 0001 wLength 001a
streaming request (req 83 cs 01)
bRequestType a1 bRequest 82 wValue 0100 wIndex 0001 wLength 001a
streaming request (req 82 cs 01)

For me to quit, I hit ctrl-c at the terminal to kill uvc-gadget. Hopefully, you would be able to guide me to tinker around.

Thanks and have a good weekend! :-)

[jasonmnemonic]

Hi Kieran,

good morning and hope you have a good weekend!

I have done more investigation but I am unsure if I am on the correct path. I have been looking at the difference between climberhunt/wlhe uvc-gadget and Ideas-On-Board, I put some printf around some function calls as I am trying to find my footing around. Here is what I can see.

I know v4l2-source is open first and then uvc is open next. I can see callbacks are set in v4l2-source where the .set_format = v4l2_source_set_format, is set and the printf I put inside v4l2.c int v4l2_set_format(struct v4l2_device *dev, struct v4l2_pix_format *format) is never called.

Would this be a problem and how do I let this part of the code run?

Ta.

[kbingham]

Oh - interesting, well if the device format is never set then it will be using which ever format it was already configured with or it's defaults. But it might well be that there is an issue there indeed.

Hard to know without some more digging, but I'd say you're not on a wrong track.

Check to see what formats are being configured by default, and if perhaps we need to explicitly get a format from one device and set it on the other.

[kbingham]

Re-reading - it might not be a problem that we don't 'set' the source format, as long as we 'get' it at some point, and it is compatible with what we are sending on the UVC gadget device.

Perhaps worth checking we 'get' the source format, and 'set' it (or similar) on the output device..?

[jasonmnemonic]

Thanks for the replies. I am going to read you again when I have a look at this hopefully later (again, I am finding my footing...), so the question how does we "get" the format? I mean who / where should be calling set format to set it?

Also, correct me if I am wrong, whatever resolutions that we want to capture from the camera to stream to UVC, are they all set externally by v4ctl say on the command line? Can we or is it possible to set it with the code in UVC? Also from say the host PC (Windows) can we send commands to UVC and trigger resolution changes that UVC receives the command and does something?

Also before running your (Ideas-On-Board) uvc-gadget, I have the following set on my RPI0W.

/usr/bin/v4l2-ctl -c auto_exposure=0
/usr/bin/v4l2-ctl -c auto_exposure_bias=8
/usr/bin/v4l2-ctl -c contrast=20
/usr/bin/v4l2-ctl -c video_bitrate=25000000

Will the above this set formatting issue? Ta.

Also thanks for all these Kieran, you are one of the best people in the community! This person appreciates it! :-)

[kbingham]

In an ideal development, uvc-gadget should query the source device, and configure the gadget to tell the host all of the configurations that the 'camera' can provide.

Unfortunately, I think quite a bit of the configuration ends up being configured in advance by the startup scripts, so that's not easy.
In the future I'd like to see a way for uvc-gadget to configure the UDC fucntions and configurations to expose what can actually be used, but we're not there now.

In terms of those v4l2-ctl controls : Ideally, those should be set from the host side., and should be passed through the gadget I would expect, but I don't think any of that has been developed yet either.

I can't be sure, but I don't think those controls will be reset by uvc-gadget so hopefully setting them manually is enough for now, but that is not handling formats.

[peterbay]

Hi, the last few weeks i'm working on an upgraded uvc-gadget.
I have implemented resolution switching, camera control from the host computer, linux framebuffer as source device, etc..
You can see it here - https://github.com/peterbay/uvc-gadget