Ivo/webrtc

[izivkov]

The Bot app initiates a WebRTC call to the controller, by sending an WebRTC offer, providing its A/V capabilities. It then waits for an answer with controller's capabilities. Note that the media is streamed only one way from the Robot to the controller.

WebRTC does not specify signaling protocol. Usually, a separate signaling server is used
witch mediates between the two WebRTC peers, and communication between the peers and the signalling server is carried over WebSocket. However, we already have a communication channel between the peers (NetworkServiceConnection) so we are using it instead. No separate signaling server is required.

It is possible in the future to factor out signaling into a separate class and provide various signalling types, such as to separate signalling server.

• Added optional filter capabilities to ControllerToBotEventBus.subscribe(). We now have multiple subscribers to this event bus, and some of them are only interested in particular message types. It is more efficient to add this filter to the subscribe() method than filtering the messages as they are received, because filtered events are not even emitted.

[izivkov]

This PR will probably break the video in the Python program, because of signaling. However, I think it works well with the Android controller. Will have to get the python video working next.

[thias15]

So does this break the python interface, or can it still be used with RTSP server?

[izivkov]

So does this break the python interface, or can it still be used with RTSP server?

It can still be used with RSTP, but at the moment, we cannot switch to RTSP at runtime. Even after we add a configuration, we need to create a different surface at the robot side, WebRTCSurfaceView for webrtc and AutoFitSurfaceGlView for RTSP. I was trying to create this surface programmatically, but run into some issues. I'm currently trying to modify the python program to work with WebRTC as well.

[izivkov]

I have to say, it is difficult to find examples of Python programs displaying WebRTC video in pygame window. Most of the examples use python as a server, and the videos are displayed in a browser using some additional JavaScript.

[izivkov]

@dhruv2295 Let's work on this together.

I added the ability to create the surface for sending video from the Bot app. programmatically. This makes it easy to switch between RTSP and WebRTC without recompiling. Also, this enables video from all fragments, because this surface is outside of the fragments. We need to add configuration and make sure we do not create problems in fragments other than FreeRoam.

[thias15]

I have to say, it is difficult to find examples of Python programs displaying WebRTC video in pygame window. Most of the examples use python as a server, and the videos are displayed in a browser using some additional JavaScript.

I guess, we could also do WebRTC video feed via browser and only send/receive commands through python.

[izivkov]

I have to say, it is difficult to find examples of Python programs displaying WebRTC video in pygame window. Most of the examples use python as a server, and the videos are displayed in a browser using some additional JavaScript.

I guess, we could also do WebRTC video feed via browser and only send/receive commands through python.

Yes, I can revisit showing the video in a browser, but if we can configure the app to send either RTSP or WebRTC, the python script can still use RTSP.

[izivkov]

This latest change should address the instability issue. This only happens if the controller is not connected to the bot app, but we have created the surface. Now we create the surface after successful connection.

• Also cleaned up some code a little related to filtering ControllerToBot Events, which is more efficient than manual filtering.
• Updated the README.md file for the controller with a new controller image showing streaming video.

[izivkov]

The instability issue was caused by BotToControllerEventBus sending new messages when the controller is not running to receive them. The back-pressure is in now handled by simply dropping massages which cannot be handled. The fix is made centrally in the BotToControllerEventBus

[izivkov]

I merged @dhruv2295 settings for RTSP/WebRTC (with minor changes due to conflicts).

[izivkov]

Added "mirror" button on controller. The controller starts in "mirrored" state. The state is kept on the server (bot app) so it can be added configuration.