Codec selection for video tunneling

[navluis]

Issue description

In our particular device which has media tunneling support, ExoPlayer is unable to list and select the tunneling decoder properly. For example, the device advertises these two components supporting avc, one without and one with tunneling support:

omx.oem.avc.decoder
and
omx.oem.avc.decoder.tunneled

ExoPlayer isn't able to select the tunneled version of the decoder but can only detect the non-tunneled version.

We believe there are two issues with the current selection logic in ExoPlayer:

1. ExoPlayer lists by default the codecs using the 'REGULAR_CODECS' filter and tunneled decoders have to be queried with 'ALL_CODECS' filter. See this in "frameworks/base/media/java/android/media/MediaCodecList.java"

/**
* Use in {@link #MediaCodecList} to enumerate all codecs, even ones that are
* not suitable for regular (buffer-to-buffer) decoding or encoding. These
* include codecs, for example, that only work with special input or output
* surfaces, such as secure-only or tunneled-only codecs.
* /
public static final int ALL_CODECS = 1;

2. When more than one decoder supporting the given mime-type is returned in the media codec list, ExoPlayer needs to iterate through the list and pick the one with the tunneling feature enabled. In our devices, the non-tunneled version decoder will be listed first so additional logic needs to be added to pick the tunneled version.

I guess another problem with the existing implementation is that there doesn't seem to be a way to tell ExoPlayer that the app prefers a tunneling decoder if the platform supports it. Having this ability would make it easier for ExoPlayer to adjust the codec selection logic as suggested above.

Reproduction steps

Play any stream with demo app on ExoPlayer v2.9.x

Link to test content

Any content, including the sample streams in the demo app

Version of ExoPlayer being used

ExoPlayer v2.9.x

Device(s) and version(s) of Android being used

Android-P on broadcom reference designs

A full bug report captured from the device

N/A ?

[andrewlewis]

(2) looks like a duplicate of #3100. I'm planning to push a limited fix over the new few weeks, which will advertise and select tunneling support if a non-primary tunneled decoder supports the same or greater capabilities of a primary non-tunneled decoder.

Thanks for reporting (1), for which I'll make a fix at the same time.

I don't understand "there doesn't seem to be a way to tell ExoPlayer that the app prefers a tunneling decoder if the platform supports it" -- if you set a tunneling audio session id on the track selector we should prefer using a tunneling decoder. Could you elaborate a bit on what is needed? Thanks.

[navluis]

Thanks for the quick response.

if you set a tunneling audio session id on the track selector we should prefer using a tunneling decoder

Yes, you're right. I should rephrase that, the statement was referring to point 1). Currently ExoPlayer decides to list the platform codecs with 'REGULAR_CODECS' or 'ALL_CODECS' filter depending on whether the app needs a secure decoder or not.

In file: library/core/src/main/java/com/google/android/exoplayer2/mediacodec/MediaCodecUtil.java
codecKind = includeSecure ? MediaCodecList.ALL_CODECS : MediaCodecList.REGULAR_CODECS;

I guess if the utility class knew whether the app preferred a tunneled decoder it could select the right filter.

Anyway, I figure these are implementation details so you guys should know better how to address it. Looking forward to try out your changes when they're ready.

Thanks,

[andrewlewis]

@navluis Thanks for the information.

Apologies for straying slightly off-topic, but in case you're working on your device's tunneling implementation: it would be great if you could verify the tunneling implementation works correctly with the use case of queuing audio/video data while the audio track is paused. In this case, the A/V synchronization logic should inspect the first A/V sync header in unplayed data in the audio track buffer and show a single video frame corresponding to that timestamp. This is to handle creating the player while paused and seeking while paused. Please have a look at #5397 (comment) for more details. Thanks!

[navluis]

I just read the discussion on #5397. I need to verify since we usually start ExoPlayer with playWhenReady(true) but my feeling is that we wouldn't display anything until the first audio frame is rendered, as we use audio to drive the playback clock. I think we would behave like the Sony Bravia TV you refer to and if we don't, I think we should since that would conform to what the framework expects from the HAL, based on my understanding.

Anyway, I'll report back once I verify it.

Thks
Luis

[andrewlewis]

Thanks Luis. The issue with that behavior is that we can't support initializing the player in the paused state, as there is no way to reliably show a single frame without playing audio. The expectation that the A/V sync logic should read ahead the first unplayed audio header and show a frame may be documented in the future [Internal: b/126155094].

[sunrise0406]

Hi All,
I'm facing the same question, "the statement was referring to point 1). Currently ExoPlayer decides to list the platform codecs with 'REGULAR_CODECS' or 'ALL_CODECS' filter depending on whether the app needs a secure decoder or not.".

Is it possible to get the tunneled codec list without a secure decoder ?
The only way to get tunneled codes is to pass a argument -'ALL_CODECS' to MediaCodecList
in terms of Google's document ,"https://developer.android.com/reference/android/media/MediaCodecList.html#ALL_CODECS".

Feel free to let me know if I have to create a new ticket on the ALL_CODECS question.
Many thanks,
sunrise0406

[andrewlewis]

@sunrise0406 You can query the ALL_CODECS list then filter out those items that support tunneling (or not). Work is in progress on this feature so I expect there will a fix available on the development branch some time over the next few weeks.

[sunrise0406]

@andrewlewis I understand, thanks for the quick response.