60 FPS Widevine DASH content stutters on some devices

[matamegger]

Content description

60 FPS Widevine encrypted DASH streams stutter and/or have many dropped frames.
The (per email) provided content shows a live ticker at the bottom, where this stuttering and frame drops can be observed very well.

There are players like the NexPlayer (https://www.nexplayersdk.com/), which shows better performance on the same content and devices, therefore we rule out a weak DRM decryption module or hardware decoders.

We also tried to find the bottleneck in the code base, tried to tweak several parameters, and implemented a decoder version using the asynchronous callback Android provides. Unfortunately without major improvements.

Additionally, on the more powerful/recent devices we observed that frames are provided to the Surface in time (ExoPlayer business logic did not drop the frames) but the stuttering was still visible. When we changed the code so that the renderTimeStamp passed to releaseOutputBuffer is not "reasonably close" to the current system time (which causes the Surface to ignore the timestamp and display the buffer at the earliest feasible time. In this mode the Surface will not drop frames), we did not see any stuttering/frame drops anymore as expected. However, over time video and audio were drifting apart as if video would not be playing at 60 FPS anymore but rather at a lower frame rate. This observation lead us to the assumption that the surface itself maybe cannot keep up with the high frame rate.

Link to test content

Provided per email.

Version of ExoPlayer being used

2.10.4 and at least back to 2.9.6

Device(s) and version(s) of Android being used

Seems unrelated to the Android version.
Seen issues with:

• OnePlus 5T
• OnePlus 5
• Galaxy Tab E (SM-T337W) | 7.1.1, 6.0.1
• Galaxy Tab 3 Lite (SM-T113) | 4.4.4
• Galaxy S8 (Canadian and "regular" version) | 9.0.0
• OnePlus 5 | 9.0.0
• Google Pixel 3 | 9.0.0
• Samsung S6 (SM-G928V) | 7.0.0
• Samsung S7 (SM-G930T1) | 8.0.0
• Galaxy Tab 3 10.1 (GT-P5210)
• Huawei P30 Lite (MAR-LX3A)
• Galaxy Tab 3 (SM-T310)
• Galaxy Tab S3 (SM-T820)
• Galaxy Tab S2 (SM-T810) | 6.0.1,7.0,7.0
• Galaxy Core (SM-G386) | 4.4.2
• Galaxy Tab 4 (SM-T230) | 4.4.2
• MiBox3 (MIBOX3) | 8.0.0
• Galaxy A8 (SM-A530W) | 7.1.1,8.0.0,9

A full bug report captured from the device

Provided per email.

[matamegger]

@ojw28 Have you had already time to check this? If not, can you maybe provide a time frame?

We are happy to support you in further investigation, just let me know.

[ojw28]

@christosts - We should probably treat this as a duplicate of #5796 (or at least investigate both at the same time). The test content and information above will be useful for helping to look at that issue.

[christosts]

Ack

[christosts]

Hi @matamegger

Checked the DASH links provided over email yesterday and they returned 404*. Can you please check the health status of the links, and if they are not alive anymore, maybe provide new ones?

Also, regarding:

When we changed the code so that the renderTimeStamp passed to releaseOutputBuffer is not "reasonably close" to the current system time

Can you please share the change you made so that we can attempt to reproduce the behavior?

Thanks.

[edit]
*I tried to access the links from a browser and got the 404.

[gvidda]

We are experiencing - what probably is - the same issue with widevine drm, but with video tracks that are 720p50fps. They work fine unencrypted.

Iv'e done several tests with one problematic device, Xiamo Mi A3 (with Android One). Changing between L1/L3 does nothing. The only thing that helps is disableing the audio renderer (for some reason)

There's also an old issue from last year about this, but it's closed with no resolution.
#4496

[matamegger]

@christosts
I will check back on the assets, and provide you the renderTimeStamp change asap.

[matamegger]

Regarding the "renderTimeStamp". It's basically something like the following patch shows.
With this change in the MediaCodecVideoRenderer the Surface will (according to the doc) not drop any frames and renders them as fast as possible.

.../library/core/src/main/java/com/google/android/exoplayer2/video/MediaCodecVideoRenderer.java
@@ -912,8 +912,8 @@ public class MediaCodecVideoRenderer extends MediaCodecRenderer {
  protected void renderOutputBufferV21(
       MediaCodec codec, int index, long presentationTimeUs, long releaseTimeNs) {
     maybeNotifyVideoSizeChanged();
     TraceUtil.beginSection("releaseOutputBuffer");
-    codec.releaseOutputBuffer(index, releaseTimeNs);
+    codec.releaseOutputBuffer(index, 0);
     TraceUtil.endSection();
     lastRenderTimeUs = SystemClock.elapsedRealtime() * 1000;
     decoderCounters.renderedOutputBufferCount++;

Note: This is only executed on devices API >= 21

[christosts]

@matamegger

thanks. If you can check the content links (they are returning 404) and provide working links, I will try to repro and come back with findings.