Large pixellation for some MPEG2 clips (using Android SW MPEG2 decoder)

[goffioul]

I'm using a Intel-based device running Android-IA (Nougat), where I've enabled the MPEG2 codec in media_codecs.xml (SW-decoder available in Android since Marshmallow). My app is using ExoPlayer r2.4.1 (with ffmpeg extension built-in). I'm testing various MPEG clips using plain HTTP. Most of the MPEG2 clips play fine, but a few of them show massive pixellation. One sample is available at https://drive.google.com/open?id=0BwljeX6541LuamdieXZzamNkNFE . Those clips play fine in any other player.

As a comparison, here's a screenshot of how it looks like on Android:
https://drive.google.com/open?id=0BwljeX6541LuSW1BYW9YamlLeGM
Here's what the same looks like in VLC on Linux:
https://drive.google.com/open?id=0BwljeX6541LuM1B5ZnoycDNndms

Is the stream not compatible with ExoPlayer? Or with Android MPEG2 decoder? Is Android MPEG2 decoder simply doing a very poor job?

[ojw28]

Sounds like you're manually enabling a decoder yourself in the platform, and observing that it doesn't work very well. Isn't that most likely to be an issue with the decoder? What decoder is it that's actually being used?

There's nothing we can actually do to look at this issue given the lack of a bug report, device details and so on (edit: ignore previous comment about test content).

[goffioul]

I've sent the result of "adb bugreport' by email. The device is an Intel/Baytrail platform running Android-IA.

The decoder is the one from Google, available in AOSP since marshmallow [1]. It used to be enabled by default [2], but for some reason this was changed on nougat [3]. The decoder is built into the firmware anyway, so I simply enabled it by including media_codecs_google_tv.xml into main media_codecs.xml.

[1] https://android.googlesource.com/platform/frameworks/av/+/android-6.0.0_r1/media/libstagefright/codecs/mpeg2dec/
[2] https://android.googlesource.com/platform/frameworks/av/+/android-6.0.0_r1/media/libstagefright/data/media_codecs_google_video.xml
[3] https://android.googlesource.com/platform/frameworks/av/+/e1c9766e23a1e5084d9fdbcc0b2005406627f1d4

[ojw28]

If it's a Google codec then you should probably file an issue on Android directly, as described here, under GfxMedia presumably.

[goffioul]

I was hoping you could inspect the sample clip to determine what special about it that can trigger pixellation, while the same decoder works fine with other clips.

[ojw28]

This is beyond what we realistically have time for, particularly given the decoder isn't enabled by default. Sorry. Please do file an issue on Android directly if you want someone to take a look. Thanks.

[goffioul]

I've made some experiments with ffmpeg, emulating the behavior of H262Reader. The strange thing was that I also got pixellation when using my experiment code, while ffplay could play the stream just fine. It appears that in the case of ffmpeg, the pixellation is due to the way the data is segmented and fed to the decoder.

Based on H262Reader, I've segmented my data on picture start code boundaries (0x00 0x00 0x01 0x00). But it appears that when doing that, the decoder ignores any sequence and GOP headers that are embedded in the data chunks. My sample stream has additional sequence and GOP headers in the middle of the stream. These contains custom decoding parameters (quantisation matrix). If these are ignored by the decoder, decoding artefacts then appear.

To make it work with ffmpeg, one must segment the data properly before feeding it to the decoder. In particular, the sequence and GOP headers must be isolated and fed separately.

Given that similar pixellation problems appear when using the MPEG2 software codec from Google, I'd say it's possible it has the same cause, aka the way H262Reader segments the data.

[ojw28]

Does this imply you know how to fix it too ;)? Feel free to send a pull request if so. It's quite low priority for us, so it'll almost certainly be much faster for you to fix it.

[goffioul]

This (temporary/experimental) patch fixes the problem. The idea is to segment data at picture, sequence and GOP header boundaries. However, I ran into an exception while playing another sample clip that contains a discontinuity.

The patch is available here:
https://drive.google.com/open?id=0BwljeX6541Luak1oMldUS3FVZjQ
The clip is available here:
https://drive.google.com/open?id=0BwljeX6541LualNuV0VTSXFWSlk

The exception is:

Internal runtime error.
java.util.NoSuchElementException
	at java.util.concurrent.LinkedBlockingDeque.removeFirst(LinkedBlockingDeque.java:424)
	at java.util.concurrent.LinkedBlockingDeque.remove(LinkedBlockingDeque.java:643)
	at com.google.android.exoplayer2.extractor.DefaultTrackOutput.dropDownstreamTo(DefaultTrackOutput.java:429)
	at com.google.android.exoplayer2.extractor.DefaultTrackOutput.readData(DefaultTrackOutput.java:295)
	at com.google.android.exoplayer2.source.ExtractorMediaPeriod.readData(ExtractorMediaPeriod.java:339)
	at com.google.android.exoplayer2.source.ExtractorMediaPeriod$SampleStreamImpl.readData(ExtractorMediaPeriod.java:573)
	at com.google.android.exoplayer2.BaseRenderer.readSource(BaseRenderer.java:274)
	at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.feedInputBuffer(MediaCodecRenderer.java:621)
	at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:511)
	at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:479)
	at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:308)
	at android.os.Handler.dispatchMessage(Handler.java:98)
	at android.os.Looper.loop(Looper.java:154)
	at android.os.HandlerThread.run(HandlerThread.java:61)