Flac in mp4 using DASH does not seem to work

[kgrevehagen]

[REQUIRED] Content description

FLAC in mp4 using DASH does not seem to work.
The same track in regular flac container works as it should.
I'm of course using the Mp4Extractor, and playing AAC in the same setup works flawlessly. I have tested with various configuration in regards to renderer, explained in the bug report section below.

[REQUIRED] Link to test content

I can probably provide if needed. Do you have any test content that should work? I tried our .mpd file in VLC on my computer, and that played as it should.

[REQUIRED] Version of ExoPlayer being used

2.10.4 forked with this change included.

[REQUIRED] Device(s) and version(s) of Android being used

Pixel 3 Android 9 and 10.

[REQUIRED] A full bug report captured from the device

Full bug report can be provided if needed.
Here are some logs from my different tries:

Using LibflacAudioRenderer: It is not fetching the dash segments, but plays without sound(time is passing, can pause, play etc) and gives me no error message.

Using default MediaCodecAudioRenderer(afaik understood from #6392 this should work because of platform decoder on Android 8.1 and above) it is fetching dash segments, but does not play like above, and I'm getting the following error:

D/CCodec: allocate(c2.android.flac.decoder)
I/Codec2Client: Available Codec2 services: "default" "software"
I/Codec2Client: Creating a Codec2 client to service "default"
I/Codec2Client: Client to Codec2 service "default" created
I/CCodec: setting up 'default' as default (vendor) store
I/Codec2Client: Creating a Codec2 client to service "default"
I/Codec2Client: Client to Codec2 service "default" created
E/Codec2Client: createComponent(c2.android.flac.decoder) -- call failed: NOT_FOUND.
I/Codec2Client: Creating a Codec2 client to service "software"
I/Codec2Client: Client to Codec2 service "software" created
I/CCodec: Created component [c2.android.flac.decoder]
D/CCodecConfig: read media type: audio/flac
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.max-count.values
D/ReflectedParamUpdater: extent() != 1 for single value type: output.subscribed-indices.values
D/ReflectedParamUpdater: extent() != 1 for single value type: input.buffers.allocator-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.allocator-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.allocator-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.pool-ids.values
D/ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.pool-ids.values
I/CCodecConfig: query failed after returning 8 values (BAD_INDEX)
D/CCodecConfig: c2 config is Dict {
      c2::u32 coded.bitrate.value = 768000
      c2::u32 input.buffers.max-size.value = 32768
      c2::u32 input.delay.value = 0
      string input.media-type.value = "audio/flac"
      string output.media-type.value = "audio/raw"
      c2::u32 raw.channel-count.value = 1
      c2::u32 raw.pcm-encoding.value = 0
      c2::u32 raw.sample-rate.value = 44100
    }
D/CCodecConfig: c2 config is Dict {
      c2::u32 coded.bitrate.value = 768000
      c2::u32 input.buffers.max-size.value = 32768
      c2::u32 input.delay.value = 0
      string input.media-type.value = "audio/flac"
      string output.media-type.value = "audio/raw"
      c2::u32 raw.channel-count.value = 2
      c2::u32 raw.pcm-encoding.value = 0
      c2::u32 raw.sample-rate.value = 44100
    }
W/Codec2Client: query -- param skipped: index = 1107298332.
D/CCodec: setup formats input: AMessage(what = 0x00000000) = {
      int32_t channel-count = 2
      int32_t max-input-size = 32768
      string mime = "audio/flac"
      int32_t pcm-encoding = 2
      int32_t sample-rate = 44100
    } and output: AMessage(what = 0x00000000) = {
      int32_t channel-count = 2
      string mime = "audio/raw"
      int32_t pcm-encoding = 2
      int32_t sample-rate = 44100
    }
W/Codec2Client: query -- param skipped: index = 1342179345.
W/Codec2Client: query -- param skipped: index = 2415921170.
D/CCodecBufferChannel: [c2.android.flac.decoder#18] Created input block pool with allocatorID 16 => poolID 17 - OK (0)
I/CCodecBufferChannel: [c2.android.flac.decoder#18] Created output block pool with allocatorID 16 => poolID 116 - OK
D/CCodecBufferChannel: [c2.android.flac.decoder#18] Configured output block pool ids 116 => OK
E/ion: ioctl c0044901 failed with code -1: Invalid argument
D/CCodecBufferChannel: [c2.android.flac.decoder#18] work failed to complete: 14
E/MediaCodec: Codec reported err 0xe, actionCode 0, while in state 6
D/CCodecBufferChannel: [c2.android.flac.decoder#18] work failed to complete: 22
E/MediaCodec: Codec reported err 0x16, actionCode 0, while in state 0
E/ExoPlayerImplInternal: Internal runtime error.
    java.lang.IllegalStateException
        at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
        at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2789)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:1437)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:653)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
W/System.err: com.google.android.exoplayer2.ExoPlaybackException: java.lang.IllegalStateException
W/System.err:     at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:397)
W/System.err:     at android.os.Handler.dispatchMessage(Handler.java:103)
W/System.err:     at android.os.Looper.loop(Looper.java:214)
W/System.err:     at android.os.HandlerThread.run(HandlerThread.java:67)
W/System.err: Caused by: java.lang.IllegalStateException
E/ExoPlayerImplInternal: Disable failed.
    java.lang.IllegalStateException
        at android.media.MediaCodec.native_flush(Native Method)
        at android.media.MediaCodec.flush(MediaCodec.java:2194)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.flushOrReleaseCodec(MediaCodecRenderer.java:702)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.onDisabled(MediaCodecRenderer.java:580)
        at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.onDisabled(MediaCodecAudioRenderer.java:625)
        at com.google.android.exoplayer2.BaseRenderer.disable(BaseRenderer.java:158)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.disableRenderer(ExoPlayerImplInternal.java:1088)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:851)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:813)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:399)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
W/System.err:     at android.media.MediaCodec.native_dequeueOutputBuffer(Native Method)
W/System.err:     at android.media.MediaCodec.dequeueOutputBuffer(MediaCodec.java:2789)
W/System.err:     at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.drainOutputBuffer(MediaCodecRenderer.java:1437)
W/System.err:     at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:653)
W/System.err:     at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
W/System.err:     at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
W/System.err: 	... 3 more
E/ExoPlayerImplInternal: Reset failed.
    java.lang.IllegalStateException
        at android.media.MediaCodec.native_stop(Native Method)
        at android.media.MediaCodec.stop(MediaCodec.java:2147)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.releaseCodec(MediaCodecRenderer.java:609)
        at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.onReset(MediaCodecRenderer.java:587)
        at com.google.android.exoplayer2.audio.MediaCodecAudioRenderer.onReset(MediaCodecAudioRenderer.java:635)
        at com.google.android.exoplayer2.BaseRenderer.reset(BaseRenderer.java:164)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.resetInternal(ExoPlayerImplInternal.java:860)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.stopInternal(ExoPlayerImplInternal.java:813)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:399)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
I/ExoPlayerImpl: Release 5e5c6b5 [ExoPlayerLib/2.10.4] [blueline, Pixel 3, Google, 29] [goog.exo.core, goog.exo.dash, goog.exo.okhttp]

[ojw28]

We were previously provided with some FLAC in DASH content in #4883, and I'm pretty sure we had it working at the time. Unfortunately it appears the samples are no longer available.

[ojw28]

@ronak2121 - Do you have any DASH/FLAC test streams that are still available?
@kgrevehagen - We probably need test content. It seems at least possible that your segmentation of FLAC into samples to put inside of the MP4 container is non-spec compliant.

As a general point, you're making life really hard for yourself by using FLAC. Do you really need to use it? Doing what nearly everyone else does (i.e. AAC) is definitely the path of least resistance, unless that's a non-started from a product perspective.

[kgrevehagen]

Here is an example of a FLAC in mp4/DASH.

[kgrevehagen]

Tested this in the ExoPlayer Sample app, and when I build the flac extension and set DefaultRenderersFactory.EXTENSION_RENDERER_MODE_PREFER on the DefaultExtractorsFactory, it gives me the following error:

V/FLACParser: FLACParser::FLACParser
E/FLACParser: FLACParser::errorCallback status=0
E/FLACParser: metadata decoding failed
E/ExoPlayerImplInternal: Playback error.
    com.google.android.exoplayer2.ExoPlaybackException: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Failed to decode StreamInfo
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.maybeInitDecoder(SimpleDecoderAudioRenderer.java:631)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.onInputFormatChanged(SimpleDecoderAudioRenderer.java:697)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:277)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Failed to decode StreamInfo
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.<init>(FlacDecoder.java:65)
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:70)
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:31)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.maybeInitDecoder(SimpleDecoderAudioRenderer.java:624)
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.onInputFormatChanged(SimpleDecoderAudioRenderer.java:697) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:277) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326) 
        at android.os.Handler.dispatchMessage(Handler.java:103) 
        at android.os.Looper.loop(Looper.java:214) 
        at android.os.HandlerThread.run(HandlerThread.java:67) 
     Caused by: com.google.android.exoplayer2.ParserException: Failed to decode stream metadata
        at com.google.android.exoplayer2.ext.flac.FlacDecoderJni.decodeStreamMetadata(FlacDecoderJni.java:149)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.<init>(FlacDecoder.java:63)
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:70) 
        at com.google.android.exoplayer2.ext.flac.LibflacAudioRenderer.createDecoder(LibflacAudioRenderer.java:31) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.maybeInitDecoder(SimpleDecoderAudioRenderer.java:624) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.onInputFormatChanged(SimpleDecoderAudioRenderer.java:697) 
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:277) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575) 
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326) 
        at android.os.Handler.dispatchMessage(Handler.java:103) 
        at android.os.Looper.loop(Looper.java:214) 
        at android.os.HandlerThread.run(HandlerThread.java:67)

which tells me that decodeStreamMetadata() fails in FlacDecoder.java.

    FlacStreamMetadata streamMetadata;
    try {
      streamMetadata = decoderJni.decodeStreamMetadata();
    } catch (ParserException e) {
      throw new FlacDecoderException("Failed to decode StreamInfo", e);
    } catch (IOException | InterruptedException e) {
      // Never happens.
      throw new IllegalStateException(e);
    }

I tried to set correct FlacStreamMetaData by doing this instead(don't know if this is correct or okay to do, but it gives me correct FlacStreamMetaData, at least):

FlacStreamMetadata streamMetadata = new FlacStreamMetadata(initializationData.get(0), 0);

but that gives me the following error:

V/FLACParser: FLACParser::FLACParser
D/EventLogger: decoderInitialized [2.51, 0.00, window=0, period=0, audio, libflac]
D/EventLogger: decoderInputFormatChanged [2.51, 0.00, window=0, period=0, audio, id=0, mimeType=audio/flac, bitrate=1046930, codecs=flac, channels=2, sample_rate=44100, language=und]
E/FLACParser: FLACParser::readBuffer write invalid blocksize 4096
E/ExoPlayerImplInternal: Playback error.
    com.google.android.exoplayer2.ExoPlaybackException: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Frame decoding failed
        at com.google.android.exoplayer2.audio.SimpleDecoderAudioRenderer.render(SimpleDecoderAudioRenderer.java:302)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:575)
        at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:326)
        at android.os.Handler.dispatchMessage(Handler.java:103)
        at android.os.Looper.loop(Looper.java:214)
        at android.os.HandlerThread.run(HandlerThread.java:67)
     Caused by: com.google.android.exoplayer2.ext.flac.FlacDecoderException: Frame decoding failed
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:101)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:32)
        at com.google.android.exoplayer2.decoder.SimpleDecoder.decode(SimpleDecoder.java:224)
        at com.google.android.exoplayer2.decoder.SimpleDecoder.run(SimpleDecoder.java:189)
        at com.google.android.exoplayer2.decoder.SimpleDecoder.access$000(SimpleDecoder.java:25)
        at com.google.android.exoplayer2.decoder.SimpleDecoder$1.run(SimpleDecoder.java:67)
     Caused by: com.google.android.exoplayer2.ext.flac.FlacDecoderJni$FlacFrameDecodeException: Cannot decode FLAC frame
        at com.google.android.exoplayer2.ext.flac.FlacDecoderJni.decodeSample(FlacDecoderJni.java:187)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:99)
        at com.google.android.exoplayer2.ext.flac.FlacDecoder.decode(FlacDecoder.java:32) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder.decode(SimpleDecoder.java:224) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder.run(SimpleDecoder.java:189) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder.access$000(SimpleDecoder.java:25) 
        at com.google.android.exoplayer2.decoder.SimpleDecoder$1.run(SimpleDecoder.java:67) 

Hopefully you can test the provided url and see if you can get some more info on what is happening here.

[ojw28]

I've figured out what's going on here. It's related to Format.initializationData, which is used to configure the decoder:

• Samsung's FLAC decoder (OMX.SEC.flac.dec) works fine with the provided stream. I.e. playback works on a Galaxy S8, without using the FLAC extension.
• Google's FLAC decoder (OMX.google.flac.decoder) requires the FLAC stream marker to be prefixed onto the initialization data, which is how FLAC codec private data is defined in the Matroska container here: "The private data contains all the header/metadata packets before the first data packet. These include the first header packet containing only the word fLaC as well as all metadata packets.". In MP4 the FLAC stream marker is not prefixed into the initialization data, which is why playback fails.
• The FLAC extension also seems to require the FLAC stream marker to be present.

I was able to get all cases working by changing this code block to add the FLAC stream marker:

int childAtomBodySize = childAtomSize - Atom.FULL_HEADER_SIZE;
initializationData = new byte[childAtomBodySize + 4];
initializationData[0] = 0x66;
initializationData[1] = 0x4C;
initializationData[2] = 0x61;
initializationData[3] = 0x43;
parent.setPosition(childPosition + Atom.FULL_HEADER_SIZE);
parent.readBytes(initializationData, /* offset= */ 4, childAtomBodySize);

I'd like to double check that the approach above is the right thing to do before submitting a fix (and, obviously, this should be for TYPE_dfLa only).

[ronak2121]

Hey sorry I’m not in the office and can’t get my test streams back up. I had previously generated the above FLAC streams using FFmpeg.

I’ll be in the office on Monday and can help then.

How are you generating them? @kgrevehagen