Adjust sample timestamps in MergingMediaPeriod for periods with different timestamp ranges

[brAzzi64]

Hi,

I have 3 video segments A, B, C with audio. I'd like to play them in sequence, while a different audio-only track plays seamlessly on top of the overall video (I want to discard the original audio of the video segments).

Is there a way to achieve this using ExoPlayer 2?

The documentation for MergingMediaSource states "The Timelines of the sources being merged must have the same number of periods", and I believe this poses a problem with taking the following approach:

a = ExtractorMediaSource(A-video-and-audio-source)
b = ExtractorMediaSource(B-video-and-audio-source)
c = ExtractorMediaSource(C-video-and-audio-source)
x = ExtractorMediaSource(X-audio-only-source)
overallSource = MergingMediaSource(ConcatenatingMediaSource(a, b ,c), x)

Is there a different approach that you think could work?

Do you think replacing the audio source by something like:

x = ConcatenatingMediaSource(ClippingMediaSource(X[0,1]), ClippingMediaSource(X[1,2]), ClippingMediaSource(X[2,3])))

where each ClippingMediaSource grabs a sequential segment of the original X-audio-only-source, should produce accurate results without any noticeable audio clipping?

Appreciate any guidance!

[tonihei]

The approach using ClippingMediaSource sounds feasible. To avoid noticeable clipping, you should ensure two things:

1. The duration of the clipping should exactly coincide with the end of an audio sample. The specified duration is in microseconds, so that should be possible. It should also be as close as possible to the corresponding video duration, so you'd need to know that in advance too.
2. The ClippingMediaSource has a constructor option to enableInitialDiscontinuity that needs to be set to false. This is safe for audio-only streams because every sample is a keyframe and no discontinuity is needed.

[brAzzi64]

Sounds good, thanks much for the pointers. I'll give this a try!

[tonihei]

Reopened as requested in #6649.

[brAzzi64]

Hi @tonihei, thanks for reopening. I finally got around to give this idea a try, but it seems like the approach of "re-assembling the audio track from its pieces" to then merge with the actual videos does not seem to be working.

I tried reducing this to a more basic, single-segment:

Merge(
  audioTrack=Clipping(Audio1, startTimeA, endTimeA)
  videoTrack=Clipping(Video1, startTimeB, endTimeB)
)  // making sure the clippings produce the same media length

but the clipping (of the same length for both tracks) does not seem to be working as expected. I'm thinking that my assumptions of how the combination of these components should behave might be wrong?

You can see my attempt on top of ExoPlayer's sample code, as a patch here, along with the results I'm observing (as comments in the code): https://gist.github.com/brAzzi64/d9aa2caedff9dbcc7b04d182add83897

Once I get this working, what I'd ultimately want to achieve is what I mentioned on my original post that you mentioned should work:

Merge(
  audioTrack=Concat(Clipping(Audio1, startTime0, endTime0), Clipping(Audio1, startTime1, endTime1), ...),
  videoTrack=Concat(Clipping(Video1), Clipping(Video2), ...)
)