IMA post-roll ad marker lost when video duration is less than audio duration

[zzzhc]

Issue description

We have some assets(dash stream) which's video duration is less than audio duration, and mediaPresentationDuration equals audio duration, for example, the audio duration is 5793364041us, video duration is 5793162375us, mediaPresentationDuration is 5793364ms.
In this case, the post-roll ad marker won't show.

Reproduction steps

The asset content is protected by drm. I will show which codes caused the issue.

in dash MPD, mediaPresentationDuration="PT1H36M33.364041S", it's 5793364ms.

in PlayerControlView#updateTimeline

// Line 874
for (int j = window.firstPeriodIndex; j <= window.lastPeriodIndex; j++) {
          timeline.getPeriod(j, period);
          int periodAdGroupCount = period.getAdGroupCount();
          for (int adGroupIndex = 0; adGroupIndex < periodAdGroupCount; adGroupIndex++) {
            long adGroupTimeInPeriodUs = period.getAdGroupTimeUs(adGroupIndex);
            if (adGroupTimeInPeriodUs == C.TIME_END_OF_SOURCE) {
              if (period.durationUs == C.TIME_UNSET) {
                // Don't show ad markers for postrolls in periods with unknown duration.
                continue;
              }
              adGroupTimeInPeriodUs = period.durationUs;
            }
            long adGroupTimeInWindowUs = adGroupTimeInPeriodUs + period.getPositionInWindowUs();
            if (adGroupTimeInWindowUs >= 0 && adGroupTimeInWindowUs <= window.durationUs) {
              if (adGroupCount == adGroupTimesMs.length) {
                int newLength = adGroupTimesMs.length == 0 ? 1 : adGroupTimesMs.length * 2;
                adGroupTimesMs = Arrays.copyOf(adGroupTimesMs, newLength);
                playedAdGroups = Arrays.copyOf(playedAdGroups, newLength);
              }
              adGroupTimesMs[adGroupCount] = C.usToMs(durationUs + adGroupTimeInWindowUs);
              playedAdGroups[adGroupCount] = period.hasPlayedAdGroup(adGroupIndex);
              adGroupCount++;
            }
          }
        }

when adGroupTimeInPeriodUs == C.TIME_END_OF_SOURCE), adGroupTimeInPeriodUs = period.durationUs;
adGroupTimeInWindowUs = adGroupTimeInPeriodUs + period.getPositionInWindowUs();

in DashMediaSource#getWindow,

    @Override
    public Window getWindow(
        int windowIndex, Window window, boolean setTag, long defaultPositionProjectionUs) {
      Assertions.checkIndex(windowIndex, 0, 1);
      long windowDefaultStartPositionUs = getAdjustedWindowDefaultStartPositionUs(
          defaultPositionProjectionUs);
      Object tag = setTag ? windowTag : null;
      boolean isDynamic =
          manifest.dynamic
              && manifest.minUpdatePeriodMs != C.TIME_UNSET
              && manifest.durationMs == C.TIME_UNSET;
      return window.set(
          tag,
          presentationStartTimeMs,
          windowStartTimeMs,
          /* isSeekable= */ true,
          isDynamic,
          windowDefaultStartPositionUs,
          windowDurationUs,
          /* firstPeriodIndex= */ 0,
          /* lastPeriodIndex= */ getPeriodCount() - 1,
          offsetInFirstPeriodUs);
    }

windowDurationUs is calcuated from lastPeriodSeekInfo.availableEndTimeUs - firstPeriodSeekInfo.availableStartTimeUs.

// DashMediaSource line 939:
SeekInfo firstPeriodSeekInfo = PeriodSeekInfo.createPeriodSeekInfo(manifest.getPeriod(0),
        manifest.getPeriodDurationUs(0));
    PeriodSeekInfo lastPeriodSeekInfo = PeriodSeekInfo.createPeriodSeekInfo(
        manifest.getPeriod(lastPeriodIndex), manifest.getPeriodDurationUs(lastPeriodIndex));
    // Get the period-relative start/end times.
long currentStartTimeUs = firstPeriodSeekInfo.availableStartTimeUs;
long currentEndTimeUs = lastPeriodSeekInfo.availableEndTimeUs;
...
long windowDurationUs = currentEndTimeUs - currentStartTimeUs;

PeriodSeekInfo.availableEndTimeUs probably equals Math.min(audioDuration, videoDuration).

// DashMediaSource line 1078: 
public static PeriodSeekInfo createPeriodSeekInfo(
        com.google.android.exoplayer2.source.dash.manifest.Period period, long durationUs) {
      int adaptationSetCount = period.adaptationSets.size();
      long availableStartTimeUs = 0;
      long availableEndTimeUs = Long.MAX_VALUE;
      boolean isIndexExplicit = false;
      boolean seenEmptyIndex = false;

      boolean haveAudioVideoAdaptationSets = false;
      for (int i = 0; i < adaptationSetCount; i++) {
        int type = period.adaptationSets.get(i).type;
        if (type == C.TRACK_TYPE_AUDIO || type == C.TRACK_TYPE_VIDEO) {
          haveAudioVideoAdaptationSets = true;
          break;
        }
      }

      for (int i = 0; i < adaptationSetCount; i++) {
        AdaptationSet adaptationSet = period.adaptationSets.get(i);
        // Exclude text adaptation sets from duration calculations, if we have at least one audio
        // or video adaptation set. See: https://github.com/google/ExoPlayer/issues/4029
        if (haveAudioVideoAdaptationSets && adaptationSet.type == C.TRACK_TYPE_TEXT) {
          continue;
        }

        DashSegmentIndex index = adaptationSet.representations.get(0).getIndex();
        if (index == null) {
          return new PeriodSeekInfo(true, 0, durationUs);
        }
        isIndexExplicit |= index.isExplicit();
        int segmentCount = index.getSegmentCount(durationUs);
        if (segmentCount == 0) {
          seenEmptyIndex = true;
          availableStartTimeUs = 0;
          availableEndTimeUs = 0;
        } else if (!seenEmptyIndex) {
          long firstSegmentNum = index.getFirstSegmentNum();
          long adaptationSetAvailableStartTimeUs = index.getTimeUs(firstSegmentNum);
          availableStartTimeUs = Math.max(availableStartTimeUs, adaptationSetAvailableStartTimeUs);
          if (segmentCount != DashSegmentIndex.INDEX_UNBOUNDED) {
            long lastSegmentNum = firstSegmentNum + segmentCount - 1;
            long adaptationSetAvailableEndTimeUs = index.getTimeUs(lastSegmentNum)
                + index.getDurationUs(lastSegmentNum, durationUs);
            availableEndTimeUs = Math.min(availableEndTimeUs, adaptationSetAvailableEndTimeUs);
          }
        }
      }
      return new PeriodSeekInfo(isIndexExplicit, availableStartTimeUs, availableEndTimeUs);
    }

period duration equals dash mediaPresentationDuration

// in DashManifest line 162
public final long getPeriodDurationMs(int index) {
    return index == periods.size() - 1
        ? (durationMs == C.TIME_UNSET ? C.TIME_UNSET : (durationMs - periods.get(index).startMs))
        : (periods.get(index + 1).startMs - periods.get(index).startMs);
  }

back to PlayerControlView, for postroll:
adGroupTimeInWindowUs=period.durationUs+period.getPositionInWindowUs()=dash mediaPresentationDuration=5793364000us,
window.durationUs=lastPeriodSeekInfo.availableEndTimeUs-firstPeriodSeekInfo.availableStartTimeUs = Math.min(audioDuration, videoDuration)=5793162375us
adGroupTimeInWindowUs > window.durationUs, the postroll won't show.

if (adGroupTimeInPeriodUs == C.TIME_END_OF_SOURCE) {
              if (period.durationUs == C.TIME_UNSET) {
                // Don't show ad markers for postrolls in periods with unknown duration.
                continue;
              }
              adGroupTimeInPeriodUs = period.durationUs;
            }
            long adGroupTimeInWindowUs = adGroupTimeInPeriodUs + period.getPositionInWindowUs();
            if (adGroupTimeInWindowUs >= 0 && adGroupTimeInWindowUs <= window.durationUs) {

Version of ExoPlayer being used

v2.10.5

Device(s) and version(s) of Android being used

emulator Nexus 5x API 28

[zzzhc]

I'm confused with availableEndTimeUs = Math.min(availableEndTimeUs, adaptationSetAvailableEndTimeUs);.
availableEndTimeUs = Math.max(availableEndTimeUs, adaptationSetAvailableEndTimeUs); is better?

[ojw28]

We have some assets(dash stream) which's video duration is less than audio duration, and mediaPresentationDuration equals audio duration, for example, the audio duration is 5793364041us, video duration is 5793162375us, mediaPresentationDuration is 5793364ms.
In this case, the post-roll ad marker won't show.

It's hard to be sure without seeing the DASH manifest, but it sounds as though this is not inter-operable DASH content according to the new DASH-IF interoperability definition. In particular:

In a static MPD a representation SHALL contain enough segment references to cover the entire time span of the period.

In the case you describe it sounds like the video representation falls into the "end not covered" case as shown in Figure 6 of that document.

Depending on how your DASH manifest is structured, it might be possible for you just to "fix" the video duration to be the same as the audio duration in the manifest.

[ojw28]

On the UI side, I'm not sure why we don't display ad markers at all in the case where they're not within the window duration. @andrewlewis - Is there a reason for doing this, as opposed to still displaying the marker at the very end of the timeline?

[ojw28]

Assigning to @andrewlewis for the UI question.