Seek succeeds but any frame read is all zeros [revisited]

[tikuma-lsuhsc]

This issue was originally discussed in Issue #349 in 2018 (moved to Discussion as #928), but it wasn't resolved then (as far as I can tell). I just came across this very issue and may be able to shed additional light.

Here is a minimal example (tested with v15.0.0 running on Python v3.13.3 in Linux):

import av

seek_first = False
with av.open(avifile) as fmt:
    vst = fmt.streams.video[0]
    if seek_first:
        fmt.seek(0, stream=vst, backward=True, any_frame=False)
    frame = next(fmt.decode(video=0))
    data = frame.to_ndarray(format="rgb24")
    assert np.any(data!=0) # errors out only if seek_first=True

The assertion appears to fail only with my AVI file (uncompressed video from highspeed video camera). If I use an MP4 file, it worked properly. Also, every frame of this AVI file is a key frame. I cannot share my video file but this info may help debugging the issue.

Thanks, and Thanks for the awesome package

P.S., I'd be happy to move this post under #928 if appropriate, but it does appear to be a real issue.

[tikuma-lsuhsc]

A follow up on generating a video file with this issue:

ffmpeg -f lavfi -i testsrc=duration=5.3:size=qcif:rate=10 -vcodec rawvideo -pix_fmt pal8 test_video.avi

The main culprit seems to be the rawvideo PAL8 format.

[tikuma-lsuhsc]

This issue is addressable with post-v15.1 releases with the PR #2003.

The cause of this behavior is that InputContainer.seek() clears the codec context's palette. This is FFmpeg behavior and not PyAV behavior. In order to restore the palette, we must (1) intercept the palette(s) which are attached to packets as a sidedata, (2) make a copy of them, and (3) attach the appropriate one to the first packet obtained after a seek.

Here is an example:

with av.open(avifile) as fmt:

    vst = fmt.streams.video[0]

    # get the palette from the first packet
    pkt = next(fmt.demux(vst))
    assert pkt.has_sidedata("palette") # expect the palette to be defined in the first packet
    palette = pkt.get_sidedata('palette')

    # seek (rewind to the beginning, which does not re-retrieve the palette)
    fmt.seek(0, stream=vst, backward=True, any_frame=False)

    # manually set the palette to the first post-seek packet so codec can re-initialize its palette
    pkt = next(fmt.demux(vst))
    pkt.set_sidedata(palette)

    # retrieve the first frame
    frame = pkt.decode()[0] # one video frame per packet guaranteed

    # thereafter, you can use the InputContainer's decode()
    frame = next(fmt.decode(vst))