Skip to content

Decoding and encoding RTSP  #777

@GustavoStahl

Description

@GustavoStahl

Overview

I'm using PyAV to read and write an RTSP link on disk (storing audio and video), but I haven't found a good way to do this since most of my implementations end up with audio and video pops. Bellow is my current test algorithm.

import av
import cv2

video_path = "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov"
rtsp = av.open(video_path)

in_streams = [stream for stream in rtsp.streams] 
a_stream = rtsp.streams.audio[0]
v_stream = rtsp.streams.video[0]

output_container = av.open('live_stream.mp4', mode='w')
out_audio = output_container.add_stream(template=a_stream)
out_video = output_container.add_stream(template=v_stream)

audio_demux = iter(rtsp.demux(a_stream))
video_demux = iter(rtsp.demux(v_stream))

frame_template = av.VideoFrame(v_stream.width, v_stream.height, format="yuv420p")

counter = 0
first_packet = True
while True:
    try:
        counter += 1

        audio_packet = next(audio_demux)
        video_packet = next(video_demux)

        if first_packet:
            audio_packet.pts = 0
            audio_packet.dts = 0
            video_packet.pts = 0
            video_packet.dts = 0
            first_packet = False

        if audio_packet.dts is None or video_packet.dts is None:
            continue

        ##### Decode/Encode frame #####
        frame = video_packet.decode()
        if len(frame) != 0:
            frame = frame[0].to_ndarray(format='yuv420p').copy()
            frame = cv2.circle(frame, (500,500), 50, (0,0,255), -1) 
            frame = frame_template.from_ndarray(frame, format="yuv420p")
            video_packet = out_video.encode(frame)
        ###############################

        audio_packet.stream = out_audio
        video_packet.stream = out_video

        output_container.mux(audio_packet)
        output_container.mux(video_packet)

        print(counter)
        if counter > 200:
            raise StopIteration

    except StopIteration:
        print("[INFO] Success")
        output_container.close()
        break

    ##### Comment this handler for more information about the raised error #####
    except Exception as e:
        print(f"[ERROR] Demuxing error: {e}")
        break

Expected behavior

Expected an mp4 output containing the whole video from the provided RTSP with the exact image and audio quality.

Actual behavior

If I disregard the part of my code that decodes the video packet as a NumPy array and encodes it again, then a video file is written on disk, but containing audio and video pops. Otherwise, an error is raised when the encoding tries to be applied on the np::ndarray.

Traceback (Error raised when maintaining the frame decoding/encoding):

  _ = out_video.encode(frame)
  File "av/stream.pyx", line 155, in av.stream.Stream.encode
  File "av/codec/context.pyx", line 476, in av.codec.context.CodecContext.encode
  File "av/codec/context.pyx", line 391, in av.codec.context.CodecContext._send_frame_and_recv
  File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument; last error log: [h264] co located POCs unavailable
Traceback (most recent call last):
  File "av/container/output.pyx", line 25, in av.container.output.close_output
TypeError: 'NoneType' object is not iterable
Exception ignored in: 'av.container.output.OutputContainer.__dealloc__'
Traceback (most recent call last):
  File "av/container/output.pyx", line 25, in av.container.output.close_output
TypeError: 'NoneType' object is not iterable

Investigation

  • Regarding only writing the RTSP on disk without decoding each frame as np::ndarray, I've tested isolating each output stream (audio and video), and in both cases the output is fine, but when they are subsequent the pops happen.
  • Regarding only decoding each frame as np::ndarray, I've tried checking the writer and frame dims, changing colorspaces, and pre-allocating a VideoFrame object to then load the array on it.

Research

I have done the following:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions