Muxing VALI video stream using FFmpeg: Timestamps and PTS Issues

[Choiuijin1125]

Hello,

I'm currently working on a video processing project that involves a specific workflow where I decode a video, process the frames through a vision AI model, and then encode the video again. My workflow is as follows:

Decode video -> Convert frame to tensor -> Process frame with a Vision AI model -> Convert tensor back to a surface -> Encode the video

After encoding the video, I obtain a raw H.265 Annex B stream. The final code snippet for the encoding process looks like this:

while True:
    success = nvEnc.FlushSinglePacket(encFrame)
    if success:
        byteArray = bytearray(encFrame)
        dstFile.write(byteArray)
    else:
        break

To mux the raw stream into an MP4 container, I'm using the following FFmpeg command:

ffmpeg -r 29.97 -i input.h265 -vcodec copy output.mp4

However, this results in the following error messages, indicating issues with unset timestamps and missing PTS values:

Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly.
[mp4 @ 0x56354b9b5a40] pts has no value

I'm looking for guidance on how to resolve these errors and successfully mux H.265 stream into an MP4 container. Any advice on how to adjust my workflow or the FFmpeg command to fix the timestamp and PTS issues would be greatly appreciated.

[RomanArzumanyan]

Hi @Choiuijin1125

You see those ffmpeg warnings because video track produced by VALI doesn't store PTS and DTS information. Reason for that is PTS and DTS are not part of video coding standards. Hence VALI has no place to insert them into compressed video frame which PyNvEncoder returns. These timestamps only exist on the container level and are to be inserted by muxer.

One way to avoid the ffmpeg warnings is to pass genpts option. It will tell ffmpeg to generate timestamps based on FPS:

ffmpeg -fflags +genpts -r 29.97 -i input.h266 -c:v copy -y output.mp4

P. S.
This is important question which is often asked so I'm moving it to Discussions to be permanently visible. I've also edited the title slightly to make it easier to search.

[RomanArzumanyan]

Hi @Choiuijin1125

A quick search shows that the missing igndts flag may be the culprit.

Original Stack Overflow and FFMpeg links:

• https://superuser.com/questions/1607636/ffmpeg-hangs-when-using-fflags-genpts
• https://ffmpeg.org/ffmpeg-formats.html

One-liner:

ffmpeg -fflags +genpts+igndts -r 29.97 -i input.h266 -c:v copy -y output.mp4

Please note that some warning messages may still be produced.

[Choiuijin1125]

@RomanArzumanyan

ffmpeg -fflags +genpts+igndts -r 29.97 -i input.h266 -c:v copy -y output.mp4 still not working.

I think the process works fine until the flush operation, which seems to introduce non-monotonically increasing DTS (Decoding Time Stamp) values, resulting in an error when flushing.

Here is a snippet of the code I'm using, based on the SampleCupy.py example:

    encFrame = np.ndarray(shape=(0), dtype=np.uint8)
    while True:
        # Decode NV12 surface
        src_surface = nvDec.DecodeSingleSurface()
        if src_surface.Empty():
            break

        # Convert to packed RGB: HWC , planar CHW
        rgb_sur = to_rgb.run(src_surface)
        if rgb_sur.Empty():
            break

        # PROCESS YOUR TENSOR HERE.
        # THIS DUMMY PROCESSING JUST ADDS RANDOM ROTATION.
        src_array = cpnvc.SurfaceToArray(rgb_sur)
        dst_array = contrast_boost(src_array)
        dst_array = grayscale(dst_array)
        surface_rgb = cpnvc.ArrayToSurface(dst_array, gpu_id)

        # Convert back to NV12
        dst_surface = to_nv12.run(surface_rgb)
        if src_surface.Empty():
            break

        # Encode
        success = nvEnc.EncodeSingleSurface(dst_surface, encFrame)
        if success:
            byteArray = bytearray(encFrame)
            dstFile.write(byteArray)

    # Encoder is asynchronous, so we need to flush it
    while True:
        success = nvEnc.FlushSinglePacket(encFrame)
        if success:
            byteArray = bytearray(encFrame)
            dstFile.write(byteArray)
        else:
            break

If I skip the flushing part, the video generates without errors and plays fine in media players.
(I didn't check about un flushed frame)

And if I use PyAV I get an error when flushing part.

        # Encoder is asynchronous, so we need to flush it
        while True:
            success = nvEnc.FlushSinglePacket(encFrame)
            if success:
                byteArray = bytearray(encFrame)
                pkt = av.packet.Packet(byteArray)
                pts_time = pdata.pts
                pkt.pts = pts_time
                pkt.dts = pts_time
                pkt.stream = out_stream  # attach pkt to the stream                
                dstFile.mux(pkt)

ERROR:libav.mp4:Application provided invalid, non monotonically increasing dts to muxer in stream 0: 17998981 >= 0

[RomanArzumanyan]

@Choiuijin1125

As I've said before, PTS and DTS have nothing to do with encoder output. E. g. you can save encoded H.264 video to disk without muxing it into container and play with VLC. It will not have any timestamp information but will be decoded and shown on display just fine.

I don't know how exactly you encode your video but https://github.com/NVIDIA/VideoProcessingFramework/blob/master/samples/SampleCupy.py shows the right approach. First you read all the raw frames you have and submit them to encod, then you flush encoder internal queue with FlushSinglePacket method.

BTW what exactly do you mean by saying ffmpeg -fflags +genpts+igndts -r 29.97 -i input.h266 -c:v copy -y output.mp4 doesn't work? Does it produce empty or broken file?

[Choiuijin1125]

@RomanArzumanyan

Oh, I'm sorry, the generated output.mp4 video does not play smoothly in some media players. While VLC manages to handle it, other players encounter issue, when trying to skip ahead by 5 seconds, leading to the playback getting stuck and then stopping.

[geheiligt]

I had some issues with the muxing as well. While the method given in [SampleCupy.py] runs, it produced output incompatible with most video players.

Adding this value to each av packet did the trick for me:
pkt.time_base = Fraction(1, fps)