On-the fly muxing of VALI encoded stream in mp4 container

[tvercaut]

Hi,

From a quick read through the material in the repo, it seems that demuxing with ffmpeg for further decoding purposes is already supported; but muxing for writing in a container such as mp4 is not.

What are the options for getting a proper mp4 while encoding with VALI? Here is what I could think of:

1. Write an Annex B stream to disc and then remux it with ffmpeg (see e.g. Muxing VALI video stream using FFmpeg: Timestamps and PTS Issues #29). It would however be nice to avoid the temporary file and additional "post-processing", especially for long videos.
2. Use a third-party python library to perform the muxing. I saw mentions of PyAV in a long discussion in VPF but the conclusion reads a bit worrisome: Ability to work along side PyAV? NVIDIA/VideoProcessingFramework#99 (comment) Does it work with VALI even if not with VPF?
3. Write the output of VALI to an ffmpeg input pipe. I couldn't find any code snippet for it but I assume it is possible. Did someone try this? Could the pipe mechanism even be abstracted with imageio-ffmpeg
4. Add ffmpeg muxing support directly in VALI. I am not sure if this is on the roadmap or is deemed out of scope.

Is there any recomendation on the best way to proceeed for this task?

Best wishes,
Tom

[RomanArzumanyan]

Hi @tvercaut

PyAV works fine with VALI as well as with VPF because it basically can take any Annex B input.

Muxing support isn’t something I’d like to avoid, it’s just not going to be much better than what PyAV does. Better out-of-the-box feature, but nothing unique anyway.

I’m now knee deep into PySurface refactoring (adding proper DLPack support to kill PyTorchVideoCodec subproject). After that I can take muxing support, that’s not too much of a job. Alternatively, I’m happy to accept your MR on this topic )

[tvercaut]

OK, thanks. I'll try PyAV then.