Sanitize videos with Dangerzone

[apyrgio]

Proposed representations

Sanitizing a video means that we need to represent the video and the audio stream in the simplest format possible, with no metadata and headers.

For videos, that would be a sequence of images. In FFmpeg terms, this is called rawvideo, and is very similar to the way we handle RGB pixel streams right now. In order to decode it, we'd need to specify the frame rate, pixel format (rgb24 is the simplest one), and video size (width x height). In Dangerzone, we standardize on RGB image formats, and can already pass the width/height of each page.

For audio, that would be a sequence of audio samples. For mono streams, this sequence is linear, whereas for stereo streams, it alternates between left and right channel. In FFmpeg terms, this is represented with the pcm_s16le audio codec by default (read more).

As for re-enconding the video, it seems that the best way to move forward is to use the H.265 codec, and the MP4 container format.

Commands

Sample file (240KiB)

Get video info (width, height, frame rate) (runs in sandbox)

$ ffprobe -v error -select_streams v:0 -show_entries stream=width,height,r_frame_rate -of default=noprint_wrappers=1 sample.mp4
width=960
height=540
r_frame_rate=25/1

Decode video to raw format (runs in sandbox)

ffmpeg -i sample.mp4 -c:v rawvideo -pix_fmt rgb24 out_video.rgb

Note

For the sample clip I posted above, the raw video took 55MiB of storage space.

Decode audio to raw format (runs in sandbox)

ffmpeg -i sample.mp4 -ar 44100 -ac 2  -c:a pcm_s16le -f s16le out_audio.raw

Encode raw audio and video streams to H.265

ffmpeg -f s16le -ar 44100 -ac 2 -i out_audio.raw -f rawvideo -pix_fmt rgb24 -s 960x540 -r 25 -i out_video.rgb -c:v libx265 -c:a aac out.mp4

Note

The output file is 337K. So, similarly to images, it seems that the video gets inflated as well. We can compress it further by tweaking the crf, pix_fmt, and preset values, but that can be done at a later stage.

[apyrgio]

The above is a way to perform the sanitization manually, using intermediate files. We don't want that, as these files can get very large, very quickly. In our case, we can take advantage of the stdout and stderr of the Dangerzone container, and pipe the video and audio stream to those pipes.

Here's a proof-of-concept using named pipes (Linux only) to show that this is indeed possible:

#!/bin/bash

set -ex

rm -f video_pipe audio_pipe out.mp4
mkfifo video_pipe
mkfifo audio_pipe

ffmpeg -i sample.mp4 -ar 44100 -ac 2 -c:a pcm_s16le -f s16le pipe:1 > audio_pipe &
ffmpeg  -i sample.mp4 -c:v rawvideo  -pix_fmt rgb24 -f rawvideo pipe:1 > video_pipe &
ffmpeg -f s16le -ar 44100 -ac 2 -i audio_pipe -f rawvideo -pix_fmt rgb24 -s 960x540 -r 25 -i video_pipe -c:v libx265 -c:a aac out.mp4

[legoktm]

As for re-enconding the video, it seems that the best way to move forward is to use the H.265 codec, and the MP4 container format.

I would avoid H.265 (aka HEVC) for now, it still doesn't have wide platform support yet, primarily no Firefox support; see https://caniuse.com/?search=h265. I also don't think it provides any real advantage for the DZ usecase over H.264, which has much broader support.

But is there a specific reason to go with mp4 in the first place? It's still a patent-encumbered format until theoretically 2030 (per Wikipedia).

I'd think we should also consider using VP8/VP9 codecs with a webm container. Is allowing people to pick between the two an option? Or using whatever the input format was (assuming ffmpeg supports encoding it)?

[apyrgio]

That's super great info Kunal. I wasn't aware that HEVC is not supported by Firefox, nor the patent about mp4. It seems we need to use a codec that is:

• Patent-free, for broader device support
• Relatively fast, so that conversions don't take too long
• Not super new, because again we risk support issues

You're saying that any of VP8/VP9 should work great, right? Should we consider H.264 for better support, or AV1 perhaps?

For container formats, anything we should have in mind, or is .webm a clear winner?

[legoktm]

You're saying that any of VP8/VP9 should work great, right? Should we consider H.264 for better support, or AV1 perhaps?

It's a bit messy, H.264/mp4 have the best platform support, but are patent encumbered. Specifically H.264 encoding, which is what DZ would be doing, has patents (decoding, i.e. playing video, does not require a license). Then is VP8/VP9/webm, which are royalty free. H.265 (patent encumbered) and AV1 (royalty-free) are still pretty new but are getting more traction.

Debian mostly ignores the patents and ships ffmpeg with full H.264 encoding+decoding; Fedora strips them out. Probably something for us to discuss internally on whether DZ should ship w/ H.264-compatible ffmpeg or not.

My anecdotal impression and gut feeling is that starting with webm is a good way to start; it'll work for most people and then we'll see if anyone comes back and asks for mp4 support. (And to be clear, I think we should ingest whatever ffmpeg can handle, just discussing what format to spit out.)