diff --git a/README.md b/README.md index 5197f80..afa4c71 100644 --- a/README.md +++ b/README.md @@ -7,12 +7,71 @@ ![Coverage](https://img.shields.io/badge/coverage-97%25-brightgreen?logo=codecov) ![Python](https://img.shields.io/badge/python->=3.10-blue?logo=python) +This project is used to encode behavior videos and, depending on the user +request, will compress those videos while ensuring that they are formatted for +correct display across many devices. This may include performing common image +pre-processing steps that are common across many commercial video products, but +have to be done post-hoc for our behavior videos. +## Goals + +This will attempt to compress videos so that the results: + +* Retain the majority of the detail of the input video +* Take up as little space as possible for a target amount of visual detail +* Are in a format that can be widely viewed across devices, and streamed by + browsers +* Have pixel data in a color space that most players can properly display + +This video compression is often lossy, and the original videos are not kept, so +this library will attempt to perform the highest-quality video compression once. +The _speed_ of this compression is strictly secondary to the _quality_ of the +compression, as measured by the visual detail retained and the compression +ratio. See +[this section](#brief-benchmarks-on-video-compression-with-cpu-based-encoders-and-gpu-based-encoders) +for more details. + + +Additionally, this package should provide an easy to use interface that: + +* Presents users with a curated set of compression settings, which have been + rigorously tested +* Allow users to also provide their own compression settings, if they have + specific requirements + +## Non-goals + +* Spending as little time and compute resources as possible to transform videos, + at the expense of compression quality. ## Usage - The BehaviorVideoJob.run_job method in the transform_videos should be the primary method to call for processing video files. - On a merge to main, this package will be published as a singularity container, which can easily be run on a SLURM cluster. +## Brief benchmarks on video compression with CPU-based encoders and GPU-based encoders + +A perhaps surprising fact is that video encoders implementing the same algorithm +written for different compute resources +_do not have the same visual performance_, in that for a given compression ratio +they do not retain the same amount of visual detail. + +This can be seen in the plot below, where the GPU encoder and CPU encoders +retain different amounts of visual detail, as assessed with VMAF. + +![visual performance vs compress ratio](/assets/compression-vs-quality.png) + +This also shows that different presets of the same encoder, like _CPU Slow_ and +_CPU Fast_, also have different performance. For comrpession ratios greater than +100, it often makes sense to take your time and use a slow preset of a CPU-based +encoder to retain as much visual information for a given amount of compression. + +While it may be tempting to select a faster preset, or faster compute resource +like GPU for dramatic speedups shown below, the lossy compression and intent to +delete the original means that taking time to do it right once might well be +worth it. + +![throughput vs compress ratio](/assets/compression-vs-speed.png) + ## Development To develop the code, run @@ -84,10 +143,10 @@ The table below, from [semantic release](https://github.com/semantic-release/sem ### Documentation To generate the rst files source files for documentation, run ```bash -sphinx-apidoc -o doc_template/source/ src +sphinx-apidoc -o doc_template/source/ src ``` Then to create the documentation HTML files, run ```bash sphinx-build -b html doc_template/source/ doc_template/build/html ``` -More info on sphinx installation can be found [here](https://www.sphinx-doc.org/en/master/usage/installation.html). + More info on sphinx installation can be found [here](https://www.sphinx-doc.org/en/master/usage/installation.html). diff --git a/assets/compression-vs-quality.png b/assets/compression-vs-quality.png new file mode 100644 index 0000000..f491bc7 Binary files /dev/null and b/assets/compression-vs-quality.png differ diff --git a/assets/compression-vs-speed.png b/assets/compression-vs-speed.png new file mode 100644 index 0000000..e4177c6 Binary files /dev/null and b/assets/compression-vs-speed.png differ