If you would like to contribute code you can do so through GitHub by forking the repository and sending a pull request. When submitting code, please make every effort to follow existing conventions and style in order to keep the code as readable as possible.
By contributing your code, you agree to license your contribution under the terms of the Apache License v2.0. Your contributions should also include the following header:
/**
* Copyright 2016-2019 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Files in the repository follow the structure below (10/4/2019 TO BE UPDATED):
├── matlab: matlab scripts
├── model: VMAF model files
├── src
│ ├── libsvm: libsvm C/C++ source code
│ ├── libvmaf: stand-alone C++ implementation of VMAF and a number of other metrics such as PSNR and SSIM
│ ├── pthreads: pthreads C library used by Windows build
│ └── ptools: PicklingTools library that allows Python/C++ data sharing
├── python
│ ├── config.py: configuration information such as global paths, parameters, etc.
│ ├── src: python source files
│ ├── script: scripts for executing specific tasks
│ ├── test: unit/functional test cases
│ └── tools: common reusable classes and functions
├── resource
│ ├── dataset: dataset files
│ ├── doc: documentation files
│ ├── feature_param: feature parameter files
│ ├── images: image files
│ ├── model: permanent trained model files
│ └── model_param: model parameter files
├── workspace (files under workspace get ignored by git)
│ ├── model: output trained model files
│ ├── model_param: output model parameter files generated by hyper-parameter search
│ ├── result_store_dir: result store files
│ └── workdir: working directory
└── libvmaf: stand-alone C++ implementation of VMAF
The core class architecture can be depicted in the diagram below:
An Asset is the most basic unit with enough information to perform an execution task. It includes basic information about a distorted video and its undistorted reference video, as well as the frame range on which to perform a task (i.e. dis_start_end_frame and ref_start_end_frame), and at what resolution to perform a task (e.g. a video frame is upscaled to the resolution specified by quality_width_hight before feature extraction).
Asset extends WorkdirEnabled mixin, which comes with a thread-safe working directory to facilitate parallel execution.
An Executor takes in a list of Assets, and run computations on them, and return a list of corresponding Results. An Executor extends the TypeVersionEnabled mixin, and must specify a unique type and version combination (by the TYPE and VERSION attribute), so that the Result generated by it can be uniquely identified.
Executor is the base class for FeatureExtractor and QualityRunner, and it provides a number of shared housekeeping functions, including storing and reusing Results, creating FIFO pipes, cleaning up log files/Results, etc.
A function run_executors_in_parallel facilitates running Executors in parallel.
A Result is a key-value store of read-only execution results generated by an Executor on an Asset. A key corresponds to a 'atom' feature type or a type of a quality score, and a value is a list of score values, each corresponding to a computation unit (i.e. in the current implementation, a frame).
The Result class also provides a number of tools for aggregating the per-unit scores into a single score. The default aggregatijon method is the mean, but Result.set_score_aggregate_method() allows customizing other methods (see test_to_score_str() in test/result_test.py for examples).
ResultStore provides capability to save and load a Result. Current implementation FileSystemResultStore persists results by a simple file system that save/load result in a directory. The directory has multiple subdirectories, each corresponding to an Executor. Each subdirectory contains multiple files, each file storing dataframe for an Asset.
FeatureExtractor subclasses Executor, and is specifically for extracting features (aka elementary quality metrics) from Assets. Any concrete feature extraction implementation should extend the FeatureExtractor base class (e.g. VmafFeatureExtractor). The TYPE field corresponds to the 'aggregate' feature name, and the 'ATOM_FEATURES'/'DERIVED_ATOM_FEATURES' field corresponds to the 'atom' feature names.
FeatureAssembler assembles features for an input list of Assets on a input list of FeatureExtractor subclasses. The constructor argument feature_dict specifies the list of FeatureExtractor subclasses (i.e. the 'aggregate' feature) and selected 'atom' features. For each asset on a FeatureExtractor, it outputs a BasicResult object. FeatureAssembler is used by a QualityRunner to assemble the vector of features to be used by a TrainTestModel.
TrainTestModel is the base class for any concrete implementation of regressor, which must provide a train() method to perform training on a set of data and their groud-truth labels, and a predict() method to predict the labels on a set of data, and a to_file() and a from_file() method to save and load trained models.
A TrainTestModel constructor must supply a dictionary of parameters (i.e. param_dict) that contains the regressor's hyper-parameters. The base class also provides shared functionalities such as input data normalization/output data denormalization, evaluating prediction performance, etc.
Like a Executor, a TrainTestModel must specify a unique type and version combination (by the TYPE and VERSION attribute).
CrossValidation provides a collection of static methods to facilitate validation of a TrainTestModel object. As such, it also provides means to search the optimal hyper-parameter set for a TrainTestModel object.
QualityRunner subclasses Executor, and is specifically for evaluating the quality score for Assets. Any concrete implementation to generate the final quality score should extend the QualityRunner base class (e.g. VmafQualityRunner, PsnrQualityRunner).
There are two ways to extend a QualityRunner base class -- either by directly implementing the quality calculation (e.g. by calling a C executable, as in PsnrQualityRunner), or by calling a FeatureAssembler (with indirectly calls a FeatureExtractor) and a TrainTestModel subclass (as in VmafQualityRunner). For more details, refer to Section 'Extending Classes'.
A derived class of FeatureExtractor must:
- Override TYPE and VERSION fields.
- Override ATOM_FEATURES field.
- Optionally, override DERIVED_FEATURES field. These are the features that are 'derived' from the ATOM_FEATURES.
- Override _generate_result(self, asset), which call a command-line executable and generate feature scores in a log file.
- Optionally, override _get_feature_scores(self, asset), which read the feature scores from the log file, and return the scores in a dictionary format. FeatureExtractor base class provides a template _get_feature_scores method, which has been used by VmafFeatureExtractor as an example. If your log file format is incompatible with VmafFeatureExtractor's, consider overriding this method for your custom case.
- Optionally, if you have override DERIVED FEATURES field, also override _post_process_result(cls, result) and put the calculation of the derived attom features here.
Follow the example of VmafFeatureExtractor.
A derived class of TrainTestModel must:
- Override TYPE and VERSION fields.
- Override _train(model_param, xys_2d), which implements the training logic.
A basic example of wrapping the scikit-learn Ensemble.RandomForest class can be found in SklearnRandomForestTrainTestModel.
The scikit-learn Python package has already provided many regressor tools to choose from. But if you do not want to call from scikit-learn but rather integrating/creating other regressor implementation, you will need some additional steps. Besides the two step above, you also need to:
- Override to_file(self, filename), which must properly save the trained model and parameters to a file(s).
- Adding to the base-class static method TrainTestModel.from_file specific treatment for your subclass.
- Override static method delete(filename) if you have more than one file to delete.
- Override _predict(cls, method, xs_2d) with your customized prediction function.
For this type of subclassing, refer to the example of LibsvmnusvrTrainTestModel.
There are two ways to create a derived class of QualityRunner:
The first way is to call a command-line executable directly, very similar to what FeatureExtractor does. A derived class must:
- Override TYPE and VERSION fields.
- Override _generate_result(self, asset), which call a command-line executable and generate quality scores in a log file.
- Override _get_quality_scores(self, asset), which read the quality scores from the log file, and return the scores in a dictionary format.
For an example, follow PsnrQualityRunner.
The second way is to override the Executor._run_on_asset(self, asset) method to bypass the regular routine, but instead, in the method construct a FeatureAssembler (which calls a FeatureExtractor or many) and assembles a list of features, followed by using a TrainTestModel (pre-trained somewhere else) to predict the final quality score. A derived class must:
- Override TYPE and VERSION fields.
- Override _run_on_asset(self, asset), which runs a FeatureAssembler, collect a feature vector, run TrainTestModel.predict() on it, and return a Result object (in this case, both Executor._run_on_asset(self, asset) and QualityRunner._read_result(self, asset) get bypassed.
- Override _remove_result(self, asset) by redirecting it to the FeatureAssembler.
For an example, follow VmafQualityRunner.