STFFNet

[ICPR2024] A Novel Encoder-Decoder Network with Multi-domain Information Fusion for Video Deblurring

by Peiqi Xie, Jinhong He, Chengyun Song, and Minglong Xue⋆

Our poster is available at here.

Visual Results

Results on BSD (Real-world)

Quantitative results on different setups of BSD:

Results on REDS and GOPRO (Synthetic)

Quantitative results on different setups of REDS and GOPRO:

Quick Start

Prerequisites

• Python 3.6
• PyTorch 1.6 with GPU
• opencv-python
• scikit-image
• lmdb
• thop
• tqdm
• tensorboard

Downloading Datasets

Please download and unzip the dataset file for each benchmark.

• BSD (Full BSD with RAW)
• GOPRO
• REDS

If you failed to download BSD from Google drive, please try the following BaiduCloudDisk version:
BSD 1ms8ms, password: bsd1
BSD 2ms16ms, password: bsd2
BSD 3ms24ms, password: bsd3

Training

Specify <path> (e.g. "./dataset/") as where you put the dataset file.

Modify the corresponding dataset configurations in the command, or change the default values in "./para/paramter.py".

Training command is as below:

python main.py --data_root <path> --dataset BSD --ds_config 2ms16ms

You can also tune the hyper-parameters such as batch size, learning rate, epoch number (P.S.: the actual batch size for ddp mode is num_gpus*batch_size):

python main.py --lr 1e-4 --batch_size 4 --num_gpus 2 --trainer_mode ddp

If you want to train on your own dataset, please refer to "/data/how_to_make_dataset_file.ipynb".

Inference

Please download checkpoints of pretrained models for different setups and unzip them under the main directory.

Dataset (Test Set) Inference

Command to run a pre-trained model on BSD (3ms-24ms):

python main.py --test_only --test_checkpoint ./checkpoints/ESTRNN_C80B15_BSD_3ms24ms.tar --dataset BSD --ds_config 3ms24ms --video

Blurry Video Inference

Specify "--src <path>" as where you put the blurry video file (e.g., "--src ./blur.mp4") or video directory (e.g., "--src ./blur/", the image files under the directory should be indexed as "./blur/00000000.png", "./blur/00000001.png", ...).

Specify "--dst <path>" as where you store the results (e.g., "--dst ./results/").

Command to run a pre-trained model for a blurry video is as below:

python inference.py --src <path> --dst <path> --ckpt ./checkpoints/ESTRNN_C80B15_BSD_2ms16ms.tar