Wenpu Li1,5* Pian Wan 1,2* Peng Wang1,3* Jinghang Li4 Yi Zhou4 Peidong Liu1†
*equal contribution † denotes corresponding author.
1Westlake University
2EPFL
3Zhejiang University
4Hunan University
5Guangdong University of Technology
This repository is an official PyTorch implementation of the paper "BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream". We explore the possibility of recovering the neural radiance fields and camera motion trajectory from a single blurry image. This allows BeNeRF to decode the underlying sharp video from a single blurred image.
2024.09.24
We have completed the construction of the project homepage. You can access it by clicking this link.2024.08.25
Testing and evaluation Code have been released.2024.08.20
Training Code and datasets have been released.2024.07.01
Our paper was accepted by ECCV2024!! Congratulates to all collaborators!!
In the path where your want to store code, enter the following terminal command:
git clone https://github.com/WU-CVGL/BeNeRF.git
cd BeNeRF
conda create -n benerf python=3.9
conda activate benerf
pip install -r requirements.txt
If the network speed is slow when using pip to download dependencies, you may consider changing the pip source:
python -m pip install --upgrade pip
pip config set global.index-url https://mirrors.bfsu.edu.cn/pypi/web/simple
You can download BeNeRF_Datasets
by clicking this link.
It contains real-world dataset(e.g. e2nerf_real) and synthetic dataset(benerf_blender, benerf_unreal and e2nerf_synthetic). For the every scene, the folder images
includes blurry images and folder events
includes event stream for entire sequence. The timestamps of the start and end of exposure for each image are stored in a txt
file. These timestamps are used to segment the entire event stream into individual event streams corresponding to each image. Additionally, we provide grounth sharp images in folder images_test
for syntehtic dataset in order to evaluate metrics.
For the scenes of benerf_blender
and benerf_unreal
, We provide two versions of the images: one in color and one in grayscale. The grayscale version is used for quantitative evaluation, while the color version is used for qualitative evaluation. Since event-enhanced baseline methods we compared to can only run with gray image due to the single channel event stream we synthesized, we compute the metrics of all methods with gray images for consistency.
First, you need to modify the path of datasets in config file:
datadir = XXXXX/BeNeRF_Datasets/real or synthetic/<dataset>/<scene>
Then, You can train model by entering terminal command as follow:
python train.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id>
We use wandb as a viewr to moniter the training process by defalut:
After training, all results including render image, render video, camera trajectory and checkpoint file will be saved in the path specified by logdir/<img_id>/
in the config file.
You can test the model by loading the checkpoint file saved in the logdir/<img_id>/
path. We provide three options to test model.
python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> \
--extract_poses --num_extract_poses <the number of poses you want to extract>
We have set the default number of extracted poses to 19, i.e., num_extract_poses = 19
.
python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> \
--render_images --num_render_images <the number of images you want to render>
We have set the default number of render images to 19, i.e., num_render_images = 19
.
python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> --render_video
Of course, you can choose all three options simultaneously to test the model.
python test.py --device <cuda_id> --config ./configs/<dataset>/<scene>.txt --index <img_id> \
--extract_poses --num_extract_poses <the number of poses you want to extract> \
--render_images --num_render_images <the number of images you want to render> \
--render_video
The test results will be saved in logdir/<img_id>/test_results
path.
To compute metrics like PSNR, SSIM, LPIPS for evaluating model performance on synthetic dataset, you can run script as follow:
python evaluate.py --dataset <dataset name> --scene <scene name> \
--result <path to folder of image results> --groundtruth <path to folder of groundtruth image>
For real-world datasets, since sharp ground truth images are not available, we choose to use the no-reference image quality metric BRISQUE
to quantitatively evaluate the model's performance on real-world datasets. You can run the following script in MATLAB.
matlab -r "eval_brisque('<path to folder of image results>')"
You can check our results at the following link.
If you find this repository useful, please consider citing our paper:
@inproceedings{li2024benerf,
author = {Wenpu Li and Pian Wan and Peng Wang and Jinghang Li and Yi Zhou and Peidong Liu},
title = {BeNeRF: Neural Radiance Fields from a Single Blurry Image and Event Stream},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2024}
}
In our work, the camera optimization and event stream integration into NeRF were inspired by BAD-NeRF and E-NeRF, respectively. The overall code framework is based on nerf-pytorch. We appreciate the effort of the contributors to these amazing repositories.