Yunhao Li, Xiaodong Wang, Ping Wang, Xin Yuan and Peidong Liu [CVPR 2024 Highlight]
.jpg)
This is an official PyTorch implementation of the paper SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image (CVPR 2024).
SCINeRF retrieves the 3D scenes, together with compressed images, from a single temporal-compressed snapshot compressive image.
⚡ [2024.02] Our paper has been accepted by CVPR 2024!
⚡ [2024.04] Our paper has been selected as CVPR 2024 Highlight (~2.8% acceptance rate)!
When capturing the scene, the snapshot compressive imaging (SCI) camera moves alongside a trajectoy and capture the scene into an SCI measurement.
We follow the real physical image formation process of snapshot compressive imaging (SCI) to synthesize SCI measurement from NeRF. Both NeRF and the motion trajectories are estimated by maximizing the photometric consistency between the synthesized sci measurement and the real measurement.
git clone [email protected]:WU-CVGL/SCINeRF.git
cd SCINeRF
pip install -r requirements.txt
You can download the data here.
Each scene contains an SCI measurement in 'meas.npy', and corresponding masks in 'mask.npy' (in some scenes they are 'mask_25.npy" and 'meas_25.npy').
Change the data path and other parameters (if needed) in configs/cozy2room.txt. We use cozy2room scene as an example.
You can test our code and render retrieved images from compressed image with the provided checkpoints. To do this, you should first download the checkpoints here, then put the weight file under the corresponding logs folder ./logs/cozy2room, and then change the parameter load_weights=True in cozy2room.txt, finally run
python train_cacti3.py --config configs/cozy2room.txt
python train_cacti3.py --config configs/cozy2room.txt
After training, you can get retrieved images from SCI measurement.
mask.npy: This file is the binary masks you used during the SCI imaging process.
meas.npy: This file is the SCI measurement, i.e., the compressed image.
Put two files under a folder at ./data. Then create a new config file in the ./configs folder, make sure that the data directory is correct. Finally, modify he h, w, and f parameters in your config file, which represents image height, width and focal length of the camera.
If you find this useful, please consider citing our paper:
@InProceedings{li2024scinerf,
author = {Li, Yunhao and Wang, Xiaodong and Wang, Ping and Yuan, Xin and Liu, Peidong},
title = {{SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image}},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024}
}The overall framework, metrics computing and camera transformation are derived from nerf-pytorch and BAD-NeRF respectively. We appreciate the effort of the contributors to these repositories.