Skip to content

Commit 1986e95

Browse files
MoyangLi00MoyangLi00
committed
first commit
1 parent 7640cde commit 1986e95

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

41 files changed

+4039
-2
lines changed

‎.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
/__pycache__
2+
/data
3+
/logs
4+
*.log
5+
/.hypothesis
6+
7+
CUDA_VISIBLE_DEVICES=0 nohup python train_usb_nerf.py --config ./configs/Unreal-RS/Adornment_CubicSpline.txt >> CubicSpline.log 2>&1 &
8+
CUDA_VISIBLE_DEVICES=0 nohup python train_usb_nerf.py --config ./configs/TUM-RS/CubicSpline.txt >> tum-rs.log 2>&1 &
9+
CUDA_VISIBLE_DEVICES=0 nohup python train_usb_nerf.py --config ./configs/TUM-RS/LinearSpline.txt >> tum-rs-linear.log 2>&1 &

‎README.md‎

Lines changed: 81 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,81 @@
1-
## [ICLR 2024] USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
2-
### Code and data will be released soon.
1+
# USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields
2+
<!-- ### [Project Page]() | [Video]() | -->
3+
### [Paper](https://arxiv.org/abs/2310.02687) | [Data](https://drive.google.com/file/d/1ndBlTCaWkQv1rAPXbGSVmUqRACCvfJOl/view?usp=sharing)<br>
4+
PyTorch implementation of rolling shutter effect correction with NeRF.<br><br>
5+
[USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields](https://arxiv.org/abs/2310.0268)
6+
[Moyang Li](https://scholar.google.com/citations?user=Qvu8bNYAAAAJ&hl)\*<sup>1,2</sup>,
7+
[Peng Wang](https://wangpeng000.github.io/)\*<sup>1,3</sup>,
8+
[Lingzhe Zhao](https://scholar.google.com/citations?user=mN764NsAAAAJ&hl=en)<sup>1</sup>,
9+
[Bangyan Liao](https://scholar.google.com/citations?user=0z2qluIAAAAJ&hl)<sup>1,3</sup>,
10+
[Peidong Liu](https://ethliup.github.io/)✝<sup>1,3</sup>,<br>
11+
<sup>1</sup>Westlake University, <sup>2</sup>ETH Zurich, <sup>3</sup>Zhejiang University
12+
\* denotes equal contribution
13+
✝ denotes corresponding author
14+
in ICLR 2024
15+
16+
USB-NeRF is able to correct rolling shutter distortions and recover accurate camera motion trajectory simultaneously under the framework of NeRF, by modeling the physical image formation process of a rolling shutter camera.
17+
18+
## Quickstart
19+
20+
### 1. Setup environment
21+
22+
```
23+
git clone https://github.com/WU-CVGL/USB-NeRF
24+
cd USB-NeRF
25+
pip install -r requirements.txt
26+
```
27+
28+
### 2. Download datasets
29+
30+
You can download the data [here](https://drive.google.com/file/d/1ndBlTCaWkQv1rAPXbGSVmUqRACCvfJOl/view?usp=sharing).
31+
32+
After acquiring the data, your folder structure should look like
33+
```
34+
Dataset/
35+
Unreal-RS/
36+
Adornment/
37+
images/
38+
start/
39+
mid/
40+
groundtruth.txt
41+
poses_bounds.npy
42+
BlueRoom/
43+
...
44+
LivingRoom/
45+
...
46+
WhiteRoom/
47+
...
48+
intrinsics.txt
49+
...
50+
```
51+
`images` folder contains captured rolling shutter images. `start` and `mid` folder contain global shutter images corresponding to the first and middle scanline, respectively. `groundtruth.txt` file saves the groundtruth poses, while `poses_bounds.npy` file saves the estimated camera poses with rolling shutter images via COLMAP. `intrinsics.txt` saves camera intrinsics (fx, fy, cx, cy).
52+
53+
### 3. Configs
54+
55+
Modify parameters of config file (eg: `configs/Unreal-RS/Adornment_CubicSpline.txt`) if needed.
56+
57+
58+
### 4. Training
59+
60+
```
61+
python train_usb_nerf.py --config ./configs/Unreal-RS/Adornment_CubicSpline.txt
62+
```
63+
64+
After training, you can get global shutter images, optimized camera poses and synthesized novel view images.
65+
66+
## Citation
67+
68+
If you find this useful, please consider citing our paper:
69+
70+
```bibtex
71+
@article{li2023usb,
72+
title={USB-NeRF: Unrolling Shutter Bundle Adjusted Neural Radiance Fields},
73+
author={Li, Moyang and Wang, Peng and Zhao, Lingzhe and Liao, Bangyan and Liu, Peidong},
74+
journal={arXiv preprint arXiv:2310.02687},
75+
year={2023}
76+
}
77+
```
78+
79+
## Acknowledgment
80+
81+
The overall framework, metrics computing and camera transformation are derived from [nerf-pytorch](https://github.com/yenchenlin/nerf-pytorch/), [CVR](https://github.com/GitCVfb/CVR) and [BAD-NeRF](https://github.com/WU-CVGL/BAD-NeRF) respectively. We appreciate the effort of the contributors to these repositories.

‎configs/TUM-RS/CubicSpline.txt‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
expname = seq1
2+
basedir = ./logs/TUM-RS/CubicSpline
3+
datadir = ./data/TUM-RS/seq1/cam1
4+
dataset_type = tum-rs
5+
6+
SplineModel = Cubic
7+
8+
factor = 2
9+
llffhold = 8
10+
11+
barf = True
12+
barf_start = 0.1
13+
barf_end = 0.5
14+
15+
total_pixel = 5000
16+
readout_time = 29.4737
17+
period = 50000
18+
19+
N_samples = 64
20+
N_importance = 64
21+
22+
use_viewdirs = True
23+
raw_noise_std = 0
24+
25+
load_weights = False
26+
weight_iter = 10000
27+
28+
tv_loss = False
29+
tv_width_nerf = 20
30+
tv_loss_rgb = True
31+
tv_loss_gray = True
32+
33+
tv_loss_lambda = 0.001
34+
35+
n_tvloss = 0
36+
37+
i_img = 10000
38+
i_video = 200000
39+
i_weights = 10000
40+
max_iter = 200000
41+
render_rolling_shutter = False
42+
43+
# training parameters
44+
lrate = 5e-4
45+
pose_lrate = 1e-3
46+
decay_rate = 0.1
47+
decay_rate_pose = 0.01
48+
49+
only_optimize_SE3 = False
50+
optimize_se3 = True
51+
optimize_nerf = True

‎configs/TUM-RS/LinearSpline.txt‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
expname = seq1
2+
basedir = ./logs/TUM-RS/LinearSpline
3+
datadir = ./data/TUM-RS/seq1/cam1
4+
dataset_type = tum-rs
5+
6+
SplineModel = Linear
7+
8+
factor = 1
9+
llffhold = 8
10+
11+
barf = True
12+
barf_start = 0.1
13+
barf_end = 0.5
14+
15+
total_pixel = 5000
16+
readout_time = 29.4737
17+
period = 50000
18+
19+
N_samples = 64
20+
N_importance = 64
21+
22+
use_viewdirs = True
23+
raw_noise_std = 0
24+
25+
load_weights = False
26+
weight_iter = 10000
27+
28+
tv_loss = False
29+
tv_width_nerf = 20
30+
tv_loss_rgb = True
31+
tv_loss_gray = True
32+
33+
tv_loss_lambda = 0.001
34+
35+
n_tvloss = 0
36+
37+
i_img = 10000
38+
i_video = 200000
39+
i_weights = 10000
40+
max_iter = 200000
41+
render_rolling_shutter = False
42+
43+
# training parameters
44+
lrate = 5e-4
45+
pose_lrate = 1e-3
46+
decay_rate = 0.1
47+
decay_rate_pose = 0.01
48+
49+
only_optimize_SE3 = False
50+
optimize_se3 = True
51+
optimize_nerf = True

‎configs/Unreal-RS/Adornment_CubicSpline.txt‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
expname = Adornment
2+
basedir = ./logs/Unreal-RS/CubicSpline
3+
datadir = ./data/Unreal-RS/Adornment
4+
dataset_type = llff
5+
6+
SplineModel = Cubic
7+
8+
factor = 1
9+
llffhold = 8
10+
11+
focal = 548.409
12+
13+
barf = True
14+
barf_start = 0.1
15+
barf_end = 0.5
16+
17+
total_pixel = 5000
18+
readout_time = 1
19+
period = 480
20+
21+
N_samples = 64
22+
N_importance = 64
23+
24+
use_viewdirs = True
25+
raw_noise_std = 0
26+
27+
load_weights = True
28+
weight_iter = 90000
29+
30+
tv_loss = False
31+
tv_width_nerf = 20
32+
tv_loss_rgb = True
33+
tv_loss_gray = True
34+
35+
tv_loss_lambda = 0.001
36+
37+
n_tvloss = 0
38+
39+
i_img = 10000
40+
i_video = 200000
41+
i_weights = 10000
42+
max_iter = 200000
43+
render_rolling_shutter = False
44+
45+
# training parameters
46+
lrate = 5e-4
47+
pose_lrate = 1e-3
48+
decay_rate = 0.1
49+
decay_rate_pose = 0.01
50+
51+
only_optimize_SE3 = False
52+
optimize_se3 = True
53+
optimize_nerf = True

‎configs/Unreal-RS/Adornment_LinearSpline.txt‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
expname = Adornment
2+
basedir = ./logs/Unreal-RS/LinearSpline
3+
datadir = ./data/Unreal-RS/Adornment
4+
dataset_type = llff
5+
6+
SplineModel = Linear
7+
8+
factor = 1
9+
llffhold = 8
10+
11+
focal = 548.409
12+
13+
barf = True
14+
barf_start = 0.1
15+
barf_end = 0.5
16+
17+
total_pixel = 5000
18+
readout_time = 1
19+
period = 480
20+
21+
N_samples = 64
22+
N_importance = 64
23+
24+
use_viewdirs = True
25+
raw_noise_std = 0
26+
27+
load_weights = False
28+
weight_iter = 10000
29+
30+
tv_loss = False
31+
tv_width_nerf = 20
32+
tv_loss_rgb = True
33+
tv_loss_gray = True
34+
35+
tv_loss_lambda = 0.001
36+
37+
n_tvloss = 0
38+
39+
i_img = 10000
40+
i_video = 200000
41+
i_weights = 10000
42+
max_iter = 200000
43+
render_rolling_shutter = False
44+
45+
# training parameters
46+
lrate = 5e-4
47+
pose_lrate = 1e-3
48+
decay_rate = 0.1
49+
decay_rate_pose = 0.01
50+
51+
only_optimize_SE3 = False
52+
optimize_se3 = True
53+
optimize_nerf = True

0 commit comments

Comments
 (0)