Poor performance on my own dataset

[qhdqhd]

My dataset contains about 100 views, the viewpoints form a ring, all viewpoints are looking towards the circle center.
The cameras are like this:

The scene size is about 15 meters.

The first row shows 5 rendered views of the training view. The second row shows 5 training views.

My parameter:
aabb_scale is set to 16 and I change the scale or offset in the camera positions as https://github.com/NVlabs/instant-ngp/blob/master/docs/nerf_dataset_tips.md, so the scene has been fully covered by the box.

The people in the center of the scene renders fine, but there are quite a few white floating objects or ghosting around the people.
I would like to ask why?
There are three possible reasons I can think of:

1. Is it because the scene has too few textures? Lots of black backgrounds around.
2. Or is it due to camera parameters? Because I used 100 different cameras here, but the cameras are all 12mm, and the intrinsic parameters are very close, but not exactly the same. Since your project does not support multiple camera intrinsics, only one common intrinsic is used.
3. Or the camera shooting position? My cameras are all on a plane without views from different directions like looking down and looking up.

[Tom94]

Hi there, first of call: cool setup!

(1) shouldn't be a problem as long as the camera poses are precise enough for the Train extrinsics option to fix any slight inaccuracies.

I suspect the problem comes more from (2) and (3).

For (2), we recently (yesterday) pushed support for per-camera metadata. You can customize it via the python bindings

testbed.nerf.training.dataset.metadata[image_id].camera_distortion = ...
testbed.nerf.training.dataset.metadata[image_id].focal_length = ...
testbed.nerf.training.dataset.metadata[image_id].principal_point = ...

More specifics are in python_api.cu. The .json-based loader unfortunately only supports a single set of camera parameters per json file.

For (3), this will largely manifest as artifacts when trying to view the scene from the top of bottom (i.e. outside of the convex hull of the training data). If you plan for the viewpoint to stay close to the ring of cameras, you should be fine.

Curious to hear whether you find the Python bindings w.r.t. (2) helpful.

Cheers!

[qhdqhd]

thank you for your fast anwser!
I still have a question. If the calibration accuracy is good enough, does the texture of the scene have a great impact on the training? Rich texture should be better than low texture, right?

[Tom94]

Rich texture is indeed better, yes, but less important for training than for pose estimation.

[slash-under]

In my experience, the "smoke" you're describing is most likely loss, which would disappear with more training data/potentially more points extracted with COLMAP.

I've tweaked some parameters in my fork, maybe applying the changes from here could help?

slash-under/instant-ngp@a8a3d76

[slash-under]

It might help to append a portion of video from the angles with the most loss to the dataset, in that case.

[slash-under]

(or one could look through here for more tweaks I've missed: https://colmap.github.io/faq.html)

[slash-under]

If you are using the run.py script to extract frames from images, it appears that the use of JPEG format for the frames is introducing some compression error. I've changed it to PNG in my fork, and am awaiting results to see if the output quality changes.

[qhdqhd]

Thank you, I generally use JPEG. But does this really matter?

[QinlongHuang]

I'm so sorry, but is there change in your improvement ? @slash-under