monosdf and surface normal implementation

[pablovela5620]

Continue geometric cues w/scannet implementation. Start with MonoSDF as simpler to implement v.s. Neuris

[pablovela5620]

Didn't know the best way to continue so I just made another PR. I'm trying to understand how to render the normal, right now I just copy pasted the NerfSystem as that's whats currently working

I'm using this as a reference https://github.com/autonomousvision/monosdf/blob/8e46ed516c80c30b73043ff163c2bd94b4b67062/code/model/network.py#L497

It looks like they're using the gradient outputs which I think are equivalent to this

instant-nsr-pl/models/geometry.py

Line 149 in c236bc3

grad = torch.autograd.grad(

I wanted to verify that I would then take those value and feed them into render_weight_from_density as you recommended as follows

ray_indices = unpack_info(packed_info)
sigmas = sigma_fn(t_starts, t_ends, ray_indices.long())
weights = render_weight_from_density(packed_info, t_starts, t_ends, sigmas)
normal_grad = grad / (grad.norm(2, -1, keepdim=True) + 1e-6)

normals = accumulate_along_rays(
    weights,
    ray_indices,
    values=normal_grad,
    n_rays=n_rays,
    )

I'm working on converting over from nerf to sdf, but wanted to start with the nerf system to ensure I was correctly loading the depth/normal maps.

Any advice on the normal map rendering would be appreciated!

[pablovela5620]

Managed to get surface normal rendering working, though it seems to significantly slow down training, I think there's probably a better way of doing it as there's a bunch of redundant computation happening but it is not being included inside of rendering

it20000-trimmed.mp4

[bennyguo]

Hi, sorry for the late reply! This is exactly what i meant for rendering the normal map. I think the slowdown mainly comes from second-order gradient calculation (which is not essentially required for NeRF). So if you experiment with NeuS, the slowdown could be negligible.

[pablovela5620]

Thanks for the reply, this is taking a bit longer than I had hoped as I'm both implementing this and learning about Neus/Neuris/MonoSDF at the same time. May be a while before I'm able to get this working!