Attempting to implement method on gsplat.

[jefequien]

Hello!
I'm trying to implement this method on top of gsplat and combine it with 3DGS-MCMC but I'm having trouble reproducing the effect. My PSNR on the cake scene is 24.32 which is worse than the baseline.
jefequien/gsplat-mcmc#7

The method fits the training images perfectly but the splats still appear blurry when rendered without the MLP. Despite clamping the scale_delta and rotation_delta between 1.0 and 1.1, the MLP seems to be able to sharpen parts of the scene. Do you guys have any tricks to avoid this solution?

[benhenryL]

Hello, thanks for your interest to our work and sorry for the late response.

I checked the codes in jefequien/gsplat-mcmc and found periodical opacity reset from 3D-GS is applied though we didn't.
However, I could not fix the problem even after the inactivating opacity reset.
If there is any recent update, would you like to share it please?
I'd like to work on it more carefully in the coming days, and will share the results.

Thanks!

[jefequien]

Hey! Thanks for responding. 3DGS-MCMC doesn't use the opacity reset. I actually spent quite a bit more time on this.
See this PR. nerfstudio-project/gsplat#469

I ended up using an MLP to predict a blur mask used to blend the all-in-focus image and blurred image together. This is a large concession because it requires rendering twice per training step. I also needed a custom loss function to prevent the blur mask MLP from degenerating. This works but the results are worse than what other papers get.

It is unusual that your method does not need to predict a blur mask. Both BAGS and DOF-GS predict a blur mask for blending.
https://nwang43jhu.github.io/BAGS/
https://arxiv.org/abs/2405.17351