We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) that leverages forward mapping volume rendering to achieve photorealistic novel view synthesis and relighting results. Unlike previous works that use implicit neural representations and volume rendering (e.g. NeRF), which suffer from low expressive power and high computational complexity, we extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. There are two main problems when introducing GS to inverse rendering: 1) GS does not support producing plausible normal natively; 2) forward mapping (e.g. rasterization and splatting) cannot trace the occlusion like backward mapping (e.g. ray tracing). To address these challenges, our GS-IR proposes an efficient optimization scheme that incorporates a depth-derivation-based regularization for normal estimation and a baking-based occlusion to model indirect lighting. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering. We demonstrate the superiority of our method over baseline methods through qualitative and quantitative evaluations on various challenging scenes.
我们提出了一种基于3D高斯溅射（GS）的新型逆向渲染方法GS-IR，该方法利用正向映射体积渲染实现逼真的新视角合成和重新照明效果。与以前使用隐式神经表示和体积渲染的工作（例如NeRF）不同，这些工作受限于表达能力低和计算复杂度高，我们扩展了GS——一种用于新视角合成的顶级性能表示，以从在未知照明条件下捕获的多视图图像中估计场景几何、表面材料和环境照明。在将GS引入逆向渲染时存在两个主要问题：1）GS本身不支持产生合理的法线；2）正向映射（例如光栅化和溅射）无法像反向映射（例如光线追踪）那样追踪遮挡。为了解决这些挑战，我们的GS-IR提出了一种有效的优化方案，该方案结合了基于深度导数的规范化用于法线估计和基于烘焙的遮挡以模拟间接照明。灵活且富有表现力的GS表示使我们能够实现快速紧凑的几何重建、逼真的新视角合成和有效的基于物理的渲染。我们通过对各种具有挑战性的场景进行定性和定量评估，展示了我们方法相对于基线方法的优越性。