Semantics-Controlled Gaussian Splatting for Outdoor Scene Reconstruction and Rendering in Virtual Reality
Advancements in 3D rendering like Gaussian Splatting (GS) allow novel view synthesis and real-time rendering in virtual reality (VR). However, GS-created 3D environments are often difficult to edit. For scene enhancement or to incorporate 3D assets, segmenting Gaussians by class is essential. Existing segmentation approaches are typically limited to certain types of scenes, e.g., ''circular'' scenes, to determine clear object boundaries. However, this method is ineffective when removing large objects in non-''circling'' scenes such as large outdoor scenes. We propose Semantics-Controlled GS (SCGS), a segmentation-driven GS approach, enabling the separation of large scene parts in uncontrolled, natural environments. SCGS allows scene editing and the extraction of scene parts for VR. Additionally, we introduce a challenging outdoor dataset, overcoming the ''circling'' setup. We outperform the state-of-the-art in visual quality on our dataset and in segmentation quality on the 3D-OVS dataset. We conducted an exploratory user study, comparing a 360-video, plain GS, and SCGS in VR with a fixed viewpoint. In our subsequent main study, users were allowed to move freely, evaluating plain GS and SCGS. Our main study results show that participants clearly prefer SCGS over plain GS. We overall present an innovative approach that surpasses the state-of-the-art both technically and in user experience.
3D 渲染技术的进步，如高斯分布（Gaussian Splatting，GS），使得虚拟现实（VR）中的新视角合成和实时渲染成为可能。然而，GS 创建的三维环境通常难以编辑。为了增强场景或融入三维资产，对高斯点进行按类别分割是至关重要的。现有的分割方法通常仅限于某些类型的场景，例如“环绕”场景，用于确定明确的物体边界。然而，当处理诸如大型户外场景等非“环绕”场景时，这种方法在移除大型物体时效果不佳。我们提出了语义控制的高斯分布（Semantics-Controlled GS，SCGS），这是一种基于分割驱动的 GS 方法，能够在不受控的自然环境中分离出大型场景部分。SCGS 允许对场景进行编辑，并可提取场景部分用于 VR。此外，我们引入了一个具有挑战性的户外数据集，克服了“环绕”设置的局限性。在我们的数据集上，我们在视觉质量方面超越了现有技术，并在 3D-OVS 数据集上的分割质量上取得了优异表现。我们还进行了一项探索性用户研究，比较了 360 视频、纯 GS 和 SCGS 在 VR 中固定视角下的表现。在随后的主要研究中，用户可以自由移动，评估纯 GS 和 SCGS。我们的主要研究结果表明，参与者明显更偏爱 SCGS 而非纯 GS。总体而言，我们提出了一种创新方法，在技术和用户体验上都超越了现有的技术水平。