NeRF-based 3D-aware Generative Adversarial Networks (GANs) like EG3D or GIRAFFE have shown very high rendering quality under large representational variety. However, rendering with Neural Radiance Fields poses challenges for 3D applications: First, the significant computational demands of NeRF rendering preclude its use on low-power devices, such as mobiles and VR/AR headsets. Second, implicit representations based on neural networks are difficult to incorporate into explicit 3D scenes, such as VR environments or video games. 3D Gaussian Splatting (3DGS) overcomes these limitations by providing an explicit 3D representation that can be rendered efficiently at high frame rates. In this work, we present a novel approach that combines the high rendering quality of NeRF-based 3D-aware GANs with the flexibility and computational advantages of 3DGS. By training a decoder that maps implicit NeRF representations to explicit 3D Gaussian Splatting attributes, we can integrate the representational diversity and quality of 3D GANs into the ecosystem of 3D Gaussian Splatting for the first time. Additionally, our approach allows for a high resolution GAN inversion and real-time GAN editing with 3D Gaussian Splatting scenes.
基于NeRF的3D感知生成对抗网络（GAN），如EG3D或GIRAFFE，已展示出在大范围表示性方面的非常高的渲染质量。然而，使用神经辐射场（NeRF）进行渲染对3D应用带来了挑战：首先，NeRF渲染的显著计算需求阻止了其在低功率设备上的使用，例如移动设备和VR/AR头显。其次，基于神经网络的隐式表征难以融入到显式3D场景中，如VR环境或视频游戏。3D高斯飞溅（3DGS）通过提供一个可以高帧率高效渲染的显式3D表征，克服了这些限制。在这项工作中，我们提出了一种新颖的方法，将基于NeRF的3D感知GAN的高渲染质量与3DGS的灵活性和计算优势结合起来。通过训练一个将隐式NeRF表征映射到显式3D高斯飞溅属性的解码器，我们首次将3D GAN的表征多样性和质量整合到3D高斯飞溅的生态系统中。此外，我们的方法允许高分辨率的GAN反演以及实时的GAN编辑与3D高斯飞溅场景。