We introduce Gaussian-Flow, a novel point-based approach for fast dynamic scene reconstruction and real-time rendering from both multi-view and monocular videos. In contrast to the prevalent NeRF-based approaches hampered by slow training and rendering speeds, our approach harnesses recent advancements in point-based 3D Gaussian Splatting (3DGS). Specifically, a novel Dual-Domain Deformation Model (DDDM) is proposed to explicitly model attribute deformations of each Gaussian point, where the time-dependent residual of each attribute is captured by a polynomial fitting in the time domain, and a Fourier series fitting in the frequency domain. The proposed DDDM is capable of modeling complex scene deformations across long video footage, eliminating the need for training separate 3DGS for each frame or introducing an additional implicit neural field to model 3D dynamics. Moreover, the explicit deformation modeling for discretized Gaussian points ensures ultra-fast training and rendering of a 4D scene, which is comparable to the original 3DGS designed for static 3D reconstruction. Our proposed approach showcases a substantial efficiency improvement, achieving a 5× faster training speed compared to the per-frame 3DGS modeling. In addition, quantitative results demonstrate that the proposed Gaussian-Flow significantly outperforms previous leading methods in novel view rendering quality.
我们介绍了 Gaussian-Flow，这是一种新颖的基于点的方法，用于快速动态场景重建和从多视角及单眼视频实时渲染。与受到训练和渲染速度缓慢困扰的流行 NeRF-based 方法不同，我们的方法利用了最新的点基三维高斯分散 (3DGS) 技术。具体来说，我们提出了一种新颖的双域变形模型 (DDDM)，用于显式建模每个高斯点的属性变形，其中每个属性的时域残差通过多项式拟合捕获，频域残差通过傅立叶级数拟合捕获。所提出的 DDDM 能够模拟长时间视频材料中复杂的场景变形，消除了为每一帧训练单独 3DGS 或引入额外隐式神经场来模拟三维动态的需求。此外，对离散高斯点的显式变形建模确保了四维场景的超快训练和渲染速度，与原始设计用于静态三维重建的 3DGS 相当。我们提出的方法展示了显著的效率提升，与每帧 3DGS 建模相比，训练速度提高了 5 倍。此外，定量结果表明，所提出的 Gaussian-Flow 在新视角渲染质量方面显著优于先前领先的方法。