Understanding how we grasp objects with our hands has important applications in areas like robotics and mixed reality. However, this challenging problem requires accurate modeling of the contact between hands and objects. To capture grasps, existing methods use skeletons, meshes, or parametric models that can cause misalignments resulting in inaccurate contacts. We present MANUS, a method for Markerless Hand-Object Grasp Capture using Articulated 3D Gaussians. We build a novel articulated 3D Gaussians representation that extends 3D Gaussian splatting for high-fidelity representation of articulating hands. Since our representation uses Gaussian primitives, it enables us to efficiently and accurately estimate contacts between the hand and the object. For the most accurate results, our method requires tens of camera views that current datasets do not provide. We therefore build MANUS-Grasps, a new dataset that contains hand-object grasps viewed from 53 cameras across 30+ scenes, 3 subjects, and comprising over 7M frames. In addition to extensive qualitative results, we also show that our method outperforms others on a quantitative contact evaluation method that uses paint transfer from the object to the hand.
理解我们如何用手抓握物体在机器人技术和混合现实等领域具有重要应用。然而，这个挑战性问题需要准确建模手和物体之间的接触。为了捕捉抓握动作，现有方法使用骨架、网格或参数模型，这可能导致不对齐，从而导致接触不准确。我们提出了一种名为 MANUS 的方法，用于无标记的手-物体抓握捕捉，使用可动的三维高斯模型。我们构建了一种新颖的可动三维高斯表示，扩展了三维高斯分散技术，以高保真度表示关节手。由于我们的表示使用高斯原始体，它使我们能够高效且准确地估计手和物体之间的接触。为了获得最准确的结果，我们的方法需要数十个摄像机视角，而当前的数据集并未提供。因此，我们构建了 MANUS-Grasps，一个新的数据集，包含从53个摄像机拍摄的30多个场景、3个主体以及超过700万帧的手-物体抓握动作。除了大量定性结果外，我们还展示了我们的方法在定量接触评估方法上的表现优于其他方法，该方法使用从物体到手的颜料转移。