Estimating physical properties for visual data is a crucial task in computer vision, graphics, and robotics, underpinning applications such as augmented reality, physical simulation, and robotic grasping. However, this area remains under-explored due to the inherent ambiguities in physical property estimation. To address these challenges, we introduce GaussianProperty, a training-free framework that assigns physical properties of materials to 3D Gaussians. Specifically, we integrate the segmentation capability of SAM with the recognition capability of GPT-4V(ision) to formulate a global-local physical property reasoning module for 2D images. Then we project the physical properties from multi-view 2D images to 3D Gaussians using a voting strategy. We demonstrate that 3D Gaussians with physical property annotations enable applications in physics-based dynamic simulation and robotic grasping. For physics-based dynamic simulation, we leverage the Material Point Method (MPM) for realistic dynamic simulation. For robot grasping, we develop a grasping force prediction strategy that estimates a safe force range required for object grasping based on the estimated physical properties. Extensive experiments on material segmentation, physics-based dynamic simulation, and robotic grasping validate the effectiveness of our proposed method, highlighting its crucial role in understanding physical properties from visual data.
对视觉数据进行物理属性估计是计算机视觉、图形学和机器人领域的一项关键任务，支撑着增强现实、物理模拟和机器人抓取等应用。然而，由于物理属性估计的内在模糊性，这一领域仍然未被充分探索。为应对这些挑战，我们提出了一种GaussianProperty框架，这是一种无需训练的方案，能够为3D高斯分配材料的物理属性。 具体来说，我们结合了 SAM 的分割能力和 GPT-4V(ision) 的识别能力，设计了一个用于2D图像的全局-局部物理属性推理模块。随后，我们通过投票策略将多视图2D图像的物理属性投射到3D高斯上。我们证明了带有物理属性标注的3D高斯能够支持基于物理的动态模拟和机器人抓取等应用。在基于物理的动态模拟中，我们利用材料点方法（Material Point Method, MPM）进行逼真的动态模拟。在机器人抓取方面，我们开发了一种抓取力预测策略，基于估计的物理属性，预测安全的抓取力范围。 在材料分割、基于物理的动态模拟以及机器人抓取方面的广泛实验验证了我们方法的有效性，凸显了其在通过视觉数据理解物理属性中的重要作用。