Skip to content

Latest commit

 

History

History
66 lines (58 loc) · 3.49 KB

README.md

File metadata and controls

66 lines (58 loc) · 3.49 KB
Raw

Splat-MOVER: Multi-Stage, Open-Vocabulary Robotic Manipulation via Editable Gaussian Splatting

Ola Shorinwa*, Johnathan Tucker*, Aliyah Smith, Aiden Swann, Timothy Chen, Roya Firoozi, Monroe Kennedy III, Mac Schwager

Stanford University

*Equal Contribution.

Project Page | arXiv

Splat-MOVER

We present Splat-MOVER, a modular robotics stack for open-vocabulary robotic manipulation,which leverages the editability of Gaussian Splatting (GSplat) scene representations to enable multi-stage manipulation tasks. Splat-MOVER consists of: (i) ASK-Splat, a GSplat representation that distills semantic and grasp affordance features into the 3D scene. ASK-Splat enables geometric, semantic, and affordance understanding of 3D scenes, which is critical for many robotics tasks; (ii) SEE-Splat, a real-time scene-editing module using 3D semantic masking and infilling to visualize the motions of objects that result from robot interactions in the real-world. SEE-Splat creates a “digital twin” of the evolving environment throughout the manipulation task; and (iii) Grasp-Splat, a grasp generation module that uses ASK-Splat and SEE-Splat to propose affordance-aligned candidate grasps for open-world objects.

Installation

This repository utilizes Nerfstudio, GraspNet, and VRB. This repo has been verified to work with the following package versions: nerfstudio==1.1.0, gsplat=0.1.13, and lang-sam with the commit SHA a1a9557. Please install these packages from source, then proceed with the following steps:

  1. Clone this repo.
git clone [email protected]:StanfordMSL/Splat-MOVER.git
  1. Install sagesplat as a Python package.
python -m pip install -e .
  1. Register sagesplat with Nerfstudio.
ns-install-cli

Now, you can run sagesplat like other models in Nerfstudio using:

ns-train sagesplat --data <path to the data directory for a given scene>

You can try out the data used in the experiments here.