This is not an officially supported Google product.

LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery (NeurIPS 2022)

Project Page | Video | Paper

Implementation for LASSIE. A novel method which estimates camera pose, 3D articulation, and part shapes of animal bodies given sparse images in-the-wild.

Chun-Han Yao², Wei-Chih Hung², Yuanzhen Li², Michael Rubinstein², Ming-Hsuan Yang²
, Varun Jampani^2
1UC Merced, ²Waymo, ²Google Research, ²Yonsei University

Setup

A python virtual environment is used for dependency management. The code is tested with Python 3.7, PyTorch 1.11.0, CUDA 11.3. First, to install PyTorch in the virtual environment, run:

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 --extra-index-url https://download.pytorch.org/whl/cu113

Then, install other required packages by running:

pip install -r requirements.txt

Preparing data and pre-trained model

Pascal-part

• Download Pascal images here and place them in data/pascal_part/JPEGImages/.
• Download Pascal-part annotations here and place them in data/pascal_part/Annotations_Part/.
• Download Pascal-part image sets here and place them in data/pascal_part/image-sets/.

Our image ensembles (web images)

• Download images here and place them in data/web_images/images/.
• Download keypoint annotations here and place them in data/web_images/annotations/.

Pre-trained primitive part decoder

• Download pre-trained model here and place it in model_dump/.

LASSIE optimization

To run LASSIE optimization on sparse images of an animal class (e.g. zebra), simply run:

python train.py --cls zebra

The supported animal classes include: zebra, giraffe, tiger, elephant, kangaroo, penguin, horse, cow, sheep. The qualitative results can be found in results/zebra/. The optimization settings and initial 3D skeleton can be changed in main/config.py and main/skeleton.py, respectively. Note that the first time running LASSIE optimization could take a few minutes in the DINO feature clustering step.

Evaluation

Once optimization is completed, quantitative evaluation can be done by running:

python eval.py --cls zebra

The results will be stored in results/eval/zebra.txt.

Citation

@inproceedings{yao2022-lassie,
  title         = {{LASSIE}: {L}earning {A}rticulated {S}hape from {S}parse {I}mage {E}nsemble via 3D Part Discovery},
  author        = {Yao, Chun-Han and Hung, Wei-Chih and Li, Yuanzhen and Rubinstein, Michael and Yang, Ming-Hsuan and Jampani, Varun},
  booktitle     = {Advances in Neural Information Processing Systems (NeurIPS)},
  year          = {2022},
}