🌟 STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting

arXiv Preprint | Project Page

Zenghao Chai | Chen Tang | Yongkang Wong | Mohan Kankanhalli

Abstract

The creation of 4D avatars (i.e., animated 3D avatars) from text description typically uses text-to-image (T2I) diffusion models to synthesize 3D avatars in the canonical space and subsequently applies animation with target motions. However, such an optimization-by-animation paradigm has several drawbacks. (1) For pose-agnostic optimization, the rendered images in canonical pose for naive Score Distillation Sampling (SDS) exhibit domain gap and cannot preserve view-consistency using only T2I priors, and (2) For post hoc animation, simply applying the source motions to target 3D avatars yields translation artifacts and misalignment. To address these issues, we propose Skeleton-aware Text-based 4D Avatar generation with in-network motion Retargeting (STAR). STAR considers the geometry and skeleton differences between the template mesh and target avatar, and corrects the mismatched source motion by resorting to the pretrained motion retargeting techniques. With the informatively retargeted and occlusion-aware skeleton, we embrace the skeleton-conditioned T2I and text-to-video (T2V) priors, and propose a hybrid SDS module to coherently provide multi-view and frame-consistent supervision signals. Hence, STAR can progressively optimize the geometry, texture, and motion in an end-to-end manner. The quantitative and qualitative experiments demonstrate our proposed STAR can synthesize high-quality 4D avatars with vivid animations that align well with the text description. Additional ablation studies shows the contributions of each component in STAR.

Installation

Prerequisite

• System requirement: Ubuntu
• Tested GPU: A100 (40GB)

Install Packages

git clone https://github.com/czh-98/STAR.git
cd STAR 

# create env
conda create -n star python=3.10

conda activate star 

# install pytorch
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge

# install pytorch3d
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
conda install pytorch3d -c pytorch3d

# install other dependencies
pip install -r requirements.txt

# install mmcv
pip install -U openmim
mim install mmcv==1.7.0

# install xformer
conda install xformers -c xformers

# install smplx
cd tada_smplx
python setup.py install 

cd ..

Data

• Follow the instruction of MotionDiffuse, download the pretrained model t2m.zip, unzip them and save at directory ./data/.

• Follow the instruction of TADA, register, and download the TADA Extra Data, unzip them and save at directory ./data/.

• Follow the instruction of FLAME, register, and download the FLAME Vertex Masks, and FLAME Mediapipe Landmark Embedding, unzip them and save at directory ./data/.

• Finally, the data should then be organized as:

data
├── FLAME_masks
│   ├── FLAME_masks.gif
│   ├── FLAME_masks.pkl
│   ├── FLAME.obj
│   └── readme
├── init_body
│   ├── data.npz
│   ├── data-remesh2.npz
│   ├── fit_smplx_dense_lbs_weights.npy
│   ├── fit_smplx_dense_unique.npy
│   ├── fit_smplx_dense_uv.npz
│   ├── fit_smplx_params.npz
│   └── mesh_uv.npz
├── mediapipe
│   └── face_landmarker.task
├── mediapipe_landmark_embedding
│   ├── mediapipe_landmark_embedding.npz
│   └── readme
├── smplx
│   ├── FLAME_masks.pkl
│   ├── FLAME_SMPLX_vertex_ids.npy
│   ├── smplx_faces.npy
│   ├── smplx_lbs_weights.npy
│   ├── SMPLX_MALE.npz
│   ├── SMPLX_NEUTRAL_2020.npz
│   ├── smplx_param.pkl
│   ├── smplx_to_smpl.pkl
│   ├── smplx_uv_map
│   ├── smplx_uv.npz
│   ├── smplx_uv.obj
│   └── smplx_vert_segementation.json
├── t2m
│   └── t2m_motiondiffuse
│       ├── meta
│       │   ├── mean.npy
│       │   └── std.npy
│       ├── model
│       │   └── latest.tar
│       └── opt.txt
└── talkshow
    ├── rich.npy
    └── rich.wav

Usage

Training

• The results will be saved in workspace (./exp in default). You may edit the default path in the config/*.yaml files.

python -m apps.run --config configs/train.yaml --text "Tyrion Lannister in Game of Thrones wearing black leather jacket, he/she is performing extreme acrobat while raising hands and kicking quickly" --description demo --t2m_model mdiffuse

Testing

• Once the avatar is optimized, it is ready to be animated of arbitrary motions to produce 4D contents.

python -m apps.demo --config configs/test.yaml --text "Tyrion Lannister in Game of Thrones wearing black leather jacket, he/she is performing extreme acrobat while raising hands and kicking quickly" --description demo --t2m_model mdiffuse

Export Fbx

• To export the 4D avatars, you need first install the FBX SDK as follows:

wget -P ./externals/fbx-python-sdk https://damassets.autodesk.net/content/dam/autodesk/www/files/fbx202037_fbxpythonsdk_linux.tar.gz
tar -xvzf ./externals/fbx-python-sdk/fbx202037_fbxpythonsdk_linux.tar.gz -C ./externals/fbx-python-sdk

# Follow the Install_FBX_Python_SDK.txt
# Install FBX Python SDK
chmod ugo+x ./externals/fbx-python-sdk/fbx202037_fbxpythonsdk_linux
./externals/fbx-python-sdk/fbx202037_fbxpythonsdk_linux ./externals/fbx-python-sdk
pip install ./externals/fbx-python-sdk/fbx-2020.3.7-cp310-cp310-manylinux1_x86_64.whl

• Then, you can save the fbx files according to:

python -m apps.export_fbx --config configs/test.yaml --text "Tyrion Lannister in Game of Thrones wearing black leather jacket, he/she is performing extreme acrobat while raising hands and kicking quickly" --description demo --t2m_model mdiffuse

Citation

If you find our work is helpful in your research, please cite:

@misc{chai2024star,
  author = {Chai, Zenghao and Tang, Chen and Wong, Yongkang and Kankanhalli, Mohan},
  title = {STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting},
  eprint={2406.04629},
  archivePrefix={arXiv},
  year={2024},
}

Acknowledgement

This repository is heavily based on TADA, DreamWaltz, ConditionVideo, R2ET, MotionDiffuse, MMHuman3D, Smplx2FBX. We would like to thank the authors of these work for publicly releasing their code.