Deep Characters Pytorch

Pytorch implementation of the components in Realtime Deep Dynamic Characters (Habermann et. al., Siggraph 2021)

---

News

2024-3-29 The inital version Deep Characters Pytorch is available, including an minimal runnable version of the skeleton and the embedded deformed clothed human character for the DynaCap, TriHuman, and ASH dataset. Moreover, we provide GCN-based deformation network mentioned in the paper.

---

Installation

Requirements

conda create -n mpiiddc python=3.9
conda activate mpiiddc

# tested to work on this combination
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
conda install -c conda-forge cudatoolkit-dev=11.3.1

# download and install pytorch3d==0.7.1
git clone https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
git checkout 995b60e
python setup.py install

# install other dependcies
pip install -r requirements.txt

Build The Customized Components

cd cuda_skeleton
TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6" python setup.py install

---

Examples

Testing the Skeleton

After initialization, the skeleton funcs takes a batch of DOFs and produces the joint transformations. The following script will dump joints as point clouds.

cd examples
python 0_test_skeleton.py

Testing the Deformable Character

After initialization, the deformable character funcs takes a batch of DOFs, Embedded Deformation Params, and Per-vertex Offset, and produces the posed/unposed, template, embedded deformed template, embedded deformed template with per-vertex offsets.

Here, the Embedded Deformation Params, and Per-vertex Offset is set to 0 by default. Moreover, you can switch between lbs/dqs, and wheather activate embedded deformation in ./examples/test_config.conf.

cd examples
python 1_test_character.py

Testing the Character with learnable deformations

Also we can apply the learnable deformation learned with SpatialGNN. You may download the sample weights Here, extract and put the checkpoint under ./examples/checkpoints/ folder. Then we can generate the posed/unposed character with learned defomration through running the following blocks.

cd examples
python 2_test_deformable_character.py

---

Todo

• Example for skeleton
• Example for the deformable character
• Example for the learnable embedded deformable character
• More related to other projects to come

---

Citation

If you find our work useful for your research, please, please, and please, consider citing TriHuman (where I build this repo for :D), and also the original papers Real-time Deep Dynamic Characters, and DeepCap.

@article{habermann2021,
    author = {Marc Habermann and Lingjie Liu and Weipeng Xu and Michael Zollhoefer and Gerard Pons-Moll and Christian Theobalt},
    title = {Real-time Deep Dynamic Characters},
    journal = {ACM Transactions on Graphics}, 
    month = {aug},
    volume = {40},
    number = {4}, 
    articleno = {94},
    year = {2021}, 
    publisher = {ACM}
}
@misc{zhu2023trihuman,
    title={TriHuman : A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis}, 
    author={Heming Zhu and Fangneng Zhan and Christian Theobalt and Marc Habermann},
    year={2023},
    eprint={2312.05161},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
@inproceedings{deepcap,
    title = {DeepCap: Monocular Human Performance Capture Using Weak Supervision},
    author = {Habermann, Marc and Xu, Weipeng and Zollhoefer, Michael and Pons-Moll, Gerard and Theobalt, Christian},
    booktitle = {{IEEE} Conference on Computer Vision and Pattern Recognition (CVPR)},
    month = {jun},
    organization = {{IEEE}},
    year = {2020},
}

---

Contact

For questions, clarifications, feel free to get in touch with:

Heming Zhu: [email protected]

Marc Habermann: [email protected]

---

License

Deep Characters Pyotrch is under CC-BY-NC license. The license applies to the pre-trained models and the metadata as well.