🚩 [Update] The face part of PyMAF-X has been updated. See face-only evaluation results on the NoW benchmark.

PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images

Hongwen Zhang · Yating Tian · Yuxiang Zhang · Mengcheng Li · Liang An · Zhenan Sun · Yebin Liu

TPAMI 2023

Project Page | Video | Paper

<sup>Frame by frame reconstruction. Video clipped from here.

Reconstruction result on a COCO validation image.</sup>
Click Here for More Results

Installation

• Python 3.8

conda create --no-default-packages -n pymafx python=3.8
conda activate pymafx

packages

• PyTorch tested on version 1.9.0

conda install pytorch==1.9.0 torchvision==0.10.0 cudatoolkit=11.1 -c pytorch -c conda-forge

• pytorch3d

pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"

• other packages listed in requirements.txt

pip install -r requirements.txt

necessary files

smpl_downsampling.npz & mano_downsampling.npz

• Run the following script to fetch necessary files.

bash fetch_data.sh

SMPL & SMPL-X model files

• Collect SMPL/MANO/FLAME/SMPL-X model files. Rename model files and put them into the ./data/smpl directory.

Download the partial_mesh files and put it into the ./data/partial_mesh directory.

Download the pre-trained model and put it into the ./data/pretrained_model directory.

After collecting the above necessary files, the directory structure of ./data is expected as follows.

./data
├── J_regressor_extra.npy
├── smpl_mean_params.npz
├── smpl_downsampling.npz
├── mano_downsampling.npz
├── flame_downsampling.npy
├── partial_mesh
│   └── ***_vids.npz
├── pretrained_model
│   └── PyMAF-X_model_checkpoint_v1.1.pt
└── smpl
    ├── FLAME2020
    │   ├── FLAME_NEUTRAL.pkl
    │   ├── flame_dynamic_embedding.npy
    │   └── flame_static_embedding.pkl
    ├── MANO_RIGHT.pkl
    ├── SMPLX_NEUTRAL_2020.npz
    ├── SMPL_NEUTRAL.pkl
    └── model_transfer
        ├── MANO_SMPLX_vertex_ids.pkl
        ├── SMPL-X__FLAME_vertex_ids.npy
        └── smplx_to_smpl.pkl

Demo

You can first give it a try on Google Colab using the notebook we have prepared, which is no need to prepare the environment yourself:

Run the demo code.

For image folder input:

python -m apps.demo_smplx --image_folder examples/coco_images --detection_threshold 0.3 --pretrained_model data/pretrained_model/PyMAF-X_model_checkpoint_v1.1.pt --misc TRAIN.BHF_MODE full_body MODEL.PyMAF.HAND_VIS_TH 0.1

For video input:

python -m apps.demo_smplx --vid_file examples/dancer_short.mp4 --pretrained_model data/pretrained_model/PyMAF-X_model_checkpoint_v1.1.pt --misc TRAIN.BHF_MODE full_body MODEL.PyMAF.HAND_VIS_TH 0.1

Results will be saved at ./output. You can set different hyperparamters in the scripts, e.g., --detection_threshold for the person detection threshold and MODEL.PyMAF.HAND_VIS_TH for the hand visibility threshold.

Training

To perform training, we need to collect preprocessed files of training datasets first. The pseudo SMPL-X labels (with keys of 'xpose'/'xshape') can be downloaded at here. Please also refer to PyMAF for more details about training. Example usage:

python -m apps.train --regressor pymaf_net --train_data h36m_coco_itw --eval_every 10 --save_every 20 --train_data h36m_coco_itw --misc TRAIN.BATCH_SIZE 64 MODEL.PyMAF.AUX_SUPV_ON True MODEL.PyMAF.TRANS.USE_ATT True MODEL.PyMAF.TRANS.ATT_HEAD 1 MODEL.PyMAF.TRANS.ATT_FEAT_IDX 2 MODEL.MESH_MODEL smplx TRAIN.USE_EFT True MODEL.PyMAF.USE_CAM_FEAT True LOSS.SHAPE_W 0.6 MODEL.PyMAF.BACKBONE res50 POSE_RES_MODEL.PRETR_SET coco

Citation

If this work is helpful in your research, please cite the following papers.

@article{pymafx2023,
  title={PyMAF-X: Towards Well-aligned Full-body Model Regression from Monocular Images},
  author={Zhang, Hongwen and Tian, Yating and Zhang, Yuxiang and Li, Mengcheng and An, Liang and Sun, Zhenan and Liu, Yebin},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year={2023}
}

@inproceedings{pymaf2021,
  title={PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop},
  author={Zhang, Hongwen and Tian, Yating and Zhou, Xinchi and Ouyang, Wanli and Liu, Yebin and Wang, Limin and Sun, Zhenan},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2021}
}

Acknowledgments

Part of the code is borrowed from the following projects, including DaNet, SPIN, VIBE, SPEC, MeshGraphormer, PIFu, DensePose, HMR, HRNet, pose_resnet. Many thanks to their contributions.