Skip to content

Latest commit

 

History

History
115 lines (76 loc) · 4.36 KB

PERSON_IMAGE_ANIMATION.md

File metadata and controls

115 lines (76 loc) · 4.36 KB
Raw

Pose-Guided Person Image Animation

The details of the person image animation task are provided here.

The temporal consistency is modeled in this task. Specifically, the noisy poses extracted by the popular pose extraction methods are first prepossessed by a Motion Extraction Network to obtain clean poses. Then we generate the final animation results in a recurrent manner. Technical Report is coming soon.

From Left to Right: Skeleton Squences. Propossessed Skeleton Seqences; Animation Results.

###Dataset

We provide the Alphapose extraction results of these datasets. Meanwhile, the prepossessed clean poses are also avaliable. Please use the following code to download these resources.

./script/download_animation_skeletons.sh

FashionVideo

  • Follow the instruction on dataset downloading here. Extracted images from the original videos and resize them as 256 x 256.
  • Extract human poses using Alphapose. Jump this step if you have downloaded our provided resources.
  • Follow the provided demo datasets ./dataset/danceFasion/val_256 (download using ./download.sh) to build your training and testing set ./dataset/danceFasion/train_256, ./dataset/danceFasion/test_256.

iPER

  • This dataset can be download from here. Extracted images from the original videos and resize them as 256 x 256.
  • Extract human poses using Alphapose. Jump this step if you have downloaded our provided resources.
  • Follow the provided demo datasets ./dataset/danceFasion/val_256 (download using ./download.sh) to build your training and testing set ./dataset/iPER/train_256, ./dataset/iPER/test_256.

Testing

Download the trained weights from FashionVideo and iPER. Put the obtained checkpoints under ./result/dance_fashion_checkpoints and ./result/dance_iper_checkpoints respectively.

Run the following codes to obtain the animation results.

# test on the fashionVideo dataset 
python test.py \
--name=dance_fashion_checkpoints \
--model=dance \
--attn_layer=2,3 \
--kernel_size=2=5,3=3 \
--gpu_id=0 \
--dataset_mode=dance \
--sub_dataset=fashion \
--dataroot=./dataset/danceFashion \
--results_dir=./eval_results/dance_fashion \
--checkpoints_dir=result

# test on the iper dataset
python test.py \
--name=dance_iper_checkpoints \
--model=dance \
--attn_layer=2,3 \
--kernel_size=2=5,3=3 \
--gpu_id=0 \
--dataset_mode=dance \
--sub_dataset=iper \
--dataroot=./dataset/iPER \
--results_dir=./eval_results/dance_iper \
--checkpoints_dir=result

Motion Extraction Net

We provide the details of the motion extraction net to support training and testing on custom datasets. You do not need this if you only want to test on the FashionVideo and iPER dataset.

This network is used to prepossess the noisy skeletons extracted by some pose extraction models. We train this model using the Human36M dataset. We download the training ground-truth label data_2d_h36m_gt.npz from here. The corresponding input label data_2d_h36m_detectron_pt_coco.npz is download from here.

Use the following code to train this model

python train.py \
--name=keypoint \
--model=keypoint \
--gpu_id=2 \
--dataset_mode=keypoint \
--continue_train

We also provide the trained weights. Assuming that you want to smooth the skeleton sequences of the iPER training set, you can use the following code

python test.py \
--name=dance_keypoint_checkpoints \
--model=keypoint \
--gpu_id=2 \
--dataset_mode=keypointtest \
--continue_train \
--dataroot=./dataset/iPER \
--sub_dataset=iper \
--results_dir=./dataset/iPER/train_256/train_video2d \
--eval_set=train