diff --git a/README.md b/README.md index 49f7309..9216861 100644 --- a/README.md +++ b/README.md @@ -1,157 +1,80 @@ -# Contact Potential Field -[[`Project Page`](https://lixiny.github.io/CPF)] - - -This repo contains model, demo, and test codes of our paper: - -> [CPF: Learning a **C**ontact **P**otential **F**ield to Model the Hand-Object Interaction](https://arxiv.org/abs/2012.00924) -> Lixin Yang, Xinyu Zhan, Kailin Li, Wenqiang Xu, Jiefeng Li, Cewu Lu -> ICCV 2021 - + +

- -

- - -# Guide to the Demo -## 1. Get our code: -```shell script -$ git clone --recursive https://github.com/lixiny/CPF.git -$ cd CPF -``` -## 2. Set up your new environment: - -```shell script -$ conda env create -f environment.yaml -$ conda activate cpf -``` - -## 3. Download asset files - -> For researchers in China, you can alternatively download our preprocessed files at this mirror: [百度盘](https://pan.baidu.com/s/1x6SvGpNZqWlLA-cBZvv8Og) (`2tqv`) - -Download our [[assets.zip](https://drive.google.com/file/d/1IP7dJimk0G-rixfDprgYE0F8kquB6PWf/view?usp=sharing)] and unzip it as an `assets/` folder. - - -Download the MANO model files from [official MANO website](https://mano.is.tue.mpg.de/), and put it into `assets/mano/`. -We currently only use the `MANO_RIGHT.pkl` - -Now your `assets/` folder should look like this: -``` -assets/ -├── anchor/ -│ ├── anchor_mapping_path.pkl -│ ├── anchor_weight.txt -│   ├── face_vertex_idx.txt -│   └── merged_vertex_assignment.txt -├── closed_hand/ -│   └── hand_mesh_close.obj -├── fhbhands_fits/ -│   ├── Subject_1/ -│   │   ├── ... -│   ├── Subject_2/ -| ├── ... -├── hand_palm_full.txt -└── mano/ - ├── fhb_skel_centeridx9.pkl - ├── info.txt - ├── LICENSE.txt - └── MANO_RIGHT.pkl -``` -## 4. Download dataset -### First-Person Hand Action Benchmark (fhb) - -Download and unzip the First-Person Hand Action Benchmark dataset following the [official instructions](https://github.com/guiggh/hand_pose_action) to the `data/fhbhands/` folder - -If everything is correct, your `data/fhbhands/` should look like this: -``` -. -├── action_object_info.txt -├── action_sequences_normalized/ -├── change_log.txt -├── data_split_action_recognition.txt -├── file_system.jpg -├── Hand_pose_annotation_v1/ -├── Object_6D_pose_annotation_v1_1/ -├── Object_models/ -├── Subjects_info/ -├── Video_files/ -├── Video_files_480/ # Optionally -``` -Optionally, resize the images (speeds up training !) based on the [handobjectconsist/reduce_fphab.py](https://github.com/hassony2/handobjectconsist/blob/master/reduce_fphab.py). -```shell -$ python reduce_fphab.py -``` -Download our [[fhbhands_supp.zip](https://drive.google.com/file/d/1hY_gyrZD_RU3nxI90oJZ6tNkwxKYhUGs/view?usp=sharing)] and unzip it as `data/fhbhands_supp`: - -Download our [[fhbhands_example.zip](https://drive.google.com/file/d/14wxN23RmVCSphHIV00qk-ht00yfdu_Hu/view?usp=sharing)] and unzip it as `data/fhbhands_example`. -This `fhbhands_example/` contains 10 samples that are designed to demonstrate our pipeline. -Currently, your `data/` folder should look like this: - -``` -data/ -├── fhbhands/ -├── fhbhands_supp/ -│   ├── Object_models/ -│   └── Object_models_binvox/ -├── fhbhands_example/ -│   ├── annotations/ -│   ├── images/ -│   ├── object_models/ -│   └── sample_list.txt -``` - -### HO3D -Download and unzip the [HO3D](https://www.tugraz.at/index.php?id=40231) dataset following the [official instructions](https://github.com/shreyashampali/ho3d?) to the `data/HO3D` folder. -if everything is correct, the HO3D & YCB folder in your `data/` folder should look like this: -``` -data/ -├── HO3D/ -│   ├── evaluation/ -│   ├── evaluation.txt -│   ├── train/ -│   └── train.txt -├── YCB_models/ -│   ├── 002_master_chef_can/ -│   ├── ... -... -``` - -Download our [[YCB_models_supp.zip](https://drive.google.com/file/d/1daSKseF-PrVLwd4wIcF2PLtAjYBF2XH1/view?usp=sharing)] and place it at `data/YCB_models_supp` - -Finally, the `data/` folder should have a structure like: -``` -data/ -├── fhbhands/ -├── fhbhands_supp/ -├── fhbhands_example/ -├── HO3D/ -├── YCB_models/ -├── YCB_models_supp/ -``` - -## 5. Download pre-trained checkpoints -download our pre-trained [[CPF_checkpoints.zip](https://drive.google.com/file/d/1JWu5TSTTIWvNrTZmmjhTEm4xGqv_cMhd/view?usp=sharing)], unzip it as `CPF_checkpoints/` folder: -``` -CPF_checkpoints/ -├── honet/ -│   ├── fhb/ -│   ├── ho3dofficial/ -│   └── ho3dv1/ -├── picr/ -│   ├── fhb/ -│   ├── ho3dofficial/ -│   └── ho3dv1/ -``` - -## 6. Launch visualization +

CPF: Learning a Contact Potential Field to Model the Hand-Object Interaction

+ +

+ Logo +

+

+ ICCV, 2021 +
+ Lixin Yang + · + Xinyu Zhan + · + Kailin Li + · + Wenqiang Xu + · + Jiefeng Li + · + Cewu Lu +

+ +

+ + + Paper PDF + + + Project Page + + Youtube Video + +

+

-> Replace the `${GPU_ID}` with a list of integers that indicates the GPU id. -> eg: `--gpu 0,1`; `--gpu 0`; `--gpu 0,1,2,3` +
+ +This repo contains model, demo, and test codes. + +
+ Table of Contents +
    +
  1. + Installation +
    2. +
  2. + Demo +
    3. +
  3. + Evaluation +
    4. +
  4. + Anatomical Constrained A-MANO +
    5. +
  5. + TODO +
    6. +
  6. + Citation +
    7. +
+
+
+
+ +## Installation +Following the [Installation Instruction](docs/Installation.md) to setup environment, assets, datasets and models. + + +## Demo +Notice: require active screen. +### visualize GeO fitting pipeline We create a `FHBExample` dataset in `hocontact/hodatasets/fhb_example.py` that only contains 10 samples to demonstrate our pipeline. -Notice: this demo requires active screen for visualizing. Press `q` in the "runtime hand" window to start fitting. ```shell # Only support single GPU ! @@ -160,166 +83,85 @@ $ python scripts/run_demo.py \ --init_ckpt CPF_checkpoints/picr/fhb/checkpoint_200.pth.tar \ --honet_mano_fhb_hand ``` - +

+
Press q in the "runtime hand" window to start fitting +

- +

-## 7. Test on full dataset (FHB, HO3D v1/v2) +### visualize A-MANO'S anchor position +This demo shows the anochrs positions on MANO hand's surface +```shell +$ python scripts/recover_anchor.py --render +``` +## Evaluation + +We provide shell srcipts for evaluating on FHB, HO3Dv1 and v2. -We provide shell srcipts to test on the full dataset to approximately reproduce our results. -### FHB +### FHB dataset dump the results of HoNet and PiCR: ```shell # recommend 2 GPUs -$ python scripts/dump_picr_res.py \ - --gpu ${GPU_ID} \ - --dist_master_addr localhost \ - --dist_master_port 12355 \ - --exp_keyword fhb \ - --train_datasets fhb \ - --train_splits train \ - --val_dataset fhb \ - --val_split test \ - --split_mode actions \ - --batch_size 8 \ - --dump_eval \ - --dump \ - --vertex_contact_thresh 0.8 \ - --filter_thresh 5.0 \ - --dump_prefix common/picr \ - --init_ckpt CPF_checkpoints/picr/fhb/checkpoint_200.pth.tar +$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_FHB.sh ``` -and reload the GeO optimizer: +and fit GeO optimizer: ```shell -# setting 1: hand-only -# CUDA_VISIBLE_DEVICES=0,1,2,3 +# setting 1: hand-only # recommend 4 GPUs -$ python scripts/eval_geo.py \ - --gpu ${GPU_ID} \ - --n_workers 16 \ - --data_path common/picr/fhbhands/test_actions_mf1.0_rf0.25_fct5.0_ec \ - --mode hand +$ export GPU_ID=0,1,2,3 && sh ./scripts/fit_GeO_handonly_FHB.sh # setting 2: hand-obj -$ python scripts/eval_geo.py \ - --gpu ${GPU_ID} \ - --n_workers 16 \ - --data_path common/picr/fhbhands/test_actions_mf1.0_rf0.25_fct5.0_ec \ - --mode hand_obj \ - --compensate_tsl +$ export GPU_ID=0,1,2,3 && sh ./scripts/fit_GeO_handobj_FHB.sh ``` ### HO3Dv1 -dump: +dump the results of HoNet and PiCR: ```shell # recommend 2 GPUs -$ python scripts/dump_picr_res.py \ - --gpu ${GPU_ID} \ - --dist_master_addr localhost \ - --dist_master_port 12356 \ - --exp_keyword ho3dv1 \ - --train_datasets ho3d \ - --train_splits train \ - --val_dataset ho3d \ - --val_split test \ - --split_mode objects \ - --batch_size 4 \ - --dump_eval \ - --dump \ - --vertex_contact_thresh 0.8 \ - --filter_thresh 5.0 \ - --dump_prefix common/picr_ho3dv1 \ - --init_ckpt CPF_checkpoints/picr/ho3dv1/checkpoint_300.pth.tar +$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_HO3Dv1.sh ``` -and reload optimizer: +and fit GeO optimizer: ```shell # hand-only -# CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 # recommend 8 GPUs -$ python scripts/eval_geo.py \ - --gpu ${GPU_ID} - --n_workers 24 \ - --data_path common/picr_ho3dv1/HO3D/test_objects_mf1_likev1_fct5.0_ec/ \ - --lr 1e-2 \ - --n_iter 500 \ - --hodata_no_use_cache \ - --lambda_contact_loss 10.0 \ - --lambda_repulsion_loss 4.0 \ - --repulsion_query 0.030 \ - --repulsion_threshold 0.080 \ - --mode hand +$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handonly_HO3Dv1.sh # hand-obj # recommend 8 GPUs -$ python scripts/eval_geo.py \ - --gpu ${GPU_ID} \ - --n_workers 24 \ - --data_path common/picr_ho3dv1/HO3D/test_objects_mf1_likev1_fct5.0_ec/ \ - --lr 1e-2 \ - --n_iter 500 \ - --hodata_no_use_cache \ - --lambda_contact_loss 10.0 \ - --lambda_repulsion_loss 6.0 \ - --repulsion_query 0.030 \ - --repulsion_threshold 0.080 \ - --mode hand_obj - +$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handobj_HO3Dv1.sh ``` -### HO3Dofficial -dump: +### HO3Dv2 (version 2) +dump the results of HoNet and PiCR: ```shell # recommend 2 GPUs -$ python scripts/dump_picr_res.py \ - --gpu 0,1 \ - --dist_master_addr localhost \ - --dist_master_port 12356 \ - --exp_keyword ho3dofficial \ - --train_datasets ho3d \ - --train_splits val \ - --val_dataset ho3d \ - --val_split test \ - --split_mode official \ - --batch_size 4 \ - --dump_eval \ - --dump \ - --test_dump \ - --vertex_contact_thresh 0.8 \ - --filter_thresh 5.0 \ - --dump_prefix common/picr_ho3dofficial \ - --init_ckpt CPF_checkpoints/picr/ho3dofficial/checkpoint_300.pth.tar +$ export GPU_ID=0,1 && sh ./scripts/dump_HoNetPiCR_HO3Dv2.sh ``` -and reload optimizer: +and fit GeO optimizer: ```shell -# CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 # recommend 8 GPUs -$ python scripts/eval_geo.py \ - --gpu ${GPU_ID} \ - --n_workers 24 \ - --data_path common/picr_ho3dofficial/HO3D/test_official_mf1_likev1_fct\(x\)_ec/ \ - --lr 1e-2 \ - --n_iter 500 \ - --hodata_no_use_cache \ - --lambda_contact_loss 10.0 \ - --lambda_repulsion_loss 2.0 \ - --repulsion_query 0.030 \ - --repulsion_threshold 0.080 \ - --mode hand_obj +$ export GPU_ID=0,1,2,3,4,5,6,7 && sh ./scripts/fit_GeO_handobj_HO3Dv2.sh ``` +### evaluation results +Above scripts may take a while ( ~ 1 day ). We also provide the results in [fitting_res](docs/fitting_res.txt). +## Anatomical Constrained A-MANO -## Results - -Testing on the full dataset may take a while ( 0.5 ~ 1.5 day ), thus we also provide our test results at [fitting_res.txt](https://github.com/lixiny/CPF/blob/main/fitting_res.txt). +We provide pytorch implementation of our Anatomical Constrained MANO in [lixiny/manopth](https://github.com/lixiny/manopth), which is modified from the original [hassony2/manopth](https://github.com/hassony2/manopth). -## Anatomically Constrained A-MANO -We provide pytorch implementation of our Anatomical Constrained MANO in [lixiny/manotorch](https://github.com/lixiny/manotorch), which is modified from the original [hassony2/manopth](https://github.com/hassony2/manopth). Thank [Yana Hasson](https://hassony2.github.io/) for providing the code. +## TODO +- [x] testing code and pretrained models + - [x] HoNet (FHB, HO3Dv1/v2) + - [x] PiCR (FHB, HO3Dv1/v2) +- [x] fitting code of GeO, both *hand-only* and *hand-object* (FHB, HO3Dv1/v2) +- [ ] training code +- [ ] contact region visualization -## Citation +# Citation If you find this work helpful, please consider citing us: ``` @inproceedings{yang2021cpf, diff --git a/scripts/dump_HoNetPiCR_FHB.sh b/scripts/dump_HoNetPiCR_FHB.sh new file mode 100644 index 0000000..0f7b87a --- /dev/null +++ b/scripts/dump_HoNetPiCR_FHB.sh @@ -0,0 +1,19 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python ./scripts/dump_picr_res.py \ + --gpu ${GPU_ID} \ + --dist_master_addr localhost \ + --dist_master_port 12355 \ + --exp_keyword fhb \ + --train_datasets fhb \ + --train_splits train \ + --val_dataset fhb \ + --val_split test \ + --split_mode actions \ + --batch_size 8 \ + --dump_eval \ + --dump \ + --vertex_contact_thresh 0.8 \ + --filter_thresh 5.0 \ + --dump_prefix common/picr \ + --init_ckpt CPF_checkpoints/picr/fhb/checkpoint_200.pth.tar \ No newline at end of file diff --git a/scripts/dump_HoNetPiCR_HO3Dv1.sh b/scripts/dump_HoNetPiCR_HO3Dv1.sh new file mode 100644 index 0000000..40c9a5d --- /dev/null +++ b/scripts/dump_HoNetPiCR_HO3Dv1.sh @@ -0,0 +1,19 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/dump_picr_res.py \ + --gpu ${GPU_ID} \ + --dist_master_addr localhost \ + --dist_master_port 12356 \ + --exp_keyword ho3dv1 \ + --train_datasets ho3d \ + --train_splits train \ + --val_dataset ho3d \ + --val_split test \ + --split_mode objects \ + --batch_size 4 \ + --dump_eval \ + --dump \ + --vertex_contact_thresh 0.8 \ + --filter_thresh 5.0 \ + --dump_prefix common/picr_ho3dv1 \ + --init_ckpt CPF_checkpoints/picr/ho3dv1/checkpoint_300.pth.tar \ No newline at end of file diff --git a/scripts/dump_HoNetPiCR_HO3Dv2.sh b/scripts/dump_HoNetPiCR_HO3Dv2.sh new file mode 100644 index 0000000..7804bba --- /dev/null +++ b/scripts/dump_HoNetPiCR_HO3Dv2.sh @@ -0,0 +1,20 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/dump_picr_res.py \ + --gpu ${GPU_ID} \ + --dist_master_addr localhost \ + --dist_master_port 12356 \ + --exp_keyword ho3dofficial \ + --train_datasets ho3d \ + --train_splits val \ + --val_dataset ho3d \ + --val_split test \ + --split_mode official \ + --batch_size 4 \ + --dump_eval \ + --dump \ + --test_dump \ + --vertex_contact_thresh 0.8 \ + --filter_thresh 5.0 \ + --dump_prefix common/picr_ho3dofficial \ + --init_ckpt CPF_checkpoints/picr/ho3dofficial/checkpoint_300.pth.tar \ No newline at end of file diff --git a/scripts/fit_GeO_handobj_FHB.sh b/scripts/fit_GeO_handobj_FHB.sh new file mode 100644 index 0000000..a226948 --- /dev/null +++ b/scripts/fit_GeO_handobj_FHB.sh @@ -0,0 +1,8 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/fit_geo.py \ + --gpu ${GPU_ID} \ + --n_workers 16 \ + --data_path common/picr/fhbhands/test_actions_mf1.0_rf0.25_fct5.0_ec \ + --mode hand_obj \ + --compensate_tsl \ No newline at end of file diff --git a/scripts/fit_GeO_handobj_HO3Dv1.sh b/scripts/fit_GeO_handobj_HO3Dv1.sh new file mode 100644 index 0000000..a215f1c --- /dev/null +++ b/scripts/fit_GeO_handobj_HO3Dv1.sh @@ -0,0 +1,14 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/fit_geo.py \ + --gpu ${GPU_ID} \ + --n_workers 24 \ + --data_path common/picr_ho3dv1/HO3D/test_objects_mf1_likev1_fct5.0_ec/ \ + --lr 1e-2 \ + --n_iter 500 \ + --hodata_no_use_cache \ + --lambda_contact_loss 10.0 \ + --lambda_repulsion_loss 6.0 \ + --repulsion_query 0.030 \ + --repulsion_threshold 0.080 \ + --mode hand_obj \ No newline at end of file diff --git a/scripts/fit_GeO_handobj_HO3Dv2.sh b/scripts/fit_GeO_handobj_HO3Dv2.sh new file mode 100644 index 0000000..6189fda --- /dev/null +++ b/scripts/fit_GeO_handobj_HO3Dv2.sh @@ -0,0 +1,14 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/fit_geo.py \ + --gpu ${GPU_ID} \ + --n_workers 24 \ + --data_path common/picr_ho3dofficial/HO3D/test_official_mf1_likev1_fct\(x\)_ec/ \ + --lr 1e-2 \ + --n_iter 500 \ + --hodata_no_use_cache \ + --lambda_contact_loss 10.0 \ + --lambda_repulsion_loss 2.0 \ + --repulsion_query 0.030 \ + --repulsion_threshold 0.080 \ + --mode hand_obj \ No newline at end of file diff --git a/scripts/fit_GeO_handonly_FHB.sh b/scripts/fit_GeO_handonly_FHB.sh new file mode 100644 index 0000000..db750ab --- /dev/null +++ b/scripts/fit_GeO_handonly_FHB.sh @@ -0,0 +1,7 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/fit_geo.py \ + --gpu ${GPU_ID} \ + --n_workers 16 \ + --data_path common/picr/fhbhands/test_actions_mf1.0_rf0.25_fct5.0_ec \ + --mode hand \ No newline at end of file diff --git a/scripts/fit_GeO_handonly_HO3Dv1.sh b/scripts/fit_GeO_handonly_HO3Dv1.sh new file mode 100644 index 0000000..c3a43c6 --- /dev/null +++ b/scripts/fit_GeO_handonly_HO3Dv1.sh @@ -0,0 +1,14 @@ +echo "Using GPU_ID: ${GPU_ID}" + +python scripts/fit_geo.py \ + --gpu ${GPU_ID} \ + --n_workers 24 \ + --data_path common/picr_ho3dv1/HO3D/test_objects_mf1_likev1_fct5.0_ec/ \ + --lr 1e-2 \ + --n_iter 500 \ + --hodata_no_use_cache \ + --lambda_contact_loss 10.0 \ + --lambda_repulsion_loss 4.0 \ + --repulsion_query 0.030 \ + --repulsion_threshold 0.080 \ + --mode hand \ No newline at end of file