SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis

A fork of the official implementation of the CVPR 2024 paper 3D Facial Expressions through Analysis-by-Neural Synthesis.

SMIRK reconstructs 3D faces from monocular images with facial geometry that faithfully recover extreme, asymmetric, and subtle expressions.

Installation

You need to have a working version of PyTorch and Pytorch3D installed. We provide a requirements.txt file that can be used to install the necessary dependencies for a Python 3.9 setup with CUDA 11.7:

conda create -n smirk python=3.9
pip install -r requirements.txt
# install pytorch3d now
pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py39_cu117_pyt201/download.html

Then, in order to download the required models, run:

bash quick_install.sh

The above installation includes downloading the FLAME model. This requires registration. If you do not have an account you can register at https://flame.is.tue.mpg.de/

This command will also download the SMIRK pretrained model which can also be found on Google Drive.

Demo

We provide two demos. One that can be used to test the model on a single image,

python demo.py --input_path samples/test_image2.png --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt --crop

and one that can be used to test the model on a video,

python demo_video.py --input_path samples/dafoe.mp4 --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt --crop --render_orig

To Run the optimization-based demo, you can use the following command:

python demo_video_optimized.py --input_path parent_dir_of_videos --out_path results/ --checkpoint pretrained_models/SMIRK_em1.pt

Training

Dataset Preparation

SMIRK was trained on a combination of the following datasets: LRS3, MEAD, CelebA, and FFHQ.

1. §§Download the LRS3 dataset from here. We are aware that currently this dataset has been removed from the website. It can be replaced with any other similar dataset, e.g. LRS2.

2. Download the MEAD dataset from here.

3. Download the CelebA dataset from here. You can download directly the aligned images img_align_celeba.zip.

4. Download the FFHQ256 dataset from here.

After downloading the datasets we need to extract the landmarks using mediapipe and FAN. We provide the scripts for preprocessing in datasets/preprocess_scripts. Example usage:

python datasets/preprocess_scripts/apply_mediapipe_to_dataset.py --input_dir PATH_TO_FFHQ256/images --output_dir PATH_TO_FFHQ256/mediapipe_landmarks

and for FAN:

python datasets/preprocess_scripts/apply_fan_to_dataset.py --input_dir PATH_TO_FFHQ256/images --output_dir PATH_TO_FFHQ256/fan_landmarks

Note that for obtaining the FAN landmarks we use the implementation in https://github.com/hhj1897/face_alignment.

Next, make sure to update the config files in configs with the correct paths to the datasets and their landmarks.

Pretraining

At the pretraining stage, we train all 3 encoders (pose, shape, and expression) using only the extracted landmarks and the output of MICA.

python train.py configs/config_pretrain.yaml train.log_path="logs/pretrain"

Training

After pretraining, at the core stage of SMIRK, we freeze the shape and pose encoders and train the expression encoder with the full SMIRK framework (reconstruction path and cycle path).

python train.py configs/config_train.yaml resume=logs/pretrain/first_stage_pretrained_encoder.pt train.loss_weights.emotion_loss=1.0

Citation

If you find this work useful, please consider citing:

@inproceedings{SMIRK:CVPR:2024,
    title = {3D Facial Expressions through Analysis-by-Neural-Synthesis},
    author = {Retsinas, George and Filntisis, Panagiotis P., and Danecek, Radek and Abrevaya, Victoria F. and Roussos, Anastasios and Bolkart, Timo and Maragos, Petros},
    booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2024}
}

Acknowledgements

We acknowledge the following repositories and papers that were used in this work:

• MICA
• EMOCA
• AutoLink