Can you animate any FLAME face model like voca does

[GrizzCMX]

Can I use the flame face model outside the vocaset(Training set, Validation set and test set) as a template to generate face animation, how do I set it up, and can I do it using a pre-trained model? Or need to retrain the model

[Doubiiu]

You can check the templates.pkl file and define your own templates.pkl to use new templates. Sure you can use the pre-trained model.

[GrizzCMX]

Thanks for your reply! I will try it agin

[LeoniusChen]

You can check the templates.pkl file and define your own templates.pkl to use new templates. Sure you can use the pre-trained model.

Hi, do you know how to get a templates.pkl from a single portrait image? Any open-source repo to be found?

[Doubiiu]

@LeoniusChen Hi, DECA is a widely used 3D face reconstruction method by estimating FLAME parameters, and then they can be converted to mesh representations. Please check the FLAME repo for more info.

[LeoniusChen]

@LeoniusChen Hi, DECA is a widely used 3D face reconstruction method by estimating FLAME parameters, and then they can be converted to mesh representations. Please check the FLAME repo for more info.

I successfully obtained the template using DECA and FLAME, and replace the template in the codes

CodeTalker/main/demo.py

Line 179 in 1a65bfe

template_file = os.path.join(cfg.data_root, "FLAME_sample.ply")

and

CodeTalker/main/demo.py

Line 136 in 1a65bfe

template_file = os.path.join(cfg.data_root, cfg.template_file)

. However, the result is not satisfying. I wonder if the subject for inference should be carefully chosen?

zh_en_12s_FaceTalk_170809_00138_TA_condition_FaceTalk_170725_00137_TA_audio.mp4

[Doubiiu]

Hi, 'subject' is used to change the template for displaying, since you have utilized your own subject, there should be no effect when you are changing 'subject' in demo.yaml. But I think chaning 'condition' in the config may influence the results, you can have a try. BTW, the unsatisfactory results may be caused by domain gap between Chinese and English. You may check if the English audio can work well.

[zk19971101]

@Doubiiu @LeoniusChen
Is the color and context information from a single portrait image can be add to tamplate or Mesh?

[hrWong]

Hi @Doubiiu @LeoniusChen I'm using DECA and can't get the .pkl template, I'd like to ask you how exactly did you get the template.pkl?