face moving while blinking

[oleg-mir]

Hi,

1. I've noticed a behavior where the whole face is moving when I'm blinking.
   can the blinking be ignored?
   what is causing this behavior?

2. when the face on the camera is stable and not moving the output is not stable and moving a little bit.
   I assume this is due to a face detection on every frame.
   how can i improve it and provide more stable output for the same input?

Thanks!

[MarekKowalski]

Hi,

1. I suspect that this is caused by the fact that the landmarks located on the eye corners move down when the person is blinking. I can see 2 easy ways to prevent that:

• Not taking the eye landmarks into account when fitting the face model. To do this you would have to remove the eye landmarks from the idxs2D array and remove the corresponding model vertices from idxs3D.
• Using a better facial landmark localisation method that might not have this behaviour. For this you can try the Deep Alignment Network method, a version of FaceSwap with that method is available, details in Readme. Whether this will help or not depends on whether the problem above is also exhibited in the 300-W training set, which the method was trained on.

2. This one is due to jitter in landmark localisation. I suggest you try a more stable method, again, you can try Deep Alignment Network. You can also try to apply some sort of an exponential smoother to the landmark locations.

Marek