Upscaling using refernce image

[whambamm]

Hi,

it is known that inswap128 is the best swapping model so far. it is also known that the fact that it uses only 128x128 pixel output requires us to use blind upscaling models (like GFPGAN) which assumes how the face should look like.

i have seen some work which try to upscale using a reference image, so the upscale is done with reference to a high res picture of the target face.

given face swaps are always uses reference image i thought maybe it will be possible to use those upscaling methods to achieve better, more realistic and more accurate results?

So far i have found 2 researches which show promising results, however the only one i was able to run was

• the first one is old and i couldn't get it to work at all (RefSR - Refenrece based face super-resolution via variational autoencoder).

• However the second one (SAIR - GAN prior based face image restoration ) is written in torch and i was able to run it. the results were not promissing but i didn't used the optional latent vector file (i have some issues creating it) so it could be my fault.

furthermore all of those models are trained and designed for different input and output size (i would assume we will need an input size of 128x128 and output size of 512x512).

my knowledge in ML is not that great so i am kind of blocked at it right now.

if anybody have any ideas or knows other upscaling models which uses reference image for upscaling and are will to share it will be amazing.

[C0untFloyd]

The included DMDNet Post Processor uses a reference image behind the curtains. For this to work, I'm re-using the face of the original image and the already detected landmarks. That's why IMHO it is the best in preserving the original identity.

[Elise96nl]

i acknowledge, and for it to work with AMD. only cpu for the dmdnet, so prepare for 10 sec per frame!

`Enhance_DMDNet
true to false fixes dmdnet on amd directml! But i quess its running on cpu. but it's nice to only have dmdnet on cpu while rest of roop is on directml mode.
takes a toll
Processing: 8%|█▋ | 24/310 [02:12<26:24, 5.54s/frames, memory_usage=07.13GB, execution_threads=1]

` def create(self, devicename):

    self.torchdevice = torch.device(devicename)

    model_dmdnet = DMDNet().to(self.torchdevice)

    weights = torch.load('./models/DMDNet.pth', map_location=self.torchdevice) 

    model_dmdnet.load_state_dict(weights, strict=False)``