v4.2.3

[C0untFloyd]

Hotfix for gradio pydantic issue with fastapi
Upgraded to Gradio 4.43 hoping it will fix remaining issues
Added new action when no face detected -> use last swapped
Specified image format for image controls - opening new tabs on preview images possible again!
Hardcoded image output format for livecam to jpeg - might be faster than previous webp
Chain events to be only executed if previous was a success

Some more infos:

Downgrading fastapi and upgrading gradio might fix the pydantic issue and the sporadic server connection error. If it still happens, a crude workaround is to just refresh the page in your browser.

The image format used for opening a new tab by right-clicking into the preview image is set in the settings tab (image output format).

Use last swapped as "no face action" is the smartest action so far. With this, if no face gets detected in a video the last successful frame is re-used. BUT only for 15 repeating frames! After 15 frames without a face, the original (unswapped) frame will be used. The counter will reset once a face is swapped again. This is very useful e.g. for faces turning at weird angles where you would have had weird flicker before.

---

This discussion was created from the release v4.2.3.

[Elise96nl]

great work my friend!

I finished a 2800p VR180 SBS 3D scene today, 3 days of rendering. 3 faces (6 because it's sbs 3D) and after pulling some broken frames out of the video it is so amazing, it actually scares me.

"With great power comes great responsibility"

[C0untFloyd]

Now that's what I call dedication! 🤣
Do you have some sort of post-process to repair the missing lense distortion? I still need to include this for a real vr swap.

[Elise96nl]

Fish-eye is a no-go due to the strange distorted faces when they get too close to the top of the image. But it's also the VR conent you use. there is soomething like rfc vr, 180 long, 180 fisheye, 360 fisheye etc. I've looked at the flat video files, and VR180 without the fish eye is realy just sbs two seamly normal frames next to eachother, perfect for swapping. I've selected a "perfect" scene with a lot of front-facing frames. First, I rendered it using Roop Unleashed in frame extraction mode and removed some bad frames from the folder. However, I encountered problems with the audio.*

Now, I process the whole scene in memory mode. Afterward, I use tools like DJV to review all the frames. Especially with 60fps VR, you can delete bad angles or mis-swaps without them being noticeable in the final video. Typically, I get about 25 good swaps, 1 "no face detected," then another 25 good swaps. If a face moves out of frame and becomes less than 50% visible, I might get 1 or 2 mis-swaps, but deleting them in a 60fps video is no issue at all. I'm curious about what the new "swap last face" function does.

The last real issue is still the mouth. The quick fix you implemented works well for projects where resemblance isn't crucial, but I was shocked to find out how much a person’s likeness depends on their mouth and chin. Sometimes it works, and the person still looks like themselves, but most of the time, the bottom 25% of the face is just as important as the other 75%. So I'm still struggling with Clip2Seg. I've found that the "open mouth" keyword helps, but on my DML (AMD), the processing time jumps from 3 seconds per frame to 40 seconds per frame when swapping 6 faces. Right now, I'm in the process of extracting the final video from Roop Unleashed into PNG frames, removing the bad angles and mis-swaps, and using Excel to track frame ranges with major obstructions. Then, I selectively re-render those ranges in Roop Unleashed with Clip2Seg, adjusting the face offset (0.25/0.30) or using manual masking to perfect those last frames.

It’s a lot of work—4 days of effort for a 10-minute, 2800p VR180 SBS 3D scene. When I was looking at the frames, I thought it was cool and resembled the people I'd used, but it wasn’t anything special. Then I put on my Quest 2 headset, and the end result was amazing. I've made two big projects—one turned out crazy good, but the other had too much post-processing. I can’t remember if I used GFPGAN or Restore++ on the really good one, so I should’ve kept a log. Lesson learned.

*If you stop the process, the temp file has no audio. I assume it's because audio gets added once the process completes, but I thought it was worth noting.

_** All this Python coding and experimentation made me a bit jealous, so I wanted to create my own fork by adding a ZLUDA option to the execution providers. ZLUDA essentially allows CUDA to run on Windows with AMD, acting as a translation layer for CUDA. It works like a charm for me in Stable Diffusion, giving me 3x the performance of DML. I was thinking about adding ZLUDA into the run command by putting ZLUDA files in the path, but I’m not sure if I should approach this by treating CUDA as the provider and adding some ZLUDA redirect code into Roop Unleashed, or if ZLUDA needs to be imported like DML with a different version of Torch and ONNX.

Maybe it’s something you and I could work on together—I can provide the info and testing, and you can help implement it into Roop Unleashed. A lot of Stable Diffusion forks already have the ZLUDA option built in, so adding it to your app could be quick. You could probably do it in no time, but for me, it would take a month of studying. Still, I’m into it! Let me know if you're interested._

[C0untFloyd]

I assume it's because audio gets added once the process completes, but I thought it was worth noting.

Yes, it's added at the very end.

About ZLUDA:
https://github.com/vosen/ZLUDA?tab=readme-ov-file#warning-important-warning

It seems that it will be rebuilt from scratch. There is already experimental support for Rocm (which also is abandonded?). I'm also not sure if I even need to code something as this sounds like some sort of API replacement.

[Elise96nl]

i have stable-diffusion-forge-amd running with zluda right now. I saw in the commits what is new to support it. Its indeed just a cuda translation tool. 3 small changes, and it should work. but i dont think it would really matter. Your app is running just fine on directml for now.

In the forge cmd we can see some details how they make it work. It's still cuda, in general. but you have to disable some options. cudnn is one of them. and when i tried making my own zluda fork of roop unleashed. i ran into the problem, you can disable cudnn in torch, but not in onnx. So it would be cpu for onnx, and zluda for the rest. I dont really think it would make a difference.

and you got the wrong zluda version

this one is up to date.

https://github.com/lshqqytiger/ZLUDA

On Windows, you don't have to build PyTorch from sources. Instead, just replace dll files of official CUDA release of PyTorch. After that, insert these codes into the application.

torch.backends.cudnn.enabled = False
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_math_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(False)

#If you are getting an error about nvrtc/hiprtc, insert

def jit_script(f, *_, **__):
    f.graph = torch._C.Graph() # pylint: disable=protected-access
    return f
torch.jit.script = jit_script
You may have an issue while running torch.topk. If so, insert

_topk = torch.topk
def topk(tensor: torch.Tensor, *args, **kwargs):
    device = tensor.device
    values, indices = _topk(tensor.cpu(), *args, **kwargs)
    return torch.return_types.topk((values.to(device), indices.to(device),))
torch.topk = topk

[C0untFloyd]

So it would be cpu for onnx, and zluda for the rest.

Yeah, there are very few things left running on torch, from the top of my head only DMDNet and Clip2Seg masking. I tried to convert everything to onnx to be able to have it running in parallel. I don't think it would be noticeable 😞

and you got the wrong zluda version

this one is up to date.

https://github.com/lshqqytiger/ZLUDA

Depends on your POV 😉
The repo I linked is the OG, which unfortunately was closed down by AMD recently -> https://www.tomshardware.com/pc-components/gpus/amd-asks-developer-to-take-down-open-source-zluda-dev-vows-to-rebuild-his-project

It doesn't look like it has a bright future.

[ItsMeTho]

Very nice upgrade!

In addition, I've noticed that sending the result through Davinci Resolve's neural denoise filter (paid version only) can smooth out ANY of the flickering, color changes and even the occasional missed frame. It can look up to 5 frames forward & back, so it caches pretty much everything. At high settings it can even smooth quite a lot of drastic ugly frame swaps, when you get some of those weird looking faces. But it's not Gen AI tech. Using high denoise settings does have a big tradeoff, you can loose too much detail so I always cut the big errors. And then with 30% denoise or so, it gives ultra smooth finish with the details. Add the skin color filter that eliminates any issues with face too red, or stuff like that. It works so well and very easy and fast. Might want to add some grain back into it before final export.

[ItsMeTho]

Davinci Resolve's neural denoise filter

What options? the UltraNR feature? Is there an equivalent in other programs? e.g. neat video plugin?

Until AI there wasn't much. D Resolve, and Neat Image is a very old name but idk much ab it. With AI more applications offer NR. But I do believe right now Resolve's NR is pretty much the best out there, especially considering how fast it is when u have the GPU power.

[ItsMeTho]

https://www.steakunderwater.com/VFXPedia/__man/Resolve18-6/DaVinciResolve18_Manual_files/part3142.htm

Here's the updated DR 19 manual (middelste kolom): https://www.blackmagicdesign.com/support/

[C0untFloyd]

This is interesting and helpful but couldn't you both just create a new discussion under show & tell or general?

[ItsMeTho]

This is interesting and helpful but couldn't you both just create a new discussion under show & tell or general?

Sure np. How's it going @Elise96nl? I'm curious. Let me know if u need any help.

[Elise96nl]

I tried it on a 30 min scene. Took forever so i cancelled. Then i found out about fusion inside resolve studio. found a flicker tutorial from corridor digital. 3 deflickers and dirt removal. took 10 days on my 30 min scene. So still not really a workflow.

Tried it on a 30 sec clip. and it does do the trick a little bit. But most important will be removing the bad frames. So extract to jpg with davinci, and check all frames by hand.

[ItsMeTho]

One thing I notice is that clear temp folder button often doesn't trigger total destruction of temp folder content. At least not immediately. Often it doesn't trigger the notification. Sometimes after restarting server it does, sometimes after clicking the button 10 times it does. Sometimes without getting the 'Temp files removed' notification, I see gigabytes gone when I keep hitting the button (maybe once was enough lol), but then still thousands of 0 bytes items remain in the folder.

[C0untFloyd]

True. That's because if there are any images still being displayed on any UI component the temporary files are still locked and can't be deleted. Unfortunately I can't tell the UI framework to release all those resources. Perhaps another message is needed to show that some files couldn't be deleted.

[Elise96nl]

a i never tried it more then 1 time. I usally just close the app, and after remove the temp file by hand before a new evening of late night rooping :P

[ItsMeTho]

Thanks, yes after restarting the server it's much better. Idk why I never made that link.

[VanHellsyng-23]

Thanks a lot!

[VanHellsyng-23]

Sorry my friend, I tried both updating the program and manually editing the two files you described in the solution, but I keep encountering the same error. It seems like the program won't run if it doesn't detect a graphics card on the PC :(

[C0untFloyd]

Damn you're right - I'm sorry! But thanks to your console dump I could find and fix this really quick.
If you don't want to update again, you can also change this line in config.yaml
provider: cuda
to
provider: cpu

[VanHellsyng-23]

It works! Great! Thank you so much! :)

[DanielTheProgrammer-18]

can anyone please help me with the error ->> conda environment creation failed
i have a NVIDA gpu