LORA

[vic-t]

Could someone make a short write up how one could make use of a LORA model in combination with a base model? If it's not too complex.

[NeusZimmer]

Hi, i think we depend on the people of Huggingface to extend the onnx pipelines, I'm monitoring their branches looking just for it, as long as for textual inversions, but still no luck...
:(, I hope they invest some time trying to include it in a future version...

[Amblyopius]

For textual inversion the best solution is to just run text encoder on CPU using torch. Saves you VRAM and as it is one shot, it works fine on CPU. Not really supported in the current flow though. I have experimental hybrid pipelines that do it. I might switch to using those instead but it does imply maintaining a separate set of pipelines for everything (rather than just controlnet and pix2pix)

LORA works if you just make combined models but that's another massive waste of disk space.

[NeusZimmer]

May we get an example of the Text inv. code??? i'm not able to figure out about it...

[Amblyopius]

Just look at the latest diffusers docs:
https://huggingface.co/docs/diffusers/training/text_inversion

My implementation was on an older diffusers version, current one seems even easier. The problem you'll have is that you need to have a hybrid pipeline where text encoder is torch and everything else is ONNX. I have one for text2img but I need to update it to latest diffusers. I'll try to have it added as a demo to the repository. It'll be just code with no UI (similar to how the pix2pix / controlnet demos were added as a reference more than anything else).