Textual inversion support

[vexCoder]

Hi,

i might be throwing some nonsense here but i really have little idea about the code, i just wanted to create a desktop frontend (typescript) using this as a base backend (im still learning python).

anyhow i wanted to know if the script supports textual inversion (embeddings) like in automatic1111? Or is there a api documentation that i can start reading so i can try and implement?

[Amblyopius]

Textual inversion is not supported as currently we've no training related code in place. It is also quite far down the list of things I'd add to the ONNX related code in the repository. That doesn't mean it's impossible as many other models can be trained on ONNX but currently beyond the scope of what's going on here.

If you want to work with training, I would suggest that for the time being you should switch to Linux if your card isn't supported on Windows. With Linux+ROCm you should be able to use AMD cards with automatic1111

[vexCoder]

Textual inversion is not supported as currently we've no training related code in place. It is also quite far down the list of things I'd add to the ONNX related code in the repository. That doesn't mean it's impossible as many other models can be trained on ONNX but currently beyond the scope of what's going on here.

If you want to work with training, I would suggest that for the time being you should switch to Linux if your card isn't supported on Windows. With Linux+ROCm you should be able to use AMD cards with automatic1111

Ah not training i meant, using .pt files. for example from civit.ai

[Amblyopius]

So you solely want the output of textual inversion to use it with the associated model?

That could be on the list at an earlier stage. As a shortcut you could probably just run the Text Encoder in torch on CPU and then run the generation in ONNX. My understanding is that it only impacts text embeddings.

If you want a pointer to what you need to understand conceptually about Stable Diffusion I would suggest you read https://towardsdatascience.com/stable-diffusion-using-hugging-face-501d8dbdd8 as it explains Text Encoder, UNET, VAE and the core related concepts. If you look at fig 15 there's a dark blue box called "CLIP model" that gives the textual embeddings. CLIP would run at an acceptable speed on CPU in torch.

To mix ONNX and torch you need to "hack" the ONNX pipeline. That is already demonstrated in code in the repo as that's how we get the lwp pipeline working.

[vexCoder]

So you solely want the output of textual inversion to use it with the associated model?

That could be on the list at an earlier stage. As a shortcut you could probably just run the Text Encoder in torch on CPU and then run the generation in ONNX. My understanding is that it only impacts text embeddings.

If you want a pointer to what you need to understand conceptually about Stable Diffusion I would suggest you read https://towardsdatascience.com/stable-diffusion-using-hugging-face-501d8dbdd8 as it explains Text Encoder, UNET, VAE and the core related concepts. If you look at fig 15 there's a dark blue box called "CLIP model" that gives the textual embeddings. CLIP would run at an acceptable speed on CPU in torch.

To mix ONNX and torch you need to "hack" the ONNX pipeline. That is already demonstrated in code in the repo as that's how we get the lwp pipeline working.

Thank you mate, your explanation is clear as day even for someone like me who know nothing about it.