IP-Adapter

[monstruosoft]

I'm currently taking a look at IP-Adapter; I don't know if I'm using it wrong but I don't get it! It's true that it follows somewhat closely the face of the control image you use but it's otherwise terrible at following text prompts. Also, I always get the exact same face I use as input, only either flipped horizontally or slightly tilted, usually badly blended in, but I think that's not different to what you could get by pasting the face and using plain img2img.

In any case, trying to add IP-Adapter support to FastSD CPU would require upgrading the diffusers library requirement to version 0.24 or 0.25, which might require some work; to begin with, it seems like FastSD CPU can't even run on version 0.25.

[rupeshs]

Yes, we need to upgrade diffusers and may be pytorch(2.2), I'm planning that but we need to ensure that It will work on windows/linux/mac/android.
Probably I can update those packages. This is not a high priority as of now.
I got better quality output for IP adapter

[rupeshs]

[monstruosoft]

Nevermind, I was using IP-Adapter with the default weight of 1.0, which is too high. Reducing the adapter weight improves the result when following text prompts.

IP-Adapter face models seem to work better with realistic models, though reducing the adapter weight also reduces how closely it will follow the facial features. Maybe the IP-Adapter FaceID model gives better results? Gonna try it.

This is not a high priority as of now.

So, I guess IP-Adapter support in FastSD CPU will have to wait, then.

[rupeshs]

Base model is important here I used dreamshaper8(I didnt use faceid for the above image).

[monstruosoft]

Yeah, dreamshaper-8, low adapter weight, does better at following text prompt.

[monstruosoft]

I still don't quite get IP-Adapter; sometimes it generates images that follow closely the input image but sometimes it generates completely unrelated images. I guess it's more meant to be used in inpainting or with image masks in multi-adapter mode, where it seems to have the most potential but unfortunately multi-adapter has probably too high requirements for me to even try it.

[rupeshs]

@monstruosoft Try this code

from diffusers import DiffusionPipeline, LCMScheduler
import torch
from diffusers.utils import load_image
from PIL import Image


model_id = "Lykon/dreamshaper-8"
lcm_lora_id = "latent-consistency/lcm-lora-sdv1-5"

pipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float32)

pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="models",
    weight_name="ip-adapter_sd15.bin",
)
pipe.load_lora_weights(lcm_lora_id)
pipe.fuse_lora()
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.set_ip_adapter_scale(0.8)
# pipe.enable_model_cpu_offload()

prompt = "a handsome man,realistic"
# image = load_image(
#     "https://user-images.githubusercontent.com/24734142/266492875-2d50d223-8475-44f0-a7c6-08b51cb53572.png"
# )
image = Image.open(r"C:\Users\Rupesh\Downloads\Popeye-Flex-e1637717888907.webp")
images = pipe(
    prompt=prompt,
    ip_adapter_image=image,
    num_inference_steps=4,
    guidance_scale=1,
).images[0]
images.save("ipdemo.png")

[monstruosoft]

Thanks, will try it.

OK, that code is not much different than the code I was using; I mean, there's not many ways you can write the code to try IP-Adapter, it's just that with ControlNet you have a clear idea of what to expect in the output image, with IP-Adapter not so much.

[monstruosoft]

Anyway, I've been making some tests and I think it might be possible to add IP-Adapter support by using the code straight from the IP-Adapter github repo instead of using the official diffusers implementation, so no need to upgrade the diffusers requirement.