about LCM onnx model

[Amin456789]

Hi!

can someone please tell how we can use the LCM model in onnx? i see u guys made an script to run it in onnx, but what about the model? can we simply use the normal stable diffusion script onnx conversation for lcm model too? or we have to wait someone make an conversation script?

or could someone upload onnx converted of LCM model on huggingface and share it with us please?

kind regards

Who can help?

@echarlaix

[echarlaix]

Hi @Amin456789,

To export latent consistency models you can either use the CLI :

optimum-cli export onnx --model SimianLuo/LCM_Dreamshaper_v7 lcm_onnx/

or directly load the model with ORTLatentConsistencyModelPipeline and convert it to ONNX on-the-fly by setting export=True

from optimum.onnxruntime import ORTLatentConsistencyModelPipeline

pipe = ORTLatentConsistencyModelPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", export=True)
prompt = "sailing ship in storm by Leonardo da Vinci"
images = pipe(prompt=prompt, num_inference_steps=4, guidance_scale=8.0).images

Also LCMs support was enabled in #1469 so you'll need to install optimum from source for now

[Amin456789]

thank u so much!

[Amin456789]

@echarlaix another question, does this version of LCM run on CPU too? can this onnx model be fp16 or quantized to int8?

[nbertagnolli]

Thanks for working on the support for latent consistency models! : ). When I run the above code I get the following error:

ValueError: Pipeline <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> expected {'unet', 'safety_checker', 'scheduler', 'text_encoder', 'tokenizer', 'vae', 'feature_extractor'}, but only {'unet', 'safety_checker', 'text_encoder', 'tokenizer', 'vae', 'feature_extractor'} were passed.

It looks like it can't find the scheduler. I've installed everything from source and when I load in the model using vanilla transformers I see that it has an associated scheulder:

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("SimianLuo/LCM_Dreamshaper_v7", custom_pipeline="latent_consistency_txt2img", custom_revision="main")
print(pipe.scheduler)

I get:

LCMScheduler {
  "_class_name": "LCMScheduler",
  "_diffusers_version": "0.22.0.dev0",
  "beta_end": 0.012,
  "beta_schedule": "scaled_linear",
  "beta_start": 0.00085,
  "clip_sample": true,
  "clip_sample_range": 1.0,
  "dynamic_thresholding_ratio": 0.995,
  "num_train_timesteps": 1000,
  "prediction_type": "epsilon",
  "rescale_betas_zero_snr": false,
  "sample_max_value": 1.0,
  "set_alpha_to_one": true,
  "steps_offset": 0,
  "thresholding": false,
  "timestep_spacing": "leading",
  "trained_betas": null
}

Any thoughts on why the ORTLatentConsistencyModelPipeline can't find the scheduler in this case?

[NeusZimmer]

@echarlaix another question, does this version of LCM run on CPU too? can this onnx model be fp16 or quantized to int8?

I've converted it to fp16 with no issues and run ok (also on CPU) will upload to civitai if allowed by the owner of the model.

[echarlaix]

@nbertagnolli, do you still have this issue with diffusers v0.22.0 or higher ? If yes could you open a new issue describing the error you are getting and ping me there?

Great news, thanks for sharing @NeusZimmer !