How can we use CPU offloading when using AutoModelForCausalLM and THUDM/cogvlm2-llama3-chat-19B

[FurkanGozukara]

It is working great with below however still not sufficient. Uses around 16 GB VRAM

I want to further lower the requirement if possible

How can I achieve that?

model path is : THUDM/cogvlm2-llama3-chat-19B

                    model = AutoModelForCausalLM.from_pretrained(
                        MODEL_PATH,
                        torch_dtype=TORCH_TYPE,
                        trust_remote_code=True,
                        quantization_config=BitsAndBytesConfig(load_in_4bit=True),
                        low_cpu_mem_usage=True
                    ).eval()

Who can help?

text models: @ArthurZucker
vision models: @amyeroberts, @qubvel
pipelines: @Rocketknight1
quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber