how to avoid oom when inference qwen2-vl 7B with batch=2?

[YSF-A]

I try to inference qwen2-vl 7B with batch = 2 on 4090 (24G), and I got a oom. How to avoid the oom? In my opinion，compared to 4090(24G), batch = 2 is not too large.

I found that the llm engine is 15G, but after build ModelRunnerCpp，memory increases by almost 20G. Is it correct?

[gujiewen]

Have you set --kv_cache_free_gpu_memory_fraction to be smaller? By default, it's 0.9.

[YSF-A]

Have you set --kv_cache_free_gpu_memory_fraction to be smaller? By default, it's 0.9.

@gujiewen thanks for your reply. I set --kv_cache_free_gpu_memory_fraction to 0.8 and it works.

But the output is empty. I check the output_ids and output_token_ids are full of eos.

[sunnyqgg]

Hi @YSF-A . pls use the latest code and you can run it with INT4/INT8/FP8 referring to examples/qwen. Pls let me know if you have any other questions about it.

Thanks.