How to Offload Models and Load in Other Ones

[jmsalvador2395]

I have some example code here

from vllm import LLM, SamplingParams

# set model name and prompt
model_name = 'mistralai/Mistral-7B-Instruct-v0.1'
prompt = 'some prompt'

# load model
llm = LLM(model_name,
          download_dir='/data/shared/llm_cache',
          gpu_memory_utilization=.9,
          tensor_parallel_size=4)

# load sampling params
sp = SamplingParams(max_tokens=512)

# generate response
response = llm.generate(
    prompt,
    sampling_params=sp
)

# print response
print(response[0].outputs[0].text)

I want to be able to load in another model and run inference using the same prompt. Is there a way for me to offload the current model safely?

For context, I've tried calling del llm and del llm.llm_engine along with calling torch.cuda.empty_cache() and saw that it causes issues.
When trying to create another llm object I get the message 2023-12-03 03:37:22,346 INFO worker.py:1507 -- Calling ray.init() again after it has already been called. Then after calling ray.shutdown() and trying to define a new llm object, i get the warning that tokenizers parallelism was disabled.

Any help is appreciated.

[roshan-gopalakrishnan]

I am also looking for an answer.

[asimj1342]

    del llm
    gc.collect()
    torch.cuda.empty_cache()
    
    try this

[roshan-gopalakrishnan]

What is gc ? how to import that ?

[lihuahua123]

import gc

What is gc ? how to import that ?

[roshan-gopalakrishnan]

This solution worked for me.

[roshan-gopalakrishnan]

Thanks @lihuahua123, I saw this link #3281 for more information.