Use Facebook LLaMa model with pyllamacpp?

[VoxanyNet]

Hello!

I want to use the Facebook LLaMA model with pyllamacpp and am looking for some help.

I followed the instructions on the llama.cpp README to convert the model to the ggml format and quantize it. I am using the 30B model.

Then, in Python, I wrote the following code:

model = Model(ggml_model="ggml-model-q4_0.bin", n_ctx=512)

generated_text = model.generate("Once upon a time ", n_predict=50)

However, upon running the code I am given the following output.

llama_model_load: loading model from 'ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx   = 512
llama_model_load: n_embd  = 6656
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 52
llama_model_load: n_layer = 60
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 17920
llama_model_load: n_parts = 4
llama_model_load: type    = 3
llama_model_load: ggml map size = 19391.80 MB
llama_model_load: ggml ctx size = 151.25 KB
llama_model_load: mem required  = 21695.95 MB (+ 6248.00 MB per state)
llama_model_load: loading tensors from 'ggml-model-q4_0.bin'
Traceback (most recent call last):
  File "/mnt/externaldrive/code/personal/python/copilot/__main__.py", line 8, in <module>
    model = Model(ggml_model="ggml-model-q4_0.bin", n_ctx=512)
  File "/home/kruz/.local/lib/python3.10/site-packages/pyllamacpp/model.py", line 60, in __init__
    self._ctx = pp.llama_init_from_file(ggml_model, self.llama_params)
ValueError: basic_string::_S_create

I'm not sure if support for LLaMa was ever a plan, but any help would be appreciated!

[abdeladim-s]

Hi @VoxanyNet,

If it the model is converted correctly, it should be working I guess.

Could you please try to run the model with llama.cpp first and check if it working ?

[VoxanyNet]

I'm sorry I just found the problem. I was using the 30B model which requires over 20GB of memory. I have 32GB on my system but I believe there wasn't enough left over. Using the 13B model works for me.

Thanks for the response!

[afxlab1]

Hi, for some reason it crashes am i doing something wrong?
model = Model(ggml_model = 'C:/Program Files/gpt4all 0.1.0/bin/ggml-gpt4all-j.bin', n_ctx=512)
response = model.generate("Once upon a time, ", n_predict=55)

[VoxanyNet]

Could you give us the error it gives you?