gpt4all not working in simple python container on low spec laptop

[rickspencer3]

I am seeing if I can get gpt4all with python working in a container on a very low spec laptop.

I'm trying to get started with the simplest possible configuration, but I'm pulling my hair out not understanding why I can't get past downloading the model. I can build and run the container just fine. I can log into it and run the python file just fine. The model downloads just fine. But then it always results in the following:

100%|████████████████████████████████████████████████████████████████████████████████████| 45.9M/45.9M [00:13<00:00, 3.34MiB/s]
LLModel ERROR: CPU does not support AVX
Traceback (most recent call last):
  File "//main.py", line 3, in <module>
    model = GPT4All(model_name="all-MiniLM-L6-v2-f16.gguf")
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gpt4all/gpt4all.py", line 102, in __init__
    self.model = _pyllmodel.LLModel(self.config["path"], n_ctx, ngl)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/gpt4all/_pyllmodel.py", line 178, in __init__
    raise ValueError(f"Unable to instantiate model: {'null' if s is None else s.decode()}")
ValueError: Unable to instantiate model: Model format not supported (no matching implementation found)

I've tried setting the device parameter. I assume "CPU does not support AVX" is the clue, but I can't find anything on line that helps me understand what it means and how to fix whatever is the problem.

requirements.txt

gpt4all

dockerfile

FROM python

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY main.py .

main.py

from gpt4all import GPT4All

model = GPT4All(model_name="all-MiniLM-L6-v2-f16.gguf")
output =model.generate("The Python progrmming language is ", max_tokens=300)
print(output)

[rickspencer3]

I ran the whole thing on a different, and much more powerful computer, and it seems to be working:

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45.9M/45.9M [00:08<00:00, 5.42MiB/s]
bert_load_from_file: gguf version     = 2
bert_load_from_file: gguf alignment   = 32
bert_load_from_file: gguf data offset = 695552
bert_load_from_file: model name           = BERT
bert_load_from_file: model architecture   = bert
bert_load_from_file: model file type      = 1
bert_load_from_file: bert tokenizer vocab = 30522
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

When using orca-mini-3b-gguf2-q4_0.gguf and removing the max_tokens argument, I get sensible results:

>>> from gpt4all import GPT4All
>>> model = GPT4All(model_name="orca-mini-3b-gguf2-q4_0.gguf")
>>> output = model.generate("hello")
print(outp>>> print(output)
@hello.com
Hi, I'm interested in buying a property in London and would like to know more about the area. Can you provide me with some information on the local community and the local amenities?
>>> output = model.generate("Python")
>>> print(output)
 3.6.2 is released!
Python 3.6.2 has been released today, October 17th, 2018. This release includes several bug fixes and performance improvements, as well as some new features.
>>> output = model.generate("what is Python")
>>> print(output)
's built-in type system?
Python has a dynamic typing system, which means that the type of an object can change during runtime. This allows for greater flexibility and adaptability in programming. In Python, you do not need to specify the type of a variable at declaration time like in other languages such as C++ or Java. Instead, Python uses a "dictionary" to represent data types, which maps each identifier to its corresponding type. For example, the string "hello" is represented as a dictionary with the key "h" and value "str". This allows for easy conversion between different types of data, such as converting an integer to a string or vice versa.

[rickspencer3]

Ok, nm, now I understand :

I found the system requirements here: https://docs.gpt4all.io/gpt4all_faq.html#does-that-mean-gpt4all-is-compatible-with-all-llamacpp-models-and-vice-versa

Your CPU needs to support [AVX or AVX2 instructions](https://en.wikipedia.org/wiki/Advanced_Vector_Extensions) and you need enough RAM to load a model into memory.