java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:200)

[laobuda]

Trying to run jlama with Mistral model fails with the following error:

jlama restapi tjake/Mistral-7B-Instruct-v0.3-JQ4 --auto-download

WARNING: Using incubator modules: jdk.incubator.vector
README.md 100% [=========================================================================================] 8/8KB (0:00:01 / 0:00:00)
config.json 100% [====================================================================================] 601/601B (0:00:01 / 0:00:00)
model.safetensors 100% [===========================================================================] 4530/4530MB (0:09:56 / 0:00:00)
tokenizer.json 100% [====================================================================================] 1/1MB (0:00:01 / 0:00:00)
tokenizer_config.json 100% [=========================================================================] 140/140KB (0:00:01 / 0:00:00)
Unknown vector type: 128
Using Native SIMD Operations (OffHeap)
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:200)
at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:129)
at com.github.tjake.jlama.cli.commands.ApiServiceCommand.run(ApiServiceCommand.java:73)
at picocli.CommandLine.executeUserObject(CommandLine.java:2026)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2264)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:2664)
at picocli.CommandLine.parseWithHandler(CommandLine.java:2599)
at com.github.tjake.jlama.cli.JlamaCli.main(JlamaCli.java:58)
at jlama.main(jlama.java:22)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:74)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486)
at com.github.tjake.jlama.model.ModelSupport.loadModel(ModelSupport.java:197)
... 12 more
Caused by: java.lang.UnsupportedOperationException
at com.github.tjake.jlama.tensor.operations.PanamaTensorOperations.quantize(PanamaTensorOperations.java:1605)
at com.github.tjake.jlama.tensor.operations.NativeTensorOperations.quantize(NativeTensorOperations.java:434)
at com.github.tjake.jlama.model.AbstractModel.(AbstractModel.java:146)
at com.github.tjake.jlama.model.llama.LlamaModel.(LlamaModel.java:58)
at com.github.tjake.jlama.model.mistral.MistralModel.(MistralModel.java:48)
at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
... 15 more

[tjake]

This is caused by lack of support for 128bit SIMD. Are you on a M-Servies mac? That seems to be the one machine that hits this if they are using the wrong JVM: see #68

[laobuda]

This is caused by lack of support for 128bit SIMD. Are you on a M-Servies mac? That seems to be the one machine that hits this if they are using the wrong JVM: see #68

I am running on a Linux distro. "Xubuntu" with an intel CPU, and I got the same error running the command in #68

[tjake]

You must have a very old intel cpu? Can you paste the output for lscpu ?

[laobuda]

Indeed it is an old pc from 2010 I am attaching the result of the lscpu command
Thanks in advance
linux_hw.txt