CPU vs GPU

[Selbyl]

Is there an option to change this platform to use CUDA or ROCM instead of CPU?

[gamersalpha]

for now no

i waiting to ...

he is tooo slowly with CPU

[Ionaut]

Yeah, CUDA gpu support would be great. That and leaving the model in the vram for the duration of the conversation.

[SurvivaLlama]

So.. Slow for everyone?

[gaby]

This is a llama.cpp issue, not Serge. May be worth creating this issue on that repo.

[lee-b]

llama.cpp is specifically intended by be a CPU-only implementation. They've said before that they won't support GPU. Someone would have to submit a patch so good that they're willing to accept it anyway, or else I suppose this project would have to find a similar, but GPU-based version of it.

[jpaodev]

@lee-b @gaby since llama.cpp does support GPU offloading and solutions such as CLBlast or ROCm now, is this up for debate? Thanks in advance!

[gaby]

@jpaodev In progress #944