Does GPT4ALL use Hardware acceleration with Intel Chips?

[Richard-Woessner]

I don't have a powerful laptop, just a 13th gen i7 with 16gb of ram. I was wondering if GPT4ALL already utilized Hardware Acceleration for Intel chips, and if not how much performace would it add.

[cebtenzzre]

I'm assuming you're talking about Intel DL Boost. That consists of AVX-512 VNNI and AVX-512 BF16. llama.cpp does not use BF16 for quantized models, so the latter is not relevant to GPT4All. The former can be enabled in llama.cpp with the GGML_AVX512_VNNI flag. But we only enable AVX2, F16C, and FMA in our GPT4All releases for best compatibility, since llama.cpp does not currently implement dynamic dispatch depending on CPU features.

Neither of these are likely to help significantly, because LLMs tend to be bottlenecked by memory bandwidth on CPU. Discrete GPUs have very fast memory, which is one of the reasons they are more optimized for LLMs than CPUs. Though you could try building GPT4All with -DGGML_AVX512_VNNI=ON or even -DLLAMA_NATIVE=ON, and see if you notice any improvement in t/s. I do not have a new enough CPU in order to test.

[Richard-Woessner]

Okay, thank you for the response. I will try it anyway just to see what difference it makes, if any, and Ill post them here in case anyone else is curious.

[cosmic-snow]

Ah this is what that was from.

I did enable that in a local build and it barely made a difference here, but YMMV.