Discrepancies in LLM Performance on CPU vs. GPU

[joaos96]

Hello everyone,

I've been working on a script for forensic analysis of messages and I've observed some intriguing discrepancies in the performance of the model when run on CPU versus GPU. Specifically, the model tends to generate more accurate and reliable responses when executed on a GPU rather than a CPU.

Has anyone else experienced similar issues? I'm curious about the technical reasons behind such differences and whether they are related to the specific LLM architecture, data handling, or perhaps the computation of certain operations that are more efficiently handled by GPUs.

[euroblaze]

@joaos96

That is indeed intriguing!
Which LLM did you test?
And which inference server are you using?

Thanks

[alew3]

@joaos96 if you train the model on a specific GPU and run it on a CPU you can get different results because of the way each architecture handles the floating point precision.