⚖️ Suprising performances : gemma2:8b 1.40 times faster than gemma2:2b on ollama 🦙

[adriens]

❔ Context

As mentionned earlier :

• feat: 🎁 🔀 Semantic Router ollama/gemma2 hotline #346

A gave a try to semantic-router and got really impressive results, see 🔀 Semantic Router w. ollama/gemma2 : real life 10ms hotline challenge 🤯 .

... but recently gemma2:2b has been released, then I switched to this model, with the hope that :

• It should be faster
• Be as good

... but surprinsngly it did as good, but slower.

👉 The goal of this issue is to understand why... and what could be done to make semantic router run even faster than 10ms.

⚖️ Data

Considering the following runs :

• Jupyter on gemma2:2b
• Jupyter on gemma2:8b

Below some performances, both with same output quality :

Cell N°	gemma2:2b	gemma2:8b
13	44.2 ms	33 ms
14	16 ms	12.7 ms
15	15.9 ms	12 ms
16	17 ms	12 ms
17	15.9 ms	11.8 ms
18	21.4 ms	11.4 ms
19	16.3 ms	12.6 ms
20	15.6 ms	11.6 ms

ℹ️ On each test, the 8b is faster than the 2b... and it's surprising:

📊 Benchmark conclusion

The average speed-up factor of gemma2:8b compared to gemma2:2b is approximately 1.40. This means that, on average, gemma2:8b is 1.40 times faster than gemma2:2b.

👉 Do you get the same performances ?

[adriens]

According to your feedbacks, I'll push an issue to ollama