Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

refactor(scripts): improve Ollama performance evaluation #139

Merged
merged 3 commits into from
Mar 27, 2024
Merged

Conversation

frgfm
Copy link
Member

@frgfm frgfm commented Mar 27, 2024

This PR introduces the following modifications:

  • add HTTP evaluation to avoid Ollama stat reporting issues
  • renamed script and benchmark result
  • improved script output
pxlksr/opencodeinterpreter-ds:6.7b-Q4_K_M (5 runs) at 2024-03-27 18:35:42.451474+00:00
HTTP: ttft 180.28ms (± 54.44) | ingestion 76.27 tok/s (± 9.39) | generation 38.00 tok/s (± 1.43)
Ollama-reported: load duration 0.25ms (± 0.35) | ingestion 77.58 tok/s (± 9.46) | generation 38.33 tok/s (± 1.49)

@frgfm frgfm added the type: improvement New feature or request label Mar 27, 2024
@frgfm frgfm added this to the 0.1.0 milestone Mar 27, 2024
@frgfm frgfm self-assigned this Mar 27, 2024
@frgfm frgfm changed the title refactor(scripts): improved Ollama performance evaluation refactor(scripts): improve Ollama performance evaluation Mar 27, 2024
@frgfm frgfm merged commit 2cacaef into main Mar 27, 2024
19 of 20 checks passed
@frgfm frgfm deleted the perf-eval branch March 27, 2024 18:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
ext: scripts type: improvement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant