New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

evals and benchmarking structure #21

Merged

tstescoTT merged 29 commits into main from tstesco/evals-benchmarking-structure

Nov 5, 2024

Contributor

tstescoTT commented Oct 24, 2024

change log

adding evals instructions for vLLM with lm-evaluation-harness
adding benchmark instructions for vLLM offline_inference_tt.py
adding locust dir placeholder for stress testing
adding docs for repo development setup
adding pre-commit configuration for ruff linting, addressing Add linting / formatting checks on PRs #7

tstescoTT force-pushed the tstesco/evals-benchmarking-structure branch from f9b4fb3 to 446e74a Compare

October 24, 2024 14:37

tstescoTT requested review from milank94 and mvanniasingheTT

October 24, 2024 14:38

tstescoTT added 11 commits

October 24, 2024 14:38


          adding vLLM dockerfile

b687ace


          adding label GHCR and sed import edit for vllm server_example_tt.py

ed34e60


          adding evals instructions and run script

fd0c87d


          move vllm llama 3.1 70b implementation to top level model impl dir

4d0cb6b


          update eval instructions

a793d35


          adding llama 3.1 70b benchmarking instructions

7ed83b1


          adding dir for locust

0e19de6


          add doc link for vllm setup

4f7a328


          adding pre-commit with ruff linting and formatting

b40c536


          add pre-commit instructions

488c19c


          add GHCR repo connection label to Dockerfile

ee888d9

tstescoTT force-pushed the tstesco/evals-benchmarking-structure branch from 446e74a to ee888d9 Compare

October 24, 2024 18:38

milank94 reviewed

View reviewed changes

docs/development.md Show resolved Hide resolved

milank94 reviewed

View reviewed changes

benchmarking/README.md Show resolved Hide resolved

milank94 reviewed

View reviewed changes

evals/README.md Outdated Show resolved Hide resolved

milank94 reviewed

View reviewed changes

evals/README.md Outdated Show resolved Hide resolved

milank94 reviewed

View reviewed changes

vllm-tt-metal-llama3-70b/README.md Show resolved Hide resolved

tstescoTT added 3 commits

October 25, 2024 12:56


          adding requirements-dev.txt

6c35b17


          adding EOF newlines

022cd77


          add ifeval to run_evals.sh, set to stream=False

f17a922

milank94 reviewed

View reviewed changes

requirements-dev.txt Outdated Show resolved Hide resolved


          pin dev dependency versions

443e930

mvanniasingheTT reviewed

View reviewed changes

evals/README.md Outdated Show resolved Hide resolved

mvanniasingheTT reviewed

View reviewed changes

evals/README.md Outdated Show resolved Hide resolved

tstescoTT added 3 commits

October 25, 2024 14:35


          update vllm eval docs and development docs

ed15e1c


          remove fork branch for tstescoTT/lm-evaluation-harness, can now use main

42bd5a6


          remove fork branch for tstescoTT/lm-evaluation-harness, can now use main

f2ac1cc

tstescoTT added 2 commits

October 28, 2024 18:48


          adding TT_VLLM_COMMIT_SHA_OR_TAG to Dockerfile

b7a9406


          uplift tt-metal and vllm commit SHA

77cb4ba

milank94 reviewed

View reviewed changes

evals/README.md Outdated Show resolved Hide resolved

milank94 reviewed

View reviewed changes

evals/run_evals.sh Show resolved Hide resolved

milank94 reviewed

View reviewed changes

vllm-tt-metal-llama3-70b/vllm.llama3.src.base.inference.v0.52.0.Dockerfile Outdated Show resolved Hide resolved

milank94 mentioned this pull request

Adding Mock Model + Test Directory #24

Merged

tstescoTT added 7 commits

October 30, 2024 13:53


          adding TT_VLLM_COMMIT_DOCKER_TAG in vllm dev docker build instructions

7de2b00


          adding comment header for ifeval

f1ad0e1


          fix comment

cbbc80e


          update vllm commit sha

dc27981


          adding additonal tools, removing symlink for tt-metal models

43dff21


          update Dockerfile version in docs

213d7ef


          remove debug tools

fb12460

milank94 reviewed

View reviewed changes

vllm-tt-metal-llama3-70b/README.md Show resolved Hide resolved

milank94 reviewed

View reviewed changes

vllm-tt-metal-llama3-70b/README.md Outdated Show resolved Hide resolved

tstescoTT added 2 commits

November 4, 2024 15:35


          update Docker Run vllm doc

ceb7516


          adding updated commit shas

19ef67a

mvanniasingheTT reviewed

View reviewed changes

evals/README.md Show resolved Hide resolved

mvanniasingheTT reviewed

View reviewed changes

evals/README.md Show resolved Hide resolved

mvanniasingheTT reviewed

View reviewed changes

vllm-tt-metal-llama3-70b/docs/development.md Show resolved Hide resolved

mvanniasingheTT reviewed

View reviewed changes

evals/README.md Show resolved Hide resolved

milank94 approved these changes

View reviewed changes

Contributor

milank94 left a comment

Pending the responses to @mvanniasingheTT's comments, PR looks good to me.

mvanniasingheTT approved these changes

View reviewed changes

tstescoTT merged commit 6af6afb into main

1 check passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet