Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add MVP LLM support to MA #783

Merged
merged 12 commits into from
Nov 3, 2023
Merged

Add MVP LLM support to MA #783

merged 12 commits into from
Nov 3, 2023

Conversation

nv-braf
Copy link
Contributor

@nv-braf nv-braf commented Nov 1, 2023

This story is to merge my LLM branch to main. Here is what the output of a vLLM run looks like:

17:54:06 [Model Analyzer] Exporting server only metrics to /swdev/foo/results/metrics-server-only.csv
17:54:06 [Model Analyzer] Exporting inference metrics to /swdev/foo/results/metrics-model-inference.csv
Models (Inference):
Model   Batch   Periodic Concurrency   Request Period   Text Input Length   Max Tokens   Model Config Path   Instance Group   Avg First Token latency (ms)   Avg Token-to-Token latency (ms)  
vllm    1       1:1:1                  32               10                  128          vllm_config_0       1:GPU            23.6                           5.2                              
vllm    1       1:2:1                  32               10                  128          vllm_config_0       1:GPU            19.2                           5.3                              
vllm    1       1:4:1                  32               10                  128          vllm_config_0       1:GPU            20.7                           5.8                              
vllm    1       1:8:1                  32               10                  128          vllm_config_0       1:GPU            19.8                           6.0                              
vllm    1       1:16:1                 32               10                  128          vllm_config_0       1:GPU            22.0                           6.0                              
vllm    1       1:32:1                 32               10                  128          vllm_config_0       1:GPU            18.3                           5.5                              
vllm    1       1:64:1                 32               10                  128          vllm_config_0       1:GPU            20.7                           6.2                              
vllm    1       1:128:1                32               10                  128          vllm_config_0       1:GPU            19.1                           5.7                              
vllm    1       2:2:1                  32               10                  128          vllm_config_0       1:GPU            23.7                           5.8                              
vllm    1       2:4:1                  32               10                  128          vllm_config_0       1:GPU            23.6                           5.1                              
vllm    1       2:8:1                  32               10                  128          vllm_config_0       1:GPU            22.3                           5.8                              
vllm    1       2:16:1                 32               10                  128          vllm_config_0       1:GPU            22.7                           6.2                              
vllm    1       2:32:1                 32               10                  128          vllm_config_0       1:GPU            19.8                           5.6

nv-braf and others added 12 commits October 3, 2023 08:05
@nv-braf @mc-nv
Adding new options for LLM (#768)
c9d467f 
* Update README and versions for 23.09 branch (#761) (#767)

* Adding new options for LLM

* Fixing codeQL issues

* Fixing codeQL issue

---------

Co-authored-by: Misha Chornyi <[email protected]>
@nv-braf @nv-hwoo
Add LLM support to Brute Search (#769)
1b7aee0 
* Initial coding complete

* First unit test passing

* Adding test for prompt length

* Refactor PACG methods

* Further refactoring

* Ensure early exit isn't enabled for LLM models

* Fix type checking errors

* Attempt at fixing codeql issue

* Revert "Attempt at fixing codeql issue"

This reverts commit 2619b83.

* Attempt at codeQL fix

* Adding deepcopy back in

* Removing deepcopy in an attempt to fix codeQL errors

* Update model_analyzer/config/input/config_command_profile.py

Co-authored-by: Hyunjae Woo <[email protected]>

* Update model_analyzer/config/generate/perf_analyzer_config_generator.py

Co-authored-by: Hyunjae Woo <[email protected]>

* Update model_analyzer/config/generate/perf_analyzer_config_generator.py

Co-authored-by: Hyunjae Woo <[email protected]>

* Update model_analyzer/config/generate/perf_analyzer_config_generator.py

Co-authored-by: Hyunjae Woo <[email protected]>

* Moving location of method

* Changing parameter to inference load

* Changing parameter to inference load

* Changing prompt length to text input length

* Changing max_tokens to use request-parameter

* Fix input-data typo

* Changing non-parameter to parameter

---------

Co-authored-by: Hyunjae Woo <[email protected]>
@nv-braf
New LLM record types (#770)
14ea528 
* New measurement fields created.

* Fixing omission in llm_metric_table

* Changing name to be avg_token_to_token...
@nv-braf
New config options based on live run (#775)
6e0fc24 
* Added new config options and modified existing options

* Refactoring model parameter setting

* Removing magic numbers
@nv-braf
Capture LLM metrics from PA (#774)
e81a369 
* Initial code for aggregation of new LLM metrics

* New measurement fields created.

* Fixing PA unit tests

* Adding hooks in metrics to capture new LLM fields

* Fixing codeQL errors

* Fixing type checking errors

* Changes needed post-merge from other branches

* Revert naming mistake (due to merge).

* Changes uncovered during live testing

* Fixes based on hwoo review

* Fixing typo

* Change to use lists and mean()

* Changes based on hwoo review
@nv-braf
Correct how periodic concurrency works in PACG (#777)
efea104 
* Created a new class ConfigRangeNumeric and using it for periodic-concurrency

* Fixes and defaults for periodic concurrency

* First unit test passing

* PACG chagnes complete. Unit tests updated and passing

* Removing uneeded class

* Fixing codeQL and hwoo's review suggestions

* Adding missing else
@nv-braf
Llm testing live run (#778)
b3aba30 
* Created a new class ConfigRangeNumeric and using it for periodic-concurrency

* Fixes and defaults for periodic concurrency

* First unit test passing

* PACG chagnes complete. Unit tests updated and passing

* Removing uneeded class

* Changes to fix live run

* Minor refactor and cleanup

* Removing json files

* Changing to use f-string

* More cleanup from hwoo CR

* Removing stale code for request period

* Fix nit
@nv-braf
Changes to get LLM summary reports working (#779)
f508625 
* Changes to get LLM summary reports working

* Addressing hwoo's CR
@nv-braf
Adding illegal LLM checks w/ unit testing + some minor cleanup (#781)
8854589 
* Adding illegal LLM checks w/ unit testing + some minor cleanup

* Updated with TMA
@nv-braf
Misc LLM cleanup (#782)
d9e075b 
* General cleanup

* Add ticket nums to todos
@nv-braf
Fix for non-LLM breaking bug introduced.
f229273
@nv-braf
summary table in progress
709531b
debermudez
debermudez approved these changes Nov 1, 2023
View reviewed changes
Copy link
Contributor

@debermudez debermudez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving under the assumption that all of the previous work was reviewed in a side branch.

@nv-braf nv-braf merged commit f15427e into main Nov 3, 2023
3 checks passed
nv-braf added a commit that referenced this pull request Jan 23, 2024
@nv-braf
Revert "Add MVP LLM support to MA (#783)"
accabf7 
This reverts commit f15427e.
nv-braf added a commit that referenced this pull request Jan 23, 2024
@nv-braf
Revert LLM changes (#818)
6019852 
* Revert "Add MVP LLM support to MA (#783)"

This reverts commit f15427e.

* Fixing merge conficts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@debermudez debermudez debermudez approved these changes

@tgerdesnv tgerdesnv Awaiting requested review from tgerdesnv

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nv-braf @debermudez