[WIP] LLaVA support #720
Merged
[WIP] LLaVA support #720
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mwawrzos
force-pushed
the
mwawrzos/openai-vision
branch
from
e1bfcb4 to
1326edb
Compare
mwawrzos
force-pushed
the
mwawrzos/openai-vision
branch
from
1326edb to
06df643
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
mwawrzos
force-pushed
the
mwawrzos/openai-vision
branch
from
a400418 to
dfb6b1d
Compare
milesial
reviewed
Jul 1, 2024
| @@ -41,6 +87,7 @@ class PromptSource(Enum): | |||
| class OutputFormat(Enum): | |||
| OPENAI_CHAT_COMPLETIONS = auto() | |||
| OPENAI_COMPLETIONS = auto() | |||
| OPENAI_VISION = auto() | |||
There was a problem hiding this comment.
The response format for chat VLMs is the same as the regular chat completion since we just have text out, why have a separate entry?
There was a problem hiding this comment.
The name of the enum is a bit misleading 😅 The OutputFormat enum is actually not about the format of the response but it's about the format of the resulting input json file by LlmInputs.
mwawrzos
force-pushed
the
mwawrzos/openai-vision
branch
from
60b658a to
8bf2710
Compare
src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/dataset_decorators.py
Fixed
Show fixed
Hide fixed
src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/dataset_decorators.py
Fixed
Show fixed
Hide fixed
nv-hwoo
force-pushed
the
mwawrzos/openai-vision
branch
from
bb5511d to
b82bf40
Compare
dyastremsky
reviewed
Jul 10, 2024
|
@dyastremsky yes, the script was originally created by Tim in the beginning of genai-perf when we didn't have CI but we never used it afterwards (it's not even part of our unit test). Since we now have CI in place, I don't think we need this script any more. |
Great job cleaning this up! |
dyastremsky
approved these changes
Jul 10, 2024
nv-hwoo
added a commit
that referenced
this pull request
Jul 11, 2024
* POC for LLaVA support * non-streaming request in VLM tests * image component sent in "image_url" field instead of HTML tag * generate sample image instead of loading from docs * add vision to endpoint mapping * fixes for handling OutputFormat * refactor - extract image preparation to a separate module * fixes to the refactor * replace match-case syntax with if-elseif-else * Update image payload format and fix tests * Few clean ups and tickets added for follow up tasks * Fix and add tests for vision format * Remove output format from profile data parser * Revert irrelevant code change * Revert changes * Remove unused dependency * Comment test_extra_inputs --------- Co-authored-by: Hyunjae Woo <[email protected]>
nv-hwoo
added a commit
that referenced
this pull request
Jul 18, 2024
* POC LLaVA VLM support (#720) * POC for LLaVA support * non-streaming request in VLM tests * image component sent in "image_url" field instead of HTML tag * generate sample image instead of loading from docs * add vision to endpoint mapping * fixes for handling OutputFormat * refactor - extract image preparation to a separate module * fixes to the refactor * replace match-case syntax with if-elseif-else * Update image payload format and fix tests * Few clean ups and tickets added for follow up tasks * Fix and add tests for vision format * Remove output format from profile data parser * Revert irrelevant code change * Revert changes * Remove unused dependency * Comment test_extra_inputs --------- Co-authored-by: Hyunjae Woo <[email protected]> * Support multi-modal input from file for OpenAI Chat Completions (#749) * add synthetic image generator (#751) * synthetic image generator * format randomization * images should be base64-encoded arbitrarly * randomized image format * randomized image shape * prepare SyntheticImageGenerator to support different image sources * read from files * python 3.10 support fixes * remove unused imports * skip sampled image sizes with negative values * formats type fix * remove unused variable * synthetic image generator encodes images to base64 * image format not randomized * sample each dimension independently Co-authored-by: Hyunjae Woo <[email protected]> * apply code-review suggestsions * update class name * deterministic synthetic image generator * add typing to SyntheticImageGenerator * SyntheticImageGenerator doesn't load files * SyntheticImageGenerator always encodes images to base64 * remove unused imports * generate gaussian noise instead of blank images --------- Co-authored-by: Hyunjae Woo <[email protected]> * Add command line arguments for synthetic image generation (#753) * Add CLI options for synthetic image generation * read image format from file when --input-file is used * move encode_image method to utils * Lazy import some modules * Support synthetic image generation in GenAI-Perf (#754) * support synthetic image generation for VLM model * add test * integrate sythetic image generator into LlmInputs * add source images for synthetic image data * use abs to get positive int --------- Co-authored-by: Marek Wawrzos <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The goal of this MR is to enable measuring VLM throughput and latency where input includes images.