Synthetic image generator #751

mwawrzos · 2024-07-11T21:09:37Z

The PR allows users to add multimodal data to the synthetic prompts.

Generated images:

are filled with uniform noise,
will have randomized shape (controlled with mean_size and dimiensions_stddev parameters),
user can choose image format from PNG or JPEG for base64 encoding.

src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/synthetic_image_generator.py

@@ -0,0 +1,87 @@
+import base64
+import os


src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

@@ -0,0 +1,95 @@
+import base64
+from io import BytesIO, StringIO


src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

+    )
+
+    # exception is raised, when PIL.Image.resize is called with negative values
+    image = next(sut)


src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/synthetic_image_generator.py

src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

nv-hwoo · 2024-07-12T17:22:16Z

src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

+@patch("pathlib.Path.exists", return_value=True)
+@patch(
+    "PIL.Image.open",
+    return_value=DUMMY_IMAGE,


Suggested change

return_value=DUMMY_IMAGE,

return_value=Image.new("RGB", (100, 100), color="blue"),

I'm testing against the DUMMY_IMAGE in an assertion below, so I prefer to keep it named, but I moved the variable definition from the global to the local scope.

src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/synthetic_image_generator.py

+from enum import Enum, auto
+from io import BytesIO
+from pathlib import Path
+from typing import List, Optional, Tuple, cast


Co-authored-by: Hyunjae Woo <[email protected]>

src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/synthetic_image_generator.py

+import base64
+from enum import Enum, auto
+from io import BytesIO
+from pathlib import Path


src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/synthetic_image_generator.py

+from enum import Enum, auto
+from io import BytesIO
+from pathlib import Path
+from typing import Generator, Optional, Tuple, cast


src/c++/perf_analyzer/genai-perf/genai_perf/llm_inputs/synthetic_image_generator.py

+from typing import Generator, Optional, Tuple, cast
+
+import numpy as np
+from genai_perf.exceptions import GenAIPerfException


src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

@@ -0,0 +1,87 @@
+import base64
+from io import BytesIO
+from pathlib import Path


src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

+import base64
+from io import BytesIO
+from pathlib import Path
+from unittest.mock import patch


src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py

+
+import numpy as np
+import pytest
+from genai_perf.exceptions import GenAIPerfException


* POC LLaVA VLM support (#720) * POC for LLaVA support * non-streaming request in VLM tests * image component sent in "image_url" field instead of HTML tag * generate sample image instead of loading from docs * add vision to endpoint mapping * fixes for handling OutputFormat * refactor - extract image preparation to a separate module * fixes to the refactor * replace match-case syntax with if-elseif-else * Update image payload format and fix tests * Few clean ups and tickets added for follow up tasks * Fix and add tests for vision format * Remove output format from profile data parser * Revert irrelevant code change * Revert changes * Remove unused dependency * Comment test_extra_inputs --------- Co-authored-by: Hyunjae Woo <[email protected]> * Support multi-modal input from file for OpenAI Chat Completions (#749) * add synthetic image generator (#751) * synthetic image generator * format randomization * images should be base64-encoded arbitrarly * randomized image format * randomized image shape * prepare SyntheticImageGenerator to support different image sources * read from files * python 3.10 support fixes * remove unused imports * skip sampled image sizes with negative values * formats type fix * remove unused variable * synthetic image generator encodes images to base64 * image format not randomized * sample each dimension independently Co-authored-by: Hyunjae Woo <[email protected]> * apply code-review suggestsions * update class name * deterministic synthetic image generator * add typing to SyntheticImageGenerator * SyntheticImageGenerator doesn't load files * SyntheticImageGenerator always encodes images to base64 * remove unused imports * generate gaussian noise instead of blank images --------- Co-authored-by: Hyunjae Woo <[email protected]> * Add command line arguments for synthetic image generation (#753) * Add CLI options for synthetic image generation * read image format from file when --input-file is used * move encode_image method to utils * Lazy import some modules * Support synthetic image generation in GenAI-Perf (#754) * support synthetic image generation for VLM model * add test * integrate sythetic image generator into LlmInputs * add source images for synthetic image data * use abs to get positive int --------- Co-authored-by: Marek Wawrzos <[email protected]>

mwawrzos added 7 commits July 11, 2024 23:01

synthetic image generator

a8656f8

format randomization

a5b6dbc

images should be base64-encoded arbitrarly

9630704

randomized image format

7915dd7

randomized image shape

6176dc4

prepare SyntheticImageGenerator to support different image sources

9f8a426

read from files

5178b27

github-advanced-security bot found potential problems Jul 11, 2024

View reviewed changes

mwawrzos added 4 commits July 12, 2024 00:09

python 3.10 support fixes

09e81a8

remove unused imports

7f5d573

skip sampled image sizes with negative values

c4e7c35

formats type fix

5673a51

github-advanced-security bot found potential problems Jul 12, 2024

View reviewed changes

src/c++/perf_analyzer/genai-perf/tests/test_synthetic_image_generator.py Outdated

)

# exception is raised, when PIL.Image.resize is called with negative values

image = next(sut)

Check notice

Code scanning / CodeQL

Unused local variable Note test

Variable image is not used.

mwawrzos added 2 commits July 12, 2024 15:31

remove unused variable

d64cd27

synthetic image generator encodes images to base64

6d5b4ea

nv-hwoo reviewed Jul 12, 2024

View reviewed changes

nv-hwoo mentioned this pull request Jul 12, 2024

Add command line arguments for synthetic image generation #753

Merged

image format not randomized

edad485

github-advanced-security bot found potential problems Jul 15, 2024

View reviewed changes

mwawrzos and others added 7 commits July 15, 2024 12:59

sample each dimension independently

935da0b

Co-authored-by: Hyunjae Woo <[email protected]>

apply code-review suggestsions

d8712eb

update class name

b5d4b64

deterministic synthetic image generator

fb6e982

add typing to SyntheticImageGenerator

287edba

SyntheticImageGenerator doesn't load files

af0a93a

SyntheticImageGenerator always encodes images to base64

e0b43fd

github-advanced-security bot found potential problems Jul 15, 2024

View reviewed changes

mwawrzos added 2 commits July 15, 2024 18:46

remove unused imports

1ef4f71

generate gaussian noise instead of blank images

ae66dc3

nv-hwoo approved these changes Jul 15, 2024

View reviewed changes

nv-hwoo merged commit 92b2f3d into vision-language Jul 15, 2024
5 checks passed

nv-hwoo deleted the synthetic-image-generator branch July 15, 2024 21:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Synthetic image generator #751

Synthetic image generator #751

mwawrzos commented Jul 11, 2024 •

edited

Loading

nv-hwoo Jul 12, 2024

mwawrzos Jul 15, 2024

		@@ -0,0 +1,95 @@
		import base64
		from io import BytesIO, StringIO

	return_value=DUMMY_IMAGE,
	return_value=Image.new("RGB", (100, 100), color="blue"),

Synthetic image generator #751

Synthetic image generator #751

Conversation

mwawrzos commented Jul 11, 2024 • edited Loading

nv-hwoo Jul 12, 2024

Choose a reason for hiding this comment

mwawrzos Jul 15, 2024

Choose a reason for hiding this comment

mwawrzos commented Jul 11, 2024 •

edited

Loading