Create-Workload Improvements: Separate operations and test procedures from workload.json #446

IanHoang · 2024-01-23T23:29:20Z

Description

Updated way create-workload feature writes operations and test procedures. This will make custom workloads look similar to official workloads in OSB workloads repository.

This is an example of what is outputted by create-workload feature now. Notice how test procedures and operations are separate from workload.json now.

hoangia@3c22fbd0d988 movies-with-custom-queries % ls -lrt
total 640
-rw-r--r--  1 hoangia  staff    1556 Jan 23 17:15 movies-2000.json
-rw-r--r--  1 hoangia  staff    8469 Jan 23 17:15 movies-2000-documents-1k.json.bz2
-rw-r--r--  1 hoangia  staff  144148 Jan 23 17:15 movies-2000-documents-1k.json
-rw-r--r--  1 hoangia  staff    8434 Jan 23 17:15 movies-2000-documents.json.bz2
-rw-r--r--  1 hoangia  staff  144148 Jan 23 17:15 movies-2000-documents.json
-rw-r--r--  1 hoangia  staff     734 Jan 23 17:15 workload.json
drwxr-xr-x  3 hoangia  staff      96 Jan 23 17:15 operations
drwxr-xr-x  3 hoangia  staff      96 Jan 23 17:15 test_procedures
-rw-r--r--  1 hoangia  staff       0 Jan 23 17:15 movies-2000-documents-1k.json.offset

Issues Resolved

#376

Testing

New functionality includes testing
Created workload with default settings
Ran workload with default settings
Created workload with custom queries
Ran workload with custom queries

# Workload with Default Settings
(.venv) hoangia@3c22fbd0d988 opensearch-benchmark % opensearch-benchmark execute-test --workload-path=/Users/hoangia/Desktop/Development/create-workload/movies-new --pipeline=benchmark-only --target-host=<cluster endpoint> --client-options="basic_auth_user:'<username>',basic_auth_password:'<password>'" --test-mode

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Preparing file offset table for [/Users/hoangia/Desktop/Development/create-workload/movies-new/movies-2000-documents-1k.json] ... [OK]
[INFO] Executing test with workload [movies-new], test_procedure [default-test-procedure] and provision_config_instance ['external'] with version [2.5.0].

[WARNING] merges_total_time is 2548 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 157751 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 15525 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 117 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running cluster-health                                                         [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running default                                                                [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |         Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |              |     2.62953 |    min |
|             Min cumulative indexing time across primary shards |              |           0 |    min |
|          Median cumulative indexing time across primary shards |              | 0.000891667 |    min |
|             Max cumulative indexing time across primary shards |              |      0.5247 |    min |
|            Cumulative indexing throttle time of primary shards |              |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |              |           0 |    min |
| Median cumulative indexing throttle time across primary shards |              |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |              |           0 |    min |
|                        Cumulative merge time of primary shards |              |   0.0424667 |    min |
|                       Cumulative merge count of primary shards |              |          12 |        |
|                Min cumulative merge time across primary shards |              |           0 |    min |
|             Median cumulative merge time across primary shards |              |           0 |    min |
|                Max cumulative merge time across primary shards |              |   0.0108167 |    min |
|               Cumulative merge throttle time of primary shards |              |           0 |    min |
|       Min cumulative merge throttle time across primary shards |              |           0 |    min |
|    Median cumulative merge throttle time across primary shards |              |           0 |    min |
|       Max cumulative merge throttle time across primary shards |              |           0 |    min |
|                      Cumulative refresh time of primary shards |              |    0.258883 |    min |
|                     Cumulative refresh count of primary shards |              |         906 |        |
|              Min cumulative refresh time across primary shards |              |           0 |    min |
|           Median cumulative refresh time across primary shards |              |    0.000175 |    min |
|              Max cumulative refresh time across primary shards |              |   0.0418833 |    min |
|                        Cumulative flush time of primary shards |              |     0.00195 |    min |
|                       Cumulative flush count of primary shards |              |          28 |        |
|                Min cumulative flush time across primary shards |              |           0 |    min |
|             Median cumulative flush time across primary shards |              |           0 |    min |
|                Max cumulative flush time across primary shards |              |     0.00195 |    min |
|                                        Total Young Gen GC time |              |           0 |      s |
|                                       Total Young Gen GC count |              |           0 |        |
|                                          Total Old Gen GC time |              |           0 |      s |
|                                         Total Old Gen GC count |              |           0 |        |
|                                                     Store size |              |    0.149438 |     GB |
|                                                  Translog size |              | 0.000391557 |     GB |
|                                         Heap used for segments |              |           0 |     MB |
|                                       Heap used for doc values |              |           0 |     MB |
|                                            Heap used for terms |              |           0 |     MB |
|                                            Heap used for norms |              |           0 |     MB |
|                                           Heap used for points |              |           0 |     MB |
|                                    Heap used for stored fields |              |           0 |     MB |
|                                                  Segment count |              |          77 |        |
|                                                 Min Throughput | index-append |     2507.83 | docs/s |
|                                                Mean Throughput | index-append |     2507.83 | docs/s |
|                                              Median Throughput | index-append |     2507.83 | docs/s |
|                                                 Max Throughput | index-append |     2507.83 | docs/s |
|                                        50th percentile latency | index-append |     398.777 |     ms |
|                                       100th percentile latency | index-append |     423.004 |     ms |
|                                   50th percentile service time | index-append |     398.777 |     ms |
|                                  100th percentile service time | index-append |     423.004 |     ms |
|                                                     error rate | index-append |           0 |      % |
|                                                 Min Throughput |      default |        3.42 |  ops/s |
|                                                Mean Throughput |      default |        3.42 |  ops/s |
|                                              Median Throughput |      default |        3.42 |  ops/s |
|                                                 Max Throughput |      default |        3.42 |  ops/s |
|                                       100th percentile latency |      default |     543.494 |     ms |
|                                  100th percentile service time |      default |     251.091 |     ms |
|                                                     error rate |      default |           0 |      % |


--------------------------------
[INFO] SUCCESS (took 19 seconds)
--------------------------------

# Workload with Custom Queries
(.venv) hoangia@3c22fbd0d988 opensearch-benchmark % opensearch-benchmark execute-test --workload-path=/Users/hoangia/Desktop/Development/create-workload/movies-with-custom-queries --pipeline=benchmark-only --target-host=<cluster endpoint> --client-options="basic_auth_user:'<username>',basic_auth_password:'<password>'" --test-mode

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[INFO] Executing test with workload [movies-with-custom-queries], test_procedure [custom-test-procedures] and provision_config_instance ['external'] with version [2.5.0].

[WARNING] merges_total_time is 2548 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] indexing_total_time is 157773 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 15536 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 192 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
Running delete-index                                                           [100% done]
Running create-index                                                           [100% done]
Running cluster-health                                                         [100% done]
Running index-append                                                           [100% done]
Running refresh-after-index                                                    [100% done]
Running default                                                                [100% done]
Running term                                                                   [100% done]

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                                                         Metric |         Task |       Value |   Unit |
|---------------------------------------------------------------:|-------------:|------------:|-------:|
|                     Cumulative indexing time of primary shards |              |     2.62918 |    min |
|             Min cumulative indexing time across primary shards |              |           0 |    min |
|          Median cumulative indexing time across primary shards |              | 0.000891667 |    min |
|             Max cumulative indexing time across primary shards |              |      0.5247 |    min |
|            Cumulative indexing throttle time of primary shards |              |           0 |    min |
|    Min cumulative indexing throttle time across primary shards |              |           0 |    min |
| Median cumulative indexing throttle time across primary shards |              |           0 |    min |
|    Max cumulative indexing throttle time across primary shards |              |           0 |    min |
|                        Cumulative merge time of primary shards |              |   0.0424667 |    min |
|                       Cumulative merge count of primary shards |              |          12 |        |
|                Min cumulative merge time across primary shards |              |           0 |    min |
|             Median cumulative merge time across primary shards |              |           0 |    min |
|                Max cumulative merge time across primary shards |              |   0.0108167 |    min |
|               Cumulative merge throttle time of primary shards |              |           0 |    min |
|       Min cumulative merge throttle time across primary shards |              |           0 |    min |
|    Median cumulative merge throttle time across primary shards |              |           0 |    min |
|       Max cumulative merge throttle time across primary shards |              |           0 |    min |
|                      Cumulative refresh time of primary shards |              |     0.25875 |    min |
|                     Cumulative refresh count of primary shards |              |         885 |        |
|              Min cumulative refresh time across primary shards |              |           0 |    min |
|           Median cumulative refresh time across primary shards |              |    0.000175 |    min |
|              Max cumulative refresh time across primary shards |              |   0.0418833 |    min |
|                        Cumulative flush time of primary shards |              |     0.00195 |    min |
|                       Cumulative flush count of primary shards |              |          28 |        |
|                Min cumulative flush time across primary shards |              |           0 |    min |
|             Median cumulative flush time across primary shards |              |           0 |    min |
|                Max cumulative flush time across primary shards |              |     0.00195 |    min |
|                                        Total Young Gen GC time |              |           0 |      s |
|                                       Total Young Gen GC count |              |           0 |        |
|                                          Total Old Gen GC time |              |           0 |      s |
|                                         Total Old Gen GC count |              |           0 |        |
|                                                     Store size |              |    0.149785 |     GB |
|                                                  Translog size |              | 0.000391557 |     GB |
|                                         Heap used for segments |              |           0 |     MB |
|                                       Heap used for doc values |              |           0 |     MB |
|                                            Heap used for terms |              |           0 |     MB |
|                                            Heap used for norms |              |           0 |     MB |
|                                           Heap used for points |              |           0 |     MB |
|                                    Heap used for stored fields |              |           0 |     MB |
|                                                  Segment count |              |          75 |        |
|                                                 Min Throughput | index-append |     2467.89 | docs/s |
|                                                Mean Throughput | index-append |     2467.89 | docs/s |
|                                              Median Throughput | index-append |     2467.89 | docs/s |
|                                                 Max Throughput | index-append |     2467.89 | docs/s |
|                                        50th percentile latency | index-append |      391.09 |     ms |
|                                       100th percentile latency | index-append |     428.779 |     ms |
|                                   50th percentile service time | index-append |      391.09 |     ms |
|                                  100th percentile service time | index-append |     428.779 |     ms |
|                                                     error rate | index-append |           0 |      % |
|                                                 Min Throughput |      default |        3.62 |  ops/s |
|                                                Mean Throughput |      default |        3.62 |  ops/s |
|                                              Median Throughput |      default |        3.62 |  ops/s |
|                                                 Max Throughput |      default |        3.62 |  ops/s |
|                                       100th percentile latency |      default |     525.854 |     ms |
|                                  100th percentile service time |      default |     249.288 |     ms |
|                                                     error rate |      default |           0 |      % |
|                                                 Min Throughput |         term |        3.46 |  ops/s |
|                                                Mean Throughput |         term |        3.46 |  ops/s |
|                                              Median Throughput |         term |        3.46 |  ops/s |
|                                                 Max Throughput |         term |        3.46 |  ops/s |
|                                       100th percentile latency |         term |     539.734 |     ms |
|                                  100th percentile service time |         term |      250.29 |     ms |
|                                                     error rate |         term |           0 |      % |


--------------------------------
[INFO] SUCCESS (took 22 seconds)
--------------------------------

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ian Hoang <[email protected]>

…s format Signed-off-by: Ian Hoang <[email protected]>

Signed-off-by: Ian Hoang <[email protected]>

osbenchmark/resources/default-test-procedures.json.j2

osbenchmark/resources/custom-test-procedures.json.j2

osbenchmark/workload_generator/workload_generator.py

osbenchmark/resources/base-workload.json.j2

osbenchmark/resources/default-operations.json.j2

osbenchmark/resources/custom-test-procedures.json.j2

Signed-off-by: Ian Hoang <[email protected]>

Ian Hoang added 4 commits January 23, 2024 14:47

Add changes for workload_generator and resources/templates

8fc07ec

Signed-off-by: Ian Hoang <[email protected]>

Fix bugs to allow workload to be created

e914c48

Signed-off-by: Ian Hoang <[email protected]>

Add working changes to custom operations and test procedures

4bda72c

Signed-off-by: Ian Hoang <[email protected]>

Add fixes for custom test procedures part 2

8eac9bb

Signed-off-by: Ian Hoang <[email protected]>

IanHoang requested a review from gkamat as a code owner January 23, 2024 23:29

Ian Hoang added 3 commits January 23, 2024 17:30

Make default test procedures format the same as custom test procedure…

235f0a4

…s format Signed-off-by: Ian Hoang <[email protected]>

Fix pylint error

bfa5afe

Signed-off-by: Ian Hoang <[email protected]>

Remove old unused workload.json templates

11aa710

Signed-off-by: Ian Hoang <[email protected]>

gkamat approved these changes Jan 24, 2024

View reviewed changes

Ian Hoang added 5 commits January 25, 2024 11:12

Add fixes for default operations and default test procedures

f75c3a5

Signed-off-by: Ian Hoang <[email protected]>

Fix spacing for jinja parameters and simplify operation parameters

74b8642

Signed-off-by: Ian Hoang <[email protected]>

Fix raw / endraw pair to be more concise

641d9fc

Signed-off-by: Ian Hoang <[email protected]>

Add suggested fixes to custom-operations and custom-test-procedures

59adc88

Signed-off-by: Ian Hoang <[email protected]>

Remove pass statement for pylint

c9e7135

Signed-off-by: Ian Hoang <[email protected]>

IanHoang merged commit 5538546 into opensearch-project:main Jan 25, 2024
8 checks passed

IanHoang deleted the test-procedures-and-operations-separation branch January 25, 2024 19:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create-Workload Improvements: Separate operations and test procedures from workload.json #446

Create-Workload Improvements: Separate operations and test procedures from workload.json #446

IanHoang commented Jan 23, 2024 •

edited

Loading

Create-Workload Improvements: Separate operations and test procedures from workload.json #446

Create-Workload Improvements: Separate operations and test procedures from workload.json #446

Conversation

IanHoang commented Jan 23, 2024 • edited Loading

Description

Issues Resolved

Testing

IanHoang commented Jan 23, 2024 •

edited

Loading