Benchmark: Add 2.0 name changes

Signed-off-by: Archer <[email protected]>
opensearch-project · Aug 1, 2024 · 6299549 · 6299549
1 parent 73ab08c
commit 6299549
Show file tree

Hide file tree

Showing 19 changed files with 137 additions and 134 deletions.
diff --git a/_benchmark/quickstart.md b/_benchmark/quickstart.md
@@ -76,7 +76,7 @@ If successful, OpenSearch returns the following response:
 
 ```bash
 $ opensearch-benchmark --help
-usage: opensearch-benchmark [-h] [--version] {execute-test,list,info,create-workload,generate,compare,download,install,start,stop} ...
+usage: opensearch-benchmark [-h] [--version] {run,list,info,create-workload,generate,compare,download,install,start,stop} ...
 
    ____                  _____                      __       ____                  __                         __
   / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
@@ -92,13 +92,13 @@ optional arguments:
   --version             show program's version number and exit
 
 subcommands:
-  {execute-test,list,info,create-workload,generate,compare,download,install,start,stop}
-    execute-test        Run a benchmark
+  {run,list,info,create-workload,generate,compare,download,install,start,stop}
+    run        Run a benchmark
     list                List configuration options
     info                Show info about a workload
     create-workload     Create a Benchmark workload from existing data
     generate            Generate artifacts
-    compare             Compare two test_executions
+    compare             Compare two test_runs
     download            Downloads an artifact
     install             Installs an OpenSearch node locally
     start               Starts an OpenSearch node locally
@@ -114,9 +114,9 @@ You can now run your first benchmark. The following benchmark uses the [percolat
 
 ### Understanding workload command flags
 
-Benchmarks are run using the [`execute-test`]({{site.url}}{{site.baseurl}}/benchmark/commands/execute-test/) command with the following command flags:
+Benchmarks are run using the [`run`]({{site.url}}{{site.baseurl}}/benchmark/commands/run/) command with the following command flags:
 
-For additional `execute_test` command flags, see the [execute-test]({{site.url}}{{site.baseurl}}/benchmark/commands/execute-test/) reference. Some commonly used options are `--workload-params`, `--exclude-tasks`, and `--include-tasks`.
+For additional `execute_test` command flags, see the [run]({{site.url}}{{site.baseurl}}/benchmark/commands/run/) reference. Some commonly used options are `--workload-params`, `--exclude-tasks`, and `--include-tasks`.
 {: .tip}
 
 * `--pipeline=benchmark-only` : Informs OSB that users wants to provide their own OpenSearch cluster.
@@ -125,14 +125,14 @@ For additional `execute_test` command flags, see the [execute-test]({{site.url}}
 * `--client-options="basic_auth_user:'<Basic Auth Username>',basic_auth_password:'<Basic Auth Password>'"`: The username and password for your OpenSearch cluster.
 * `--test-mode`: Allows a user to run the workload without running it for the entire duration. When this flag is present, Benchmark runs the first thousand operations of each task in the workload. This is only meant for sanity checks---the metrics produced are meaningless.
 
-The `--distribution-version`, which indicates which OpenSearch version Benchmark will use when provisioning. When run, the `execute-test` command will parse the correct distribution version when it connects to the OpenSearch cluster.
+The `--distribution-version`, which indicates which OpenSearch version Benchmark will use when provisioning. When run, the `run` command will parse the correct distribution version when it connects to the OpenSearch cluster.
 
 ### Running the workload
 
-To run the [percolator](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/percolator) workload with OpenSearch Benchmark, use the following `execute-test` command: 
+To run the [percolator](https://github.com/opensearch-project/opensearch-benchmark-workloads/tree/main/percolator) workload with OpenSearch Benchmark, use the following `run` command: 
 
 ```bash
-opensearch-benchmark execute-test --pipeline=benchmark-only --workload=percolator --target-host=https://localhost:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false --test-mode
+opensearch-benchmark run --pipeline=benchmark-only --workload=percolator --target-host=https://localhost:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false --test-mode
 ```
 {% include copy.html %}
 
@@ -260,7 +260,7 @@ For more details about how the summary report is generated, see [Summary report]
 
 ## Running OpenSearch Benchmark on your own cluster
 
-Now that you're familiar with running OpenSearch Benchmark on a cluster, you can run OpenSearch Benchmark on your own cluster, using the same `execute-test` command, replacing the following settings.
+Now that you're familiar with running OpenSearch Benchmark on a cluster, you can run OpenSearch Benchmark on your own cluster, using the same `run` command, replacing the following settings.
  
   * Replace `https://localhost:9200` with your target cluster endpoint.  This could be a URI like `https://search.mydomain.com` or a `HOST:PORT` specification.
   * If the cluster is configured with basic authentication, replace the username and password in the command line with the appropriate credentials.
@@ -271,7 +271,7 @@ Now that you're familiar with running OpenSearch Benchmark on a cluster, you can
 You can copy the following command template to use in your own terminal:
 
 ```bash
-opensearch-benchmark execute-test --pipeline=benchmark-only --workload=percolator --target-host=<OpenSearch Cluster Endpoint> --client-options=basic_auth_user:admin,basic_auth_password:admin
+opensearch-benchmark run --pipeline=benchmark-only --workload=percolator --target-host=<OpenSearch Cluster Endpoint> --client-options=basic_auth_user:admin,basic_auth_password:admin
 ```
 {% include copy.html %}
 

diff --git a/_benchmark/reference/commands/command-flags.md b/_benchmark/reference/commands/command-flags.md
@@ -19,7 +19,7 @@ opensearch-benchmark <command> --<command-flag>
 
 Flags that accept comma-separated values, such `--telemetry`, can also accept a JSON array. This can be defined by passing a file path ending in `.json` or inline as a JSON string.
 
-- Comma-seperated values: `opensearch-benchmark ... --test-procedure="ingest-only,search-aggregations"`
+- Comma-seperated values: `opensearch-benchmark ... --scenario="ingest-only,search-aggregations"`
 - JSON file: `opensearch-benchmark ... --workload-params="params.json"`
 - JSON inline string: `opensearch-benchmark  ... --telemetry='["node-stats", "recovery-stats"]'`
 
@@ -54,13 +54,13 @@ Defines the workload to use based on the workload's name. You can find a list of
 Defines which variables to inject into the workload. Variables injected must be available in the workload. To see which parameters are valid in the official workloads, select the workload from [the workloads repository](https://github.com/opensearch-project/opensearch-benchmark-workloads).
 
 <!-- vale off -->
-## test-procedure
+## scenario
 <!-- vale on -->
 
-Defines the test procedures to use with each workload. You can find a list of test procedures that the workload supports by specifying the workload in the `info` command, for example, `opensearch-benchmark info --workload=<workload_name>`. To look up information on a specific test procedure, use the command `opensearch-benchmark info --workload=<workload_name> --test-procedure=<test-procedure>`.
+Defines the scenarios to use with each workload. You can find a list of scenarios that the workload supports by specifying the workload in the `info` command, for example, `opensearch-benchmark info --workload=<workload_name>`. To look up information on a specific scenario, use the command `opensearch-benchmark info --workload=<workload_name> --scenario=<scenario>`.
 
 <!-- vale off -->
-## test-execution-id
+## test-run-id
 <!-- vale on -->
 
 Defines a unique ID for the test run.
@@ -69,29 +69,29 @@ Defines a unique ID for the test run.
 ## include-tasks
 <!-- vale on -->
 
-Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run.
+Defines a comma-separated list of scenario tasks to run. By default, all tasks listed in a scenario array are run.
 
-Tests are executed in the order they are defined in `test-procedure`---not in the order they are defined in the command. 
+Tests are executed in the order they are defined in `scenario`---not in the order they are defined in the command. 
 
 All task filters are case sensitive.
 
 <!-- vale off -->
 ## exclude-tasks
 <!-- vale on -->
 
-Defines a comma-separated list of test procedure tasks not to run.
+Defines a comma-separated list of scenario tasks not to run.
 
 <!-- vale off -->
 ## baseline
 <!-- vale on -->
 
-The baseline TestExecution ID used to compare the contender TestExecution.  
+The baseline TestRun ID used to compare the contender TestRun.  
 
 <!-- vale off -->
 ## contender
 <!-- vale on -->
 
-The TestExecution ID for the contender being compared to the baseline. 
+The TestRun ID for the contender being compared to the baseline. 
 
 <!-- vale off -->
 ## results-format
@@ -217,7 +217,7 @@ The major version of JDK to use.
 Defines a comma-separated list of clients to use. All options are passed to the OpenSearch Python client. Default is `timeout:60`.
 
 <!-- vale off -->
-## load-worker-coordinator-hosts
+## worker-ips
 <!-- vale on -->
 
 Defines a comma-separated list of hosts that coordinate loads. Default is `localhost`.

diff --git a/_benchmark/reference/commands/compare.md b/_benchmark/reference/commands/compare.md
@@ -12,11 +12,11 @@ redirect_from:
 # compare
 <!-- vale on -->
 
-The `compare` command helps you analyze the difference between two benchmark tests. This can help you analyze the performance impact of changes made from a previous test based on a specific Git revision. 
+The `compare` command helps you analyze the difference between two benchmark runs. This can help you analyze the performance impact of changes made from a previous run based on a specific Git revision. 
 
 ## Usage
 
-You can compare two different workload tests using their `TestExecution IDs`. To find a list of tests run from a specific workload, use `opensearch-benchmark list test_executions`. You should receive an output similar to the following: 
+You can compare two different workload runs using their `TestRun IDs`. To find a list of tests run from a specific workload, use `opensearch-benchmark list test_runs`. You should receive an output similar to the following: 
 
 
 ```
@@ -26,11 +26,11 @@ You can compare two different workload tests using their `TestExecution IDs`. To
 / /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
 \____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
     /_/
-Recent test-executions:
+Recent test-runs:
 
-Recent test_executions:
+Recent test_runs:
 
-TestExecution ID                      TestExecution Timestamp    Workload    Workload Parameters    TestProcedure        ProvisionConfigInstance    User Tags    workload Revision    Provision Config Revision
+TestRun ID                      TestRun Timestamp    Workload    Workload Parameters    TestProcedure        ProvisionConfigInstance    User Tags    workload Revision    Provision Config Revision
 ------------------------------------  -------------------------  ----------  ---------------------  -------------------  -------------------------  -----------  -------------------  ---------------------------
 729291a0-ee87-44e5-9b75-cc6d50c89702  20230524T181718Z           geonames                           append-no-conflicts  4gheap                                  30260cf
 f91c33d0-ec93-48e1-975e-37476a5c9fe5  20230524T170134Z           geonames                           append-no-conflicts  4gheap                                  30260cf
@@ -56,12 +56,12 @@ You should receive the following response comparing the final benchmark metrics
     /_/
 
 Comparing baseline
-  TestExecution ID: 729291a0-ee87-44e5-9b75-cc6d50c89702
-  TestExecution timestamp: 2023-05-24 18:17:18 
+  TestRun ID: 729291a0-ee87-44e5-9b75-cc6d50c89702
+  TestRun timestamp: 2023-05-24 18:17:18 
 
 with contender
-  TestExecution ID: a33845cc-c2e5-4488-a2db-b0670741ff9b
-  TestExecution timestamp: 2023-05-23 21:31:45
+  TestRun ID: a33845cc-c2e5-4488-a2db-b0670741ff9b
+  TestRun timestamp: 2023-05-23 21:31:45
 
 
 ------------------------------------------------------
@@ -127,8 +127,8 @@ Query latency country_agg_cached (100.0 percentile) [ms]     3.42547      2.8681
 
 You can use the following options to customize the results of your test comparison: 
 
-- `--baseline`: The baseline TestExecution ID used to compare the contender TestExecution.  
-- `--contender`: The TestExecution ID for the contender being compared to the baseline. 
+- `--baseline`: The baseline TestRun ID used to compare the contender TestRun.  
+- `--contender`: The TestRun ID for the contender being compared to the baseline. 
 - `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`.
 - `--results-numbers-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`.
 - `--results-file`: When provided a file path, writes the compare results to the file indicated in the path. 

diff --git a/_benchmark/reference/commands/index.md b/_benchmark/reference/commands/index.md
@@ -9,11 +9,11 @@ redirect_from: /benchmark/commands/index/
 
 # OpenSearch Benchmark command reference
 
-This section provides a list of commands supported by OpenSearch Benchmark, including commonly used commands such as `execute-test` and `list`.
+This section provides a list of commands supported by OpenSearch Benchmark, including commonly used commands such as `run` and `list`.
 
 - [compare]({{site.url}}{{site.baseurl}}/benchmark/commands/compare/)
 - [download]({{site.url}}{{site.baseurl}}/benchmark/commands/download/)
-- [execute-test]({{site.url}}{{site.baseurl}}/benchmark/commands/execute-test/)
+- [run]({{site.url}}{{site.baseurl}}/benchmark/commands/run/)
 - [info]({{site.url}}{{site.baseurl}}/benchmark/commands/info/)
 - [list]({{site.url}}{{site.baseurl}}/benchmark/commands/list/)
 

diff --git a/_benchmark/reference/commands/info.md b/_benchmark/reference/commands/info.md
@@ -43,7 +43,7 @@ Showing details for workload [nyc_taxis]:
 TestProcedure [searchable-snapshot]
 ===================================
 
-Measuring performance for Searchable Snapshot feature. Based on the default test procedure 'append-no-conflicts'.
+Measuring performance for Searchable Snapshot feature. Based on the default scenario 'append-no-conflicts'.
 
 Schedule: 
 ----------
@@ -158,6 +158,6 @@ You can use the following options with the `info` command:
 - `--workload-path`: Defines the path to a downloaded or custom workload. 
 - `--workload-revision`: Defines a specific revision from the workload source tree that OpenSearch Benchmark should use.
 - `--workload`: Defines the workload to use based on the workload's name. You can find a list of preloaded workloads using `opensearch-benchmark list workloads`. 
-- `--test-procedure`: Defines a test procedure to use. You can find a list of test procedures using `opensearch-benchmark list test_procedures`.
-- `--include-tasks`: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run.
-- `--exclude-tasks`: Defines a comma-separated list of test procedure tasks not to run. 
+- `--scenario`: Defines a scenario to use. You can find a list of scenarios using `opensearch-benchmark list scenarios`.
+- `--include-tasks`: Defines a comma-separated list of scenario tasks to run. By default, all tasks listed in a scenario array are run.
+- `--exclude-tasks`: Defines a comma-separated list of scenario tasks not to run. 
diff --git a/_benchmark/reference/commands/list.md b/_benchmark/reference/commands/list.md
@@ -17,8 +17,8 @@ The `list` command lists the following elements used by OpenSearch Benchmark:
 - `telemetry`: Telemetry devices  
 - `workloads`: Workloads 
 - `pipelines`: Pipelines
-- `test_executions`: Single run of a workload 
-- `provision_config_instances`: Provisioned configuration instances
+- `test_runs`: Single run of a workload 
+- `cluster_configs`: Provisioned configuration instances
 - `opensearch-plugins`: OpenSearch plugins 
 
 
@@ -27,13 +27,13 @@ The `list` command lists the following elements used by OpenSearch Benchmark:
 The following example lists any workload test runs and detailed information about each test:
 
 ```
-`opensearch-benchmark list test_executions
+`opensearch-benchmark list test_runs
 ```
 
 OpenSearch Benchmark returns information about each test. 
 
 ```
-benchmark list test_executions
+benchmark list test_runs
 
    ____                  _____                      __       ____                  __                         __
   / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
@@ -43,9 +43,9 @@ benchmark list test_executions
     /_/
 
 
-Recent test_executions:
+Recent test_runs:
 
-TestExecution ID                      TestExecution Timestamp    Workload    Workload Parameters    TestProcedure        ProvisionConfigInstance    User Tags    workload Revision    Provision Config Revision
+TestRun ID                      TestRun Timestamp    Workload    Workload Parameters    TestProcedure        ProvisionConfigInstance    User Tags    workload Revision    Provision Config Revision
 ------------------------------------  -------------------------  ----------  ---------------------  -------------------  -------------------------  -----------  -------------------  ---------------------------
 729291a0-ee87-44e5-9b75-cc6d50c89702  20230524T181718Z           geonames                           append-no-conflicts  4gheap                                  30260cf
 f91c33d0-ec93-48e1-975e-37476a5c9fe5  20230524T170134Z           geonames                           append-no-conflicts  4gheap                                  30260cf