From 15f2cb1ad16fc75fb36b25e6c10d2dc356ec0a4e Mon Sep 17 00:00:00 2001 From: AkshathRaghav Date: Tue, 10 Oct 2023 21:49:28 -0400 Subject: [PATCH 1/3] Adding documentation Signed-off-by: AkshathRaghav --- DEVELOPER_GUIDE.md | 84 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 80 insertions(+), 4 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 55291c4b6..3897f2d67 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -24,6 +24,13 @@ This document will walk you through on what's needed to start contributing code - **Pyenv** : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding. For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation). + Install the following modules to continue with the next steps: + ``` + sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ + libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ + xz-utils tk-dev libffi-dev liblzma-dev git + ``` + - **JDK**: Although OSB is a Python application, it optionally builds and provisions OpenSearch clusters. JDK version 17 is used to build the current version of OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites). Note that the `javadoc` executable should be available in the JDK installation. An earlier version of the JDK can be used, but not all the integration tests will pass. @@ -38,7 +45,9 @@ This document will walk you through on what's needed to start contributing code ### Setup -To develop OSB properly, it is recommended that you fork the official OpenSearch Benchmark repository. +To develop OSB properly, it is recommended that you fork the official OpenSearch Benchmark repository. + +For those working on WSL2, it is recommended to clone the repository and set up the working environment within the Linux subsystem. Refer to the guide for setting up WSL2 on [Visual Studio Code](https://code.visualstudio.com/docs/remote/wsl) or [PyCharm](https://www.jetbrains.com/help/pycharm/using-wsl-as-a-remote-interpreter.html#create-wsl-interpreter). After you git cloned the forked copy of OpenSearch Benchmark, use the following command-line instructions to set up OpenSearch Benchmark for development: ``` @@ -74,6 +83,74 @@ This is typically created in PyCharm IDE by visiting the `Python Interpreter`, s ` In order to run tests within the PyCharm IDE, ensure the `Python Integrated Tools` / `Testing` / `Default Test Runner` is set to `pytest`. +## Running Workloads + +### Installation + +Download the latest release of OpenSearch from https://opensearch.org/downloads.html. If you are using WSL, make sure to download it into your `/home/` directory instead of `/mnt/c`. +``` +wget https://artifacts.opensearch.org/releases/bundle/opensearch//opensearch--linux-x64.tar.gz +tar -xf opensearch-x.x.x-linux-x64.tar.gz +cd opensearch-x.x.x +``` +NOTE: Have Docker running in the background for the next steps. Refer to the installation instructions [here](https://docs.docker.com/compose/install/). + +### Setup + +Add the following settings to the `opensearch.yml` file under the config directory +``` +vim config/opensearch.yml +``` +``` +# +discovery.type: single-node +plugins.security.disabled: true +# +``` +Run the opensearch-tar-install.sh script to install and setup a cluster for our use. +``` +bash opensearch-tar-install.sh +``` +Check the output of `curl.exe "http://localhost:9200/_cluster/health?pretty"`. Output should be similar to this: +``` +{ + "cluster_name" : "", + "status" : "green", + "timed_out" : false, + "number_of_nodes" : 1, + "number_of_data_nodes" : 1, + "discovered_master" : true, + "discovered_cluster_manager" : true, + "active_primary_shards" : 3, + "active_shards" : 3, + "relocating_shards" : 0, + "initializing_shards" : 0, + "unassigned_shards" : 0, + "delayed_unassigned_shards" : 0, + "number_of_pending_tasks" : 0, + "number_of_in_flight_fetch" : 0, + "task_max_waiting_in_queue_millis" : 0, + "active_shards_percent_as_number" : 100.0 +} +``` +Now, you have a local cluster running! You can connect to this and run the workload for the next step. + +### Running the workload + +Here's a sample executation of the geonames benchmark which can be found from the [workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) repo. +``` +opensearch-benchmark execute-test --pipeline=benchmark-only --workload=geonames --target-host=127.0.0.1:9200 --test-mode --workload-params '{"number_of_shards":"1","number_of_replicas":"0"}' +``` + +And we're done! You should be seeing the performance metrics soon enough! + +### Debugging + +**If you are not seeing any results, it should be an indicator that there is an issue with your cluster setup or the way the manager is accessing it**. Use the command below to view the logs. +``` +tail -f ~/.bencmark/logs/bechmark.log +``` + ## Executing tests Once setup is complete, you may run the unit and integration tests. @@ -87,10 +164,10 @@ make test ### Integration Tests -Integration tests can be run on the following operating systems: +Integration tests are expected to run for approximately **20-30 mins** and can be run on the following operating systems: * RedHat * CentOS - * Ubuntu + * Ubuntu (and WSL) * Amazon Linux 2 * MacOS @@ -100,7 +177,6 @@ Invoke integration tests by running the following command within the root direct make it ``` -Integration tests are expected to run for approximately 20-30 mins. ## Submitting your changes for a pull request From 0f2af675e76e9b224d5ffb1711c8a74f70a0c2b6 Mon Sep 17 00:00:00 2001 From: AkshathRaghav Date: Thu, 12 Oct 2023 21:51:10 -0400 Subject: [PATCH 2/3] Making changes to documentation Signed-off-by: AkshathRaghav --- DEVELOPER_GUIDE.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 3897f2d67..406a8da06 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -24,7 +24,7 @@ This document will walk you through on what's needed to start contributing code - **Pyenv** : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding. For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation). - Install the following modules to continue with the next steps: + **Optional Step:** For Debian-based systems, install the following modules to continue with the next steps: ``` sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ @@ -83,9 +83,9 @@ This is typically created in PyCharm IDE by visiting the `Python Interpreter`, s ` In order to run tests within the PyCharm IDE, ensure the `Python Integrated Tools` / `Testing` / `Default Test Runner` is set to `pytest`. -## Running Workloads +## Setting Up an Local OpenSearch Cluster For OSB Development (Optional) -### Installation +### OpenSearch Installation Download the latest release of OpenSearch from https://opensearch.org/downloads.html. If you are using WSL, make sure to download it into your `/home/` directory instead of `/mnt/c`. ``` @@ -95,7 +95,7 @@ cd opensearch-x.x.x ``` NOTE: Have Docker running in the background for the next steps. Refer to the installation instructions [here](https://docs.docker.com/compose/install/). -### Setup +### OpenSearch cluster setup Add the following settings to the `opensearch.yml` file under the config directory ``` @@ -135,7 +135,7 @@ Check the output of `curl.exe "http://localhost:9200/_cluster/health?pretty"`. O ``` Now, you have a local cluster running! You can connect to this and run the workload for the next step. -### Running the workload +### Running workloads on a locally installed cluster Here's a sample executation of the geonames benchmark which can be found from the [workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) repo. ``` From dbf40c86fe8698bce52ef8d5727398254f043d03 Mon Sep 17 00:00:00 2001 From: AkshathRaghav Date: Fri, 13 Oct 2023 12:49:32 -0400 Subject: [PATCH 3/3] Updating contents Signed-off-by: AkshathRaghav --- DEVELOPER_GUIDE.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 406a8da06..47d771fe2 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -7,6 +7,7 @@ This document will walk you through on what's needed to start contributing code - [Prerequisites](#prerequisites) - [Setup](#setup) - [Importing the project into an IDE](#importing-the-project-into-an-ide) +- [Setting Up a Local OpenSearch Cluster For OSB Development (Optional)](#setting-up-a-local-opensearch-cluster-for-osb-development-optional) - [Executing tests](#executing-tests) - [Unit tests](#unit-tests) - [Integration tests](#integration-tests) @@ -83,7 +84,7 @@ This is typically created in PyCharm IDE by visiting the `Python Interpreter`, s ` In order to run tests within the PyCharm IDE, ensure the `Python Integrated Tools` / `Testing` / `Default Test Runner` is set to `pytest`. -## Setting Up an Local OpenSearch Cluster For OSB Development (Optional) +## Setting Up a Local OpenSearch Cluster For OSB Development (Optional) ### OpenSearch Installation @@ -95,7 +96,7 @@ cd opensearch-x.x.x ``` NOTE: Have Docker running in the background for the next steps. Refer to the installation instructions [here](https://docs.docker.com/compose/install/). -### OpenSearch cluster setup +### OpenSearch Cluster setup Add the following settings to the `opensearch.yml` file under the config directory ``` @@ -135,7 +136,7 @@ Check the output of `curl.exe "http://localhost:9200/_cluster/health?pretty"`. O ``` Now, you have a local cluster running! You can connect to this and run the workload for the next step. -### Running workloads on a locally installed cluster +### Running Workloads on a locally installed Cluster Here's a sample executation of the geonames benchmark which can be found from the [workloads](https://github.com/opensearch-project/opensearch-benchmark-workloads) repo. ```