From 4f2a458d4dc2d077d0c517729eb26048d89e79c3 Mon Sep 17 00:00:00 2001 From: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> Date: Mon, 6 May 2024 15:03:21 -0400 Subject: [PATCH 1/6] Add missing OPENSEARCH_INITIAL_ADMIN_PASSWORD for both apt/deb and yum/rpm (#7079) * Add missing OPENSEARCH_INITIAL_ADMIN_PASSWORD for apt install on debian.md This page's install command when installing via rep package is missing `OPENSEARCH_INITIAL_ADMIN_PASSWORD` Signed-off-by: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> * More update on deb and rpm installation steps on 2.12 security changes Signed-off-by: Peter Zhu * Move 2.12 top Signed-off-by: Peter Zhu * fix formatting Signed-off-by: Peter Zhu * More tweaks Signed-off-by: Peter Zhu * More tweaks 2 Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/rpm.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu * Add more things Signed-off-by: Peter Zhu * Add more things Signed-off-by: Peter Zhu * Update _install-and-configure/install-opensearch/debian.md Signed-off-by: Peter Zhu Co-authored-by: Nathan Bower Signed-off-by: Peter Zhu --------- Signed-off-by: Drew Miranda <107503402+drewmiranda-gl@users.noreply.github.com> Signed-off-by: Peter Zhu Co-authored-by: Peter Zhu Co-authored-by: Nathan Bower --- .../install-opensearch/debian.md | 27 ++++++++--- .../install-opensearch/rpm.md | 47 ++++++++++++------- 2 files changed, 51 insertions(+), 23 deletions(-) diff --git a/_install-and-configure/install-opensearch/debian.md b/_install-and-configure/install-opensearch/debian.md index 72ae05d87c..0f7d674d7b 100644 --- a/_install-and-configure/install-opensearch/debian.md +++ b/_install-and-configure/install-opensearch/debian.md @@ -37,21 +37,23 @@ This guide assumes that you are comfortable working from the Linux command line ### Install OpenSearch from a package 1. Download the Debian package for the desired version directly from the [OpenSearch downloads page](https://opensearch.org/downloads.html){:target='\_blank'}. The Debian package can be downloaded for both **x64** and **arm64** architectures. -1. From the CLI, install using `dpkg`. +1. From the CLI, install the package using `dpkg`: + + For new installations of OpenSearch 2.12 and later, you must define a custom admin password in order to set up a demo security configuration. Use one of the following commands to define a custom admin password: ```bash # x64 - sudo dpkg -i opensearch-{{site.opensearch_version}}-linux-x64.deb + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= dpkg -i opensearch-{{site.opensearch_version}}-linux-x64.deb # arm64 - sudo dpkg -i opensearch-{{site.opensearch_version}}-linux-arm64.deb + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= dpkg -i opensearch-{{site.opensearch_version}}-linux-arm64.deb ``` - For OpenSearch 2.12 and greater, a custom admin password is required in order to set up a security demo configuration. To set a custom admin password, use one the following commands: + Use the following command for OpenSearch versions 2.11 and earlier: ```bash # x64 - sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= dpkg -i opensearch-{{site.opensearch_version}}-linux-x64.deb + sudo dpkg -i opensearch-{{site.opensearch_version}}-linux-x64.deb # arm64 - sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= dpkg -i opensearch-{{site.opensearch_version}}-linux-arm64.deb + sudo dpkg -i opensearch-{{site.opensearch_version}}-linux-arm64.deb ``` 1. After the installation succeeds, enable OpenSearch as a service. @@ -136,14 +138,27 @@ APT, the primary package management tool for Debian–based operating systems, a 1. Choose the version of OpenSearch you want to install: - Unless otherwise indicated, the latest available version of OpenSearch is installed. + ```bash + # For new installations of OpenSearch 2.12 and later, you must define a custom admin password in order to set up a demo security configuration. + # Use one of the following commands to define a custom admin password: + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= apt-get install opensearch + + # Use the following command for OpenSearch versions 2.11 and earlier: sudo apt-get install opensearch ``` {% include copy.html %} - To install a specific version of OpenSearch: + ```bash # Specify the version manually using opensearch= + + # For new installations of OpenSearch 2.12 and later, you must define a custom admin password in order to set up a demo security configuration. + # Use one of the following commands to define a custom admin password: + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= apt-get install opensearch={{site.opensearch_version}} + + # Use the following command for OpenSearch versions 2.11 and earlier: sudo apt-get install opensearch={{site.opensearch_version}} ``` diff --git a/_install-and-configure/install-opensearch/rpm.md b/_install-and-configure/install-opensearch/rpm.md index a22ea96d61..1810273fb1 100644 --- a/_install-and-configure/install-opensearch/rpm.md +++ b/_install-and-configure/install-opensearch/rpm.md @@ -44,36 +44,37 @@ This guide assumes that you are comfortable working from the Linux command line ``` {% include copy.html %} -1. For OpenSearch 2.12 and greater, a custom admin password is required in order to set up a security demo configuration. To set a custom admin password, use one the following commands: +1. From the CLI, you can install the package with `rpm` or `yum`: ```bash - # Install the x64 package using yum. + # For new installations of OpenSearch 2.12 and later, you must define a custom admin password in order to set up a demo security configuration. + # Use one of the following commands to define a custom admin password: + + ## Install the x64 package using yum. sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= yum install opensearch-{{site.opensearch_version}}-linux-x64.rpm - # Install the x64 package using rpm. + ## Install the x64 package using rpm. sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= rpm -ivh opensearch-{{site.opensearch_version}}-linux-x64.rpm - # Install the arm64 package using yum. - sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= yum install opensearch-{{site.opensearch_version}}-linux-x64.rpm + ## Install the arm64 package using yum. + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= yum install opensearch-{{site.opensearch_version}}-linux-arm64.rpm - # Install the arm64 package using rpm. - sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= rpm -ivh opensearch-{{site.opensearch_version}}-linux-x64.rpm - ``` + ## Install the arm64 package using rpm. + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= rpm -ivh opensearch-{{site.opensearch_version}}-linux-arm64.rpm -1. From the CLI, you can install the package with `rpm` or `yum`. + # Use the following command for OpenSearch versions 2.11 and earlier: - ```bash - # Install the x64 package using yum. + ## Install the x64 package using yum. sudo yum install opensearch-{{site.opensearch_version}}-linux-x64.rpm - # Install the x64 package using rpm. + ## Install the x64 package using rpm. sudo rpm -ivh opensearch-{{site.opensearch_version}}-linux-x64.rpm - # Install the arm64 package using yum. - sudo yum install opensearch-{{site.opensearch_version}}-linux-x64.rpm + ## Install the arm64 package using yum. + sudo yum install opensearch-{{site.opensearch_version}}-linux-arm64.rpm - # Install the arm64 package using rpm. - sudo rpm -ivh opensearch-{{site.opensearch_version}}-linux-x64.rpm + ## Install the arm64 package using rpm. + sudo rpm -ivh opensearch-{{site.opensearch_version}}-linux-arm64.rpm ``` 1. After the installation succeeds, enable OpenSearch as a service. @@ -125,15 +126,27 @@ YUM, the primary package management tool for Red Hat–based operating systems, ``` {% include copy.html %} -1. Choose the version of OpenSearch you want to install: +1. Choose the version of OpenSearch you want to install: - Unless otherwise indicated, the latest available version of OpenSearch is installed. + ```bash + # For OpenSearch versions 2.12 and later, a custom admin password is required in order to set up a demo security configuration for a new installation. + # To set a custom admin password, use the following commands: + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= yum install opensearch + + # Use the following command for OpenSearch versions 2.11 and earlier: sudo yum install opensearch ``` {% include copy.html %} - To install a specific version of OpenSearch: + ```bash + # For OpenSearch versions 2.12 and later, a custom admin password is required in order to set up a demo security configuration for a new installation. + # To set a custom admin password, use the following commands: + sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD= yum install 'opensearch-{{site.opensearch_version}}' + + # Use the following command for OpenSearch versions 2.11 and earlier: sudo yum install 'opensearch-{{site.opensearch_version}}' ``` {% include copy.html %} From 5ed69127b6797bab4a653dec774f782fd55d652b Mon Sep 17 00:00:00 2001 From: Daniel Widdis Date: Tue, 7 May 2024 08:47:41 -0700 Subject: [PATCH 2/6] Flow Framework documentation for no-op step delay (#7080) * Flow Framework documentation for no-op step delay Signed-off-by: Daniel Widdis * Update _automating-configurations/workflow-steps.md Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Daniel Widdis * Update workflow-steps.md * tools_order is for agent step Signed-off-by: Daniel Widdis * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Daniel Widdis Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- _automating-configurations/workflow-steps.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/_automating-configurations/workflow-steps.md b/_automating-configurations/workflow-steps.md index 99c1f57993..2fba435ec7 100644 --- a/_automating-configurations/workflow-steps.md +++ b/_automating-configurations/workflow-steps.md @@ -25,11 +25,11 @@ The following table lists the workflow step types. The `user_inputs` fields for |Step type |Corresponding API |Description | |--- |--- |--- | -|`noop` |No API | A no-operation (no-op) step that does nothing. It may be useful in some cases for synchronizing parallel steps. | +| `noop` | No API | A no-operation (no-op) step that does nothing, which is useful for synchronizing parallel steps. If the `user_inputs` field contains a `delay` key, this step will wait for the specified amount of time. | |`create_connector` |[Create Connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/connector-apis/create-connector/) |Creates a connector to a model hosted on a third-party platform. | |`delete_connector` |[Delete Connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/connector-apis/delete-connector/) |Deletes a connector to a model hosted on a third-party platform. | |`register_model_group` |[Register Model Group]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-group-apis/register-model-group/) |Registers a model group. The model group will be deleted automatically once no model is present in the group. | -|`register_remote_model` |[Register Model (remote)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/#register-a-model-hosted-on-a-third-party-platform) |Registers a model hosted on a third-party platform. If the `user_inputs` field contains a `deploy` key that is set to `true`, also deploys the model. | +|`register_remote_model` |[Register Model (remote)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/#register-a-model-hosted-on-a-third-party-platform) | Registers a model hosted on a third-party platform. If the `user_inputs` field contains a `deploy` key that is set to `true`, the model is also deployed. | |`register_local_pretrained_model` |[Register Model (pretrained)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/#register-a-pretrained-text-embedding-model) | Registers an OpenSearch-provided pretrained text embedding model that is hosted on your OpenSearch cluster. If the `user_inputs` field contains a `deploy` key that is set to `true`, also deploys the model. | |`register_local_sparse_encoding_model` |[Register Model (sparse)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/#register-a-pretrained-sparse-encoding-model) | Registers an OpenSearch-provided pretrained sparse encoding model that is hosted on your OpenSearch cluster. If the `user_inputs` field contains a `deploy` key that is set to `true`, also deploys the model. | |`register_local_custom_model` |[Register Model (custom)]({{site.url}}{{site.baseurl}}/ml-commons-plugin/api/model-apis/register-model/#register-a-custom-model) | Registers a custom model that is hosted on your OpenSearch cluster. If the `user_inputs` field contains a `deploy` key that is set to `true`, also deploys the model. | @@ -45,13 +45,14 @@ The following table lists the workflow step types. The `user_inputs` fields for ## Additional fields -You can include the following additional fields in the `user_inputs` field when indicated. +You can include the following additional fields in the `user_inputs` field if the field is supported by the indicated step type. -|Field |Data type |Description | +|Field |Data type | Step type | Description | |--- |--- |--- | -|`node_timeout` |Time units |A user-provided timeout for this step. For example, `20s` for a 20-second timeout. | -|`deploy` |Boolean |Applicable to the Register Model step type. If set to `true`, also executes the Deploy Model step. | -|`tools_order` |List |Applicable only to the Register Agent step type. Specifies the ordering of `tools`. For example, specify `["foo_tool", "bar_tool"]` to sequence those tools in that order. | +|`node_timeout` | Time units | All | A user-provided timeout for this step. For example, `20s` for a 20-second timeout. | +|`deploy` | Boolean | Register model | If set to `true`, also deploys the model. | +|`tools_order` | List | Register agent | Specifies the ordering of `tools`. For example, specify `["foo_tool", "bar_tool"]` to sequence those tools in that order. | +|`delay` | Time units | No-op | Waits for the specified amount of time. For example, `250ms` specifies to wait for 250 milliseconds before continuing the workflow. | You can include the following additional fields in the `previous_node_inputs` field when indicated. @@ -61,4 +62,4 @@ You can include the following additional fields in the `previous_node_inputs` fi ## Example workflow steps -For example workflow step implementations, see the [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/). \ No newline at end of file +For example workflow step implementations, see the [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/). From 979564ea578e307e8d17df09f71f628de62bfb67 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 7 May 2024 10:48:09 -0500 Subject: [PATCH 3/6] Add finetuning a workload (#7029) * Add refresh API. Signed-off-by: Archer * Add finetuning a workload Signed-off-by: Archer * Add additonal tweaks Signed-off-by: Archer * Delete _api-reference/index-apis/refresh.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Archer Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- _benchmark/user-guide/finetine-workloads.md | 55 +++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 _benchmark/user-guide/finetine-workloads.md diff --git a/_benchmark/user-guide/finetine-workloads.md b/_benchmark/user-guide/finetine-workloads.md new file mode 100644 index 0000000000..4fc0a284db --- /dev/null +++ b/_benchmark/user-guide/finetine-workloads.md @@ -0,0 +1,55 @@ +--- +layout: default +title: Fine-tuning custom workloads +nav_order: 12 +parent: User guide +--- + +# Fine-tuning custom workloads + +While custom workloads can help make benchmarks more specific to your application's needs, they sometimes require additional adjustments to make sure they closely resemble a production cluster. + +You can fine-tune your custom workloads to more closely match your benchmarking needs by using the `create-workload` feature. `create-workload` can extract documents from all indexes or specific indexes selected by the user. + +## Characteristics to consider + +When beginning to use `create-workload` to fine-tune a custom workload, consider the following characteristics of the workload: + +1. **Queries** -- Consider the kinds of documents and as well as indexes targeted by the queries and the fields the queries call. +2. **Shard Size** -- Match the shard size of the workload with the shard size of your cluster, or the benchmark will not simulate the behavior of your application. Lucene operates based on shard sizes and does not have indexes. Calculate the shard size for any index that you want to include in the custom workload. +3. **Shard count** -- Choose the number of shards according to how you want to size the workload and improve query performance. Because each use case is different, you can determine the shard count in two ways: + 1. Divide the ideal index size by the shard size found in step 2. + 2. Multiply the ideal number of shards by the shard size found in step 2. +4. **Decide how many documents to extract (Recommended)** -- Now that the shard size is set and the number of shards needed in the final index decided, you can determine how many documents you want to extract. In some cases, users aren’t interested in retrieving the entire document corpus from an index because the corpus might be too big. Instead, you might want to generate a smaller corpus. However, the corpus of the generated index should be *representative* of the index in the production workload. In other words, it should contain documents from across the index and not only from a certain area, for example, the first half of the index or the last third of the index. To decide how many documents to extract: + 1. Multiply the number of shards by the shard size. Because every document is created unequally, add a buffer to the number---an arbitrary number of additional documents. The buffer provides assistance when the number of documents extracted is lower than the expected number, which will not help to retain your shard size. The shard size should be lower than the number of documents extracted. + 2. Divide the store size by the product of the previous step. The value of the multiple is used to set the number of sample documents from the reference index. +5. **Target cluster configuration** -- Factor the configuration and characteristics of the target cluster into how you generate the workload. + + +## Example + +The following example contains an index named `stocks`. The `stocks` index includes documents containing statistics on all stocks being traded on the New York Stock Exchange (NYSE). OpenSearch Dashboards provides information about the `stocks` index, as shown in the following code example: + +``` +health status index uuid pri rep docs.count docs.deleted store.size pri.store.size +green open stocks asjdkfjacklajldf 12 1 997818020 232823733 720gb 360gb +``` + +Using this information, you can start adjusting the workload to your specifications, as shown in the following steps: + +1. **Fetch queries associated with this index** -- Obtain the queries needed to make requests to the `stocks` index. +2. **Find the shard size for the index** -- To get the shard size of the index, divide the store size by the number of shards in the index: `720 / (12 + (12 * 1)) = 30`. 30 GB is the shard size. You can verify this by dividing the primary store size value by the number of primary shards. +3. **Determine the number of index shards** -- Determine the number of shards needed in the index to represent your application under a production load. For example, if you want your index to hold 300 GB of documents, but 300 GB is too much for the benchmark, determine a number that makes sense. For example, 300 GB of documents divided by the 30 GB shard size determined in the last step, or `300 / 30 = 10`, produces 10 shards. These 10 shards can either be 10 primary shards and 0 replicas, 5 primary shards and 1 replica, or 2 primary shards and 4 replicas. The shard configuration depends on your cluster's index needs. +4. **Decide how many documents to extract** -- To retain 30 GB and have 10 shards, you need to extract at least 300 GB of documents. To determine the number of documents to extract, divide the store size value by the index size value, in this example, `720 / 300 = 2.4`. Because you want to make sure you reach a value of 30 GB per shard, it is best to round down and choose 2 as the extraction multiple, which means that OpenSearch Benchmark will extract every other document. +5. **Think about the target cluster configuration** -- Assess the cluster you're planning to work on. Consider the use case, the size of the cluster, and the number of nodes. While fine-tuning the workload, this could be an iterative process where small adjustments to the cluster are made according to the results of the workload runs. + + +## Replicating metrics + +In many cases, a workload will not be able to exactly replicate the exact metrics of a production cluster. However, you can aim to get as close as possible to your ideal cluster metrics by replicating the following metrics: + +* CPU utilization +* Search request rates +* Indexing rates + + From 939fa165a64b413f8e2a45c73d1ed08a13d4964c Mon Sep 17 00:00:00 2001 From: Andriy Redko Date: Tue, 7 May 2024 13:03:15 -0400 Subject: [PATCH 4/6] Minor documentation correction for 2.14.0 (#7101) * Minor documentation correction for 2.14.0 Signed-off-by: Andriy Redko * Address code review comments Signed-off-by: Andriy Redko * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Andriy Redko Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _field-types/supported-field-types/date.md | 2 +- .../configuring-opensearch/network-settings.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/_field-types/supported-field-types/date.md b/_field-types/supported-field-types/date.md index e0b99f3a90..fb008d1512 100644 --- a/_field-types/supported-field-types/date.md +++ b/_field-types/supported-field-types/date.md @@ -62,7 +62,7 @@ OpenSearch has built-in date formats, but you can also create your own custom fo ## Default format -As of OpenSearch 2.12, the default date format is `strict_date_time_no_millis||strict_date_optional_time||epoch_millis`. To revert the default format back to `strict_date_optional_time||epoch_millis` (the default format for OpenSearch 2.11 and earlier), set the `opensearch.experimental.optimization.datetime_formatter_caching.enabled` feature flag to `false`. For more information about enabling and disabling feature flags, see [Enabling experimental features]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/). +As of OpenSearch 2.12, you can to choose to use an experimental default date format, `strict_date_time_no_millis||strict_date_optional_time||epoch_millis`. To use the experimental default, set the `opensearch.experimental.optimization.datetime_formatter_caching.enabled` feature flag to `true`. For more information about enabling and disabling feature flags, see [Enabling experimental features]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/). ## Built-in formats diff --git a/_install-and-configure/configuring-opensearch/network-settings.md b/_install-and-configure/configuring-opensearch/network-settings.md index 3728c55bc2..f96dde97e1 100644 --- a/_install-and-configure/configuring-opensearch/network-settings.md +++ b/_install-and-configure/configuring-opensearch/network-settings.md @@ -56,4 +56,4 @@ The default OpenSearch transport is provided by the `transport-netty4` module an Plugin | Description :---------- | :-------- `transport-nio` | The OpenSearch transport based on Java NIO.
Installation: `./bin/opensearch-plugin install transport-nio`
Configuration (using `opensearch.yml`):
`transport.type: nio-transport`
`http.type: nio-http-transport` -`transport-reactor-netty4` | The OpenSearch HTTP transport based on [Project Reactor](https://github.com/reactor/reactor-netty) and Netty 4 (**experimental**)
Installation: `./bin/opensearch-plugin install transport-reactor-netty4`
Configuration (using `opensearch.yml`):
`http.type: reactor-netty4` +`transport-reactor-netty4` | The OpenSearch HTTP transport based on [Project Reactor](https://github.com/reactor/reactor-netty) and Netty 4 (**experimental**)
Installation: `./bin/opensearch-plugin install transport-reactor-netty4`
Configuration (using `opensearch.yml`):
`http.type: reactor-netty4`
`http.type: reactor-netty4-secure` From 23193dc1ac256d532eb120b1d4dfa36fb0209d67 Mon Sep 17 00:00:00 2001 From: Ashish Singh Date: Tue, 7 May 2024 22:33:55 +0530 Subject: [PATCH 5/6] Add documentation for settings introduced in 2.14 (#7057) * Add documentation for settings introduced in 2.14 Signed-off-by: Ashish Singh * Update _install-and-configure/configuring-opensearch/index-settings.md Co-authored-by: Nathan Bower Signed-off-by: Ashish Singh * Apply suggestions from code review Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Signed-off-by: Ashish Singh * Apply suggestions from code review Co-authored-by: Nathan Bower Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Ashish Singh Signed-off-by: Ashish Singh Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuring-opensearch/index-settings.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/_install-and-configure/configuring-opensearch/index-settings.md b/_install-and-configure/configuring-opensearch/index-settings.md index aaa933bb9e..3bd4285efb 100644 --- a/_install-and-configure/configuring-opensearch/index-settings.md +++ b/_install-and-configure/configuring-opensearch/index-settings.md @@ -39,6 +39,22 @@ OpenSearch supports the following cluster-level index settings. All settings in - `indices.fielddata.cache.size` (String): The maximum size of the field data cache. May be specified as an absolute value (for example, `8GB`) or a percentage of the node heap (for example, `50%`). This value is static so you must specify it in the `opensearch.yml` file. If you don't specify this setting, the maximum size is unlimited. This value should be smaller than the `indices.breaker.fielddata.limit`. For more information, see [Field data circuit breaker]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/circuit-breaker/#field-data-circuit-breaker-settings). +- `cluster.remote_store.index.path.type` (String): The path strategy for the data stored in the remote store. This setting is effective only for remote-store-enabled clusters. This setting supports the following values: + - `fixed`: Stores the data in path structure `///`. + - `hashed_prefix`: Stores the data in path structure `hash()////`. + - `hashed_infix`: Stores the data in path structure `/hash()///`. + `shard-data-idenitifer` is characterized by the index_uuid, shard_id, kind of data (translog, segments), and type of data (data, metadata, lock_files). + Default is `fixed`. + +- `cluster.remote_store.index.path.hash_algorithm` (String): The hash function used to derive the hash value when `cluster.remote_store.index.path.type` is set to `hashed_prefix` or `hashed_infix`. This setting is effective only for remote-store-enabled clusters. This setting supports the following values: + - `fnv_1a_base64`: Uses the FNV1a hash function and generates a url-safe 20-bit base64-encoded hash value. + - `fnv_1a_composite_1`: Uses the FNV1a hash function and generates a custom encoded hash value that scales well with most remote store options. The FNV1a function generates 64-bit value. The custom encoding uses the most significant 6 bits to create a url-safe base64 character and the next 14 bits to create a binary string. Default is `fnv_1a_composite_1`. + +- `cluster.remote_store.translog.transfer_timeout` (Time unit): Controls the timeout value while uploading translog and checkpoint files during a sync to the remote store. This setting is applicable only for remote-store-enabled clusters. Default is `30s`. + +- `cluster.remote_store.index.segment_metadata.retention.max_count` (Integer): Controls the minimum number of metadata files to keep in the segment repository on a remote store. A value below `1` disables the deletion of stale segment metadata files. Default is `10`. + + ## Index-level index settings You can specify index settings at index creation. There are two types of index settings: From 203c22acf74b5585620287c61b00ce29d0fbd3c7 Mon Sep 17 00:00:00 2001 From: Shubh Sahu Date: Tue, 7 May 2024 23:27:33 +0530 Subject: [PATCH 6/6] =?UTF-8?q?Adding=20document=20for=20setting=20introdu?= =?UTF-8?q?ced=20:=20cluster.remote=5Fstore.translo=E2=80=A6=20(#7078)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Adding document for setting introduced : cluster.remote_store.translog.max_readers Signed-off-by: Shubh Sahu * resolving vale error Signed-off-by: Shubh Sahu * Addressed comment on PR Signed-off-by: Shubh Sahu * Minor correction Signed-off-by: Shubh Sahu * Update _tuning-your-cluster/availability-and-recovery/remote-store/index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --------- Signed-off-by: Shubh Sahu Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Shubh Sahu Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../availability-and-recovery/remote-store/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/index.md b/_tuning-your-cluster/availability-and-recovery/remote-store/index.md index 5fd19f5cc2..d5dc99d5fe 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/index.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/index.md @@ -67,7 +67,7 @@ You can use the following [cluster settings]({{site.url}}{{site.baseurl}}//api-r | cluster.default.index.refresh_interval | Time unit | Sets the refresh interval when the `index.refresh_interval` setting is not provided. This setting can be useful when you want to set a default refresh interval across all indexes in a cluster and also support the `searchIdle` setting. You cannot set the interval lower than the `cluster.minimum.index.refresh_interval` setting. | | cluster.minimum.index.refresh_interval | Time unit | Sets the minimum refresh interval and applies it to all indexes in the cluster. The `cluster.default.index.refresh_interval` setting should be higher than this setting's value. If, during index creation, the `index.refresh_interval` setting is lower than the minimum, index creation fails. | | cluster.remote_store.translog.buffer_interval | Time unit | The default value of the translog buffer interval used when performing periodic translog updates. This setting is only effective when the index setting `index.remote_store.translog.buffer_interval` is not present. | - +| cluster.remote_store.translog.max_readers | Integer | Sets the maximum number of open translog files for remote-backed indexes. This limits the total number of translog files per shard. After reaching this limit, the remote store flushes the translog files. Default is `1000`. The minimum required is `100`. | ## Restoring from a backup