diff --git a/.github/workflows/pr_checklist.yml b/.github/workflows/pr_checklist.yml index accce0f882..4130f5e2bd 100644 --- a/.github/workflows/pr_checklist.yml +++ b/.github/workflows/pr_checklist.yml @@ -1,7 +1,7 @@ name: PR Checklist on: - pull_request: + pull_request_target: types: [opened] permissions: diff --git a/.gitignore b/.gitignore index 446d1deda6..da3cf9d144 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ Gemfile.lock .idea *.iml .jekyll-cache +.project diff --git a/API_STYLE_GUIDE.md b/API_STYLE_GUIDE.md index 6dc40df017..a6e0551f17 100644 --- a/API_STYLE_GUIDE.md +++ b/API_STYLE_GUIDE.md @@ -115,7 +115,7 @@ Include a table with these columns: Field | Data type | Description :--- | :--- | :--- -#### Example request +## Example request Provide a sentence that describes what is shown in the example, followed by a cut-and-paste-ready API request in JSON format. Make sure that you test the request yourself in the Dashboards Dev Tools console to make sure it works. See the following examples. @@ -139,7 +139,7 @@ POST _reindex } ``` -#### Example response +## Example response Include a JSON example response to show what the API returns. See the following examples. diff --git a/_about/version-history.md b/_about/version-history.md index 0d6d844951..09f331b235 100644 --- a/_about/version-history.md +++ b/_about/version-history.md @@ -30,6 +30,7 @@ OpenSearch version | Release highlights | Release date [2.0.1](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.1.md) | Includes bug fixes and maintenance updates for Alerting and Anomaly Detection. | 16 June 2022 [2.0.0](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.0.md) | Includes document-level monitors for alerting, OpenSearch Notifications plugins, and Geo Map Tiles in OpenSearch Dashboards. Also adds support for Lucene 9 and bug fixes for all OpenSearch plugins. For a full list of release highlights, see the Release Notes. | 26 May 2022 [2.0.0-rc1](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-2.0.0-rc1.md) | The Release Candidate for 2.0.0. This version allows you to preview the upcoming 2.0.0 release before the GA release. The preview release adds document-level alerting, support for Lucene 9, and the ability to use term lookup queries in document level security. | 03 May 2022 +[1.3.18](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.18.md) | Includes maintenance updates for OpenSearch security. | 16 July 2024 [1.3.17](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.17.md) | Includes maintenance updates for OpenSearch security and OpenSearch Dashboards security. | 06 June 2024 [1.3.16](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.16.md) | Includes bug fixes and maintenance updates for OpenSearch security, index management, performance analyzer, and reporting. | 23 April 2024 [1.3.15](https://github.com/opensearch-project/opensearch-build/blob/main/release-notes/opensearch-release-notes-1.3.15.md) | Includes bug fixes and maintenance updates for cross-cluster replication, SQL, OpenSearch Dashboards reporting, and alerting. | 05 March 2024 diff --git a/_aggregations/bucket/adjacency-matrix.md b/_aggregations/bucket/adjacency-matrix.md index fd521f8510..cf62295763 100644 --- a/_aggregations/bucket/adjacency-matrix.md +++ b/_aggregations/bucket/adjacency-matrix.md @@ -2,7 +2,6 @@ layout: default title: Adjacency matrix parent: Bucket aggregations -grand_parent: Aggregations nav_order: 10 redirect_from: - /query-dsl/aggregations/bucket/adjacency-matrix/ diff --git a/_aggregations/bucket/date-histogram.md b/_aggregations/bucket/date-histogram.md index e308104e16..db3fe6884e 100644 --- a/_aggregations/bucket/date-histogram.md +++ b/_aggregations/bucket/date-histogram.md @@ -2,7 +2,6 @@ layout: default title: Date histogram parent: Bucket aggregations -grand_parent: Aggregations nav_order: 20 redirect_from: - /query-dsl/aggregations/bucket/date-histogram/ diff --git a/_aggregations/bucket/date-range.md b/_aggregations/bucket/date-range.md index c7d66d729d..d2498a53da 100644 --- a/_aggregations/bucket/date-range.md +++ b/_aggregations/bucket/date-range.md @@ -2,7 +2,6 @@ layout: default title: Date range parent: Bucket aggregations -grand_parent: Aggregations nav_order: 30 redirect_from: - /query-dsl/aggregations/bucket/date-range/ diff --git a/_aggregations/bucket/diversified-sampler.md b/_aggregations/bucket/diversified-sampler.md index 7249ac3555..a62410cc8c 100644 --- a/_aggregations/bucket/diversified-sampler.md +++ b/_aggregations/bucket/diversified-sampler.md @@ -2,7 +2,6 @@ layout: default title: Diversified sampler parent: Bucket aggregations -grand_parent: Aggregations nav_order: 40 redirect_from: - /query-dsl/aggregations/bucket/diversified-sampler/ diff --git a/_aggregations/bucket/filter.md b/_aggregations/bucket/filter.md index 0768ea1148..58624c222c 100644 --- a/_aggregations/bucket/filter.md +++ b/_aggregations/bucket/filter.md @@ -2,7 +2,6 @@ layout: default title: Filter parent: Bucket aggregations -grand_parent: Aggregations nav_order: 50 redirect_from: - /query-dsl/aggregations/bucket/filter/ diff --git a/_aggregations/bucket/filters.md b/_aggregations/bucket/filters.md index b3977da7c1..2e9270b30d 100644 --- a/_aggregations/bucket/filters.md +++ b/_aggregations/bucket/filters.md @@ -2,7 +2,6 @@ layout: default title: Filters parent: Bucket aggregations -grand_parent: Aggregations nav_order: 60 redirect_from: - /query-dsl/aggregations/bucket/filters/ diff --git a/_aggregations/bucket/geo-distance.md b/_aggregations/bucket/geo-distance.md index a111015ac1..7b8e660630 100644 --- a/_aggregations/bucket/geo-distance.md +++ b/_aggregations/bucket/geo-distance.md @@ -2,7 +2,6 @@ layout: default title: Geodistance parent: Bucket aggregations -grand_parent: Aggregations nav_order: 70 redirect_from: - /query-dsl/aggregations/bucket/geo-distance/ diff --git a/_aggregations/bucket/geohash-grid.md b/_aggregations/bucket/geohash-grid.md index 13f89799ba..3969ea9a13 100644 --- a/_aggregations/bucket/geohash-grid.md +++ b/_aggregations/bucket/geohash-grid.md @@ -2,7 +2,6 @@ layout: default title: Geohash grid parent: Bucket aggregations -grand_parent: Aggregations nav_order: 80 redirect_from: - /query-dsl/aggregations/bucket/geohash-grid/ diff --git a/_aggregations/bucket/geohex-grid.md b/_aggregations/bucket/geohex-grid.md index 03fd45e369..eef2ed0b21 100644 --- a/_aggregations/bucket/geohex-grid.md +++ b/_aggregations/bucket/geohex-grid.md @@ -2,7 +2,6 @@ layout: default title: Geohex grid parent: Bucket aggregations -grand_parent: Aggregations nav_order: 85 redirect_from: - /opensearch/geohexgrid-agg/ diff --git a/_aggregations/bucket/geotile-grid.md b/_aggregations/bucket/geotile-grid.md index dd0c4f8a1f..e8e80451f5 100644 --- a/_aggregations/bucket/geotile-grid.md +++ b/_aggregations/bucket/geotile-grid.md @@ -2,7 +2,6 @@ layout: default title: Geotile grid parent: Bucket aggregations -grand_parent: Aggregations nav_order: 87 redirect_from: - /query-dsl/aggregations/bucket/geotile-grid/ diff --git a/_aggregations/bucket/global.md b/_aggregations/bucket/global.md index bfd516b8a3..483c28a69e 100644 --- a/_aggregations/bucket/global.md +++ b/_aggregations/bucket/global.md @@ -2,7 +2,6 @@ layout: default title: Global parent: Bucket aggregations -grand_parent: Aggregations nav_order: 90 redirect_from: - /query-dsl/aggregations/bucket/global/ diff --git a/_aggregations/bucket/histogram.md b/_aggregations/bucket/histogram.md index 0d9f2bb964..b97755bcae 100644 --- a/_aggregations/bucket/histogram.md +++ b/_aggregations/bucket/histogram.md @@ -2,7 +2,6 @@ layout: default title: Histogram parent: Bucket aggregations -grand_parent: Aggregations nav_order: 100 redirect_from: - /query-dsl/aggregations/bucket/histogram/ diff --git a/_aggregations/bucket/ip-range.md b/_aggregations/bucket/ip-range.md index 897827d412..bef4eb8f1f 100644 --- a/_aggregations/bucket/ip-range.md +++ b/_aggregations/bucket/ip-range.md @@ -2,7 +2,6 @@ layout: default title: IP range parent: Bucket aggregations -grand_parent: Aggregations nav_order: 110 redirect_from: - /query-dsl/aggregations/bucket/ip-range/ diff --git a/_aggregations/bucket/missing.md b/_aggregations/bucket/missing.md index 547076859d..e9bd5981a1 100644 --- a/_aggregations/bucket/missing.md +++ b/_aggregations/bucket/missing.md @@ -2,7 +2,6 @@ layout: default title: Missing parent: Bucket aggregations -grand_parent: Aggregations nav_order: 120 redirect_from: - /query-dsl/aggregations/bucket/missing/ diff --git a/_aggregations/bucket/multi-terms.md b/_aggregations/bucket/multi-terms.md index 62a4d264e0..8c99a450fe 100644 --- a/_aggregations/bucket/multi-terms.md +++ b/_aggregations/bucket/multi-terms.md @@ -2,7 +2,6 @@ layout: default title: Multi-terms parent: Bucket aggregations -grand_parent: Aggregations nav_order: 130 redirect_from: - /query-dsl/aggregations/bucket/multi-terms/ diff --git a/_aggregations/bucket/nested.md b/_aggregations/bucket/nested.md index 94a0f4416a..89c44c6457 100644 --- a/_aggregations/bucket/nested.md +++ b/_aggregations/bucket/nested.md @@ -2,7 +2,6 @@ layout: default title: Nested parent: Bucket aggregations -grand_parent: Aggregations nav_order: 140 redirect_from: - /query-dsl/aggregations/bucket/nested/ diff --git a/_aggregations/bucket/range.md b/_aggregations/bucket/range.md index f4e19f188d..7b17b64ed4 100644 --- a/_aggregations/bucket/range.md +++ b/_aggregations/bucket/range.md @@ -2,7 +2,6 @@ layout: default title: Range parent: Bucket aggregations -grand_parent: Aggregations nav_order: 150 redirect_from: - /query-dsl/aggregations/bucket/date-range/ diff --git a/_aggregations/bucket/reverse-nested.md b/_aggregations/bucket/reverse-nested.md index bfd04986fa..3757ec7ab7 100644 --- a/_aggregations/bucket/reverse-nested.md +++ b/_aggregations/bucket/reverse-nested.md @@ -2,7 +2,6 @@ layout: default title: Reverse nested parent: Bucket aggregations -grand_parent: Aggregations nav_order: 160 redirect_from: - /query-dsl/aggregations/bucket/reverse-nested/ diff --git a/_aggregations/bucket/sampler.md b/_aggregations/bucket/sampler.md index 5411052d45..7d8d71e04a 100644 --- a/_aggregations/bucket/sampler.md +++ b/_aggregations/bucket/sampler.md @@ -2,7 +2,6 @@ layout: default title: Sampler parent: Bucket aggregations -grand_parent: Aggregations nav_order: 170 redirect_from: - /query-dsl/aggregations/bucket/diversified-sampler/ diff --git a/_aggregations/bucket/significant-terms.md b/_aggregations/bucket/significant-terms.md index 34a4354a73..c255379dd8 100644 --- a/_aggregations/bucket/significant-terms.md +++ b/_aggregations/bucket/significant-terms.md @@ -2,7 +2,6 @@ layout: default title: Significant terms parent: Bucket aggregations -grand_parent: Aggregations nav_order: 180 redirect_from: - /query-dsl/aggregations/bucket/significant-terms/ diff --git a/_aggregations/bucket/significant-text.md b/_aggregations/bucket/significant-text.md index 6f1c7ebeca..b30b3aba95 100644 --- a/_aggregations/bucket/significant-text.md +++ b/_aggregations/bucket/significant-text.md @@ -2,7 +2,6 @@ layout: default title: Significant text parent: Bucket aggregations -grand_parent: Aggregations nav_order: 190 redirect_from: - /query-dsl/aggregations/bucket/significant-text/ diff --git a/_aggregations/bucket/terms.md b/_aggregations/bucket/terms.md index 5d05c328d4..b36214e3f6 100644 --- a/_aggregations/bucket/terms.md +++ b/_aggregations/bucket/terms.md @@ -2,7 +2,6 @@ layout: default title: Terms parent: Bucket aggregations -grand_parent: Aggregations nav_order: 200 redirect_from: - /query-dsl/aggregations/bucket/terms/ diff --git a/_aggregations/index.md b/_aggregations/index.md index 385c7a09d8..a1c457be02 100644 --- a/_aggregations/index.md +++ b/_aggregations/index.md @@ -1,7 +1,7 @@ --- layout: default title: Aggregations -has_children: true +has_children: false nav_order: 5 nav_exclude: true permalink: /aggregations/ diff --git a/_aggregations/metric/average.md b/_aggregations/metric/average.md index 428f1e76b6..9ad0c582fe 100644 --- a/_aggregations/metric/average.md +++ b/_aggregations/metric/average.md @@ -2,7 +2,6 @@ layout: default title: Average parent: Metric aggregations -grand_parent: Aggregations nav_order: 10 redirect_from: - /query-dsl/aggregations/metric/average/ diff --git a/_aggregations/metric/cardinality.md b/_aggregations/metric/cardinality.md index c40dbb4497..e03a561adb 100644 --- a/_aggregations/metric/cardinality.md +++ b/_aggregations/metric/cardinality.md @@ -2,7 +2,6 @@ layout: default title: Cardinality parent: Metric aggregations -grand_parent: Aggregations nav_order: 20 redirect_from: - /query-dsl/aggregations/metric/cardinality/ diff --git a/_aggregations/metric/extended-stats.md b/_aggregations/metric/extended-stats.md index 633407dab0..467fa348b7 100644 --- a/_aggregations/metric/extended-stats.md +++ b/_aggregations/metric/extended-stats.md @@ -2,7 +2,6 @@ layout: default title: Extended stats parent: Metric aggregations -grand_parent: Aggregations nav_order: 30 redirect_from: - /query-dsl/aggregations/metric/extended-stats/ diff --git a/_aggregations/metric/geobounds.md b/_aggregations/metric/geobounds.md index 27b7646ca5..9489c6b18e 100644 --- a/_aggregations/metric/geobounds.md +++ b/_aggregations/metric/geobounds.md @@ -2,7 +2,6 @@ layout: default title: Geobounds parent: Metric aggregations -grand_parent: Aggregations nav_order: 40 redirect_from: - /query-dsl/aggregations/metric/geobounds/ diff --git a/_aggregations/metric/geocentroid.md b/_aggregations/metric/geocentroid.md index 711f49862a..14a2d179bb 100644 --- a/_aggregations/metric/geocentroid.md +++ b/_aggregations/metric/geocentroid.md @@ -2,7 +2,6 @@ layout: default title: Geocentroid parent: Metric aggregations -grand_parent: Aggregations nav_order: 45 --- diff --git a/_aggregations/metric/matrix-stats.md b/_aggregations/metric/matrix-stats.md index 475e0caa24..188f8745fb 100644 --- a/_aggregations/metric/matrix-stats.md +++ b/_aggregations/metric/matrix-stats.md @@ -2,7 +2,6 @@ layout: default title: Matrix stats parent: Metric aggregations -grand_parent: Aggregations nav_order: 50 redirect_from: - /query-dsl/aggregations/metric/matrix-stats/ diff --git a/_aggregations/metric/maximum.md b/_aggregations/metric/maximum.md index 63b4d62a7b..1a1aaff607 100644 --- a/_aggregations/metric/maximum.md +++ b/_aggregations/metric/maximum.md @@ -2,7 +2,6 @@ layout: default title: Maximum parent: Metric aggregations -grand_parent: Aggregations nav_order: 60 redirect_from: - /query-dsl/aggregations/metric/maximum/ diff --git a/_aggregations/metric/median-absolute-deviation.md b/_aggregations/metric/median-absolute-deviation.md index 7332d7eb2f..a882475158 100644 --- a/_aggregations/metric/median-absolute-deviation.md +++ b/_aggregations/metric/median-absolute-deviation.md @@ -2,7 +2,6 @@ layout: default title: Median absolute deviation parent: Metric aggregations -grand_parent: Aggregations nav_order: 65 redirect_from: - /query-dsl/aggregations/metric/median-absolute-deviation/ diff --git a/_aggregations/metric/minimum.md b/_aggregations/metric/minimum.md index dd17c854a9..9455c71fea 100644 --- a/_aggregations/metric/minimum.md +++ b/_aggregations/metric/minimum.md @@ -2,7 +2,6 @@ layout: default title: Minimum parent: Metric aggregations -grand_parent: Aggregations nav_order: 70 redirect_from: - /query-dsl/aggregations/metric/minimum/ diff --git a/_aggregations/metric/percentile-ranks.md b/_aggregations/metric/percentile-ranks.md index 33ccb3d291..660cb01bd1 100644 --- a/_aggregations/metric/percentile-ranks.md +++ b/_aggregations/metric/percentile-ranks.md @@ -2,7 +2,6 @@ layout: default title: Percentile ranks parent: Metric aggregations -grand_parent: Aggregations nav_order: 80 redirect_from: - /query-dsl/aggregations/metric/percentile-ranks/ diff --git a/_aggregations/metric/percentile.md b/_aggregations/metric/percentile.md index c68b0e0ec7..0f241306d1 100644 --- a/_aggregations/metric/percentile.md +++ b/_aggregations/metric/percentile.md @@ -2,7 +2,6 @@ layout: default title: Percentile parent: Metric aggregations -grand_parent: Aggregations nav_order: 90 redirect_from: - /query-dsl/aggregations/metric/percentile/ diff --git a/_aggregations/metric/scripted-metric.md b/_aggregations/metric/scripted-metric.md index d1807efbc0..4247f9aa0e 100644 --- a/_aggregations/metric/scripted-metric.md +++ b/_aggregations/metric/scripted-metric.md @@ -2,7 +2,6 @@ layout: default title: Scripted metric parent: Metric aggregations -grand_parent: Aggregations nav_order: 100 redirect_from: - /query-dsl/aggregations/metric/scripted-metric/ diff --git a/_aggregations/metric/stats.md b/_aggregations/metric/stats.md index 0a54831522..d8ba5963e0 100644 --- a/_aggregations/metric/stats.md +++ b/_aggregations/metric/stats.md @@ -2,7 +2,6 @@ layout: default title: Stats parent: Metric aggregations -grand_parent: Aggregations nav_order: 110 redirect_from: - /query-dsl/aggregations/metric/stats/ diff --git a/_aggregations/metric/sum.md b/_aggregations/metric/sum.md index 0320de63fc..2e0b32cb3d 100644 --- a/_aggregations/metric/sum.md +++ b/_aggregations/metric/sum.md @@ -2,7 +2,6 @@ layout: default title: Sum parent: Metric aggregations -grand_parent: Aggregations nav_order: 120 redirect_from: - /query-dsl/aggregations/metric/sum/ diff --git a/_aggregations/metric/top-hits.md b/_aggregations/metric/top-hits.md index b6752300b2..cead3f77f2 100644 --- a/_aggregations/metric/top-hits.md +++ b/_aggregations/metric/top-hits.md @@ -2,7 +2,6 @@ layout: default title: Top hits parent: Metric aggregations -grand_parent: Aggregations nav_order: 130 redirect_from: - /query-dsl/aggregations/metric/top-hits/ diff --git a/_aggregations/metric/value-count.md b/_aggregations/metric/value-count.md index dfddaf9417..596fb5d806 100644 --- a/_aggregations/metric/value-count.md +++ b/_aggregations/metric/value-count.md @@ -2,7 +2,6 @@ layout: default title: Value count parent: Metric aggregations -grand_parent: Aggregations nav_order: 140 redirect_from: - /query-dsl/aggregations/metric/value-count/ diff --git a/_aggregations/metric/weighted-avg.md b/_aggregations/metric/weighted-avg.md index 268f78bfdc..6f67939d6e 100644 --- a/_aggregations/metric/weighted-avg.md +++ b/_aggregations/metric/weighted-avg.md @@ -2,7 +2,6 @@ layout: default title: Weighted average parent: Metric aggregations -grand_parent: Aggregations nav_order: 150 --- diff --git a/_api-reference/analyze-apis.md b/_api-reference/analyze-apis.md index 10af71c1ad..ac8e9e249f 100644 --- a/_api-reference/analyze-apis.md +++ b/_api-reference/analyze-apis.md @@ -61,7 +61,7 @@ Field | Data type | Description :--- | :--- | :--- text | String or Array of Strings | Text to analyze. If you provide an array of strings, the text is analyzed as a multi-value field. -#### Example requests +## Example requests [Analyze array of text strings](#analyze-array-of-text-strings) diff --git a/_api-reference/cat/cat-aliases.md b/_api-reference/cat/cat-aliases.md index b0c2d7184e..2d5c5c300a 100644 --- a/_api-reference/cat/cat-aliases.md +++ b/_api-reference/cat/cat-aliases.md @@ -15,26 +15,6 @@ has_children: false The CAT aliases operation lists the mapping of aliases to indexes, plus routing and filtering information. -## Example - -```json -GET _cat/aliases?v -``` -{% include copy-curl.html %} - -To limit the information to a specific alias, add the alias name after your query: - -```json -GET _cat/aliases/?v -``` -{% include copy-curl.html %} - -If you want to get information for more than one alias, separate the alias names with commas: - -```json -GET _cat/aliases/alias1,alias2,alias3 -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -55,7 +35,28 @@ Parameter | Type | Description local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. expand_wildcards | Enum | Expands wildcard expressions to concrete indexes. Combine multiple values with commas. Supported values are `all`, `open`, `closed`, `hidden`, and `none`. Default is `open`. -## Response +## Example requests + +```json +GET _cat/aliases?v +``` +{% include copy-curl.html %} + +To limit the information to a specific alias, add the alias name after your query: + +```json +GET _cat/aliases/?v +``` +{% include copy-curl.html %} + +If you want to get information for more than one alias, separate the alias names with commas: + +```json +GET _cat/aliases/alias1,alias2,alias3 +``` +{% include copy-curl.html %} + +## Example response The following response shows that `alias1` refers to a `movies` index and has a configured filter: diff --git a/_api-reference/cat/cat-allocation.md b/_api-reference/cat/cat-allocation.md index 23ebed79ff..085a755dc1 100644 --- a/_api-reference/cat/cat-allocation.md +++ b/_api-reference/cat/cat-allocation.md @@ -14,26 +14,6 @@ has_children: false The CAT allocation operation lists the allocation of disk space for indexes and the number of shards on each node. -## Example - -```json -GET _cat/allocation?v -``` -{% include copy-curl.html %} - -To limit the information to a specific node, add the node name after your query: - -```json -GET _cat/allocation/ -``` -{% include copy-curl.html %} - -If you want to get information for more than one node, separate the node names with commas: - -```json -GET _cat/allocation/node_name_1,node_name_2,node_name_3 -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -54,7 +34,28 @@ bytes | Byte size | Specify the units for byte size. For example, `7kb` or `6gb` local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. -## Response +## Example requests + +```json +GET _cat/allocation?v +``` +{% include copy-curl.html %} + +To limit the information to a specific node, add the node name after your query: + +```json +GET _cat/allocation/ +``` +{% include copy-curl.html %} + +If you want to get information for more than one node, separate the node names with commas: + +```json +GET _cat/allocation/node_name_1,node_name_2,node_name_3 +``` +{% include copy-curl.html %} + +## Example response The following response shows that eight shards are allocated to each of the two nodes available: diff --git a/_api-reference/cat/cat-cluster_manager.md b/_api-reference/cat/cat-cluster_manager.md index abf204ce16..d81e334009 100644 --- a/_api-reference/cat/cat-cluster_manager.md +++ b/_api-reference/cat/cat-cluster_manager.md @@ -14,12 +14,6 @@ has_children: false The CAT cluster manager operation lists information that helps identify the elected cluster manager node. -## Example - -``` -GET _cat/cluster_manager?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -37,7 +31,14 @@ Parameter | Type | Description :--- | :--- | :--- cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. -## Response +## Example requests + +``` +GET _cat/cluster_manager?v +``` +{% include copy-curl.html %} + +## Example response ```json id | host | ip | node diff --git a/_api-reference/cat/cat-count.md b/_api-reference/cat/cat-count.md index 34baa04dd4..8d0b4fbad2 100644 --- a/_api-reference/cat/cat-count.md +++ b/_api-reference/cat/cat-count.md @@ -15,7 +15,19 @@ redirect_from: The CAT count operation lists the number of documents in your cluster. -## Example + +## Path and HTTP methods + +``` +GET _cat/count?v +GET _cat/count/?v +``` + +## URL parameters + +All CAT count URL parameters are optional. You can specify any of the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index). + +## Example requests ```json GET _cat/count?v @@ -36,19 +48,7 @@ GET _cat/count/index_or_alias_1,index_or_alias_2,index_or_alias_3 ``` {% include copy-curl.html %} -## Path and HTTP methods - -``` -GET _cat/count?v -GET _cat/count/?v -``` - -## URL parameters - -All CAT count URL parameters are optional. You can specify any of the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index). - - -## Response +## Example response The following response shows the overall document count as 1625: diff --git a/_api-reference/cat/cat-field-data.md b/_api-reference/cat/cat-field-data.md index 6481e5cea1..05c720b952 100644 --- a/_api-reference/cat/cat-field-data.md +++ b/_api-reference/cat/cat-field-data.md @@ -8,13 +8,31 @@ redirect_from: - /opensearch/rest-api/cat/cat-field-data/ --- -# CAT fielddata +# CAT Field Data **Introduced 1.0** {: .label .label-purple } -The CAT fielddata operation lists the memory size used by each field per node. +The CAT Field Data operation lists the memory size used by each field per node. -## Example + +## Path and HTTP methods + +``` +GET _cat/fielddata?v +GET _cat/fielddata/?v +``` + +## URL parameters + +All CAT fielddata URL parameters are optional. + +In addition to the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index), you can specify the following parameter: + +Parameter | Type | Description +:--- | :--- | :--- +bytes | Byte size | Specify the units for byte size. For example, `7kb` or `6gb`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). + +## Example requests ```json GET _cat/fielddata?v @@ -35,24 +53,7 @@ GET _cat/fielddata/field_name_1,field_name_2,field_name_3 ``` {% include copy-curl.html %} -## Path and HTTP methods - -``` -GET _cat/fielddata?v -GET _cat/fielddata/?v -``` - -## URL parameters - -All CAT fielddata URL parameters are optional. - -In addition to the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index), you can specify the following parameter: - -Parameter | Type | Description -:--- | :--- | :--- -bytes | Byte size | Specify the units for byte size. For example, `7kb` or `6gb`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). - -## Response +## Example response The following response shows the memory size for all fields as 284 bytes: diff --git a/_api-reference/cat/cat-health.md b/_api-reference/cat/cat-health.md index 7767cfbc46..1c400916ad 100644 --- a/_api-reference/cat/cat-health.md +++ b/_api-reference/cat/cat-health.md @@ -15,12 +15,6 @@ redirect_from: The CAT health operation lists the status of the cluster, how long the cluster has been up, the number of nodes, and other useful information that helps you analyze the health of your cluster. -## Example - -```json -GET _cat/health?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -38,7 +32,15 @@ Parameter | Type | Description time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). ts | Boolean | If true, returns HH:MM:SS and Unix epoch timestamps. Default is `true`. -## Response +## Example request + +The following example request give cluster health information for the past 5 days: + +```json +GET _cat/health?v&time=5d +``` + +## Example response ```json GET _cat/health?v&time=5d diff --git a/_api-reference/cat/cat-indices.md b/_api-reference/cat/cat-indices.md index fe9556899e..16c57e5791 100644 --- a/_api-reference/cat/cat-indices.md +++ b/_api-reference/cat/cat-indices.md @@ -14,26 +14,6 @@ redirect_from: The CAT indices operation lists information related to indexes, that is, how much disk space they are using, how many shards they have, their health status, and so on. -## Example - -``` -GET _cat/indices?v -``` -{% include copy-curl.html %} - -To limit the information to a specific index, add the index name after your query. - -``` -GET _cat/indices/?v -``` -{% include copy-curl.html %} - -If you want to get information for more than one index, separate the indexes with commas: - -```json -GET _cat/indices/index1,index2,index3 -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -58,8 +38,29 @@ pri | Boolean | Whether to return information only from the primary shards. Defa time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). expand_wildcards | Enum | Expands wildcard expressions to concrete indexes. Combine multiple values with commas. Supported values are `all`, `open`, `closed`, `hidden`, and `none`. Default is `open`. +## Example requests + +``` +GET _cat/indices?v +``` +{% include copy-curl.html %} + +To limit the information to a specific index, add the index name after your query. + +``` +GET _cat/indices/?v +``` +{% include copy-curl.html %} + +If you want to get information for more than one index, separate the indexes with commas: + +```json +GET _cat/indices/index1,index2,index3 +``` +{% include copy-curl.html %} + -## Response +## Example response ```json health | status | index | uuid | pri | rep | docs.count | docs.deleted | store.size | pri.store.size diff --git a/_api-reference/cat/cat-nodeattrs.md b/_api-reference/cat/cat-nodeattrs.md index 6b4cc6d92e..b09e164698 100644 --- a/_api-reference/cat/cat-nodeattrs.md +++ b/_api-reference/cat/cat-nodeattrs.md @@ -14,12 +14,6 @@ redirect_from: The CAT nodeattrs operation lists the attributes of custom nodes. -## Example - -``` -GET _cat/nodeattrs?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -38,8 +32,17 @@ Parameter | Type | Description local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. +## Example request + +The following example request returns attributes about custom nodes: + +``` +GET _cat/nodeattrs?v +``` +{% include copy-curl.html %} + -## Response +## Example response ```json node | host | ip | attr | value diff --git a/_api-reference/cat/cat-nodes.md b/_api-reference/cat/cat-nodes.md index 864e5dfdd5..5e7238a0d0 100644 --- a/_api-reference/cat/cat-nodes.md +++ b/_api-reference/cat/cat-nodes.md @@ -16,12 +16,6 @@ The CAT nodes operation lists node-level information, including node roles and l A few important node metrics are `pid`, `name`, `cluster_manager`, `ip`, `port`, `version`, `build`, `jdk`, along with `disk`, `heap`, `ram`, and `file_desc`. -## Example - -``` -GET _cat/nodes?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -43,8 +37,17 @@ cluster_manager_timeout | Time | The amount of time to wait for a connection to time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). include_unloaded_segments | Boolean | Whether to include information from segments not loaded into memory. Default is `false`. +## Example request + +The following example request lists node level information: + +``` +GET _cat/nodes?v +``` +{% include copy-curl.html %} + -## Response +## Example response ```json ip | heap.percent | ram.percent | cpu load_1m | load_5m | load_15m | node.role | node.roles | cluster_manager | name diff --git a/_api-reference/cat/cat-pending-tasks.md b/_api-reference/cat/cat-pending-tasks.md index 748defd06e..ea224670ac 100644 --- a/_api-reference/cat/cat-pending-tasks.md +++ b/_api-reference/cat/cat-pending-tasks.md @@ -15,12 +15,6 @@ redirect_from: The CAT pending tasks operation lists the progress of all pending tasks, including task priority and time in queue. -## Example - -``` -GET _cat/pending_tasks?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -40,7 +34,16 @@ local | Boolean | Whether to return information from the local node only instead cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). -## Response +## Example request + +The following example request lists the progress of all pending node tasks: + +``` +GET _cat/pending_tasks?v +``` +{% include copy-curl.html %} + +## Example response ```json insertOrder | timeInQueue | priority | source diff --git a/_api-reference/cat/cat-plugins.md b/_api-reference/cat/cat-plugins.md index 519c77f27f..358eb70fbf 100644 --- a/_api-reference/cat/cat-plugins.md +++ b/_api-reference/cat/cat-plugins.md @@ -15,12 +15,6 @@ redirect_from: The CAT plugins operation lists the names, components, and versions of the installed plugins. -## Example - -``` -GET _cat/plugins?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -39,7 +33,16 @@ Parameter | Type | Description local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. -## Response +## Example requests + +The following example request lists all installed plugins: + +``` +GET _cat/plugins?v +``` +{% include copy-curl.html %} + +## Example response ```json name component version diff --git a/_api-reference/cat/cat-recovery.md b/_api-reference/cat/cat-recovery.md index da66aa7272..8f251a94e0 100644 --- a/_api-reference/cat/cat-recovery.md +++ b/_api-reference/cat/cat-recovery.md @@ -15,26 +15,6 @@ redirect_from: The CAT recovery operation lists all completed and ongoing index and shard recoveries. -## Example - -``` -GET _cat/recovery?v -``` -{% include copy-curl.html %} - -To see only the recoveries of a specific index, add the index name after your query. - -``` -GET _cat/recovery/?v -``` -{% include copy-curl.html %} - -If you want to get information for more than one index, separate the indexes with commas: - -```json -GET _cat/recovery/index1,index2,index3 -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -55,7 +35,28 @@ bytes | Byte size | Specify the units for byte size. For example, `7kb` or `6gb` detailed | Boolean | Whether to include detailed information about shard recoveries. Default is `false`. time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). -## Response +## Example requests + +``` +GET _cat/recovery?v +``` +{% include copy-curl.html %} + +To see only the recoveries of a specific index, add the index name after your query. + +``` +GET _cat/recovery/?v +``` +{% include copy-curl.html %} + +If you want to get information for more than one index, separate the indexes with commas: + +```json +GET _cat/recovery/index1,index2,index3 +``` +{% include copy-curl.html %} + +## Example response ```json index | shard | time | type | stage | source_host | source_node | target_host | target_node | repository | snapshot | files | files_recovered | files_percent | files_total | bytes | bytes_recovered | bytes_percent | bytes_total | translog_ops | translog_ops_recovered | translog_ops_percent diff --git a/_api-reference/cat/cat-repositories.md b/_api-reference/cat/cat-repositories.md index c6d62c9c62..f0fc4bb622 100644 --- a/_api-reference/cat/cat-repositories.md +++ b/_api-reference/cat/cat-repositories.md @@ -15,13 +15,6 @@ redirect_from: The CAT repositories operation lists all snapshot repositories for a cluster. -## Example - -``` -GET _cat/repositories?v -``` -{% include copy-curl.html %} - ## Path and HTTP methods ``` @@ -39,8 +32,17 @@ Parameter | Type | Description local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. +## Example request + +The following example request lists all snapshot repositories in the cluster: + +``` +GET _cat/repositories?v +``` +{% include copy-curl.html %} + -## Response +## Example response ```json id type diff --git a/_api-reference/cat/cat-segment-replication.md b/_api-reference/cat/cat-segment-replication.md index e22012ea66..5900b97a7c 100644 --- a/_api-reference/cat/cat-segment-replication.md +++ b/_api-reference/cat/cat-segment-replication.md @@ -47,11 +47,11 @@ Parameter | Data type | Description `v` | Boolean | If `true`, the response includes column headings. Defaults to `false`. `s` | String | Specifies to sort the results. For example, `s=shardId:desc` sorts by shardId in descending order. -## Example +## Example requests The following examples illustrate various segment replication responses. -#### Example 1: No active segment replication events +### No active segment replication events The following query requests segment replication metrics with column headings for all indexes: @@ -67,7 +67,7 @@ shardId target_node target_host checkpoints_behind bytes_behind current_lag last [index-1][0] runTask-1 127.0.0.1 0 0b 0s 7ms 0 ``` -#### Example 2: Shard ID specified +### Shard ID specified The following query requests segment replication metrics with column headings for shards with the ID `0` from indexes `index1` and `index2`: @@ -84,7 +84,7 @@ shardId target_node target_host checkpoints_behind bytes_behind current_lag last [index-2][0] runTask-1 127.0.0.1 0 0b 0s 5ms 0 ``` -#### Example 3: Detailed response +### Detailed response The following query requests detailed segment replication metrics with column headings for all indexes: @@ -101,7 +101,7 @@ shardId target_node target_host checkpoints_behind bytes_behind current_lag last [index-2][0] runTask-1 127.0.0.1 0 0b 0s 5ms 0 done 7ms 3 100.0% 3664 100.0% 2023-03-16T13:53:33.466Z 2023-03-16T13:53:33.474Z 3 3 3.5kb 3.5kb 0s 1ms 0s 2ms 2ms ``` -#### Example 4: Sorting the results +### Sorting the results The following query requests segment replication metrics with column headings for all indexes, sorted by shard ID in descending order: @@ -118,7 +118,7 @@ shardId target_node target_host checkpoints_behind bytes_behind current_lag [test6][0] runTask-2 127.0.0.1 0 0b 0s 4ms 0 ``` -#### Example 5: Using a metric alias +### Using a metric alias In a request, you can either use a metric's full name or one of its aliases. The following query is the same as the preceding query, but it uses the alias `s` instead of `shardID` for sorting: @@ -127,9 +127,9 @@ GET /_cat/segment_replication?v&s=s:desc ``` {% include copy-curl.html %} -## Response metrics +## Example response metrics -The following table lists the response metrics that are returned for all requests. When referring to a metric in a query parameter, you can provide either the metric's full name or any of its aliases, as shown in the previous [example](#example-5-using-a-metric-alias). +The following table lists the response metrics that are returned for all requests. When referring to a metric in a query parameter, you can provide either the metric's full name or any of its aliases, as shown in the previous [example](#using-a-metric-alias). Metric | Alias | Description :--- | :--- | :--- diff --git a/_api-reference/cat/cat-segments.md b/_api-reference/cat/cat-segments.md index b860486692..cd9eda38be 100644 --- a/_api-reference/cat/cat-segments.md +++ b/_api-reference/cat/cat-segments.md @@ -15,7 +15,25 @@ redirect_from: The cat segments operation lists Lucene segment-level information for each index. -## Example + +## Path and HTTP methods + +``` +GET _cat/segments +``` + +## URL parameters + +All CAT segments URL parameters are optional. + +In addition to the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index), you can specify the following parameters: + +Parameter | Type | Description +:--- | :--- | :--- +bytes | Byte size | Specify the units for byte size. For example, `7kb` or `6gb`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).. +cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. + +## Example requests ``` GET _cat/segments?v @@ -36,25 +54,7 @@ GET _cat/segments/index1,index2,index3 ``` {% include copy-curl.html %} -## Path and HTTP methods - -``` -GET _cat/segments -``` - -## URL parameters - -All CAT segments URL parameters are optional. - -In addition to the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index), you can specify the following parameters: - -Parameter | Type | Description -:--- | :--- | :--- -bytes | Byte size | Specify the units for byte size. For example, `7kb` or `6gb`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/).. -cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. - - -## Response +## Example response ```json index | shard | prirep | ip | segment | generation | docs.count | docs.deleted | size | size.memory | committed | searchable | version | compound diff --git a/_api-reference/cat/cat-shards.md b/_api-reference/cat/cat-shards.md index 9a727b5b11..b07f11aca3 100644 --- a/_api-reference/cat/cat-shards.md +++ b/_api-reference/cat/cat-shards.md @@ -15,26 +15,6 @@ redirect_from: The CAT shards operation lists the state of all primary and replica shards and how they are distributed. -## Example - -``` -GET _cat/shards?v -``` -{% include copy-curl.html %} - -To see only the information about shards of a specific index, add the index name after your query. - -``` -GET _cat/shards/?v -``` -{% include copy-curl.html %} - -If you want to get information for more than one index, separate the indexes with commas: - -``` -GET _cat/shards/index1,index2,index3 -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -55,8 +35,30 @@ local | Boolean | Whether to return information from the local node only instead cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). +## Example requests + +The following example requests returns information about shards: + +``` +GET _cat/shards?v +``` +{% include copy-curl.html %} + +To see only the information about shards of a specific index, add the index name after your query. + +``` +GET _cat/shards/?v +``` +{% include copy-curl.html %} + +If you want to get information for more than one index, separate the indexes with commas: + +``` +GET _cat/shards/index1,index2,index3 +``` +{% include copy-curl.html %} -## Response +## Example response ```json index | shard | prirep | state | docs | store | ip | | node diff --git a/_api-reference/cat/cat-snapshots.md b/_api-reference/cat/cat-snapshots.md index 82cb5c1b1f..2e1bd514bf 100644 --- a/_api-reference/cat/cat-snapshots.md +++ b/_api-reference/cat/cat-snapshots.md @@ -15,12 +15,6 @@ redirect_from: The CAT snapshots operation lists all snapshots for a repository. -## Example - -``` -GET _cat/snapshots?v -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -39,8 +33,17 @@ Parameter | Type | Description cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). +## Example request + +The following example request lists all snapshots: + +``` +GET _cat/snapshots?v +``` +{% include copy-curl.html %} + -## Response +## Example response ```json index | shard | prirep | state | docs | store | ip | | node diff --git a/_api-reference/cat/cat-tasks.md b/_api-reference/cat/cat-tasks.md index 4d2a06cced..7a71b592e7 100644 --- a/_api-reference/cat/cat-tasks.md +++ b/_api-reference/cat/cat-tasks.md @@ -15,13 +15,6 @@ redirect_from: The CAT tasks operation lists the progress of all tasks currently running on your cluster. -## Example - -``` -GET _cat/tasks?v -``` -{% include copy-curl.html %} - ## Path and HTTP methods ``` @@ -41,8 +34,17 @@ detailed | Boolean | Returns detailed task information. (Default: false) parent_task_id | String | Returns tasks with a specified parent task ID (node_id:task_number). Keep empty or set to -1 to return all. time | Time | Specify the units for time. For example, `5d` or `7h`. For more information, see [Supported units]({{site.url}}{{site.baseurl}}/opensearch/units/). +## Example request + +The following example request lists all tasks in progress: + +``` +GET _cat/tasks?v +``` +{% include copy-curl.html %} + -## Response +## Example response ```json action | task_id | parent_task_id | type | start_time | timestamp | running_time | ip | node diff --git a/_api-reference/cat/cat-templates.md b/_api-reference/cat/cat-templates.md index d7c7aac90f..ba47ae711d 100644 --- a/_api-reference/cat/cat-templates.md +++ b/_api-reference/cat/cat-templates.md @@ -15,19 +15,6 @@ redirect_from: The CAT templates operation lists the names, patterns, order numbers, and version numbers of index templates. -## Example - -``` -GET _cat/templates?v -``` -{% include copy-curl.html %} - -If you want to get information for a specific template or pattern: - -``` -GET _cat/templates/ -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -47,8 +34,24 @@ Parameter | Type | Description local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. +## Example requests + +The following example request returns information about all templates: + +``` +GET _cat/templates?v +``` +{% include copy-curl.html %} + +If you want to get information for a specific template or pattern: + +``` +GET _cat/templates/ +``` +{% include copy-curl.html %} + -## Response +## Example response ``` name | index_patterns order version composed_of diff --git a/_api-reference/cat/cat-thread-pool.md b/_api-reference/cat/cat-thread-pool.md index 491b523092..de24052175 100644 --- a/_api-reference/cat/cat-thread-pool.md +++ b/_api-reference/cat/cat-thread-pool.md @@ -14,7 +14,27 @@ redirect_from: The CAT thread pool operation lists the active, queued, and rejected threads of different thread pools on each node. -## Example + +## Path and HTTP methods + +``` +GET _cat/thread_pool +``` + +## URL parameters + +All CAT thread pool URL parameters are optional. + +In addition to the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index), you can specify the following parameters: + +Parameter | Type | Description +:--- | :--- | :--- +local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. +cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. + +## Example requests + +The following example request gives information about thread pools on all nodes: ``` GET _cat/thread_pool?v @@ -35,25 +55,8 @@ GET _cat/thread_pool/?v ``` {% include copy-curl.html %} -## Path and HTTP methods - -``` -GET _cat/thread_pool -``` - -## URL parameters - -All CAT thread pool URL parameters are optional. - -In addition to the [common URL parameters]({{site.url}}{{site.baseurl}}/api-reference/cat/index), you can specify the following parameters: - -Parameter | Type | Description -:--- | :--- | :--- -local | Boolean | Whether to return information from the local node only instead of from the cluster manager node. Default is `false`. -cluster_manager_timeout | Time | The amount of time to wait for a connection to the cluster manager node. Default is 30 seconds. - -## Response +## Example response ```json node_name name active queue rejected diff --git a/_api-reference/cluster-api/cluster-allocation.md b/_api-reference/cluster-api/cluster-allocation.md index b1b1c266d6..4ec6e27f2b 100644 --- a/_api-reference/cluster-api/cluster-allocation.md +++ b/_api-reference/cluster-api/cluster-allocation.md @@ -17,17 +17,6 @@ The most basic cluster allocation explain request finds an unassigned shard and If you add some options, you can instead get information on a specific shard, including why OpenSearch assigned it to its current node. -## Example - -```json -GET _cluster/allocation/explain?include_yes_decisions=true -{ - "index": "movies", - "shard": 0, - "primary": true -} -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -58,8 +47,20 @@ index | String | The name of the shard's index. primary | Boolean | Whether to provide an explanation for the primary shard (true) or its first replica (false), which share the same shard ID. shard | Integer | The shard ID that you want an explanation for. +## Example request + +```json +GET _cluster/allocation/explain?include_yes_decisions=true +{ + "index": "movies", + "shard": 0, + "primary": true +} +``` +{% include copy-curl.html %} + -## Response +## Example response ```json { diff --git a/_api-reference/cluster-api/cluster-decommission.md b/_api-reference/cluster-api/cluster-decommission.md index 867f58eda0..c707e5390a 100644 --- a/_api-reference/cluster-api/cluster-decommission.md +++ b/_api-reference/cluster-api/cluster-decommission.md @@ -54,7 +54,7 @@ DELETE /_cluster/decommission/awareness ``` {% include copy-curl.html %} -#### Response +#### Example response ```json @@ -74,7 +74,7 @@ GET /_cluster/decommission/awareness/zone/_status ``` {% include copy-curl.html %} -#### Response +#### Example response ```json { diff --git a/_api-reference/cluster-api/cluster-health.md b/_api-reference/cluster-api/cluster-health.md index 73c83d5ee6..7081a1a587 100644 --- a/_api-reference/cluster-api/cluster-health.md +++ b/_api-reference/cluster-api/cluster-health.md @@ -17,16 +17,6 @@ The most basic cluster health request returns a simple status of the health of y To get the status of a specific index, provide the index name. -## Example - -This request waits 50 seconds for the cluster to reach the yellow status or better: - -``` -GET _cluster/health?wait_for_status=yellow&timeout=50s -``` -{% include copy-curl.html %} - -If the cluster health becomes yellow or green before 50 seconds elapse, it returns a response immediately. Otherwise it returns a response as soon as it exceeds the timeout. ## Path and HTTP methods @@ -55,7 +45,18 @@ wait_for_no_initializing_shards | Boolean | Whether to wait until there are no i wait_for_status | Enum | Wait until the cluster health reaches the specified status or better. Supported values are `green`, `yellow`, and `red`. weights | JSON object | Assigns weights to attributes within the request body of the PUT request. Weights can be set in any ration, for example, 2:3:5. In a 2:3:5 ratio with three zones, for every 100 requests sent to the cluster, each zone would receive either 20, 30, or 50 search requests in a random order. When assigned a weight of `0`, the zone does not receive any search traffic. -#### Example request +## Example requests + +The following examples show how to use the cluster health API. + +This request waits 50 seconds for the cluster to reach the yellow status or better: + +``` +GET _cluster/health?wait_for_status=yellow&timeout=50s +``` +{% include copy-curl.html %} + +If the cluster health becomes yellow or green before 50 seconds elapse, it returns a response immediately. Otherwise it returns a response as soon as it exceeds the timeout. The following example request retrieves cluster health for all indexes in the cluster: @@ -64,7 +65,7 @@ GET _cluster/health ``` {% include copy-curl.html %} -#### Example response +## Example response The response contains cluster health information: diff --git a/_api-reference/cluster-api/cluster-settings.md b/_api-reference/cluster-api/cluster-settings.md index 3538339001..ec682ecdbd 100644 --- a/_api-reference/cluster-api/cluster-settings.md +++ b/_api-reference/cluster-api/cluster-settings.md @@ -32,25 +32,6 @@ include_defaults (GET only) | Boolean | Whether to include default settings as p cluster_manager_timeout | Time unit | The amount of time to wait for a response from the cluster manager node. Default is `30 seconds`. timeout (PUT only) | Time unit | The amount of time to wait for a response from the cluster. Default is `30 seconds`. - -#### Example request - -```json -GET _cluster/settings?include_defaults=true -``` -{% include copy-curl.html %} - -#### Example response - -```json -PUT _cluster/settings -{ - "persistent":{ - "action.auto_create_index": false - } -} -``` - ## Request fields The GET operation has no request body options. All cluster setting field parameters are optional. @@ -60,9 +41,24 @@ Not all cluster settings can be updated using the cluster settings API. You will For a listing of all cluster settings, see [Configuring OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/). -#### Example request -For a PUT operation, the request body must contain `transient` or `persistent`, along with the setting you want to update: +## Example requests + +The following example request show how to use the cluster settings API. + +### Check default cluster settings + +The following example request checks for default cluster settings: + +```json +GET _cluster/settings?include_defaults=true +``` +{% include copy-curl.html %} + +### Update cluster setting + +The following example updates the `cluster.max_shards_per_node` setting. For a PUT operation, the request body must contain `transient` or `persistent`, along with the setting you want to update: + ```json PUT _cluster/settings @@ -76,7 +72,9 @@ PUT _cluster/settings For more information about transient settings, persistent settings, and precedence, see [OpenSearch configuration]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/). -#### Example response +## Example response + +The following example response shows that the persistent cluster setting, `max_shard_per_node`, has been updated: ```json { diff --git a/_api-reference/cluster-api/cluster-stats.md b/_api-reference/cluster-api/cluster-stats.md index 8f8b585a6a..fb0ade2c6b 100644 --- a/_api-reference/cluster-api/cluster-stats.md +++ b/_api-reference/cluster-api/cluster-stats.md @@ -15,12 +15,6 @@ redirect_from: The cluster stats API operation returns statistics about your cluster. -## Example - -```json -GET _cluster/stats/nodes/_cluster_manager -``` -{% include copy-curl.html %} ## Path and HTTP methods @@ -41,7 +35,16 @@ Parameter | Type | Description Although the `master` node is now called `cluster_manager` for version 2.0, we retained the `master` field for backwards compatibility. If you have a node that has either a `master` role or a `cluster_manager` role, the `count` increases for both fields by 1. To see an example node count increase, see the Response sample. {: .note } -## Response +## Example request + +The following example requests returns information about the cluster manager node: + +```json +GET _cluster/stats/nodes/_cluster_manager +``` +{% include copy-curl.html %} + +## Example response ```json { diff --git a/_api-reference/document-apis/bulk.md b/_api-reference/document-apis/bulk.md index a869bad6dd..0475aa573d 100644 --- a/_api-reference/document-apis/bulk.md +++ b/_api-reference/document-apis/bulk.md @@ -134,7 +134,7 @@ All actions support the same metadata: `_index`, `_id`, and `_require_alias`. If { "script" : { "source": "ctx._source.title = \"World War Z\"" } } ``` -## Response +## Example response In the response, pay particular attention to the top-level `errors` boolean. If true, you can iterate over the individual actions for more detailed information. diff --git a/_api-reference/document-apis/delete-by-query.md b/_api-reference/document-apis/delete-by-query.md index 6f4104c254..64da909aad 100644 --- a/_api-reference/document-apis/delete-by-query.md +++ b/_api-reference/document-apis/delete-by-query.md @@ -88,7 +88,7 @@ To search your index for specific documents, you must include a [query]({{site.u } ``` -## Response +## Example response ```json { "took": 143, diff --git a/_api-reference/document-apis/delete-document.md b/_api-reference/document-apis/delete-document.md index c3dea2f7e1..ece99a28ca 100644 --- a/_api-reference/document-apis/delete-document.md +++ b/_api-reference/document-apis/delete-document.md @@ -42,7 +42,7 @@ version_type | Enum | Retrieves a specifically typed document. Available options wait_for_active_shards | String | The number of active shards that must be available before OpenSearch processes the delete request. Default is 1 (only the primary shard). Set to `all` or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the operation to succeed. | No -## Response +## Example response ```json { "_index": "sample-index1", diff --git a/_api-reference/document-apis/get-documents.md b/_api-reference/document-apis/get-documents.md index 3eaeb507d4..d493df136b 100644 --- a/_api-reference/document-apis/get-documents.md +++ b/_api-reference/document-apis/get-documents.md @@ -49,7 +49,7 @@ version | Integer | The version of the document to return, which must match the version_type | Enum | Retrieves a specifically typed document. Available options are `external` (retrieve the document if the specified version number is greater than the document's current version) and `external_gte` (retrieve the document if the specified version number is greater than or equal to the document's current version). For example, to retrieve version 3 of a document, use `/_doc/1?version=3&version_type=external`. -## Response +## Example response ```json { "_index": "sample-index1", diff --git a/_api-reference/document-apis/index-document.md b/_api-reference/document-apis/index-document.md index d131a2f50e..a506e2d9d8 100644 --- a/_api-reference/document-apis/index-document.md +++ b/_api-reference/document-apis/index-document.md @@ -110,7 +110,7 @@ Your request body must contain the information you want to index. } ``` -## Response +## Example response ```json { "_index": "sample-index", diff --git a/_api-reference/document-apis/multi-get.md b/_api-reference/document-apis/multi-get.md index 2d3246fa58..b267b8f3ac 100644 --- a/_api-reference/document-apis/multi-get.md +++ b/_api-reference/document-apis/multi-get.md @@ -54,7 +54,11 @@ _source.excludes | Array | Specifies which fields to exclude in the query respon ids | Array | IDs of the documents to retrieve. Only allowed when an index is specified in the URL. | No -#### Example without specifying index in URL +## Example requests + +### Specify an index in the request body + +The following example requests does specifies an index in the request body: ```json GET _mget @@ -76,7 +80,9 @@ GET _mget ``` {% include copy-curl.html %} -#### Example of specifying index in URL +### Specify an index the URL + +The following example specifies an index in the URL: ```json GET sample-index1/_mget @@ -95,7 +101,10 @@ GET sample-index1/_mget ``` {% include copy-curl.html %} -#### Example Response +## Example response + +The following example response returns information about multiple documents: + ```json { "docs": [ diff --git a/_api-reference/document-apis/reindex.md b/_api-reference/document-apis/reindex.md index 48f14923f5..8ac1c48be4 100644 --- a/_api-reference/document-apis/reindex.md +++ b/_api-reference/document-apis/reindex.md @@ -79,10 +79,9 @@ version_type | The indexing operation's version type. Valid values are `internal op_type | Whether to copy over documents that are missing in the destination index. Valid values are `create` (ignore documents with the same ID from the source index) and `index` (copy everything from the source index). pipeline | Which ingest pipeline to utilize during the reindex. script | A script that OpenSearch uses to apply transformations to the data during the reindex operation. -source | The actual script that OpenSearch runs. lang | The scripting language. Valid options are `painless`, `expression`, `mustache`, and `java`. -## Response +## Example response ```json { "took": 28829, diff --git a/_api-reference/document-apis/update-by-query.md b/_api-reference/document-apis/update-by-query.md index 217ae69550..09b3bd599f 100644 --- a/_api-reference/document-apis/update-by-query.md +++ b/_api-reference/document-apis/update-by-query.md @@ -102,7 +102,7 @@ To update your indexes and documents by query, you must include a [query]({{site } ``` -## Response +## Example response ```json { "took": 21, diff --git a/_api-reference/document-apis/update-document.md b/_api-reference/document-apis/update-document.md index 3da7030fa5..3f951b5adf 100644 --- a/_api-reference/document-apis/update-document.md +++ b/_api-reference/document-apis/update-document.md @@ -195,7 +195,7 @@ After the upsert operation, the document's `first_name` and `last_name` fields a } ``` -## Response +## Example response ```json { "_index": "sample-index1", diff --git a/_api-reference/index-apis/alias.md b/_api-reference/index-apis/alias.md index a38a3798a4..ebd7bdedfd 100644 --- a/_api-reference/index-apis/alias.md +++ b/_api-reference/index-apis/alias.md @@ -75,7 +75,7 @@ routing | String | Used to assign a custom value to a shard for specific operati index_routing | String | Assigns a custom value to a shard only for index operations. | No search_routing | String | Assigns a custom value to a shard only for search operations. | No -## Response +## Example response ```json { diff --git a/_api-reference/index-apis/clear-index-cache.md b/_api-reference/index-apis/clear-index-cache.md index 55a5ce85d5..9bf873301d 100644 --- a/_api-reference/index-apis/clear-index-cache.md +++ b/_api-reference/index-apis/clear-index-cache.md @@ -38,11 +38,11 @@ All query parameters are optional. | query | Boolean | If `true`, clears the query cache. Defaults to `true`. | | request | Boolean | If `true`, clears the request cache. Defaults to `true`. | -#### Example requests +## Example requests The following example requests show multiple clear cache API uses. -##### Clear a specific cache +### Clear a specific cache The following request clears the fields cache only: @@ -69,7 +69,7 @@ POST /my-index/_cache/clear?request=true ``` {% include copy-curl.html %} -#### Clear the cache for specific fields +### Clear the cache for specific fields The following request clears the fields caches of `fielda` and `fieldb`: @@ -78,7 +78,7 @@ POST /my-index/_cache/clear?fields=fielda,fieldb ``` {% include copy-curl.html %} -#### Clear caches for specific data streams or indexes +### Clear caches for specific data streams or indexes The following request clears the cache for two specific indexes: @@ -96,14 +96,14 @@ POST /_cache/clear ``` {% include copy-curl.html %} -#### Clear unused entries from the cache on search-capable nodes +### Clear unused entries from the cache on search-capable nodes ```json POST /*/_cache/clear?file=true ``` {% include copy-curl.html %} -#### Example response +## Example response The `POST /books,hockey/_cache/clear` request returns the following fields: diff --git a/_api-reference/index-apis/clone.md b/_api-reference/index-apis/clone.md index 60228b5894..c1496cbaf8 100644 --- a/_api-reference/index-apis/clone.md +++ b/_api-reference/index-apis/clone.md @@ -66,7 +66,25 @@ task_execution_timeout | Time | The explicit task execution timeout. Only useful The clone index API operation creates a new target index, so you can specify any [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) to apply to the target index. -## Response +## Example request + +```json +PUT /sample-index1/_clone/cloned-index1 +{ + "settings": { + "index": { + "number_of_shards": 2, + "number_of_replicas": 1 + } + }, + "aliases": { + "sample-alias1": {} + } +} +``` +{% include copy-curl.html %} + +## Example response ```json { diff --git a/_api-reference/index-apis/close-index.md b/_api-reference/index-apis/close-index.md index 7e43198d37..865d17d90a 100644 --- a/_api-reference/index-apis/close-index.md +++ b/_api-reference/index-apis/close-index.md @@ -41,7 +41,7 @@ cluster_manager_timeout | Time | How long to wait for a connection to the cluste timeout | Time | How long to wait for a response from the cluster. Default is `30s`. -## Response +## Example response ```json { "acknowledged": true, diff --git a/_api-reference/index-apis/create-index.md b/_api-reference/index-apis/create-index.md index 53d2dc28f9..ff5d7dbda5 100644 --- a/_api-reference/index-apis/create-index.md +++ b/_api-reference/index-apis/create-index.md @@ -52,7 +52,7 @@ timeout | Time | How long to wait for the request to return. Default is `30s`. As part of your request, you can optionally specify [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/), [mappings]({{site.url}}{{site.baseurl}}/field-types/index/), and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) for your newly created index. -#### Example request +## Example request ```json PUT /sample-index1 diff --git a/_api-reference/index-apis/delete-index.md b/_api-reference/index-apis/delete-index.md index 20e5c51c93..ad00eb7eca 100644 --- a/_api-reference/index-apis/delete-index.md +++ b/_api-reference/index-apis/delete-index.md @@ -38,7 +38,7 @@ cluster_manager_timeout | Time | How long to wait for a connection to the cluste timeout | Time | How long to wait for the response to return. Default is `30s`. -## Response +## Example response ```json { "acknowledged": true diff --git a/_api-reference/index-apis/exists.md b/_api-reference/index-apis/exists.md index 429ac40745..351e2f2088 100644 --- a/_api-reference/index-apis/exists.md +++ b/_api-reference/index-apis/exists.md @@ -40,6 +40,6 @@ ignore_unavailable | Boolean | If true, OpenSearch does not search for missing o local | Boolean | Whether to return information from only the local node instead of from the cluster manager node. Default is `false`. -## Response +## Example response The index exists API operation returns only one of two possible response codes: `200` -- the index exists, and `404` -- the index does not exist. diff --git a/_api-reference/index-apis/force-merge.md b/_api-reference/index-apis/force-merge.md index 6c2a61bef3..ce7501ebe3 100644 --- a/_api-reference/index-apis/force-merge.md +++ b/_api-reference/index-apis/force-merge.md @@ -74,42 +74,56 @@ The following table lists the available query parameters. All query parameters a | `only_expunge_deletes` | Boolean | If `true`, the merge operation only expunges segments containing a certain percentage of deleted documents. The percentage is 10% by default and is configurable in the `index.merge.policy.expunge_deletes_allowed` setting. Prior to OpenSearch 2.12, `only_expunge_deletes` ignored the `index.merge.policy.max_merged_segment` setting. Starting with OpenSearch 2.12, using `only_expunge_deletes` does not produce segments larger than `index.merge.policy.max_merged_segment` (by default, 5 GB). For more information, see [Deleted documents](#deleted-documents). Default is `false`. | | `primary_only` | Boolean | If set to `true`, then the merge operation is performed only on the primary shards of an index. This can be useful when you want to take a snapshot of the index after the merge is complete. Snapshots only copy segments from the primary shards. Merging the primary shards can reduce resource consumption. Default is `false`. | -#### Example request: Force merge a specific index +## Example requests + +The following examples show how to use the Force merge API. + +### Force merge a specific index + +The following example force merges a specific index: ```json POST /testindex1/_forcemerge ``` {% include copy-curl.html %} -#### Example request: Force merge multiple indexes +### Force merge multiple indexes + +The following example force merges multiple indexes: ```json POST /testindex1,testindex2/_forcemerge ``` {% include copy-curl.html %} -#### Example request: Force merge all indexes +### Force merge all indexes + +The following example force merges all indexes: ```json POST /_forcemerge ``` {% include copy-curl.html %} -#### Example request: Force merge a data stream's backing indexes into one segment +### Force merge a data stream's backing indexes into one segment + +The following example force merges a data stream's backing indexes into one segment: ```json POST /.testindex-logs/_forcemerge?max_num_segments=1 ``` {% include copy-curl.html %} -#### Example request: Force merge primary shards +### Force merge primary shards + +The following example force merges an index's primary shards: ```json POST /.testindex-logs/_forcemerge?primary_only=true ``` {% include copy-curl.html %} -#### Example response +## Example response ```json { diff --git a/_api-reference/index-apis/get-index.md b/_api-reference/index-apis/get-index.md index 733110d63a..e2d2d85c65 100644 --- a/_api-reference/index-apis/get-index.md +++ b/_api-reference/index-apis/get-index.md @@ -41,7 +41,7 @@ local | Boolean | Whether to return information from only the local node instead cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`. -## Response +## Example response ```json { "sample-index1": { diff --git a/_api-reference/index-apis/get-settings.md b/_api-reference/index-apis/get-settings.md index c41b25b4f5..94cb4a7c6c 100644 --- a/_api-reference/index-apis/get-settings.md +++ b/_api-reference/index-apis/get-settings.md @@ -45,7 +45,7 @@ ignore_unavailable | Boolean | If true, OpenSearch does not include missing or c local | Boolean | Whether to return information from the local node only instead of the cluster manager node. Default is `false`. cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`. -## Response +## Example response ```json { diff --git a/_api-reference/index-apis/open-index.md b/_api-reference/index-apis/open-index.md index 12381aa8c6..0d8ef62282 100644 --- a/_api-reference/index-apis/open-index.md +++ b/_api-reference/index-apis/open-index.md @@ -43,7 +43,7 @@ wait_for_completion | Boolean | When set to `false`, the request returns immedia task_execution_timeout | Time | The explicit task execution timeout. Only useful when wait_for_completion is set to `false`. Default is `1h`. -## Response +## Example response ```json { "acknowledged": true, diff --git a/_api-reference/index-apis/recover.md b/_api-reference/index-apis/recover.md new file mode 100644 index 0000000000..dc2df1e5a2 --- /dev/null +++ b/_api-reference/index-apis/recover.md @@ -0,0 +1,291 @@ +--- +layout: default +title: Recovery +parent: Index APIs +nav_order: 40 +--- + +# Recovery API +Introduced 1.0 +{: .label .label-purple } + +The Recovery API provides information about any completed or ongoing shard recoveries for one or more indexes. If a data stream is listed, the API returns information about that data stream's backing indexes. + +Shard recovery involves creating a shard copy to restore a primary shard from a snapshot or to synchronize a replica shard. After the shard recovery process completes, the recovered shard becomes available for use in search and index operations. + +Shard recovery occurs automatically in the following scenarios: + +- Node startup, known as a local store recovery +- Replication of a primary shard +- Relocation of a shard to a different node within the same cluster +- Restoration of a snapshot +- Clone, shrink, or split operations + +The Recovery API reports solely on completed recoveries for shard copies presently stored in the cluster. It reports only the most recent recovery for each shard copy and does not include historical information about previous recoveries or information about recoveries of shard copies that no longer exist. Consequently, if a shard copy completes a recovery and is subsequently relocated to a different node, then the information about the original recovery is not displayed in the Recovery API. + + +## Path and HTTP methods + +```json +GET /_recovery +GET //recovery/ +``` + +## Path parameters + +Parameter | Data type | Description +:--- | :--- +`index-name` | String | A comma-separated list of indexes, data streams, or index aliases to which the operation is applied. Supports wildcard expressions (`*`). Use `_all` or `*` to specify all indexes and data streams in a cluster. | + + +## Query parameters + +All of the following query parameters are optional. + +Parameter | Data type | Description +:--- | :--- | :--- +`active_only` | Boolean | When `true`, the response only includes active shard recoveries. Default is `false`. +`detailed` | Boolean | When `true`, provides detailed information about shard recoveries. Default is `false`. +`index` | String | A comma-separated list or wildcard expression of index names used to limit the request. + +## Response fields + +The API responds with the following information about the recovery shard. + +Parameter | Data type | Description +:--- | :--- | :--- +`id` | Integer | The ID of the shard. +`type` | String | The recovery source for the shard. Returned values include:
- `EMPTY_STORE`: An empty store. Indicates a new primary shard or the forced allocation of an empty primary shard using the Cluster Reroute API.
- `EXISTING_STORE`: The store of an existing primary shard. Indicates that the recovery is related to node startup or the allocation of an existing primary shard.
- `LOCAL_SHARDS`: Shards belonging to another index on the same node. Indicates that the recovery is related to a clone, shrink, or split operation.
- `PEER`: A primary shard on another node. Indicates that the recovery is related to shard replication.
- `SNAPSHOT`: A snapshot. Indicates that the recovery is related to a snapshot restore operation. +`STAGE` | String | The recovery stage. Returned values can include:
- `INIT`: Recovery has not started.
- `INDEX`: Reading index metadata and copying bytes from the source to the destination.
- `VERIFY_INDEX`: Verifying the integrity of the index.
- `TRANSLOG`: Replaying the transaction log.
- `FINALIZE`: Cleanup.
- `DONE`: Complete. +`primary` | Boolean | When `true`, the shard is a primary shard. +`start_time` | String | The timestamp indicating when the recovery started. +`stop_time` | String | The timestamp indicating when the recovery completed. +`total_time_in_millis` | String | The total amount of time taken to recover a shard, in milliseconds. +`source` | Object | The recovery source. This can include a description of the repository (if the recovery is from a snapshot) or a description of the source node. +`target` | Object | The destination node. +`index` | Object | Statistics about the physical index recovery. +`translog` | Object | Statistics about the translog recovery. + `start` | Object | Statistics about the amount of time taken to open and start the index. + +## Example requests + +The following examples demonstrate how to recover information using the Recovery API. + +### Recover information from several or all indexes + +The following example request returns recovery information about several indexes in a [human-readable format](https://opensearch.org/docs/latest/api-reference/common-parameters/#human-readable-output): + +```json +GET index1,index2/_recovery?human +``` +{% include copy-curl.html %} + +The following example request returns recovery information about all indexes in a human-readable format: + +```json +GET /_recovery?human +``` +{% include copy-curl.html %} + +### Recover detailed information + +The following example request returns detailed recovery information: + +```json +GET _recovery?human&detailed=true +``` +{% include copy-curl.html %} + +## Example response + +The following response returns detailed recovery information about an index named `shakespeare`: + +```json +{ + "shakespeare": { + "shards": [ + { + "id": 0, + "type": "EXISTING_STORE", + "stage": "DONE", + "primary": true, + "start_time": "2024-07-01T18:06:47.415Z", + "start_time_in_millis": 1719857207415, + "stop_time": "2024-07-01T18:06:47.538Z", + "stop_time_in_millis": 1719857207538, + "total_time": "123ms", + "total_time_in_millis": 123, + "source": { + "bootstrap_new_history_uuid": false + }, + "target": { + "id": "uerS7REgRQCbBF3ImY8wOQ", + "host": "172.18.0.3", + "transport_address": "172.18.0.3:9300", + "ip": "172.18.0.3", + "name": "opensearch-node2" + }, + "index": { + "size": { + "total": "17.8mb", + "total_in_bytes": 18708764, + "reused": "17.8mb", + "reused_in_bytes": 18708764, + "recovered": "0b", + "recovered_in_bytes": 0, + "percent": "100.0%" + }, + "files": { + "total": 7, + "reused": 7, + "recovered": 0, + "percent": "100.0%", + "details": [ + { + "name": "_1.cfs", + "length": "9.8mb", + "length_in_bytes": 10325945, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + }, + { + "name": "_0.cfe", + "length": "479b", + "length_in_bytes": 479, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + }, + { + "name": "_0.si", + "length": "333b", + "length_in_bytes": 333, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + }, + { + "name": "_1.cfe", + "length": "479b", + "length_in_bytes": 479, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + }, + { + "name": "_1.si", + "length": "333b", + "length_in_bytes": 333, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + }, + { + "name": "_0.cfs", + "length": "7.9mb", + "length_in_bytes": 8380790, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + }, + { + "name": "segments_3", + "length": "405b", + "length_in_bytes": 405, + "reused": true, + "recovered": "0b", + "recovered_in_bytes": 0 + } + ] + }, + "total_time": "6ms", + "total_time_in_millis": 6, + "source_throttle_time": "-1", + "source_throttle_time_in_millis": 0, + "target_throttle_time": "-1", + "target_throttle_time_in_millis": 0 + }, + "translog": { + "recovered": 0, + "total": 0, + "percent": "100.0%", + "total_on_start": 0, + "total_time": "113ms", + "total_time_in_millis": 113 + }, + "verify_index": { + "check_index_time": "0s", + "check_index_time_in_millis": 0, + "total_time": "0s", + "total_time_in_millis": 0 + } + }, + { + "id": 0, + "type": "PEER", + "stage": "DONE", + "primary": false, + "start_time": "2024-07-01T18:06:47.693Z", + "start_time_in_millis": 1719857207693, + "stop_time": "2024-07-01T18:06:47.744Z", + "stop_time_in_millis": 1719857207744, + "total_time": "50ms", + "total_time_in_millis": 50, + "source": { + "id": "uerS7REgRQCbBF3ImY8wOQ", + "host": "172.18.0.3", + "transport_address": "172.18.0.3:9300", + "ip": "172.18.0.3", + "name": "opensearch-node2" + }, + "target": { + "id": "HFYKietmTO6Ud9COgP0k9Q", + "host": "172.18.0.2", + "transport_address": "172.18.0.2:9300", + "ip": "172.18.0.2", + "name": "opensearch-node1" + }, + "index": { + "size": { + "total": "0b", + "total_in_bytes": 0, + "reused": "0b", + "reused_in_bytes": 0, + "recovered": "0b", + "recovered_in_bytes": 0, + "percent": "0.0%" + }, + "files": { + "total": 0, + "reused": 0, + "recovered": 0, + "percent": "0.0%", + "details": [] + }, + "total_time": "1ms", + "total_time_in_millis": 1, + "source_throttle_time": "-1", + "source_throttle_time_in_millis": 0, + "target_throttle_time": "-1", + "target_throttle_time_in_millis": 0 + }, + "translog": { + "recovered": 0, + "total": 0, + "percent": "100.0%", + "total_on_start": -1, + "total_time": "42ms", + "total_time_in_millis": 42 + }, + "verify_index": { + "check_index_time": "0s", + "check_index_time_in_millis": 0, + "total_time": "0s", + "total_time_in_millis": 0 + } + } + ] + } +} +``` diff --git a/_api-reference/index-apis/rollover.md b/_api-reference/index-apis/rollover.md new file mode 100644 index 0000000000..722dfe196c --- /dev/null +++ b/_api-reference/index-apis/rollover.md @@ -0,0 +1,195 @@ +--- +layout: default +title: Rollover Index +parent: Index APIs +nav_order: 63 +--- + +# Rollover Index +Introduced 1.0 +{: .label .label-purple } + +The Rollover Index API creates a new index for a data stream or index alias based on the `wait_for_active_shards` setting. + +## Path and HTTP methods + +```json +POST //_rollover/ +POST //_rollover/ +``` + +## Rollover types + +You can roll over a data stream, an index alias with one index, or an index alias with a write index. + +### Data stream + +When you perform a rollover operation on a data stream, the API generates a fresh write index for that stream. Simultaneously, the stream's preceding write index transforms into a regular backing index. Additionally, the rollover process increments the generation count of the data stream. Data stream rollovers do not support specifying index settings in the request body. + +### Index alias with one index + +When initiating a rollover on an index alias associated with a single index, the API generates a new index and disassociates the original index from the alias. + +### Index alias with a write index + +When an index alias references multiple indexes, one index must be designated as the write index. During a rollover, the API creates a new write index with its `is_write_index` property set to `true` while updating the previous write index by setting its `is_write_index property` to `false.` + +## Incrementing index names for an alias + +During the index alias rollover process, if you don't specify a custom name and the current index's name ends with a hyphen followed by a number (for example, `my-index-000001` or `my-index-3`), then the rollover operation will automatically increment that number for the new index's name. For instance, rolling over `my-index-000001` will generate `my-index-000002`. The numeric portion is always padded with leading zeros to ensure a consistent length of six characters. + +## Using date math with index rollovers + +When using an index alias for time-series data, you can leverage [date math](https://opensearch.org/docs/latest/field-types/supported-field-types/date/) in the index name to track the rollover date. For example, you can create an alias pointing to `my-index-{now/d}-000001`. If you create an alias on June 11, 2029, then the index name would be `my-index-2029.06.11-000001`. For a rollover on June 12, 2029, the new index would be named `my-index-2029.06.12-000002`. See [Roll over an index alias with a write index](#rolling-over-an-index-alias-with-a-write-index) for a practical example. + +## Path parameters + +The Rollover Index API supports the parameters listed in the following table. + +Parameter | Type | Description +:--- | :--- | :--- +`` | String | The name of the data stream or index alias to roll over. Required. | +`` | String | The name of the index to create. Supports date math. Data streams do not support this parameter. If the name of the alias's current write index does not end with `-` and a number, such as `my-index-000001` or `my-index-2`, then the parameter is required. + +## Query parameters + +The following table lists the supported query parameters. + +Parameter | Type | Description +:--- | :--- | :--- +`cluster_manager_timeout` | Time | The amount of time to wait for a connection to the cluster manager node. Default is `30s`. +`timeout` | Time | The amount of time to wait for a response. Default is `30s`. +`wait_for_active_shards` | String | The number of active shards that must be available before OpenSearch processes the request. Default is `1` (only the primary shard). You can also set to `all` or a positive integer. Values greater than `1` require replicas. For example, if you specify a value of `3`, then the index must have two replicas distributed across two additional nodes in order for the operation to succeed. + +## Request body + +The following request body parameters are supported. + +### `alias` + +The `alias` parameter specifies the alias name as the key. It is required when the `template` option exists in the request body. The object body contains the following optional parameters. + + +Parameter | Type | Description +:--- | :--- | :--- +`filter` | Query DSL object | The query that limits the number of documents that the alias can access. +`index_routing` | String | The value that routes indexing operations to a specific shard. When specified, overwrites the `routing` value for indexing operations. +`is_hidden` | Boolean | Hides or unhides the alias. When `true`, the alias is hidden. Default is `false`. Indexes for the alias must have matching values for this setting. +`is_write_index` | Boolean | Specifies the write index. When `true`, the index is the write index for the alias. Default is `false`. +`routing` | String | The value used to route index and search operations to a specific shard. +`search_routing` | String | Routes search operations to a specific shard. When specified, it overwrites `routing` for search operations. + +### `mappings` + +The `mappings` parameter specifies the index field mappings. It is optional. See [Mappings and field types](https://opensearch.org/docs/latest/field-types/) for more information. + +### `conditions` + +The `conditions` parameter is an optional object defining criteria for triggering the rollover. When provided, OpenSearch only rolls over if the current index satisfies one or more specified conditions. If omitted, then the rollover occurs unconditionally without prerequisites. + +The object body supports the following parameters. + +Parameter | Type | Description +:--- | :--- | :--- +| `max_age` | Time units | Triggers a rollover after the maximum elapsed time since index creation is reached. The elapsed time is always calculated since the index creation time, even if the index origination date is configured to a custom date, such as when using the `index.lifecycle.parse_origination_date` or `index.lifecycle.origination_date` settings. Optional. | +`max_docs` | Integer | Triggers a rollover after the specified maximum number of documents, excluding documents added since the last refresh and documents in replica shards. Optional. +`max_size` | Byte units | Triggers a rollover when the index reaches a specified size, calculated as the total size of all primary shards. Replicas are not counted. Use the `_cat indices` API and check the `pri.store.size` value to see the current index size. Optional. +`max_primary_shard_size` | Byte units | Triggers a rollover when the largest primary shard in the index reaches a certain size. This is the maximum size of the primary shards in the index. As with `max_size`, replicas are ignored. To see the current shard size, use the `_cat shards` API. The `store` value shows the size of each shard, and `prirep` indicates whether a shard is a primary (`p`) or a replica (`r`). Optional. + +### `settings` + +The `settings` parameter specifies the index configuration options. See [Index settings](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/) for more information. + +## Example requests + +The following examples illustrate using the Rollover Index API. A rollover occurs when one or more of the specified conditions are met: + +- The index was created 5 or more days ago. +- The index contains 500 or more documents. +- The index's largest primary shard is 100 GB or larger. + +### Rolling over a data stream + +The following request rolls over the data stream if the current write index meets any of the specified conditions: + +```json +POST my-data-stream/_rollover +{ + "conditions": { + "max_age": "5d", + "max_docs": 500, + "max_primary_shard_size": "100gb" + } +} +``` +{% include copy-curl.html %} + +### Rolling over an index alias with a write index + +The following request creates a date-time index and sets it as the write index for `my-alias`: + +```json +PUT +PUT %3Cmy-index-%7Bnow%2Fd%7D-000001%3E +{ + "aliases": { + "my-alias": { + "is_write_index": true + } + } +} +``` +{% include copy-curl.html %} + +The next request performs a rollover using the alias: + +```json +POST my-alias/_rollover +{ + "conditions": { + "max_age": "5d", + "max_docs": 500, + "max_primary_shard_size": "100gb" + } +} +``` +{% include copy-curl.html %} + +### Specifying settings during a rollover + +In most cases, you can use an index template to automatically configure the indexes created during a rollover operation. However, when rolling over an index alias, you can use the Rollover Index API to introduce additional index settings or override the settings defined in the template by sending the following request: + +```json +POST my-alias/_rollover +{ + "settings": { + "index.number_of_shards": 4 + } +} +``` +{% include copy-curl.html %} + + +## Example response + +OpenSearch returns the following response confirming that all conditions except `max_primary_shard_size` were met: + +```json +{ + "acknowledged": true, + "shards_acknowledged": true, + "old_index": ".ds-my-data-stream-2029.06.11-000001", + "new_index": ".ds-my-data-stream-2029.06.12-000002", + "rolled_over": true, + "dry_run": false, + "conditions": { + "[max_age: 5d]": true, + "[max_docs: 500]": true, + "[max_primary_shard_size: 100gb]": false + } +} +``` + + + + diff --git a/_api-reference/index-apis/segment.md b/_api-reference/index-apis/segment.md new file mode 100644 index 0000000000..a8a7ccaee1 --- /dev/null +++ b/_api-reference/index-apis/segment.md @@ -0,0 +1,122 @@ +--- +layout: default +title: Segment +parent: Index APIs +nav_order: 64 +--- + +# Segment +Introduced 1.0 +{: .label .label-purple } + +The Segment API provides details about the Lucene segments within index shards as well as information about the backing indexes of data streams. + + +## Path and HTTP methods + +```json +GET //_segments +GET /_segments +``` + +## Path parameters + +The following table lists the available path parameters. All path parameters are optional. + +Parameter | Data type | Description +:--- | :--- | :--- +`` | String | A comma-separated list of indexes, data streams, or index aliases to which the operation is applied. Supports wildcard expressions (`*`). Use `_all` or `*` to specify all indexes and data streams in a cluster. | + +## Query parameters + +The Segment API supports the following optional query parameters. + +Parameter | Data type | Description +:--- | :--- | :--- +`allow_no_indices` | Boolean | Whether to ignore wildcards that don't match any indexes. Default is `true`. +`allow_partial_search_results` | Boolean | Whether to return partial results if the request encounters an error or times out. Default is `true`. +`expand_wildcards` | String | Specifies the type of index that wildcard expressions can match. Supports comma-separated values. Valid values are `all` (match any index), `open` (match open, non-hidden indexes), `closed` (match closed, non-hidden indexes), `hidden` (match hidden indexes), and `none` (deny wildcard expressions). Default is `open`. +`ignore_unavailable` | Boolean | When `true`, OpenSearch ignores missing or closed indexes. If `false`, OpenSearch returns an error if the force merge operation encounters missing or closed indexes. Default is `false`. +`verbose` | Boolean | When `true`, provides information about Lucene's memory usage. Default is `false`. + +## Response body fields + +Parameter | Data type | Description + :--- | :--- | :--- +`` | String | The name of the segment used to create internal file names in the shard directory. +`generation` | Integer | The generation number, such as `0`, incremented for each written segment and used to name the segment. +`num_docs` | Integer | The number of documents, obtained from Lucene. Nested documents are counted separately from their parents. Deleted documents, as well as recently indexed documents that are not yet assigned to a segment, are excluded. +`deleted_docs` | Integer | The number of deleted documents, obtained from Lucene, which may not match the actual number of delete operations performed. Recently deleted documents that are not yet assigned to a segment are excluded. Deleted documents are automatically merged when appropriate. OpenSearch will occasionally delete extra documents in order to track recent shard operations. +`size_in_bytes` | Integer | The amount of disk space used by the segment, for example, `50kb`. +`memory_in_bytes` | Integer | The amount of segment data, measured in bytes, that is kept in memory to facilitate efficient search operations, such as `1264`. A value of `-1` indicates that OpenSearch was unable to compute this number. +`committed` | Boolean | When `true`, the segments are synced to disk. Segments synced to disk can survive a hard reboot. If `false`, then uncommitted segment data is stored in the transaction log as well so that changes can be replayed at the next startup. +`search` | Boolean | When `true`, segment search is enabled. When `false`, the segment may have already been written to disk and require a refresh in order to be searchable. +`version` | String | The Lucene version used to write the segment. +`compound` | Boolean | When `true`, indicates that Lucene merged all segment files into one file in order to save any file descriptions. +`attributes` | Object | Shows if high compression was enabled. + +## Example requests + +The following example requests show you how to use the Segment API. + +### Specific data stream or index + +```json +GET /index1/_segments +``` +{% include copy-curl.html %} + +### Several data streams and indexes + +```json +GET /index1,index2/_segments +``` +{% include copy-curl.html %} + +### All data streams and indexes in a cluster + +```json +GET /_segments +``` +{% include copy-curl.html %} + +## Example response + +```json +{ + "_shards": ... + "indices": { + "test": { + "shards": { + "0": [ + { + "routing": { + "state": "STARTED", + "primary": true, + "node": "zDC_RorJQCao9xf9pg3Fvw" + }, + "num_committed_segments": 0, + "num_search_segments": 1, + "segments": { + "_0": { + "generation": 0, + "num_docs": 1, + "deleted_docs": 0, + "size_in_bytes": 3800, + "memory_in_bytes": 1410, + "committed": false, + "search": true, + "version": "7.0.0", + "compound": true, + "attributes": { + } + } + } + } + ] + } + } + } +} +``` + diff --git a/_api-reference/index-apis/split.md b/_api-reference/index-apis/split.md index 03b2f742d1..ad13bffbba 100644 --- a/_api-reference/index-apis/split.md +++ b/_api-reference/index-apis/split.md @@ -66,7 +66,7 @@ task_execution_timeout | Time | The explicit task execution timeout. Only useful The split index API operation creates a new target index, so you can specify any [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) to apply to the target index. -## Response +## Example response ```json { diff --git a/_api-reference/index-apis/stats.md b/_api-reference/index-apis/stats.md index 8ccd624de1..7310298594 100644 --- a/_api-reference/index-apis/stats.md +++ b/_api-reference/index-apis/stats.md @@ -71,49 +71,65 @@ Parameter | Data type | Description `include_segment_file_sizes` | Boolean | Specifies whether to report the aggregated disk usage of each Lucene index file. Only applies to `segments` statistics. Default is `false`. `include_unloaded_segments` | Boolean | Specifies whether to include information from segments that are not loaded into memory. Default is `false`. -#### Example request: One index +## Example requests + +The following example requests show how to use the Index Stats API. + +### One index + +The following example returns index stats for a single index: ```json GET /testindex/_stats ``` {% include copy-curl.html %} -#### Example request: Comma-separated list of indexes +### Comma-separated list of indexes + +The following example returns stats for multiple indexes: ```json GET /testindex1,testindex2/_stats ``` {% include copy-curl.html %} -#### Example request: Wildcard expression +### Wildcard expression + +The following example returns starts about any index that starts with `testindex`: ```json GET /testindex*/_stats ``` {% include copy-curl.html %} -#### Example request: Specific stats +### Specific stats + +The following example returns index stats related to the index and flush operations: ```json GET /testindex/_stats/refresh,flush ``` {% include copy-curl.html %} -#### Example request: Expand wildcards +### Expand wildcards + +The following example expands all wildcards related to index stats: ```json GET /testindex*/_stats?expand_wildcards=open,hidden ``` {% include copy-curl.html %} -#### Example request: Shard-level statistics +### Shard-level statistics + +The following example returns shard level stats about a test index: ```json GET /testindex/_stats?level=shards ``` {% include copy-curl.html %} -#### Example response +## Example response By default, the returned statistics are aggregated in the `primaries` and `total` aggregations. The `primaries` aggregation contains statistics for the primary shards. The `total` aggregation contains statistics for both primary and replica shards. The following is an example Index Stats API response: diff --git a/_api-reference/index-apis/update-alias.md b/_api-reference/index-apis/update-alias.md new file mode 100644 index 0000000000..f32d34025e --- /dev/null +++ b/_api-reference/index-apis/update-alias.md @@ -0,0 +1,84 @@ +--- +layout: default +title: Create or update alias +parent: Index APIs +nav_order: 5 +--- + +# Create or Update Alias + +**Introduced 1.0** +{: .label .label-purple } + +The Create or Update Alias API adds a data stream or index to an alias or updates the settings for an existing alias. For more alias API operations, see [Index aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/). + +The Create or Update Alias API is distinct from the [Alias API]({{site.url}}{{site.baseurl}}/opensearch/rest-api/alias/), which supports the addition and removal of aliases and the removal of alias indexes. In contrast, the following API only supports adding or updating an alias without updating the index itself. Each API also uses different request body parameters. +{: .note} + +## Path and HTTP methods + +```json +POST //_alias/ +PUT //_alias/ +POST /_alias/ +PUT /_alias/ +POST //_aliases/ +PUT //_aliases/ +POST /_aliases/ +PUT /_aliases/ +PUT //_alias +PUT //_aliases +PUT /_alias +``` + +## Path parameters + +| Parameter | Type | Description | +:--- | :--- | :--- +| `target` | String | A comma-delimited list of data streams and indexes. Wildcard expressions (`*`) are supported. To target all data streams and indexes in a cluster, use `_all` or `*`. Optional. | +| `alias-name` | String | The alias name to be created or updated. Optional. | + +## Query parameters + +All query parameters are optional. + +Parameter | Type | Description +:--- | :--- | :--- +`cluster_manager_timeout` | Time | The amount of time to wait for a response from the cluster manager node. Default is `30s`. +`timeout` | Time | The amount of time to wait for a response from the cluster. Default is `30s`. + +## Request body + +In the request body, you can specify the index name, the alias name, and the settings for the alias. All fields are optional. + +Field | Type | Description +:--- | :--- | :--- | :--- +`index` | String | A comma-delimited list of data streams or indexes that you want to associate with the alias. If this field is set, it will override the index name specified in the URL path. +`alias` | String | The name of the alias. If this field is set, it will override the alias name specified in the URL path. +`is_write_index` | Boolean | Specifies whether the index should be a write index. An alias can only have one write index at a time. If a write request is submitted to an alias that links to multiple indexes, then OpenSearch runs the request only on the write index. +`routing` | String | Assigns a custom value to a shard for specific operations. +`index_routing` | String | Assigns a custom value to a shard only for index operations. +`search_routing` | String | Assigns a custom value to a shard only for search operations. +`filter` | Object | A filter to use with the alias so that the alias points to a filtered part of the index. + +## Example request + +The following example request adds a sample alias with a custom routing value: + +```json +POST sample-index/_alias/sample-alias +{ + "routing":"test" +} +``` +{% include copy-curl.html %} + +## Example response + +```json +{ + "acknowledged": true +} +``` + +For more alias API operations, see [Index aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/). diff --git a/_api-reference/nodes-apis/nodes-hot-threads.md b/_api-reference/nodes-apis/nodes-hot-threads.md index 3fb6ff65ea..f5e014dd6d 100644 --- a/_api-reference/nodes-apis/nodes-hot-threads.md +++ b/_api-reference/nodes-apis/nodes-hot-threads.md @@ -46,14 +46,14 @@ ignore_idle_threads | Boolean | Don’t show threads that are in known idle st type | String | Supported thread types are `cpu`, `wait`, or `block`. Defaults to `cpu`. timeout | Time | Sets the time limit for node response. Default value is `30s`. -#### Example request +## Example request ```json GET /_nodes/hot_threads ``` {% include copy-curl.html %} -#### Example response +## Example response ```bash ::: {opensearch}{F-ByTQzVQ3GQeYzQJArJGQ}{GxbcLdCATPWggOuQHJAoCw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true} @@ -65,7 +65,7 @@ GET /_nodes/hot_threads org.opensearch.performanceanalyzer.collectors.ScheduledMetricCollectorsExecutor.run(ScheduledMetricCollectorsExecutor.java:100) ``` -## Response +## Example response Unlike the majority of OpenSearch API responses, this response is in a text format. diff --git a/_api-reference/nodes-apis/nodes-info.md b/_api-reference/nodes-apis/nodes-info.md index d7c810410e..a8953505ff 100644 --- a/_api-reference/nodes-apis/nodes-info.md +++ b/_api-reference/nodes-apis/nodes-info.md @@ -79,7 +79,7 @@ Parameter | Type | Description flat_settings| Boolean | Specifies whether to return the `settings` object of the response in flat format. Default is `false`. timeout | Time | Sets the time limit for node response. Default value is `30s`. -#### Example request +## Example request The following query requests the `process` and `transport` metrics from the cluster manager node: @@ -88,7 +88,7 @@ GET /_nodes/cluster_manager:true/process,transport ``` {% include copy-curl.html %} -#### Example response +## Example response The response contains the metric groups specified in the `` request parameter (in this case, `process` and `transport`): diff --git a/_api-reference/nodes-apis/nodes-reload-secure.md b/_api-reference/nodes-apis/nodes-reload-secure.md index 52b2ef67ab..b4be66ddd4 100644 --- a/_api-reference/nodes-apis/nodes-reload-secure.md +++ b/_api-reference/nodes-apis/nodes-reload-secure.md @@ -36,7 +36,7 @@ The request may include an optional object containing the password for the OpenS } ``` -#### Example request +## Example request The following is an example API request: @@ -45,7 +45,7 @@ POST _nodes/reload_secure_settings ``` {% include copy-curl.html %} -#### Example response +## Example response The following is an example response: diff --git a/_api-reference/nodes-apis/nodes-stats.md b/_api-reference/nodes-apis/nodes-stats.md index 145b3d5b24..2ccea54390 100644 --- a/_api-reference/nodes-apis/nodes-stats.md +++ b/_api-reference/nodes-apis/nodes-stats.md @@ -44,7 +44,7 @@ thread_pool | Statistics about each thread pool for the node. fs | File system statistics, such as read/write statistics, data path, and free disk space. transport | Transport layer statistics about send/receive in cluster communication. http | Statistics about the HTTP layer. -breaker | Statistics about the field data circuit breakers. +breakers | Statistics about the field data circuit breakers. script | Statistics about scripts, such as compilations and cache evictions. discovery | Statistics about cluster states. ingest | Statistics about ingest pipelines. @@ -118,14 +118,14 @@ level | String | Specifies whether statistics for the `indices` metric are aggre timeout | Time | Sets the time limit for node response. Default is `30s`. include_segment_file_sizes | Boolean | If segment statistics are requested, this field specifies to return the aggregated disk usage of every Lucene index file. Default is `false`. -#### Example request +## Example request ```json GET _nodes/stats/ ``` {% include copy-curl.html %} -#### Example response +## Example response Select the arrow to view the example response. diff --git a/_api-reference/nodes-apis/nodes-usage.md b/_api-reference/nodes-apis/nodes-usage.md index 532ddb626b..355b7f8ff2 100644 --- a/_api-reference/nodes-apis/nodes-usage.md +++ b/_api-reference/nodes-apis/nodes-usage.md @@ -38,7 +38,7 @@ Parameter | Type | Description timeout | Time | Sets the time limit for a response from the node. Default is `30s`. cluster_manager_timeout | Time | Sets the time limit for a response from the cluster manager. Default is `30s`. -#### Example request +## Example request The following request returns usage details for all nodes: @@ -47,7 +47,7 @@ GET _nodes/usage ``` {% include copy-curl.html %} -#### Example response +## Example response The following is an example response: diff --git a/_api-reference/profile.md b/_api-reference/profile.md index 94c7857b80..4f8c69db9c 100644 --- a/_api-reference/profile.md +++ b/_api-reference/profile.md @@ -26,7 +26,7 @@ A slice is the unit of work that can be executed by a thread. Each query can be In general, the max/min/avg slice time captures statistics across all slices for a timing type. For example, when profiling aggregations, the `max_slice_time_in_nanos` field in the `aggregations` section shows the maximum time consumed by the aggregation operation and its children across all slices. -#### Example request: Non-concurrent search +## Example request: Non-concurrent search To use the Profile API, include the `profile` parameter set to `true` in the search request sent to the `_search` endpoint: diff --git a/_api-reference/rank-eval.md b/_api-reference/rank-eval.md index 04fd3cf5c0..881ff3a22b 100644 --- a/_api-reference/rank-eval.md +++ b/_api-reference/rank-eval.md @@ -45,7 +45,7 @@ ignore_unlabeled | Defaults to `false`. Unlabeled documents are ignored when set template_id | Template ID. params | Parameters used in the template. -#### Example request +## Example request ````json GET shakespeare/_rank_eval @@ -76,7 +76,7 @@ GET shakespeare/_rank_eval ```` {% include copy-curl.html %} -#### Example response +## Example response ````json { diff --git a/_api-reference/render-template.md b/_api-reference/render-template.md index 16bada0290..409fde5e4a 100644 --- a/_api-reference/render-template.md +++ b/_api-reference/render-template.md @@ -44,7 +44,7 @@ Both of the following request examples use the search template with the template "source": { "query": { "match": { - "play_name": "{{play_name}}" + "play_name": "{% raw %}{{play_name}}{% endraw %}" } } }, @@ -76,11 +76,11 @@ If you don't want to use a saved template, or want to test a template before sav ``` { "source": { - "from": "{{from}}{{^from}}10{{/from}}", - "size": "{{size}}{{^size}}10{{/size}}", + "from": "{% raw %}{{from}}{{^from}}0{{/from}}{% endraw %}", + "size": "{% raw %}{{size}}{{^size}}10{{/size}}{% endraw %}", "query": { "match": { - "play_name": "{{play_name}}" + "play_name": "{% raw %}{{play_name}}{% endraw %}" } } }, diff --git a/_api-reference/script-apis/create-stored-script.md b/_api-reference/script-apis/create-stored-script.md index 04a73a205a..0a915cd836 100644 --- a/_api-reference/script-apis/create-stored-script.md +++ b/_api-reference/script-apis/create-stored-script.md @@ -47,7 +47,7 @@ All parameters are optional. | lang | String | Scripting language. Required. | | source | String or Object | Required.

For scripts, a string with the contents of the script.

For search templates, an object that defines the search template. Supports the same parameters as the [Search]({{site.url}}{{site.baseurl}}/api-reference/search) API request body. Search templates also support Mustache variables. | -#### Example request +## Example request The sample uses an index called `books` with the following documents: @@ -117,7 +117,7 @@ PUT _scripts/my-first-script See [Execute Painless stored script]({{site.url}}{{site.baseurl}}/api-reference/script-apis/exec-stored-script/) for information about running the script. -#### Example response +## Example response The `PUT _scripts/my-first-script` request returns the following field: diff --git a/_api-reference/script-apis/delete-script.md b/_api-reference/script-apis/delete-script.md index 363b0152df..fe9c272acc 100644 --- a/_api-reference/script-apis/delete-script.md +++ b/_api-reference/script-apis/delete-script.md @@ -26,7 +26,7 @@ Path parameters are optional. | cluster_manager_timeout | Time | Amount of time to wait for a connection to the cluster manager. Optional, defaults to `30s`. | | timeout | Time | The period of time to wait for a response. If a response is not received before the timeout value, the request will be dropped. -#### Example request +## Example request The following request deletes the `my-first-script` script: @@ -35,7 +35,7 @@ DELETE _scripts/my-script ```` {% include copy-curl.html %} -#### Example response +## Example response The `DELETE _scripts/my-first-script` request returns the following field: diff --git a/_api-reference/script-apis/exec-script.md b/_api-reference/script-apis/exec-script.md index 4ecb6a37fc..b6476be980 100644 --- a/_api-reference/script-apis/exec-script.md +++ b/_api-reference/script-apis/exec-script.md @@ -26,7 +26,7 @@ POST /_scripts/painless/_execute | context | A context for the script. Optional. Default is `painless_test`. | | context_setup | Specifies additional parameters for the context. Optional.| -#### Example request +## Example request The following request uses the default `painless_context` for the script: @@ -44,7 +44,7 @@ GET /_scripts/painless/_execute ``` {% include copy-curl.html %} -#### Example response +## Example response The response contains the average of two script parameters: diff --git a/_api-reference/script-apis/exec-stored-script.md b/_api-reference/script-apis/exec-stored-script.md index 7525ec81a4..a7de3b5274 100644 --- a/_api-reference/script-apis/exec-stored-script.md +++ b/_api-reference/script-apis/exec-stored-script.md @@ -21,7 +21,7 @@ OpenSearch provides several ways to run a script; the following sections show ho | script_fields | Object | Fields to include in output. | | script | Object | ID of the script that produces a value for a field. | -#### Example request +## Example request The following request runs the stored script that was created in [Create or update stored script]({{site.url}}{{site.baseurl}}/api-reference/script-apis/create-stored-script/). The script sums the ratings for each book and displays the sum in the `total_ratings` field in the output. diff --git a/_api-reference/script-apis/get-script-contexts.md b/_api-reference/script-apis/get-script-contexts.md index 40a155955f..85421128a1 100644 --- a/_api-reference/script-apis/get-script-contexts.md +++ b/_api-reference/script-apis/get-script-contexts.md @@ -11,14 +11,14 @@ nav_order: 5 Retrieves all contexts for stored scripts. -#### Example request +## Example request ````json GET _script_context ```` {% include copy-curl.html %} -#### Example response +## Example response The `GET _script_context` request returns the following fields: diff --git a/_api-reference/script-apis/get-script-language.md b/_api-reference/script-apis/get-script-language.md index f84b7db4a0..76414d52ea 100644 --- a/_api-reference/script-apis/get-script-language.md +++ b/_api-reference/script-apis/get-script-language.md @@ -11,14 +11,14 @@ nav_order: 6 The get script language API operation retrieves all supported script languages and the contexts in which they may be used. -#### Example request +## Example request ```json GET _script_language ``` {% include copy-curl.html %} -#### Example response +## Example response The `GET _script_language` request returns the available contexts for each language: diff --git a/_api-reference/script-apis/get-stored-script.md b/_api-reference/script-apis/get-stored-script.md index cc681cd0f4..d7987974d3 100644 --- a/_api-reference/script-apis/get-stored-script.md +++ b/_api-reference/script-apis/get-stored-script.md @@ -23,7 +23,7 @@ Retrieves a stored script. :--- | :--- | :--- | cluster_manager_timeout | Time | Amount of time to wait for a connection to the cluster manager. Optional, defaults to `30s`. | -#### Example request +## Example request The following retrieves the `my-first-script` stored script. @@ -32,7 +32,7 @@ GET _scripts/my-first-script ```` {% include copy-curl.html %} -#### Example response +## Example response The `GET _scripts/my-first-script` request returns the following fields: diff --git a/_api-reference/scroll.md b/_api-reference/scroll.md index cee599902d..b940c90d86 100644 --- a/_api-reference/scroll.md +++ b/_api-reference/scroll.md @@ -106,7 +106,7 @@ scroll | Time | Specifies the amount of time the search context is maintained. scroll_id | String | The scroll ID for the search. rest_total_hits_as_int | Boolean | Whether the `hits.total` property is returned as an integer (`true`) or an object (`false`). Default is `false`. -## Response +## Example response ```json { diff --git a/_api-reference/snapshots/create-repository.md b/_api-reference/snapshots/create-repository.md index 54807b85d1..8ee7885ca8 100644 --- a/_api-reference/snapshots/create-repository.md +++ b/_api-reference/snapshots/create-repository.md @@ -48,7 +48,7 @@ Request field | Description `remote_store_index_shallow_copy` | Boolean | Determines whether the snapshot of the remote store indexes are captured as a shallow copy. Default is `false`. `readonly` | Whether the repository is read-only. Useful when migrating from one cluster (`"readonly": false` when registering) to another cluster (`"readonly": true` when registering). Optional. -#### Example request +## Example request The following example registers an `fs` repository using the local directory `/mnt/snapshots` as `location`. @@ -85,7 +85,7 @@ Request field | Description For the `base_path` parameter, do not enter the `s3://` prefix when entering your S3 bucket details. Only the name of the bucket is required. {: .note} -#### Example request +## Example request The following request registers a new S3 repository called `my-opensearch-repo` in an existing bucket called `my-open-search-bucket`. By default, all snapshots are stored in the `my/snapshot/directory`. @@ -101,7 +101,7 @@ PUT /_snapshot/my-opensearch-repo ``` {% include copy-curl.html %} -#### Example response +## Example response Upon success, the following JSON object is returned: diff --git a/_api-reference/snapshots/create-snapshot.md b/_api-reference/snapshots/create-snapshot.md index 6334878d8c..d4c9ef8219 100644 --- a/_api-reference/snapshots/create-snapshot.md +++ b/_api-reference/snapshots/create-snapshot.md @@ -46,7 +46,7 @@ Field | Data type | Description `include_global_state` | Boolean | Whether to include cluster state in the snapshot. Default is `true`. `partial` | Boolean | Whether to allow partial snapshots. Default is `false`, which fails the entire snapshot if one or more shards fails to stor -#### Example requests +## Example requests ##### Request without a body @@ -72,7 +72,7 @@ PUT _snapshot/my-s3-repository/2 ``` {% include copy-curl.html %} -#### Example responses +## Example responses Upon success, the response content depends on whether you include the `wait_for_completion` query parameter. diff --git a/_api-reference/snapshots/delete-snapshot-repository.md b/_api-reference/snapshots/delete-snapshot-repository.md index 385205a5df..1fadc21207 100644 --- a/_api-reference/snapshots/delete-snapshot-repository.md +++ b/_api-reference/snapshots/delete-snapshot-repository.md @@ -21,7 +21,7 @@ Parameter | Data type | Description :--- | :--- | :--- repository | String | Repository to delete. | -#### Example request +## Example request The following request deletes the `my-opensearch-repo` repository: @@ -30,7 +30,7 @@ DELETE _snapshot/my-opensearch-repo ```` {% include copy-curl.html %} -#### Example response +## Example response Upon success, the response returns the following JSON object: diff --git a/_api-reference/snapshots/delete-snapshot.md b/_api-reference/snapshots/delete-snapshot.md index e4232c20ec..d231adf74a 100644 --- a/_api-reference/snapshots/delete-snapshot.md +++ b/_api-reference/snapshots/delete-snapshot.md @@ -24,7 +24,7 @@ Parameter | Data type | Description repository | String | Repostory that contains the snapshot. | snapshot | String | Snapshot to delete. | -#### Example request +## Example request The following request deletes a snapshot called `my-first-snapshot` from the `my-opensearch-repo` repository: @@ -33,7 +33,7 @@ DELETE _snapshot/my-opensearch-repo/my-first-snapshot ``` {% include copy-curl.html %} -#### Example response +## Example response Upon success, the response returns the following JSON object: diff --git a/_api-reference/snapshots/get-snapshot-repository.md b/_api-reference/snapshots/get-snapshot-repository.md index 501d0785dd..6617106059 100644 --- a/_api-reference/snapshots/get-snapshot-repository.md +++ b/_api-reference/snapshots/get-snapshot-repository.md @@ -29,7 +29,7 @@ You can also get details about a snapshot during and after snapshot creation. Se | local | Boolean | Whether to get information from the local node. Optional, defaults to `false`.| | cluster_manager_timeout | Time | Amount of time to wait for a connection to the cluster manager node. Optional, defaults to 30 seconds. | -#### Example request +## Example request The following request retrieves information for the `my-opensearch-repo` repository: @@ -38,7 +38,7 @@ GET /_snapshot/my-opensearch-repo ```` {% include copy-curl.html %} -#### Example response +## Example response Upon success, the response returns repositry information. This sample is for an `s3` repository type. diff --git a/_api-reference/snapshots/get-snapshot-status.md b/_api-reference/snapshots/get-snapshot-status.md index 6f8320d0b0..c7f919bcb3 100644 --- a/_api-reference/snapshots/get-snapshot-status.md +++ b/_api-reference/snapshots/get-snapshot-status.md @@ -42,7 +42,7 @@ Using the API to return state for other than currently running snapshots can be :--- | :--- | :--- | ignore_unavailable | Boolean | How to handles requests for unavailable snapshots. If `false`, the request returns an error for unavailable snapshots. If `true`, the request ignores unavailable snapshots, such as those that are corrupted or temporarily cannot be returned. Defaults to `false`.| -#### Example request +## Example request The following request returns the status of `my-first-snapshot` in the `my-opensearch-repo` repository. Unavailable snapshots are ignored. @@ -54,7 +54,7 @@ GET _snapshot/my-opensearch-repo/my-first-snapshot/_status ```` {% include copy-curl.html %} -#### Example response +## Example response The example that follows corresponds to the request above in the [Example request](#example-request) section. diff --git a/_api-reference/snapshots/get-snapshot.md b/_api-reference/snapshots/get-snapshot.md index da44c1f23d..ac55c0370f 100644 --- a/_api-reference/snapshots/get-snapshot.md +++ b/_api-reference/snapshots/get-snapshot.md @@ -25,7 +25,7 @@ Retrieves information about a snapshot. | verbose | Boolean | Whether to show all, or just basic snapshot information. If `true`, returns all information. If `false`, omits information like start/end times, failures, and shards. Optional, defaults to `true`.| | ignore_unavailable | Boolean | How to handle snapshots that are unavailable (corrupted or otherwise temporarily can't be returned). If `true` and the snapshot is unavailable, the request does not return the snapshot. If `false` and the snapshot is unavailable, the request returns an error. Optional, defaults to `false`.| -#### Example request +## Example request The following request retrieves information for the `my-first-snapshot` located in the `my-opensearch-repo` repository: @@ -34,7 +34,7 @@ GET _snapshot/my-opensearch-repo/my-first-snapshot ```` {% include copy-curl.html %} -#### Example response +## Example response Upon success, the response returns snapshot information: diff --git a/_api-reference/snapshots/restore-snapshot.md b/_api-reference/snapshots/restore-snapshot.md index 7b82f72256..cdb9948c28 100644 --- a/_api-reference/snapshots/restore-snapshot.md +++ b/_api-reference/snapshots/restore-snapshot.md @@ -57,7 +57,7 @@ All request body parameters are optional. * Ingest pipelines * Index lifecycle policies -#### Example request +## Example request The following request restores the `opendistro-reports-definitions` index from `my-first-snapshot`. The `rename_pattern` and `rename_replacement` combination causes the index to be renamed to `opendistro-reports-definitions_restored` because duplicate open index names in a cluster are not allowed. @@ -73,7 +73,7 @@ POST /_snapshot/my-opensearch-repo/my-first-snapshot/_restore } ```` -#### Example response +## Example response Upon success, the response returns the following JSON object: diff --git a/_api-reference/snapshots/verify-snapshot-repository.md b/_api-reference/snapshots/verify-snapshot-repository.md index 12fada3303..e5e6337196 100644 --- a/_api-reference/snapshots/verify-snapshot-repository.md +++ b/_api-reference/snapshots/verify-snapshot-repository.md @@ -32,7 +32,7 @@ Path parameters are optional. | cluster_manager_timeout | Time | Amount of time to wait for a connection to the cluster manager node. Optional, defaults to `30s`. | | timeout | Time | The period of time to wait for a response. If a response is not received before the timeout value, the request fails and returns an error. Defaults to `30s`. | -#### Example request +## Example request The following request verifies that the my-opensearch-repo is functional: @@ -40,7 +40,7 @@ The following request verifies that the my-opensearch-repo is functional: POST /_snapshot/my-opensearch-repo/_verify?timeout=0s&cluster_manager_timeout=50s ```` -#### Example response +## Example response The example that follows corresponds to the request above in the [Example request](#example-request) section. diff --git a/_api-reference/tasks.md b/_api-reference/tasks.md index 5c3a41fd34..e4ca0b6049 100644 --- a/_api-reference/tasks.md +++ b/_api-reference/tasks.md @@ -28,59 +28,6 @@ GET _tasks/ Note that if a task finishes running, it won't be returned as part of your request. For an example of a task that takes a little longer to finish, you can run the [`_reindex`]({{site.url}}{{site.baseurl}}/opensearch/reindex-data) API operation on a larger document, and then run `tasks`. -#### Example response -```json -{ - "nodes": { - "Mgqdm0r9SEGClWxp_RbnaQ": { - "name": "opensearch-node1", - "transport_address": "172.18.0.3:9300", - "host": "172.18.0.3", - "ip": "172.18.0.3:9300", - "roles": [ - "data", - "ingest", - "master", - "remote_cluster_client" - ], - "tasks": { - "Mgqdm0r9SEGClWxp_RbnaQ:17416": { - "node": "Mgqdm0r9SEGClWxp_RbnaQ", - "id": 17416, - "type": "transport", - "action": "cluster:monitor/tasks/lists", - "start_time_in_millis": 1613599752458, - "running_time_in_nanos": 994000, - "cancellable": false, - "headers": {} - } - }, - "Mgqdm0r9SEGClWxp_RbnaQ:17413": { - "node": "Mgqdm0r9SEGClWxp_RbnaQ", - "id": 17413, - "type": "transport", - "action": "indices:data/write/bulk", - "start_time_in_millis": 1613599752286, - "running_time_in_nanos": 172846500, - "cancellable": false, - "parent_task_id": "Mgqdm0r9SEGClWxp_RbnaQ:17366", - "headers": {} - }, - "Mgqdm0r9SEGClWxp_RbnaQ:17366": { - "node": "Mgqdm0r9SEGClWxp_RbnaQ", - "id": 17366, - "type": "transport", - "action": "indices:data/write/reindex", - "start_time_in_millis": 1613599750929, - "running_time_in_nanos": 1529733100, - "cancellable": true, - "headers": {} - } - } - } - } -} -``` You can also use the following parameters with your query. @@ -97,14 +44,29 @@ Parameter | Data type | Description | For example, this request returns tasks currently running on a node named `opensearch-node1`: -#### Example request +## Example requests + +### Return information about running tasks + +The following request returns tasks currently running on a node named `opensearch-node1`: ```json GET /_tasks?nodes=opensearch-node1 ``` {% include copy-curl.html %} -#### Example response +### Return information about active search tasks + +The following request returns detailed information about active search tasks: + +```bash +curl -XGET "localhost:9200/_tasks?actions=*search&detailed +``` +{% include copy.html %} + +## Example response + +The following example response shows information about running tasks: ```json { @@ -148,76 +110,6 @@ GET /_tasks?nodes=opensearch-node1 } ``` -The following request returns detailed information about active search tasks: - -#### Example request - -```bash -curl -XGET "localhost:9200/_tasks?actions=*search&detailed -``` -{% include copy.html %} - -#### Example response - -```json -{ - "nodes" : { - "CRqNwnEeRXOjeTSYYktw-A" : { - "name" : "runTask-0", - "transport_address" : "127.0.0.1:9300", - "host" : "127.0.0.1", - "ip" : "127.0.0.1:9300", - "roles" : [ - "cluster_manager", - "data", - "ingest", - "remote_cluster_client" - ], - "attributes" : { - "testattr" : "test", - "shard_indexing_pressure_enabled" : "true" - }, - "tasks" : { - "CRqNwnEeRXOjeTSYYktw-A:677" : { - "node" : "CRqNwnEeRXOjeTSYYktw-A", - "id" : 677, - "type" : "transport", - "action" : "indices:data/read/search", - "description" : "indices[], search_type[QUERY_THEN_FETCH], source[{\"query\":{\"query_string\":}}]", - "start_time_in_millis" : 1660106254525, - "running_time_in_nanos" : 1354236, - "cancellable" : true, - "cancelled" : false, - "headers" : { }, - "resource_stats" : { - "average" : { - "cpu_time_in_nanos" : 0, - "memory_in_bytes" : 0 - }, - "total" : { - "cpu_time_in_nanos" : 0, - "memory_in_bytes" : 0 - }, - "min" : { - "cpu_time_in_nanos" : 0, - "memory_in_bytes" : 0 - }, - "max" : { - "cpu_time_in_nanos" : 0, - "memory_in_bytes" : 0 - }, - "thread_info" : { - "thread_executions" : 0, - "active_threads" : 0 - } - } - } - } - } - } -} - -``` ### The `resource_stats` object diff --git a/_automating-configurations/api/deprovision-workflow.md b/_automating-configurations/api/deprovision-workflow.md index e9219536ce..98c944a9d4 100644 --- a/_automating-configurations/api/deprovision-workflow.md +++ b/_automating-configurations/api/deprovision-workflow.md @@ -9,7 +9,9 @@ nav_order: 70 When you no longer need a workflow, you can deprovision its resources. Most workflow steps that create a resource have corresponding workflow steps to reverse that action. To retrieve all resources currently created for a workflow, call the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/). When you call the Deprovision Workflow API, resources included in the `resources_created` field of the Get Workflow Status API response will be removed using a workflow step corresponding to the one that provisioned them. -The workflow executes the provisioning workflow steps in reverse order. If failures occur because of resource dependencies, such as preventing deletion of a registered model if it is still deployed, the workflow attempts retries. +The workflow executes the provisioning steps in reverse order. If a failure occurs because of a resource dependency, such as trying to delete a registered model that is still deployed, then the workflow retries the failing step as long as at least one resource was deleted. + +To prevent data loss, resources created using the `create_index`, `create_search_pipeline`, and `create_ingest_pipeline` steps require the resource ID to be included in the `allow_delete` parameter. ## Path and HTTP methods @@ -24,6 +26,7 @@ The following table lists the available path parameters. | Parameter | Data type | Description | | :--- | :--- | :--- | | `workflow_id` | String | The ID of the workflow to be deprovisioned. Required. | +| `allow-delete` | String | A comma-separated list of resource IDs to be deprovisioned. Required if deleting resources of type `index_name` or `pipeline_id`. | ### Example request @@ -53,6 +56,14 @@ If deprovisioning did not completely remove all resources, OpenSearch responds w In some cases, the failure happens because of another dependent resource that took some time to be removed. In this case, you can attempt to send the same request again. {: .tip} +If deprovisioning required the `allow_delete` parameter, then OpenSearch responds with a `403 (FORBIDDEN)` status and identifies the resources that were not deprovisioned: + +```json +{ + "error": "These resources require the allow_delete parameter to deprovision: [index_name my-index]." +} +``` + To obtain a more detailed deprovisioning status than is provided by the summary in the error response, query the [Get Workflow Status API]({{site.url}}{{site.baseurl}}/automating-configurations/api/get-workflow-status/). On success, the workflow returns to a `NOT_STARTED` state. If some resources have not yet been removed, they are provided in the response. \ No newline at end of file diff --git a/_benchmark/reference/commands/compare.md b/_benchmark/reference/commands/compare.md index a9eb8bcc10..35bafe0704 100644 --- a/_benchmark/reference/commands/compare.md +++ b/_benchmark/reference/commands/compare.md @@ -130,7 +130,7 @@ You can use the following options to customize the results of your test comparis - `--baseline`: The baseline TestExecution ID used to compare the contender TestExecution. - `--contender`: The TestExecution ID for the contender being compared to the baseline. - `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`. -- `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. +- `--results-numbers-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. - `--results-file`: When provided a file path, writes the compare results to the file indicated in the path. - `--show-in-results`: Determines whether or not to include the comparison in the results file. diff --git a/_data-prepper/pipelines/configuration/processors/convert_entry_type.md b/_data-prepper/pipelines/configuration/processors/convert_entry_type.md index 2fc9fdb9bd..c2c46260ed 100644 --- a/_data-prepper/pipelines/configuration/processors/convert_entry_type.md +++ b/_data-prepper/pipelines/configuration/processors/convert_entry_type.md @@ -14,10 +14,22 @@ The `convert_entry_type` processor converts a value type associated with the spe You can configure the `convert_entry_type` processor with the following options. + + | Option | Required | Description | | :--- | :--- | :--- | -| `key`| Yes | Keys whose value needs to be converted to a different type. | -| `type` | No | Target type for the key-value pair. Possible values are `integer`, `double`, `string`, and `Boolean`. Default value is `integer`. | +| `key`| Yes | Key whose value needs to be converted to a different type. | +| `keys`| Yes | Keys whose value needs to be converted to a different type. | +| `type` | No | Target type for the key-value pair. Possible values are `integer`, `long`, `double`, `big_decimal`, `string`, and `boolean`. Default value is `integer`. | +| `null_values` | No | String representation of what constitutes a `null` value. If the field value equals one of these strings, then the value is considered `null` and is converted to `null`. | +| `scale` | No | Modifies the scale of the `big_decimal` when converting to a `big_decimal`. The default value is `0`. | +| `tags_on_failure` | No | A list of tags to be added to the event metadata when the event fails to convert. | +| `convert_when` | No | Specifies a condition using a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) for performing the `convert_entry_type` operation. If specified, the `convert_entry_type` operation runs only when the expression evaluates to `true`. | ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/delay.md b/_data-prepper/pipelines/configuration/processors/delay.md new file mode 100644 index 0000000000..c4e9d8e973 --- /dev/null +++ b/_data-prepper/pipelines/configuration/processors/delay.md @@ -0,0 +1,27 @@ +--- +layout: default +title: delay +parent: Processors +grand_parent: Pipelines +nav_order: 41 +--- + +# delay + +This processor will add a delay into the processor chain. Typically, you should use this only for testing, experimenting, and debugging. + +## Configuration + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +`for` | No | Duration | The duration of time to delay. Defaults to `1s`. + +## Usage + +The following example shows using the `delay` processor to delay for 2 seconds. + +```yaml +processor: + - delay: + for: 2s +``` diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete_entries.md similarity index 99% rename from _data-prepper/pipelines/configuration/processors/delete-entries.md rename to _data-prepper/pipelines/configuration/processors/delete_entries.md index 33c54a0b29..c9a93a1f3e 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete_entries.md @@ -3,7 +3,7 @@ layout: default title: delete_entries parent: Processors grand_parent: Pipelines -nav_order: 41 +nav_order: 43 --- # delete_entries diff --git a/_data-prepper/pipelines/configuration/processors/mutate-event.md b/_data-prepper/pipelines/configuration/processors/mutate-event.md index 9b3b2afb33..1afb34a970 100644 --- a/_data-prepper/pipelines/configuration/processors/mutate-event.md +++ b/_data-prepper/pipelines/configuration/processors/mutate-event.md @@ -13,7 +13,7 @@ Mutate event processors allow you to modify events in Data Prepper. The followin * [add_entries]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/add-entries/) allows you to add entries to an event. * [convert_entry_type]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/convert_entry_type/) allows you to convert value types in an event. * [copy_values]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/copy-values/) allows you to copy values within an event. -* [delete_entries]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/delete-entries/) allows you to delete entries from an event. +* [delete_entries]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/delete_entries/) allows you to delete entries from an event. * [list_to_map]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/list-to-map) allows you to convert list of objects from an event where each object contains a `key` field into a map of target keys. * `map_to_list` allows you to convert a map of objects from an event, where each object contains a `key` field, into a list of target keys. * [rename_keys]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/rename-keys/) allows you to rename keys in an event. diff --git a/_data-prepper/pipelines/configuration/processors/parse-ion.md b/_data-prepper/pipelines/configuration/processors/parse_ion.md similarity index 61% rename from _data-prepper/pipelines/configuration/processors/parse-ion.md rename to _data-prepper/pipelines/configuration/processors/parse_ion.md index 0edd446c42..8360eaa296 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-ion.md +++ b/_data-prepper/pipelines/configuration/processors/parse_ion.md @@ -14,12 +14,22 @@ The `parse_ion` processor parses [Amazon Ion](https://amazon-ion.github.io/ion-d You can configure the `parse_ion` processor with the following options. + + | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `event` that is parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. | | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. | -| `tags_on_failure` | No | String | A list of strings that specify the tags to be set in the event that the processors fails or an unknown exception occurs while parsing. +| `parse_when` | No | String | Specifies under which conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). | +| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults is `true`. | +| `delete_source` | No | Boolean | If set to `true`, then the source field is deleted. Defaults is `false`. | +| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/parse-json.md b/_data-prepper/pipelines/configuration/processors/parse_json.md similarity index 70% rename from _data-prepper/pipelines/configuration/processors/parse-json.md rename to _data-prepper/pipelines/configuration/processors/parse_json.md index 2cbce4782e..894d5dba42 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json.md +++ b/_data-prepper/pipelines/configuration/processors/parse_json.md @@ -15,11 +15,22 @@ The `parse_json` processor parses JSON data for an event, including any nested f You can configure the `parse_json` processor with the following options. + + | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. | | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. | +| `parse_when` | No | String | Specifies under which conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). | +| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. | +| `delete_source` | No | Boolean | If set to `true` then this will delete the source field. Defaults to `false`. | +| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/parse-xml.md b/_data-prepper/pipelines/configuration/processors/parse_xml.md similarity index 70% rename from _data-prepper/pipelines/configuration/processors/parse-xml.md rename to _data-prepper/pipelines/configuration/processors/parse_xml.md index 861705da2b..c8c9f3eebf 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-xml.md +++ b/_data-prepper/pipelines/configuration/processors/parse_xml.md @@ -14,13 +14,22 @@ The `parse_xml` processor parses XML data for an event. You can configure the `parse_xml` processor with the following options. + + | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | | `source` | No | String | Specifies which `event` field to parse. | | `destination` | No | String | The destination field of the parsed XML. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only string because these are not valid `event` fields. | | `pointer` | No | String | A JSON pointer to the field to be parsed. The value is null by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event` object. If the key that is pointed to already exists in the `event` object and the `destination` is the root, then the pointer uses the entire path of the key. | | `parse_when` | No | String | Specifies under what conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Data Prepper Expression Syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). | -| `tags_on_failure` | No | String | A list of strings that specify the tags to be set if the processor fails or an unknown exception occurs while parsing. +| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. | +| `delete_source` | No | Boolean | If set to `true` then this will delete the source field. Defaults to `false`. | +| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/write_json.md b/_data-prepper/pipelines/configuration/processors/write_json.md index 8f1e6851da..20414b4672 100644 --- a/_data-prepper/pipelines/configuration/processors/write_json.md +++ b/_data-prepper/pipelines/configuration/processors/write_json.md @@ -11,8 +11,15 @@ nav_order: 56 The `write_json` processor converts an object in an event into a JSON string. You can customize the processor to choose the source and target field names. -Option | Description | Example -:--- | :--- | :--- + + +Option | Description | Example +:--- | :--- | :--- source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}}`, then the `write_json` processor outputs the event as `"{\"key1\":\"value1\",\"key2\":{\"key3\":\"value3\"}}"`. target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. | `key1` diff --git a/_data-prepper/pipelines/configuration/sources/kafka.md b/_data-prepper/pipelines/configuration/sources/kafka.md index 4df72cfdd6..e8452a93c3 100644 --- a/_data-prepper/pipelines/configuration/sources/kafka.md +++ b/_data-prepper/pipelines/configuration/sources/kafka.md @@ -120,7 +120,7 @@ Use the following options when setting SSL encryption. Option | Required | Type | Description :--- | :--- | :--- | :--- `type` | No | String | The encryption type. Use `none` to disable encryption. Default is `ssl`. -`Insecure` | No | Boolean | A Boolean flag used to turn off SSL certificate verification. If set to `true`, certificate authority (CA) certificate verification is turned off and insecure HTTP requests are sent. Default is `false`. +`insecure` | No | Boolean | A Boolean flag used to turn off SSL certificate verification. If set to `true`, certificate authority (CA) certificate verification is turned off and insecure HTTP requests are sent. Default is `false`. #### AWS diff --git a/_data-prepper/pipelines/configuration/sources/s3.md b/_data-prepper/pipelines/configuration/sources/s3.md index 5a7d9986e5..7b1599f838 100644 --- a/_data-prepper/pipelines/configuration/sources/s3.md +++ b/_data-prepper/pipelines/configuration/sources/s3.md @@ -138,7 +138,7 @@ The `codec` determines how the `s3` source parses each Amazon S3 object. For inc ### `newline` codec -The `newline` codec parses each single line as a single log event. This is ideal for most application logs because each event parses per single line. It can also be suitable for S3 objects that have individual JSON objects on each line, which matches well when used with the [parse_json]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) processor to parse each line. +The `newline` codec parses each single line as a single log event. This is ideal for most application logs because each event parses per single line. It can also be suitable for S3 objects that have individual JSON objects on each line, which matches well when used with the [parse_json]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse_json/) processor to parse each line. Use the following options to configure the `newline` codec. diff --git a/_field-types/dynamic.md b/_field-types/dynamic.md new file mode 100644 index 0000000000..59f59bfe3d --- /dev/null +++ b/_field-types/dynamic.md @@ -0,0 +1,342 @@ +--- +layout: default +title: Dynamic parameter +nav_order: 10 +redirect_from: + - /opensearch/dynamic/ +--- + +# Dynamic parameter + +The `dynamic` parameter specifies whether newly detected fields can be added dynamically to a mapping. It accepts the parameters listed in the following table. + +Parameter | Description +:--- | :--- +`true` | Specfies that new fields can be added dynamically to the mapping. Default is `true`. +`false` | Specifies that new fields cannot be added dynamically to the mapping. If a new field is detected, then it is not indexed or searchable but can be retrieved from the `_source` field. +`strict` | Throws an exception. The indexing operation fails when new fields are detected. +`strict_allow_templates` | Adds new fields if they match predefined dynamic templates in the mapping. + +--- + +## Example: Create an index with `dynamic` set to `true` + +1. Create an index with `dynamic` set to `true` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": true + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + } +} +``` +{% include copy-curl.html %} + +3. Confirm the mapping works as expected by sending the following request: + +```json +GET testindex1/_mapping +``` +{% include copy-curl.html %} + +The object field `patient` and two subfields `name` and `id` are added to the mapping, as shown in the following response: + +```json +{ + "testindex1": { + "mappings": { + "dynamic": "true", + "properties": { + "patient": { + "properties": { + "id": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } + } + } + } + } + } +} +``` + +--- + +## Example: Create an index with `dynamic` set to `false` + +1. Create an index with explicit mappings and `dynamic` set to `false` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": false, + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields and additional unmapped fields by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` +{% include copy-curl.html %} + +3. Confirm the mapping works as expected by sending the following request: + +```json +GET testindex1/_mapping +``` +{% include copy-curl.html %} + +The following response shows that the new fields `room` and `floor` were not added to the mapping, which remained unchanged: + +```json +{ + "testindex1": { + "mappings": { + "dynamic": "false", + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } + } +} +``` + +4. Get the unmapped fields `room` and `floor` from the document by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` + +The following request searches for the fields `room` and `floor`: + +```json +POST testindex1/_search +{ + "query": { + "term": { + "room": "room1" + } + } +} +``` + +The response returns no results: + +```json +{ + "took": 3, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 0, + "relation": "eq" + }, + "max_score": null, + "hits": [] + } +} +``` + +--- + +## Example: Create an index with `dynamic` set to `strict` + +1. Create an index with explicit mappings and `dynamic` set to `strict` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": strict, + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields and additional unmapped fields by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` +{% include copy-curl.html %} + +Note that an exception is thrown, as shown in the following response: + +```json +{ + "error": { + "root_cause": [ + { + "type": "strict_dynamic_mapping_exception", + "reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed" + } + ], + "type": "strict_dynamic_mapping_exception", + "reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed" + }, + "status": 400 +} +``` + +--- + +## Example: Create an index with `dynamic` set to `strict_allow_templates` + +1. Create an index with predefined dynamic templates and `dynamic` set to `strict_allow_templates` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": "strict_allow_templates", + "dynamic_templates": [ + { + "strings": { + "match": "room*", + "match_mapping_type": "string", + "mapping": { + "type": "keyword" + } + } + } + ], + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields and a new field `room` that matches one of the dynamic templates by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1" +} +``` +{% include copy-curl.html %} + +Indexing succeeds because the new field `room` matches the dynamic templates. However, indexing fails for the new field `floor` because it does not match one of the dynamic templates and is not explicitly mapped, as shown in the following response: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` diff --git a/_field-types/supported-field-types/geo-shape.md b/_field-types/supported-field-types/geo-shape.md index cbf63551df..b7b06a0d04 100644 --- a/_field-types/supported-field-types/geo-shape.md +++ b/_field-types/supported-field-types/geo-shape.md @@ -68,7 +68,7 @@ PUT testindex/_doc/1 { "location" : { "type" : "point", - "coordinates" : [74.00, 40.71] + "coordinates" : [74.0060, 40.7128] } } ``` @@ -126,10 +126,12 @@ PUT testindex/_doc/3 "location" : { "type" : "polygon", "coordinates" : [ - [[74.0060, 40.7128], - [71.0589, 42.3601], - [73.7562, 42.6526], - [74.0060, 40.7128]] + [ + [74.0060, 40.7128], + [73.7562, 42.6526], + [71.0589, 42.3601], + [74.0060, 40.7128] + ] ] } } @@ -159,15 +161,18 @@ PUT testindex/_doc/4 "location" : { "type" : "polygon", "coordinates" : [ - [[74.0060, 40.7128], - [71.0589, 42.3601], - [73.7562, 42.6526], - [74.0060, 40.7128]], - - [[72.6734,41.7658], - [72.6506, 41.5623], - [73.0515, 41.5582], - [72.6734, 41.7658]] + [ + [74.0060, 40.7128], + [73.7562, 42.6526], + [71.0589, 42.3601], + [74.0060, 40.7128] + ], + [ + [72.6734,41.7658], + [73.0515, 41.5582], + [72.6506, 41.5623], + [72.6734, 41.7658] + ] ] } } @@ -179,12 +184,12 @@ Index a polygon (triangle) with a triangular hole in WKT format: ```json PUT testindex/_doc/4 { - "location" : "POLYGON ((40.7128 74.0060, 42.3601 71.0589, 42.6526 73.7562, 40.7128 74.0060), (41.7658 72.6734, 41.5623 72.6506, 41.5582 73.0515, 41.7658 72.6734))" + "location" : "POLYGON ((74.0060 40.7128, 71.0589 42.3601, 73.7562 42.6526, 74.0060 40.7128), (72.6734 41.7658, 72.6506 41.5623, 73.0515 41.5582, 72.6734 41.7658))" } ``` {% include copy-curl.html %} -In OpenSearch, you can specify a polygon by listing its vertices clockwise or counterclockwise. This works well for polygons that do not cross the date line (are narrower than 180°). However, a polygon that crosses the date line (is wider than 180°) might be ambiguous because WKT does not impose a specific order on vertices. Thus, you must specify polygons that cross the date line by listing their vertices counterclockwise. +You can specify a polygon in OpenSearch by listing its vertices in clockwise or counterclockwise order. This works well for polygons that do not cross the date line (that are narrower than 180°). However, a polygon that crosses the date line (is wider than 180°) might be ambiguous because WKT does not impose a specific order on vertices. Thus, you must specify polygons that cross the date line by listing their vertices in counterclockwise order. You can define an [`orientation`](#parameters) parameter to specify the vertex traversal order at mapping time: @@ -295,23 +300,28 @@ PUT testindex/_doc/4 "type" : "multipolygon", "coordinates" : [ [ - [[74.0060, 40.7128], - [71.0589, 42.3601], - [73.7562, 42.6526], - [74.0060, 40.7128]], - - [[72.6734,41.7658], - [72.6506, 41.5623], - [73.0515, 41.5582], - [72.6734, 41.7658]] + [ + [74.0060, 40.7128], + [73.7562, 42.6526], + [71.0589, 42.3601], + [74.0060, 40.7128] + ], + [ + [73.0515, 41.5582], + [72.6506, 41.5623], + [72.6734, 41.7658], + [73.0515, 41.5582] + ] ], [ - [[73.9776, 40.7614], - [73.9554, 40.7827], - [73.9631, 40.7812], - [73.9776, 40.7614]] + [ + [73.9146, 40.8252], + [73.8871, 41.0389], + [73.6853, 40.9747], + [73.9146, 40.8252] ] ] + ] } } ``` @@ -322,7 +332,7 @@ Index a multipolygon in WKT format: ```json PUT testindex/_doc/4 { - "location" : "MULTIPOLYGON (((40.7128 74.0060, 42.3601 71.0589, 42.6526 73.7562, 40.7128 74.0060), (41.7658 72.6734, 41.5623 72.6506, 41.5582 73.0515, 41.7658 72.6734)), ((73.9776 40.7614, 73.9554 40.7827, 73.9631 40.7812, 73.9776 40.7614)))" + "location" : "MULTIPOLYGON (((74.0060 40.7128, 71.0589 42.3601, 73.7562 42.6526, 74.0060 40.7128), (72.6734 41.7658, 72.6506 41.5623, 73.0515 41.5582, 72.6734 41.7658)), ((73.9146 40.8252, 73.6853 40.9747, 73.8871 41.0389, 73.9146 40.8252)))" } ``` {% include copy-curl.html %} @@ -400,5 +410,5 @@ Parameter | Description :--- | :--- `coerce` | A Boolean value that specifies whether to automatically close unclosed linear rings. Default is `false`. `ignore_malformed` | A Boolean value that specifies to ignore malformed GeoJSON or WKT geoshapes and not to throw an exception. Default is `false` (throw an exception when geoshapes are malformed). -`ignore_z_value` | Specific to points with three coordinates. If `ignore_z_value` is `true`, the third coordinate is not indexed but is still stored in the _source field. If `ignore_z_value` is `false`, an exception is thrown. Default is `true`. +`ignore_z_value` | Specific to points with three coordinates. If `ignore_z_value` is `true`, then the third coordinate is not indexed but is still stored in the `_source` field. If `ignore_z_value` is `false`, then an exception is thrown. Default is `true`. `orientation` | Specifies the traversal order of the vertices in the geoshape's list of coordinates. `orientation` takes the following values:
1. RIGHT: counterclockwise. Specify RIGHT orientation by using one of the following strings (uppercase or lowercase): `right`, `counterclockwise`, `ccw`.
2. LEFT: clockwise. Specify LEFT orientation by using one of the following strings (uppercase or lowercase): `left`, `clockwise`, `cw`. This value can be overridden by individual documents.
Default is `RIGHT`. diff --git a/_field-types/supported-field-types/nested.md b/_field-types/supported-field-types/nested.md index d61ccd53df..90d09177d1 100644 --- a/_field-types/supported-field-types/nested.md +++ b/_field-types/supported-field-types/nested.md @@ -308,7 +308,7 @@ The following table lists the parameters accepted by object field types. All par Parameter | Description :--- | :--- -[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`. +[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to the object. Valid values are `true`, `false`, `strict`, and `strict_allow_templates`. Default is `true`. `include_in_parent` | A Boolean value that specifies whether all fields in the child nested object should also be added to the parent document in flattened form. Default is `false`. `include_in_root` | A Boolean value that specifies whether all fields in the child nested object should also be added to the root document in flattened form. Default is `false`. `properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`. diff --git a/_field-types/supported-field-types/object.md b/_field-types/supported-field-types/object.md index 372a5c46d9..db539a9608 100644 --- a/_field-types/supported-field-types/object.md +++ b/_field-types/supported-field-types/object.md @@ -73,7 +73,7 @@ The following table lists the parameters accepted by object field types. All par Parameter | Description :--- | :--- -[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`. +[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to the object. Valid values are `true`, `false`, `strict`, and `strict_allow_templates`. Default is `true`. `enabled` | A Boolean value that specifies whether the JSON contents of the object should be parsed. If `enabled` is set to `false`, the object's contents are not indexed or searchable, but they are still retrievable from the _source field. Default is `true`. `properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`. @@ -149,6 +149,7 @@ Value | Description `true` | New fields can be added to the mapping dynamically. This is the default. `false` | New fields cannot be added to the mapping dynamically. If a new field is detected, it is not indexed or searchable. However, it is still retrievable from the _source field. `strict` | When new fields are added to the mapping dynamically, an exception is thrown. To add a new field to an object, you have to add it to the mapping first. +`strict_allow_templates` | If the newly detected fields match any of the predefined dynamic templates in the mapping, then they are added to the mapping; if they do not match any of them, then an exception is thrown. Inner objects inherit the `dynamic` parameter value from their parent unless they declare their own `dynamic` parameter value. {: .note } diff --git a/_im-plugin/ism/api.md b/_im-plugin/ism/api.md index 441f737e6f..e0fbb904cd 100644 --- a/_im-plugin/ism/api.md +++ b/_im-plugin/ism/api.md @@ -553,13 +553,13 @@ Introduced 2.4 ISM allows you to run an action automatically. However, running an action can fail for a variety of reasons. You can use error prevention validation to test an action in order to rule out failures. -To enable error prevention validation, set the `plugins.index_state_management.validation_service.enabled` setting to `true`: +To enable error prevention validation, set the `plugins.index_state_management.action_validation.enabled` setting to `true`: ```bash PUT _cluster/settings { "persistent":{ - "plugins.index_state_management.validation_action.enabled": true + "plugins.index_state_management.action_validation.enabled": true } } ``` @@ -692,4 +692,4 @@ GET _plugins/_ism/explain/test-000001 }, "total_managed_indices" : 1 } -``` \ No newline at end of file +``` diff --git a/_im-plugin/ism/error-prevention/api.md b/_im-plugin/ism/error-prevention/api.md index a273d25cfb..c03a62d868 100644 --- a/_im-plugin/ism/error-prevention/api.md +++ b/_im-plugin/ism/error-prevention/api.md @@ -12,7 +12,7 @@ The ISM Error Prevention API allows you to enable Index State Management (ISM) e ## Enable error prevention validation -You can configure error prevention validation by setting the `plugins.index_state_management.validation_service.enabled` parameter. +You can configure error prevention validation by setting the `plugins.index_state_management.action_validation.enabled` parameter. #### Example request @@ -20,7 +20,7 @@ You can configure error prevention validation by setting the `plugins.index_stat PUT _cluster/settings { "persistent":{ - "plugins.index_state_management.validation_action.enabled": true + "plugins.index_state_management.action_validation.enabled": true } } ``` diff --git a/_im-plugin/refresh-analyzer.md b/_im-plugin/refresh-analyzer.md index 2e50f06dc0..bff54b739f 100644 --- a/_im-plugin/refresh-analyzer.md +++ b/_im-plugin/refresh-analyzer.md @@ -10,7 +10,7 @@ redirect_from: # Refresh search analyzer -With ISM installed, you can refresh search analyzers in real time with the following API: +You can refresh search analyzers in real time using the following API. This requires the [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/ism/index/) (ISM) plugin to be installed. For more information, see [Installing plugins]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/). ```json POST /_plugins/_refresh_search_analyzers/ diff --git a/_ingest-pipelines/processors/split.md b/_ingest-pipelines/processors/split.md index c424ef671c..cdb0cfe3de 100644 --- a/_ingest-pipelines/processors/split.md +++ b/_ingest-pipelines/processors/split.md @@ -30,7 +30,7 @@ Parameter | Required/Optional | Description :--- | :--- | :--- `field` | Required | The field containing the string to be split. `separator` | Required | The delimiter used to split the string. This can be a regular expression pattern. -`preserve_field` | Optional | If set to `true`, preserves empty trailing fields (for example, `''`) in the resulting array. If set to `false`, empty trailing fields are removed from the resulting array. Default is `false`. +`preserve_trailing` | Optional | If set to `true`, preserves empty trailing fields (for example, `''`) in the resulting array. If set to `false`, then empty trailing fields are removed from the resulting array. Default is `false`. `target_field` | Optional | The field where the array of substrings is stored. If not specified, then the field is updated in-place. `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. If set to `true`, then the processor ignores missing values in the field and leaves the `target_field` unchanged. Default is `false`. `description` | Optional | A brief description of the processor. diff --git a/_install-and-configure/configuring-opensearch/security-settings.md b/_install-and-configure/configuring-opensearch/security-settings.md index 244d601449..b9c375d208 100644 --- a/_install-and-configure/configuring-opensearch/security-settings.md +++ b/_install-and-configure/configuring-opensearch/security-settings.md @@ -15,7 +15,7 @@ The following sections describe security-related settings in `opensearch.yml`. T The Security plugin supports the following common settings: -- `plugins.security.nodes_dn` (Static): Specifies a list of distinguished names (DNs) that denote the other nodes in the cluster. This setting supports wildcards and regular expressions. The list of DNs are also read from the security index **in addition** to the YAML configuration when `plugins.security.nodes_dn_dynamic_config_enabled` is `true`. +- `plugins.security.nodes_dn` (Static): Specifies a list of distinguished names (DNs) that denote the other nodes in the cluster. This setting supports wildcards and regular expressions. The list of DNs are also read from the security index **in addition** to the YAML configuration when `plugins.security.nodes_dn_dynamic_config_enabled` is `true`. If this setting is not configured correctly, the cluster will fail to form as the nodes will not be able to trust each other and will result in the following error: `Transport client authentication no longer supported`. - `plugins.security.nodes_dn_dynamic_config_enabled` (Static): Relevant for `cross_cluster` use cases where there is a need to manage the allow listed `nodes_dn` without having to restart the nodes every time a new `cross_cluster` remote is configured. Setting `nodes_dn_dynamic_config_enabled` to `true` enables **super-admin callable** Distinguished Names APIs, which provide means to update or retrieve `nodes_dn` dynamically. This setting only has effect if `plugins.security.cert.intercluster_request_evaluator_class` is not set. Default is `false`. @@ -122,6 +122,41 @@ The Security plugin supports the following expert-level settings: - `plugins.security.check_snapshot_restore_write_privileges` (Static): Enforces write privilege evaluation when creating snapshots. Default is `true`. +If you change any of the following password hashing properties, you must rehash all internal passwords to ensure compatibility and security. +{: .warning} + +- `plugins.security.password.hashing.algorithm`: (Static): Specifies the password hashing algorithm to use. + + Valid values are: + + - `BCrypt` (Default) + - `PBKDF2` + +- `plugins.security.password.hashing.bcrypt.rounds` (Static): Specifies the number of rounds to use for password hashing with `BCrypt`. Valid values are between `4` and `31`, inclusive. Default is `12`. + +- `plugins.security.password.hashing.bcrypt.minor` (Static): Specifies the minor version of the `BCrypt` algorithm to use for password hashing. + + Valid values are: + + - `A` + - `B` + - `Y` (Default) + +- `plugins.security.password.hashing.pbkdf2.function` (Static): Specifies the pseudo-random function applied to the password. + + Valid values are: + + - `SHA1` + - `SHA224` + - `SHA256` (Default) + - `SHA384` + - `SHA512` + +- `plugins.security.password.hashing.pbkdf2.iterations` (Static): Specifies the number of times that the pseudo-random function is applied to the password. Default is `600,000`. + +- `plugins.security.password.hashing.pbkdf2.length` (Static): Specifies the desired length of the final derived key. Default is `256`. + + ## Audit log settings The Security plugin supports the following audit log settings: @@ -357,6 +392,7 @@ The Security plugin supports the following transport layer security settings: plugins.security.nodes_dn: - "CN=*.example.com, OU=SSL, O=Test, L=Test, C=DE" - "CN=node.other.com, OU=SSL, O=Test, L=Test, C=DE" + - "CN=node.example.com, OU=SSL\, Inc., L=Test, C=DE" # escape additional comma with `\` plugins.security.authcz.admin_dn: - CN=kirk,OU=client,O=client,L=test, C=de plugins.security.roles_mapping_resolution: MAPPING_ONLY diff --git a/_install-and-configure/install-opensearch/index.md b/_install-and-configure/install-opensearch/index.md index 1afe12f6a5..541321bcdd 100644 --- a/_install-and-configure/install-opensearch/index.md +++ b/_install-and-configure/install-opensearch/index.md @@ -121,5 +121,5 @@ Property | Description `opensearch.xcontent.string.length.max=` | By default, OpenSearch does not impose any limits on the maximum length of the JSON/YAML/CBOR/Smile string fields. To protect your cluster against potential distributed denial-of-service (DDoS) or memory issues, you can set the `opensearch.xcontent.string.length.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.string.length.max=5000000`. | `opensearch.xcontent.fast_double_writer=[true|false]` | By default, OpenSearch serializes floating-point numbers using the default implementation provided by the Java Runtime Environment. Set this value to `true` to use the Schubfach algorithm, which is faster but may lead to small differences in precision. Default is `false`. | `opensearch.xcontent.name.length.max=` | By default, OpenSearch does not impose any limits on the maximum length of the JSON/YAML/CBOR/Smile field names. To protect your cluster against potential DDoS or memory issues, you can set the `opensearch.xcontent.name.length.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.name.length.max=50000`. | -`opensearch.xcontent.depth.max=` | By default, OpenSearch does not impose any limits on the maximum nesting depth for JSON/YAML/CBOR/Smile documents. To protect your cluster against potential DDoS or memory issues, you can set the `opensearch.xcontent.depth.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.name.length.max=1000`. | +`opensearch.xcontent.depth.max=` | By default, OpenSearch does not impose any limits on the maximum nesting depth for JSON/YAML/CBOR/Smile documents. To protect your cluster against potential DDoS or memory issues, you can set the `opensearch.xcontent.depth.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.depth.max=1000`. | `opensearch.xcontent.codepoint.max=` | By default, OpenSearch imposes a limit of `52428800` on the maximum size of the YAML documents (in code points). To protect your cluster against potential DDoS or memory issues, you can change the `opensearch.xcontent.codepoint.max` system property to a reasonable limit (the maximum is 2,147,483,647). For example, `-Dopensearch.xcontent.codepoint.max=5000000`. | diff --git a/_ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md b/_ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md new file mode 100644 index 0000000000..b6fae8131c --- /dev/null +++ b/_ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md @@ -0,0 +1,169 @@ +--- +layout: default +title: CreateAnomalyDetectorTool +has_children: false +has_toc: false +nav_order: 70 +parent: Tools +grand_parent: Agents and tools +--- + + +# CreateAnomalyDetectorTool +**Introduced 2.16** +{: .label .label-purple } + + +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/skills/issues/337). +{: .warning} + +The `CreateAnomalyDetectorTool` helps create anomaly detectors based on your provided index. This tool retrieves index mappings and enables a large language model (LLM) to recommend category fields, aggregation fields, and their corresponding aggregation methods, which are required by the Create Anomaly Detector API. + +For comprehensive information about anomaly detectors, see [Anomaly detection]({{site.url}}{{site.baseurl}}/observing-your-data/ad/index/). +{: .tip} + +## Step 1: Register a flow agent that runs the CreateAnomalyDetectorTool + +A flow agent runs a sequence of tools in order, returning the output of the last tool. To create a flow agent, send the following register agent request: + +```json +POST /_plugins/_ml/agents/_register +{ + "name": "Test_Agent_For_Create_Anomaly_Detector_Tool", + "type": "flow", + "description": "this is a test agent for the CreateAnomalyDetectorTool", + "memory": { + "type": "demo" + }, + "tools": [ + { + "type": "CreateAnomalyDetectorTool", + "name": "DemoCreateAnomalyDetectorTool", + "parameters": { + "model_id": "" + } + } + ] +} +``` +{% include copy-curl.html %} + +OpenSearch responds with an agent ID, for example, as follows: + +```json +{ + "agent_id": "EuJYYo0B9RaBCvhuy1q8" +} +``` +{% include copy-curl.html %} + +## Step 2: Run the agent + +Run the agent by sending the following request: + +```json +POST /_plugins/_ml/agents/EuJYYo0B9RaBCvhuy1q8/_execute +{ + "parameters": { + "index": "sample_weblogs_test" + } +} +``` +{% include copy-curl.html %} + +OpenSearch responds with a JSON string containing all of the recommended parameters for creating an anomaly detector, such as the string shown in the following example repsonse: + +```json +{ + "inference_results": [ + { + "output": [ + { + "name": "response", + "result":"""{"index":"sample_weblogs_test","categoryField":"ip.keyword","aggregationField":"bytes,response,responseLatency","aggregationMethod":"sum,avg,avg","dateFields":"utc_time,timestamp"}""" + } + ] + } + ] +} +``` +{% include copy-curl.html %} + +You can then create an anomaly detector containing the recommended parameters by sending a request similar to the following: + +```json +POST _plugins/_anomaly_detection/detectors +{ + "name": "test-detector", + "description": "Test detector", + "time_field": "timestamp", + "indices": [ + "sample_weblogs_test" + ], + "feature_attributes": [ + { + "feature_name": "feature_bytes", + "feature_enabled": true, + "aggregation_query": { + "agg1": { + "sum": { + "field": "bytes" + } + } + } + }, + { + "feature_name": "feature_response", + "feature_enabled": true, + "aggregation_query": { + "agg2": { + "avg": { + "field": "response" + } + } + } + }, + { + "feature_name": "feature_responseLatency", + "feature_enabled": true, + "aggregation_query": { + "agg3": { + "avg": { + "field": "responseLatency" + } + } + } + } + ], + "detection_interval": { + "period": { + "interval": 1, + "unit": "Minutes" + } + }, + "window_delay": { + "period": { + "interval": 1, + "unit": "Minutes" + } + } +} +``` +{% include copy-curl.html %} + +## Register parameters + +The following table lists the available tool parameters for agent registration. + +Parameter | Type | Required/Optional | Description +:--- | :--- | :--- | :--- +`model_id` | String | Required | The LLM model ID used for suggesting required Create Anomaly Detector API parameters. +`model_type` | String | Optional | The model type. Valid values are `CLAUDE` (Anthropic Claude models) and `OPENAI` (OpenAI models). + +## Execute parameters + +The following table lists the available tool parameters for running the agent. + +Parameter | Type | Required/Optional | Description +:--- | :--- | :--- | :--- +`index` | String | Required | The index name. Supports wildcards (for example, `weblogs-*`). If wildcards are used, then the tool fetches mappings from the first resolved index and sends them to the LLM. diff --git a/_ml-commons-plugin/agents-tools/tools/index.md b/_ml-commons-plugin/agents-tools/tools/index.md index fba47b63da..bc71122949 100644 --- a/_ml-commons-plugin/agents-tools/tools/index.md +++ b/_ml-commons-plugin/agents-tools/tools/index.md @@ -35,6 +35,7 @@ Each tool takes a list of parameters specific to that tool. In the preceding exa |[`CatIndexTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/cat-index-tool/) |Retrieves index information for the OpenSearch cluster. | |[`ConnectorTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/connector-tool/) | Uses a [connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/) to call any REST API function. | |[`IndexMappingTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/index-mapping-tool/) |Retrieves index mapping and setting information for an index. | +|[`CreateAnomalyDetectorTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/create-anomaly-detector/) | Enables an LLM to suggest required parameters for creating an anomaly detector. | |[`MLModelTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/ml-model-tool/) |Runs machine learning models. | |[`NeuralSparseSearchTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/neural-sparse-tool/) | Performs sparse vector retrieval. | |[`PPLTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/ppl-tool/) |Translates natural language into a Piped Processing Language (PPL) query. | @@ -49,4 +50,4 @@ Each tool takes a list of parameters specific to that tool. In the preceding exa ## Developer information -The agents and tools framework is flexible and extensible. You can find the list of tools provided by OpenSearch in the [Tools library](https://github.com/opensearch-project/skills/tree/main/src/main/java/org/opensearch/agent/tools). For a different use case, you can build your own tool by implementing the [_Tool_ interface](https://github.com/opensearch-project/ml-commons/blob/2.x/spi/src/main/java/org/opensearch/ml/common/spi/tools/Tool.java). \ No newline at end of file +The agents and tools framework offers flexibility and extensibility. See the [tools library](https://github.com/opensearch-project/skills/tree/main/src/main/java/org/opensearch/agent/tools) for OpenSearch-provided tools. Implement the [**Tool** interface](https://github.com/opensearch-project/ml-commons/blob/2.x/spi/src/main/java/org/opensearch/ml/common/spi/tools/Tool.java) to build custom tools for different use cases. diff --git a/_ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md b/_ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md index 9fee4dcbd2..9014c585c8 100644 --- a/_ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md +++ b/_ml-commons-plugin/agents-tools/tools/neural-sparse-tool.md @@ -212,6 +212,7 @@ Parameter | Type | Required/Optional | Description `name` | String | Optional | The tool name. Useful when an LLM needs to select an appropriate tool for a task. `description` | String | Optional | A description of the tool. Useful when an LLM needs to select an appropriate tool for a task. `doc_size` | Integer | Optional | The number of documents to fetch. Default is `2`. +`nested_path` | String | Optional | The path to the nested object for the nested query. Only used for nested fields. Default is `null`. ## Execute parameters diff --git a/_ml-commons-plugin/agents-tools/tools/rag-tool.md b/_ml-commons-plugin/agents-tools/tools/rag-tool.md index 1f6fafe49a..c88c2d047b 100644 --- a/_ml-commons-plugin/agents-tools/tools/rag-tool.md +++ b/_ml-commons-plugin/agents-tools/tools/rag-tool.md @@ -136,6 +136,7 @@ Parameter | Type | Required/Optional | Description `prompt` | String | Optional | The prompt to provide to the LLM. `k` | Integer | Optional | The number of nearest neighbors to search for when performing neural search. Default is 10. `enable_Content_Generation` | Boolean | Optional | If `true`, returns results generated by an LLM. If `false`, returns results directly without LLM-assisted content generation. Default is `true`. +`nested_path` | String | Optional | The path to the nested object for the nested query. Only used for nested fields. Default is `null`. ## Execute parameters diff --git a/_ml-commons-plugin/agents-tools/tools/vector-db-tool.md b/_ml-commons-plugin/agents-tools/tools/vector-db-tool.md index 9093541cbb..70d7e19321 100644 --- a/_ml-commons-plugin/agents-tools/tools/vector-db-tool.md +++ b/_ml-commons-plugin/agents-tools/tools/vector-db-tool.md @@ -225,6 +225,7 @@ Parameter | Type | Required/Optional | Description `input` | String | Required for flow agent | Runtime input sourced from flow agent parameters. If using a large language model (LLM), this field is populated with the LLM response. `doc_size` | Integer | Optional | The number of documents to fetch. Default is `2`. `k` | Integer | Optional | The number of nearest neighbors to search for when performing neural search. Default is `10`. +`nested_path` | String | Optional | The path to the nested object for the nested query. Only used for nested fields. Default is `null`. ## Execute parameters diff --git a/_ml-commons-plugin/cluster-settings.md b/_ml-commons-plugin/cluster-settings.md index 0c1f433bf2..efb13dd73a 100644 --- a/_ml-commons-plugin/cluster-settings.md +++ b/_ml-commons-plugin/cluster-settings.md @@ -256,6 +256,23 @@ plugins.ml_commons.jvm_heap_memory_threshold: 85 - Default value: 85 - Value range: [0, 100] +## Set a disk free space threshold + +Sets a disk circuit breaker that checks disk usage before running an ML task. If the amount of disk free space exceeds the threshold, then OpenSearch triggers a circuit breaker and throws an exception to maintain optimal performance. + +Valid values are in byte units. To disable the circuit breaker, set this value to -1. + +### Setting + +``` +plugins.ml_commons.disk_free_space_threshold: 5G +``` + +### Values + +- Default value: 5G +- Value range: [-1, Long.MAX_VALUE] + ## Exclude node names Use this setting to specify the names of nodes on which you don't want to run ML tasks. The value should be a valid node name or a comma-separated node name list. diff --git a/_observing-your-data/alerting/per-cluster-metrics-monitors.md b/_observing-your-data/alerting/per-cluster-metrics-monitors.md index baea9c626b..bcaa03cc0c 100644 --- a/_observing-your-data/alerting/per-cluster-metrics-monitors.md +++ b/_observing-your-data/alerting/per-cluster-metrics-monitors.md @@ -91,7 +91,7 @@ The `script` parameter points the `source` to the Painless script `for (cluster "path": "_cluster/health/", "path_params": "", "url": "http://localhost:9200/_cluster/health/", - "cluster": ["cluster-1", "cluster-2"] + "clusters": ["cluster-1", "cluster-2"] } } ], diff --git a/_query-dsl/compound/bool.md b/_query-dsl/compound/bool.md index 12caea2f8d..4479094214 100644 --- a/_query-dsl/compound/bool.md +++ b/_query-dsl/compound/bool.md @@ -2,7 +2,6 @@ layout: default title: Boolean parent: Compound queries -grand_parent: Query DSL nav_order: 10 redirect_from: - /opensearch/query-dsl/compound/bool/ diff --git a/_query-dsl/compound/boosting.md b/_query-dsl/compound/boosting.md index 7aa9c6c035..ede7eb3d51 100644 --- a/_query-dsl/compound/boosting.md +++ b/_query-dsl/compound/boosting.md @@ -2,7 +2,6 @@ layout: default title: Boosting parent: Compound queries -grand_parent: Query DSL nav_order: 30 redirect_from: - /query-dsl/query-dsl/compound/boosting/ diff --git a/_query-dsl/compound/constant-score.md b/_query-dsl/compound/constant-score.md index ed12e33ec0..bb5af3800a 100644 --- a/_query-dsl/compound/constant-score.md +++ b/_query-dsl/compound/constant-score.md @@ -2,7 +2,6 @@ layout: default title: Constant score parent: Compound queries -grand_parent: Query DSL nav_order: 40 redirect_from: - /query-dsl/query-dsl/compound/constant-score/ diff --git a/_query-dsl/compound/disjunction-max.md b/_query-dsl/compound/disjunction-max.md index 8dd9e41d2c..09a7e0e729 100644 --- a/_query-dsl/compound/disjunction-max.md +++ b/_query-dsl/compound/disjunction-max.md @@ -2,7 +2,6 @@ layout: default title: Disjunction max parent: Compound queries -grand_parent: Query DSL nav_order: 50 redirect_from: - /query-dsl/query-dsl/compound/disjunction-max/ diff --git a/_query-dsl/compound/function-score.md b/_query-dsl/compound/function-score.md index 98568e0965..b28a6abed6 100644 --- a/_query-dsl/compound/function-score.md +++ b/_query-dsl/compound/function-score.md @@ -2,7 +2,6 @@ layout: default title: Function score parent: Compound queries -grand_parent: Query DSL nav_order: 60 has_math: true redirect_from: diff --git a/_query-dsl/compound/hybrid.md b/_query-dsl/compound/hybrid.md index 22b3a17fc1..69ce89ce17 100644 --- a/_query-dsl/compound/hybrid.md +++ b/_query-dsl/compound/hybrid.md @@ -2,7 +2,6 @@ layout: default title: Hybrid parent: Compound queries -grand_parent: Query DSL nav_order: 70 --- diff --git a/_query-dsl/full-text/intervals.md b/_query-dsl/full-text/intervals.md index 082f8fbe46..a31f7434b3 100644 --- a/_query-dsl/full-text/intervals.md +++ b/_query-dsl/full-text/intervals.md @@ -3,7 +3,6 @@ layout: default title: Intervals nav_order: 80 parent: Full-text queries -grand_parent: Query DSL --- # Intervals query diff --git a/_query-dsl/full-text/match-bool-prefix.md b/_query-dsl/full-text/match-bool-prefix.md index 3a0d304ce4..3964dc5ee8 100644 --- a/_query-dsl/full-text/match-bool-prefix.md +++ b/_query-dsl/full-text/match-bool-prefix.md @@ -2,7 +2,6 @@ layout: default title: Match Boolean prefix parent: Full-text queries -grand_parent: Query DSL nav_order: 20 --- diff --git a/_query-dsl/full-text/match-phrase-prefix.md b/_query-dsl/full-text/match-phrase-prefix.md index 354dd35c61..9e05f034d7 100644 --- a/_query-dsl/full-text/match-phrase-prefix.md +++ b/_query-dsl/full-text/match-phrase-prefix.md @@ -2,7 +2,6 @@ layout: default title: Match phrase prefix parent: Full-text queries -grand_parent: Query DSL nav_order: 40 --- diff --git a/_query-dsl/full-text/match-phrase.md b/_query-dsl/full-text/match-phrase.md index 18dd6a858c..747c4814d9 100644 --- a/_query-dsl/full-text/match-phrase.md +++ b/_query-dsl/full-text/match-phrase.md @@ -2,7 +2,6 @@ layout: default title: Match phrase parent: Full-text queries -grand_parent: Query DSL nav_order: 30 --- diff --git a/_query-dsl/full-text/match.md b/_query-dsl/full-text/match.md index 746a4cf5b6..b4db30ec1f 100644 --- a/_query-dsl/full-text/match.md +++ b/_query-dsl/full-text/match.md @@ -2,7 +2,6 @@ layout: default title: Match parent: Full-text queries -grand_parent: Query DSL nav_order: 10 --- diff --git a/_query-dsl/full-text/multi-match.md b/_query-dsl/full-text/multi-match.md index 7450b74721..ab1496fdd3 100644 --- a/_query-dsl/full-text/multi-match.md +++ b/_query-dsl/full-text/multi-match.md @@ -2,7 +2,6 @@ layout: default title: Multi-match parent: Full-text queries -grand_parent: Query DSL nav_order: 50 --- diff --git a/_query-dsl/full-text/query-string.md b/_query-dsl/full-text/query-string.md index 12609e29c0..47180e3f6d 100644 --- a/_query-dsl/full-text/query-string.md +++ b/_query-dsl/full-text/query-string.md @@ -2,9 +2,7 @@ layout: default title: Query string parent: Full-text queries -grand_parent: Query DSL nav_order: 60 - redirect_from: - /opensearch/query-dsl/full-text/query-string/ - /query-dsl/query-dsl/full-text/query-string/ diff --git a/_query-dsl/full-text/simple-query-string.md b/_query-dsl/full-text/simple-query-string.md index fbf37f588d..58780cfdb4 100644 --- a/_query-dsl/full-text/simple-query-string.md +++ b/_query-dsl/full-text/simple-query-string.md @@ -2,7 +2,6 @@ layout: default title: Simple query string parent: Full-text queries -grand_parent: Query DSL nav_order: 70 --- diff --git a/_query-dsl/geo-and-xy/geo-bounding-box.md b/_query-dsl/geo-and-xy/geo-bounding-box.md index df697e2ce5..1112a4278e 100644 --- a/_query-dsl/geo-and-xy/geo-bounding-box.md +++ b/_query-dsl/geo-and-xy/geo-bounding-box.md @@ -2,7 +2,6 @@ layout: default title: Geo-bounding box parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 10 redirect_from: - /opensearch/query-dsl/geo-and-xy/geo-bounding-box/ diff --git a/_query-dsl/geo-and-xy/geodistance.md b/_query-dsl/geo-and-xy/geodistance.md index 7a36b0c933..b272cad81e 100644 --- a/_query-dsl/geo-and-xy/geodistance.md +++ b/_query-dsl/geo-and-xy/geodistance.md @@ -2,7 +2,6 @@ layout: default title: Geodistance parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 20 --- diff --git a/_query-dsl/geo-and-xy/geopolygon.md b/_query-dsl/geo-and-xy/geopolygon.md index c53b1379cf..980a0c5a63 100644 --- a/_query-dsl/geo-and-xy/geopolygon.md +++ b/_query-dsl/geo-and-xy/geopolygon.md @@ -2,7 +2,6 @@ layout: default title: Geopolygon parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 30 --- diff --git a/_query-dsl/geo-and-xy/xy.md b/_query-dsl/geo-and-xy/xy.md index 88a22448c3..d0ed61c050 100644 --- a/_query-dsl/geo-and-xy/xy.md +++ b/_query-dsl/geo-and-xy/xy.md @@ -2,9 +2,7 @@ layout: default title: xy parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 50 - redirect_from: - /opensearch/query-dsl/geo-and-xy/xy/ - /query-dsl/query-dsl/geo-and-xy/xy/ diff --git a/_query-dsl/specialized/neural-sparse.md b/_query-dsl/specialized/neural-sparse.md index 47f77fa95d..8de3eaf693 100644 --- a/_query-dsl/specialized/neural-sparse.md +++ b/_query-dsl/specialized/neural-sparse.md @@ -2,7 +2,6 @@ layout: default title: Neural sparse parent: Specialized queries -grand_parent: Query DSL nav_order: 55 --- diff --git a/_query-dsl/specialized/neural.md b/_query-dsl/specialized/neural.md index fea949cf52..14b930cdb6 100644 --- a/_query-dsl/specialized/neural.md +++ b/_query-dsl/specialized/neural.md @@ -2,7 +2,6 @@ layout: default title: Neural parent: Specialized queries -grand_parent: Query DSL nav_order: 50 --- diff --git a/_query-dsl/specialized/script-score.md b/_query-dsl/specialized/script-score.md index d09158f20a..f65c266cc5 100644 --- a/_query-dsl/specialized/script-score.md +++ b/_query-dsl/specialized/script-score.md @@ -2,7 +2,6 @@ layout: default title: Script score parent: Specialized queries -grand_parent: Query DSL nav_order: 60 --- diff --git a/_query-dsl/term/exists.md b/_query-dsl/term/exists.md index 1d52744c91..95573a36f8 100644 --- a/_query-dsl/term/exists.md +++ b/_query-dsl/term/exists.md @@ -2,7 +2,6 @@ layout: default title: Exists parent: Term-level queries -grand_parent: Query DSL nav_order: 10 --- diff --git a/_query-dsl/term/fuzzy.md b/_query-dsl/term/fuzzy.md index 9afa85ea93..bf2bd43bba 100644 --- a/_query-dsl/term/fuzzy.md +++ b/_query-dsl/term/fuzzy.md @@ -2,7 +2,6 @@ layout: default title: Fuzzy parent: Term-level queries -grand_parent: Query DSL nav_order: 20 --- diff --git a/_query-dsl/term/ids.md b/_query-dsl/term/ids.md index 0c3b5393fb..f895745c97 100644 --- a/_query-dsl/term/ids.md +++ b/_query-dsl/term/ids.md @@ -2,7 +2,6 @@ layout: default title: IDs parent: Term-level queries -grand_parent: Query DSL nav_order: 30 --- diff --git a/_query-dsl/term/prefix.md b/_query-dsl/term/prefix.md index eda5307d14..2a429c9f0e 100644 --- a/_query-dsl/term/prefix.md +++ b/_query-dsl/term/prefix.md @@ -2,7 +2,6 @@ layout: default title: Prefix parent: Term-level queries -grand_parent: Query DSL nav_order: 40 --- diff --git a/_query-dsl/term/range.md b/_query-dsl/term/range.md index 8a8f53c480..ceb264db76 100644 --- a/_query-dsl/term/range.md +++ b/_query-dsl/term/range.md @@ -2,7 +2,6 @@ layout: default title: Range parent: Term-level queries -grand_parent: Query DSL nav_order: 50 --- diff --git a/_query-dsl/term/regexp.md b/_query-dsl/term/regexp.md index 65d6953516..4a038729c0 100644 --- a/_query-dsl/term/regexp.md +++ b/_query-dsl/term/regexp.md @@ -2,7 +2,6 @@ layout: default title: Regexp parent: Term-level queries -grand_parent: Query DSL nav_order: 60 --- diff --git a/_query-dsl/term/term.md b/_query-dsl/term/term.md index c1c296b9a0..a33146f6aa 100644 --- a/_query-dsl/term/term.md +++ b/_query-dsl/term/term.md @@ -2,7 +2,6 @@ layout: default title: Term parent: Term-level queries -grand_parent: Query DSL nav_order: 70 --- diff --git a/_query-dsl/term/terms-set.md b/_query-dsl/term/terms-set.md index ea0251ddff..6652d15979 100644 --- a/_query-dsl/term/terms-set.md +++ b/_query-dsl/term/terms-set.md @@ -2,7 +2,6 @@ layout: default title: Terms set parent: Term-level queries -grand_parent: Query DSL nav_order: 90 --- diff --git a/_query-dsl/term/terms.md b/_query-dsl/term/terms.md index fd15126255..42c74c0436 100644 --- a/_query-dsl/term/terms.md +++ b/_query-dsl/term/terms.md @@ -2,7 +2,6 @@ layout: default title: Terms parent: Term-level queries -grand_parent: Query DSL nav_order: 80 --- diff --git a/_query-dsl/term/wildcard.md b/_query-dsl/term/wildcard.md index 0652581941..b2d7238758 100644 --- a/_query-dsl/term/wildcard.md +++ b/_query-dsl/term/wildcard.md @@ -2,7 +2,6 @@ layout: default title: Wildcard parent: Term-level queries -grand_parent: Query DSL nav_order: 100 --- diff --git a/_search-plugins/cross-cluster-search.md b/_search-plugins/cross-cluster-search.md index 947097e8b3..7d3ff72efb 100644 --- a/_search-plugins/cross-cluster-search.md +++ b/_search-plugins/cross-cluster-search.md @@ -9,7 +9,7 @@ redirect_from: # Cross-cluster search -You can use the cross-cluster search feature in OpenSearch to search and analyze data across multiple clusters, enabling you to gain insights from distributed data sources. Cross-cluster search is available by default with the Security plugin, but you need to configure each cluster to allow remote connections from other clusters. This involves setting up remote cluster connections and configuring access permissions. +You can use cross-cluster search (CCS) in OpenSearch to search and analyze data across multiple clusters, enabling you to gain insights from distributed data sources. Cross-cluster search is available by default with the Security plugin, but you need to configure each cluster to allow remote connections from other clusters. This involves setting up remote cluster connections and configuring access permissions. --- diff --git a/_search-plugins/knn/api.md b/_search-plugins/knn/api.md index 23678063a8..c7314f7ae2 100644 --- a/_search-plugins/knn/api.md +++ b/_search-plugins/knn/api.md @@ -3,7 +3,6 @@ layout: default title: k-NN plugin API nav_order: 30 parent: k-NN search -grand_parent: Search methods has_children: false --- diff --git a/_search-plugins/knn/approximate-knn.md b/_search-plugins/knn/approximate-knn.md index c0a9557728..fa1b4096c7 100644 --- a/_search-plugins/knn/approximate-knn.md +++ b/_search-plugins/knn/approximate-knn.md @@ -3,7 +3,6 @@ layout: default title: Approximate k-NN search nav_order: 15 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- @@ -142,7 +141,7 @@ The following table provides examples of the number of results returned by vario 10 | 1 | 1 | 4 | 4 | 1 10 | 10 | 1 | 4 | 10 | 10 10 | 1 | 2 | 4 | 8 | 2 - + The number of results returned by Faiss/NMSLIB differs from the number of results returned by Lucene only when `k` is smaller than `size`. If `k` and `size` are equal, all engines return the same number of results. Starting in OpenSearch 2.14, you can use `k`, `min_score`, or `max_distance` for [radial search]({{site.url}}{{site.baseurl}}/search-plugins/knn/radial-search-knn/). @@ -254,7 +253,54 @@ POST _bulk ... ``` -After data is ingested, it can be search just like any other `knn_vector` field! +After data is ingested, it can be searched in the same way as any other `knn_vector` field. + +### Additional query parameters + +Starting with version 2.16, you can provide `method_parameters` in a search request: + +```json +GET my-knn-index-1/_search +{ + "size": 2, + "query": { + "knn": { + "my_vector2": { + "vector": [2, 3, 5, 6], + "k": 2, + "method_parameters" : { + "ef_search": 100 + } + } + } + } +} +``` +{% include copy-curl.html %} + +These parameters are dependent on the combination of engine and method used to create the index. The following sections provide information about the supported `method_parameters`. + +#### `ef_search` + +You can provide the `ef_search` parameter when searching an index created using the `hnsw` method. The `ef_search` parameter specifies the number of vectors to examine in order to find the top k nearest neighbors. Higher `ef_search` values improve recall at the cost of increased search latency. The value must be positive. + +The following table provides information about the `ef_search` parameter for the supported engines. + +Engine | Radial query support | Notes +:--- | :--- | :--- +`nmslib` | No | If `ef_search` is present in a query, it overrides the `index.knn.algo_param.ef_search` index setting. +`faiss` | Yes | If `ef_search` is present in a query, it overrides the `index.knn.algo_param.ef_search` index setting. +`lucene` | No | When creating a search query, you must specify `k`. If you provide both `k` and `ef_search`, then the larger value is passed to the engine. If `ef_search` is larger than `k`, you can provide the `size` parameter to limit the final number of results to `k`. + +#### `nprobes` + +You can provide the `nprobes` parameter when searching an index created using the `ivf` method. The `nprobes` parameter specifies the number of `nprobes` clusters to examine in order to find the top k nearest neighbors. Higher `nprobes` values improve recall at the cost of increased search latency. The value must be positive. + +The following table provides information about the `nprobes` parameter for the supported engines. + +Engine | Notes +:--- | :--- +`faiss` | If `nprobes` is present in a query, it overrides the value provided when creating the index. ### Using approximate k-NN with filters diff --git a/_search-plugins/knn/filter-search-knn.md b/_search-plugins/knn/filter-search-knn.md index 309cf6850e..2f0c4aa072 100644 --- a/_search-plugins/knn/filter-search-knn.md +++ b/_search-plugins/knn/filter-search-knn.md @@ -3,7 +3,6 @@ layout: default title: k-NN search with filters nav_order: 20 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/jni-libraries.md b/_search-plugins/knn/jni-libraries.md index 59a5b7a1e2..4dbdb2da56 100644 --- a/_search-plugins/knn/jni-libraries.md +++ b/_search-plugins/knn/jni-libraries.md @@ -3,7 +3,6 @@ layout: default title: JNI libraries nav_order: 35 parent: k-NN search -grand_parent: Search methods has_children: false redirect_from: - /search-plugins/knn/jni-library/ diff --git a/_search-plugins/knn/knn-index.md b/_search-plugins/knn/knn-index.md index ab24a0c097..ed8b9217f5 100644 --- a/_search-plugins/knn/knn-index.md +++ b/_search-plugins/knn/knn-index.md @@ -3,7 +3,6 @@ layout: default title: k-NN index nav_order: 5 parent: k-NN search -grand_parent: Search methods has_children: false --- diff --git a/_search-plugins/knn/knn-score-script.md b/_search-plugins/knn/knn-score-script.md index cc79e90850..1696bd4cad 100644 --- a/_search-plugins/knn/knn-score-script.md +++ b/_search-plugins/knn/knn-score-script.md @@ -3,7 +3,6 @@ layout: default title: Exact k-NN with scoring script nav_order: 10 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/knn-vector-quantization.md b/_search-plugins/knn/knn-vector-quantization.md index 549437f346..fe4833ee47 100644 --- a/_search-plugins/knn/knn-vector-quantization.md +++ b/_search-plugins/knn/knn-vector-quantization.md @@ -3,7 +3,6 @@ layout: default title: k-NN vector quantization nav_order: 27 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/nested-search-knn.md b/_search-plugins/knn/nested-search-knn.md index bdc1045387..d947ebc6e6 100644 --- a/_search-plugins/knn/nested-search-knn.md +++ b/_search-plugins/knn/nested-search-knn.md @@ -3,7 +3,6 @@ layout: default title: k-NN search with nested fields nav_order: 21 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/painless-functions.md b/_search-plugins/knn/painless-functions.md index 1f27cc29a6..09eb989702 100644 --- a/_search-plugins/knn/painless-functions.md +++ b/_search-plugins/knn/painless-functions.md @@ -3,7 +3,6 @@ layout: default title: k-NN Painless extensions nav_order: 25 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/performance-tuning.md b/_search-plugins/knn/performance-tuning.md index 24d92bd67d..123b1daef1 100644 --- a/_search-plugins/knn/performance-tuning.md +++ b/_search-plugins/knn/performance-tuning.md @@ -2,7 +2,6 @@ layout: default title: Performance tuning parent: k-NN search -grand_parent: Search methods nav_order: 45 --- diff --git a/_search-plugins/knn/radial-search-knn.md b/_search-plugins/knn/radial-search-knn.md index 48aaac034d..1a4a223294 100644 --- a/_search-plugins/knn/radial-search-knn.md +++ b/_search-plugins/knn/radial-search-knn.md @@ -3,7 +3,6 @@ layout: default title: Radial search nav_order: 28 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/settings.md b/_search-plugins/knn/settings.md index 4d84cc80bb..1b9aa3608c 100644 --- a/_search-plugins/knn/settings.md +++ b/_search-plugins/knn/settings.md @@ -2,7 +2,6 @@ layout: default title: Settings parent: k-NN search -grand_parent: Search methods nav_order: 40 --- diff --git a/_search-plugins/neural-sparse-search.md b/_search-plugins/neural-sparse-search.md index b2b4fc33d6..8aa2ff7dbf 100644 --- a/_search-plugins/neural-sparse-search.md +++ b/_search-plugins/neural-sparse-search.md @@ -16,8 +16,8 @@ Introduced 2.11 When selecting a model, choose one of the following options: -- Use a sparse encoding model at both ingestion time and search time (high performance, relatively high latency). -- Use a sparse encoding model at ingestion time and a tokenizer at search time for relatively low performance and low latency. The tokenism doesn't conduct model inference, so you can deploy and invoke a tokenizer using the ML Commons Model API for a more consistent experience. +- Use a sparse encoding model at both ingestion time and search time for better search relevance at the expense of relatively high latency. +- Use a sparse encoding model at ingestion time and a tokenizer at search time for lower search latency at the expense of relatively lower search relevance. Tokenization doesn't involve model inference, so you can deploy and invoke a tokenizer using the ML Commons Model API for a more streamlined experience. **PREREQUISITE**
Before using neural sparse search, make sure to set up a [pretrained sparse embedding model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/pretrained-models/#sparse-encoding-models) or your own sparse embedding model. For more information, see [Choosing a model]({{site.url}}{{site.baseurl}}/ml-commons-plugin/integrating-ml-models/#choosing-a-model). diff --git a/_search-plugins/search-pipelines/deleting-search-pipeline.md b/_search-plugins/search-pipelines/deleting-search-pipeline.md new file mode 100644 index 0000000000..3f113f7688 --- /dev/null +++ b/_search-plugins/search-pipelines/deleting-search-pipeline.md @@ -0,0 +1,26 @@ +--- +layout: default +title: Deleting search pipelines +nav_order: 30 +has_children: false +parent: Search pipelines +grand_parent: Search +--- + +# Deleting search pipelines + +Use the following request to delete a pipeline. + +To delete a specific search pipeline, pass the pipeline ID as a parameter: + +```json +DELETE /_search/pipeline/ +``` +{% include copy-curl.html %} + +To delete all search pipelines in a cluster, use the wildcard character (`*`): + +```json +DELETE /_search/pipeline/* +``` +{% include copy-curl.html %} diff --git a/_search-plugins/search-pipelines/search-processors.md b/_search-plugins/search-pipelines/search-processors.md index 4630ab950c..ad515cc541 100644 --- a/_search-plugins/search-pipelines/search-processors.md +++ b/_search-plugins/search-pipelines/search-processors.md @@ -37,13 +37,16 @@ The following table lists all supported search response processors. Processor | Description | Earliest available version :--- | :--- | :--- +[`collapse`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/collapse-processor/)| Deduplicates search hits based on a field value, similarly to `collapse` in a search request. | 2.12 [`personalize_search_ranking`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/personalize-search-ranking/) | Uses [Amazon Personalize](https://aws.amazon.com/personalize/) to rerank search results (requires setting up the Amazon Personalize service). | 2.9 -[`retrieval_augmented_generation`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rag-processor/) | Used for retrieval-augmented generation (RAG) in [conversational search]({{site.url}}{{site.baseurl}}/search-plugins/conversational-search/). | 2.10 (generally available in 2.12) [`rename_field`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rename-field-processor/)| Renames an existing field. | 2.8 [`rerank`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rerank-processor/)| Reranks search results using a cross-encoder model. | 2.12 -[`collapse`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/collapse-processor/)| Deduplicates search hits based on a field value, similarly to `collapse` in a search request. | 2.12 +[`retrieval_augmented_generation`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/rag-processor/) | Used for retrieval-augmented generation (RAG) in [conversational search]({{site.url}}{{site.baseurl}}/search-plugins/conversational-search/). | 2.10 (generally available in 2.12) +[`sort`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/sort-processor/)| Sorts an array of items in either ascending or descending order. | 2.16 +[`split`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/split-processor/)| Splits a string field into an array of substrings based on a specified delimiter. | 2.16 [`truncate_hits`]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/truncate-hits-processor/)| Discards search hits after a specified target count is reached. Can undo the effect of the `oversample` request processor. | 2.12 + ## Search phase results processors A search phase results processor runs between search phases at the coordinating node level. It intercepts the results retrieved from one search phase and transforms them before passing them to the next search phase. diff --git a/_search-plugins/search-pipelines/sort-processor.md b/_search-plugins/search-pipelines/sort-processor.md new file mode 100644 index 0000000000..dde05c1b3a --- /dev/null +++ b/_search-plugins/search-pipelines/sort-processor.md @@ -0,0 +1,251 @@ +--- +layout: default +title: Sort +nav_order: 32 +has_children: false +parent: Search processors +grand_parent: Search pipelines +--- + +# Sort processor + +The `sort` processor sorts an array of items in either ascending or descending order. Numeric arrays are sorted numerically, while string or mixed arrays (strings and numbers) are sorted lexicographically. The processor throws an error if the input is not an array. + +## Request fields + +The following table lists all available request fields. + +Field | Data type | Description +:--- | :--- | :--- +`field` | String | The field to be sorted. Must be an array. Required. +`order` | String | The sort order to apply. Accepts `asc` for ascending or `desc` for descending. Default is `asc`. +`target_field` | String | The name of the field in which the sorted array is stored. If not specified, then the sorted array is stored in the same field as the original array (the `field` variable). +`tag` | String | The processor's identifier. +`description` | String | A description of the processor. +`ignore_failure` | Boolean | If `true`, then OpenSearch [ignores any failure]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/creating-search-pipeline/#ignoring-processor-failures) of this processor and continues to run the remaining processors in the search pipeline. Optional. Default is `false`. + +## Example + +The following example demonstrates using a search pipeline with a `sort` processor. + +### Setup + +Create an index named `my_index` and index a document with the field `message` that contains an array of strings: + +```json +POST /my_index/_doc/1 +{ + "message": ["one", "two", "three", "four"], + "visibility": "public" +} +``` +{% include copy-curl.html %} + +### Creating a search pipeline + +Create a search pipeline with a `sort` response processor that sorts the `message` field and stores the sorted results in the `sorted_message` field: + +```json +PUT /_search/pipeline/my_pipeline +{ + "response_processors": [ + { + "sort": { + "field": "message", + "target_field": "sorted_message" + } + } + ] +} +``` +{% include copy-curl.html %} + +### Using a search pipeline + +Search for documents in `my_index` without a search pipeline: + +```json +GET /my_index/_search +``` +{% include copy-curl.html %} + +The response contains the field `message`: + +
+ + Response + + {: .text-delta} + +```json +{ + "took": 1, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1, + "_source": { + "message": [ + "one", + "two", + "three", + "four" + ], + "visibility": "public" + } + } + ] + } +} +``` +
+ +To search with a pipeline, specify the pipeline name in the `search_pipeline` query parameter: + +```json +GET /my_index/_search?search_pipeline=my_pipeline +``` +{% include copy-curl.html %} + +The `sorted_message` field contains the strings from the `message` field sorted alphabetically: + +
+ + Response + + {: .text-delta} + +```json +{ + "took": 3, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1, + "_source": { + "visibility": "public", + "sorted_message": [ + "four", + "one", + "three", + "two" + ], + "message": [ + "one", + "two", + "three", + "four" + ] + } + } + ] + } +} +``` +
+ +You can also use the `fields` option to search for specific fields in a document: + +```json +POST /my_index/_search?pretty&search_pipeline=my_pipeline +{ + "fields": ["visibility", "message"] +} +``` +{% include copy-curl.html %} + +In the response, the `message` field is sorted and the results are stored in the `sorted_message` field: + +
+ + Response + + {: .text-delta} + +```json +{ + "took": 2, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1, + "_source": { + "visibility": "public", + "sorted_message": [ + "four", + "one", + "three", + "two" + ], + "message": [ + "one", + "two", + "three", + "four" + ] + }, + "fields": { + "visibility": [ + "public" + ], + "sorted_message": [ + "four", + "one", + "three", + "two" + ], + "message": [ + "one", + "two", + "three", + "four" + ] + } + } + ] + } +} +``` +
\ No newline at end of file diff --git a/_search-plugins/search-pipelines/split-processor.md b/_search-plugins/search-pipelines/split-processor.md new file mode 100644 index 0000000000..6830f81ec3 --- /dev/null +++ b/_search-plugins/search-pipelines/split-processor.md @@ -0,0 +1,234 @@ +--- +layout: default +title: Split +nav_order: 33 +has_children: false +parent: Search processors +grand_parent: Search pipelines +--- + +# Split processor + +The `split` processor splits a string field into an array of substrings based on a specified delimiter. + +## Request fields + +The following table lists all available request fields. + +Field | Data type | Description +:--- | :--- | :--- +`field` | String | The field containing the string to be split. Required. +`separator` | String | The delimiter used to split the string. Specify either a single separator character or a regular expression pattern. Required. +`preserve_trailing` | Boolean | If set to `true`, preserves empty trailing fields (for example, `''`) in the resulting array. If set to `false`, then empty trailing fields are removed from the resulting array. Default is `false`. +`target_field` | String | The field in which the array of substrings is stored. If not specified, then the field is updated in place. +`tag` | String | The processor's identifier. +`description` | String | A description of the processor. +`ignore_failure` | Boolean | If `true`, then OpenSearch [ignores any failure]({{site.url}}{{site.baseurl}}/search-plugins/search-pipelines/creating-search-pipeline/#ignoring-processor-failures) of this processor and continues to run the remaining processors in the search pipeline. Optional. Default is `false`. + +## Example + +The following example demonstrates using a search pipeline with a `split` processor. + +### Setup + +Create an index named `my_index` and index a document containing the field `message`: + +```json +POST /my_index/_doc/1 +{ + "message": "ingest, search, visualize, and analyze data", + "visibility": "public" +} +``` +{% include copy-curl.html %} + +### Creating a search pipeline + +The following request creates a search pipeline with a `split` response processor that splits the `message` field and stores the results in the `split_message` field: + +```json +PUT /_search/pipeline/my_pipeline +{ + "response_processors": [ + { + "split": { + "field": "message", + "separator": ", ", + "target_field": "split_message" + } + } + ] +} +``` +{% include copy-curl.html %} + +### Using a search pipeline + +Search for documents in `my_index` without a search pipeline: + +```json +GET /my_index/_search +``` +{% include copy-curl.html %} + +The response contains the field `message`: + +
+ + Response + + {: .text-delta} +```json +{ + "took": 3, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1, + "_source": { + "message": "ingest, search, visualize, and analyze data", + "visibility": "public" + } + } + ] + } +} +``` +
+ +To search with a pipeline, specify the pipeline name in the `search_pipeline` query parameter: + +```json +GET /my_index/_search?search_pipeline=my_pipeline +``` +{% include copy-curl.html %} + +The `message` field is split and the results are stored in the `split_message` field: + +
+ + Response + + {: .text-delta} + +```json +{ + "took": 6, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1, + "_source": { + "visibility": "public", + "message": "ingest, search, visualize, and analyze data", + "split_message": [ + "ingest", + "search", + "visualize", + "and analyze data" + ] + } + } + ] + } +} +``` +
+ +You can also use the `fields` option to search for specific fields in a document: + +```json +POST /my_index/_search?pretty&search_pipeline=my_pipeline +{ + "fields": ["visibility", "message"] +} +``` +{% include copy-curl.html %} + +In the response, the `message` field is split and the results are stored in the `split_message` field: + +
+ + Response + + {: .text-delta} + +```json +{ + "took": 7, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 1, + "relation": "eq" + }, + "max_score": 1, + "hits": [ + { + "_index": "my_index", + "_id": "1", + "_score": 1, + "_source": { + "visibility": "public", + "message": "ingest, search, visualize, and analyze data", + "split_message": [ + "ingest", + "search", + "visualize", + "and analyze data" + ] + }, + "fields": { + "visibility": [ + "public" + ], + "message": [ + "ingest, search, visualize, and analyze data" + ], + "split_message": [ + "ingest", + "search", + "visualize", + "and analyze data" + ] + } + } + ] + } +} +``` +
\ No newline at end of file diff --git a/_search-plugins/sql/ppl/index.md b/_search-plugins/sql/ppl/index.md index 602255d126..bda67aba36 100644 --- a/_search-plugins/sql/ppl/index.md +++ b/_search-plugins/sql/ppl/index.md @@ -18,7 +18,7 @@ redirect_from: # PPL -Piped Processing Language (PPL) is a query language that focuses on processing data in a sequential, step-by-step manner. PPL uses the pipe (`|`) operator to combine commands to find and retrieve data. It is the primary language used with observability in OpenSearch and supports multi-data queries. +Piped Processing Language (PPL) is a query language that focuses on processing data in a sequential, step-by-step manner. PPL uses the pipe (`|`) operator to combine commands to find and retrieve data. It is particularly well suited for analyzing observability data, such as logs, metrics, and traces, due to its ability to handle semi-structured data efficiently. ## PPL syntax diff --git a/_search-plugins/sql/settings.md b/_search-plugins/sql/settings.md index d4aaac7f40..4842f98449 100644 --- a/_search-plugins/sql/settings.md +++ b/_search-plugins/sql/settings.md @@ -78,6 +78,7 @@ Setting | Default | Description `plugins.sql.cursor.keep_alive` | 1 minute | Configures how long the cursor context is kept open. Cursor contexts are resource-intensive, so we recommend a low value. `plugins.query.memory_limit` | 85% | Configures the heap memory usage limit for the circuit breaker of the query engine. `plugins.query.size_limit` | 200 | Sets the default size of index that the query engine fetches from OpenSearch. +`plugins.query.datasources.enabled` | true | Change to `false` to disable support for data sources in the plugin. ## Spark connector settings diff --git a/_security/configuration/demo-configuration.md b/_security/configuration/demo-configuration.md index 33794b0d67..be188169ad 100644 --- a/_security/configuration/demo-configuration.md +++ b/_security/configuration/demo-configuration.md @@ -9,7 +9,6 @@ nav_order: 4 Welcome to the OpenSearch Security plugin demo configuration setup guide. This tool provides a quick and easy way to replicate a production environment for testing purposes. The demo configuration includes the setup of security-related components, such as internal users, roles, role mappings, audit configuration, basic authentication, tenants, and allow lists. - The demo configuration tool performs the following tasks: 1. Configures security settings, which are then loaded into the security index. @@ -49,9 +48,10 @@ If you want to disable the Security plugin when using Docker, set the `DISABLE_S - One digit [0--9] - One special character -4. Run `docker-compose up`. +4. Make sure that Docker is running on your local machine +5. Run `docker-compose up` from the file directory where your `docker-compose.yml` file and `.env` file are located. -### TAR (Linux) +### TAR (Linux) and Mac OS For TAR distributions on Linux, download the Linux setup files from the OpenSearch [Download & Get Started](https://opensearch.org/downloads.html) page. Then use the following command to run the demo configuration: diff --git a/assets/js/_version-selector.js b/assets/js/_version-selector.js index e5efcaaf2c..23730e67f0 100644 --- a/assets/js/_version-selector.js +++ b/assets/js/_version-selector.js @@ -192,7 +192,10 @@ class VersionSelector extends HTMLElement { frag.querySelector('#selected').textContent = `${PREFIX}${this.getAttribute('selected')}`; const pathName = location.pathname.replace(/\/docs(\/((latest|\d+\.\d+)\/?)?)?/, ''); - const versionsDOMNodes = DOC_VERSIONS.map((v, idx) => `${PREFIX}${v}`); + const versionsDOMNodes = DOC_VERSIONS.map((v, idx) => v === DOC_VERSION_LATEST + ? `${PREFIX}${v}` + : `${PREFIX}${v}`, + ); if (Array.isArray(DOC_VERSIONS_ARCHIVED) && DOC_VERSIONS_ARCHIVED.length) { versionsDOMNodes.push( `Show archived`, diff --git a/templates/API_TEMPLATE.md b/templates/API_TEMPLATE.md index c4c46fc5ce..02c0f341d9 100644 --- a/templates/API_TEMPLATE.md +++ b/templates/API_TEMPLATE.md @@ -30,13 +30,13 @@ The following table lists the available path parameters. All path parameters are The following table lists the available query parameters. All query parameters are optional. -| Parameter | Data type | Description | +| Parameter | Data type | Description | | :--- | :--- | :--- | | `query_parameter` | String | Example query parameter description. Default is ... | -## Request fields +## Request body fields -The following table lists the available request fields. +The following table lists the available request body fields. | Field | Data type | Description | | :--- | :--- | :--- | @@ -44,7 +44,13 @@ The following table lists the available request fields. | `example_object.required_request_field` | Type | Required request field description. Required. | | `example_object.optional_request_field` | Type | Optional request field description. Optional. Default is ... | -#### Example request +## Example request(s) + +**TIP:** If multiple examples exist for the request, seperate those examples using an `h3` header underneath this section. + +### Request with an example object + +The following example shows an API request with an example object: ```json POST /_example/endpoint/ @@ -57,7 +63,21 @@ POST /_example/endpoint/ ``` {% include copy-curl.html %} -#### Example response +## Request without an example object + +The following example shows an API request without an example object: + +```json +POST /_example/endpoint/ +``` +{% include copy-curl.html %} + + +## Example response + +**TIP:** If multiple response examples exist for the request, seperate those examples using an `h3` header underneath this section, similar to the [Example requests](#example-requests). + +The following example shows an API response:
@@ -76,9 +96,9 @@ POST /_example/endpoint/ ```
-## Response fields +## Response body fields -The following table lists all response fields. +The following table lists all response body fields. | Field | Data type | Description | | :--- | :--- | :--- | @@ -87,3 +107,5 @@ The following table lists all response fields. ## Required permissions If you use the Security plugin, make sure you have the appropriate permissions: `cluster:example/permission/name`. + +