From e05b4004a77c0b5208a339e6abef98121e43b690 Mon Sep 17 00:00:00 2001 From: Heather Halter Date: Wed, 17 Jul 2024 11:41:44 -0700 Subject: [PATCH 01/13] Updates PPL description (#7637) * Update index.md Updated description of PPL. Signed-off-by: Heather Halter * Update index.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> * Update search-plugins/sql/ppl/index/ Signed-off-by: Heather Halter * Update _search-plugins/sql/ppl/index.md Co-authored-by: Nathan Bower Signed-off-by: Heather Halter --------- Signed-off-by: Heather Halter Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Co-authored-by: Nathan Bower --- _search-plugins/sql/ppl/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_search-plugins/sql/ppl/index.md b/_search-plugins/sql/ppl/index.md index 602255d126..bda67aba36 100644 --- a/_search-plugins/sql/ppl/index.md +++ b/_search-plugins/sql/ppl/index.md @@ -18,7 +18,7 @@ redirect_from: # PPL -Piped Processing Language (PPL) is a query language that focuses on processing data in a sequential, step-by-step manner. PPL uses the pipe (`|`) operator to combine commands to find and retrieve data. It is the primary language used with observability in OpenSearch and supports multi-data queries. +Piped Processing Language (PPL) is a query language that focuses on processing data in a sequential, step-by-step manner. PPL uses the pipe (`|`) operator to combine commands to find and retrieve data. It is particularly well suited for analyzing observability data, such as logs, metrics, and traces, due to its ability to handle semi-structured data efficiently. ## PPL syntax From d2da95d9cbec2d325df083bca817880af32d3180 Mon Sep 17 00:00:00 2001 From: Miki Date: Wed, 17 Jul 2024 12:41:35 -0700 Subject: [PATCH 02/13] Redirect to `latest` when the latest version is picked from the version selector (#7759) Signed-off-by: Miki --- assets/js/_version-selector.js | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/assets/js/_version-selector.js b/assets/js/_version-selector.js index e5efcaaf2c..23730e67f0 100644 --- a/assets/js/_version-selector.js +++ b/assets/js/_version-selector.js @@ -192,7 +192,10 @@ class VersionSelector extends HTMLElement { frag.querySelector('#selected').textContent = `${PREFIX}${this.getAttribute('selected')}`; const pathName = location.pathname.replace(/\/docs(\/((latest|\d+\.\d+)\/?)?)?/, ''); - const versionsDOMNodes = DOC_VERSIONS.map((v, idx) => `${PREFIX}${v}`); + const versionsDOMNodes = DOC_VERSIONS.map((v, idx) => v === DOC_VERSION_LATEST + ? `${PREFIX}${v}` + : `${PREFIX}${v}`, + ); if (Array.isArray(DOC_VERSIONS_ARCHIVED) && DOC_VERSIONS_ARCHIVED.length) { versionsDOMNodes.push( `Show archived`, From d8f1b7b95d3d90f130b6b590ee7526ca0c8a186a Mon Sep 17 00:00:00 2001 From: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Date: Wed, 17 Jul 2024 16:56:02 -0400 Subject: [PATCH 03/13] Fix breadcrumbs by excluding collection index pages from parent relationship (#7758) Signed-off-by: Fanit Kolchina --- _aggregations/bucket/adjacency-matrix.md | 1 - _aggregations/bucket/date-histogram.md | 1 - _aggregations/bucket/date-range.md | 1 - _aggregations/bucket/diversified-sampler.md | 1 - _aggregations/bucket/filter.md | 1 - _aggregations/bucket/filters.md | 1 - _aggregations/bucket/geo-distance.md | 1 - _aggregations/bucket/geohash-grid.md | 1 - _aggregations/bucket/geohex-grid.md | 1 - _aggregations/bucket/geotile-grid.md | 1 - _aggregations/bucket/global.md | 1 - _aggregations/bucket/histogram.md | 1 - _aggregations/bucket/ip-range.md | 1 - _aggregations/bucket/missing.md | 1 - _aggregations/bucket/multi-terms.md | 1 - _aggregations/bucket/nested.md | 1 - _aggregations/bucket/range.md | 1 - _aggregations/bucket/reverse-nested.md | 1 - _aggregations/bucket/sampler.md | 1 - _aggregations/bucket/significant-terms.md | 1 - _aggregations/bucket/significant-text.md | 1 - _aggregations/bucket/terms.md | 1 - _aggregations/index.md | 2 +- _aggregations/metric/average.md | 1 - _aggregations/metric/cardinality.md | 1 - _aggregations/metric/extended-stats.md | 1 - _aggregations/metric/geobounds.md | 1 - _aggregations/metric/geocentroid.md | 1 - _aggregations/metric/matrix-stats.md | 1 - _aggregations/metric/maximum.md | 1 - _aggregations/metric/median-absolute-deviation.md | 1 - _aggregations/metric/minimum.md | 1 - _aggregations/metric/percentile-ranks.md | 1 - _aggregations/metric/percentile.md | 1 - _aggregations/metric/scripted-metric.md | 1 - _aggregations/metric/stats.md | 1 - _aggregations/metric/sum.md | 1 - _aggregations/metric/top-hits.md | 1 - _aggregations/metric/value-count.md | 1 - _aggregations/metric/weighted-avg.md | 1 - _query-dsl/compound/bool.md | 1 - _query-dsl/compound/boosting.md | 1 - _query-dsl/compound/constant-score.md | 1 - _query-dsl/compound/disjunction-max.md | 1 - _query-dsl/compound/function-score.md | 1 - _query-dsl/compound/hybrid.md | 1 - _query-dsl/full-text/intervals.md | 1 - _query-dsl/full-text/match-bool-prefix.md | 1 - _query-dsl/full-text/match-phrase-prefix.md | 1 - _query-dsl/full-text/match-phrase.md | 1 - _query-dsl/full-text/match.md | 1 - _query-dsl/full-text/multi-match.md | 1 - _query-dsl/full-text/query-string.md | 2 -- _query-dsl/full-text/simple-query-string.md | 1 - _query-dsl/geo-and-xy/geo-bounding-box.md | 1 - _query-dsl/geo-and-xy/geodistance.md | 1 - _query-dsl/geo-and-xy/geopolygon.md | 1 - _query-dsl/geo-and-xy/xy.md | 2 -- _query-dsl/specialized/neural-sparse.md | 1 - _query-dsl/specialized/neural.md | 1 - _query-dsl/specialized/script-score.md | 1 - _query-dsl/term/exists.md | 1 - _query-dsl/term/fuzzy.md | 1 - _query-dsl/term/ids.md | 1 - _query-dsl/term/prefix.md | 1 - _query-dsl/term/range.md | 1 - _query-dsl/term/regexp.md | 1 - _query-dsl/term/term.md | 1 - _query-dsl/term/terms-set.md | 1 - _query-dsl/term/terms.md | 1 - _query-dsl/term/wildcard.md | 1 - _search-plugins/knn/api.md | 1 - _search-plugins/knn/approximate-knn.md | 1 - _search-plugins/knn/filter-search-knn.md | 1 - _search-plugins/knn/jni-libraries.md | 1 - _search-plugins/knn/knn-index.md | 1 - _search-plugins/knn/knn-score-script.md | 1 - _search-plugins/knn/knn-vector-quantization.md | 1 - _search-plugins/knn/nested-search-knn.md | 1 - _search-plugins/knn/painless-functions.md | 1 - _search-plugins/knn/performance-tuning.md | 1 - _search-plugins/knn/radial-search-knn.md | 1 - _search-plugins/knn/settings.md | 1 - 83 files changed, 1 insertion(+), 85 deletions(-) diff --git a/_aggregations/bucket/adjacency-matrix.md b/_aggregations/bucket/adjacency-matrix.md index fd521f8510..cf62295763 100644 --- a/_aggregations/bucket/adjacency-matrix.md +++ b/_aggregations/bucket/adjacency-matrix.md @@ -2,7 +2,6 @@ layout: default title: Adjacency matrix parent: Bucket aggregations -grand_parent: Aggregations nav_order: 10 redirect_from: - /query-dsl/aggregations/bucket/adjacency-matrix/ diff --git a/_aggregations/bucket/date-histogram.md b/_aggregations/bucket/date-histogram.md index e308104e16..db3fe6884e 100644 --- a/_aggregations/bucket/date-histogram.md +++ b/_aggregations/bucket/date-histogram.md @@ -2,7 +2,6 @@ layout: default title: Date histogram parent: Bucket aggregations -grand_parent: Aggregations nav_order: 20 redirect_from: - /query-dsl/aggregations/bucket/date-histogram/ diff --git a/_aggregations/bucket/date-range.md b/_aggregations/bucket/date-range.md index c7d66d729d..d2498a53da 100644 --- a/_aggregations/bucket/date-range.md +++ b/_aggregations/bucket/date-range.md @@ -2,7 +2,6 @@ layout: default title: Date range parent: Bucket aggregations -grand_parent: Aggregations nav_order: 30 redirect_from: - /query-dsl/aggregations/bucket/date-range/ diff --git a/_aggregations/bucket/diversified-sampler.md b/_aggregations/bucket/diversified-sampler.md index 7249ac3555..a62410cc8c 100644 --- a/_aggregations/bucket/diversified-sampler.md +++ b/_aggregations/bucket/diversified-sampler.md @@ -2,7 +2,6 @@ layout: default title: Diversified sampler parent: Bucket aggregations -grand_parent: Aggregations nav_order: 40 redirect_from: - /query-dsl/aggregations/bucket/diversified-sampler/ diff --git a/_aggregations/bucket/filter.md b/_aggregations/bucket/filter.md index 0768ea1148..58624c222c 100644 --- a/_aggregations/bucket/filter.md +++ b/_aggregations/bucket/filter.md @@ -2,7 +2,6 @@ layout: default title: Filter parent: Bucket aggregations -grand_parent: Aggregations nav_order: 50 redirect_from: - /query-dsl/aggregations/bucket/filter/ diff --git a/_aggregations/bucket/filters.md b/_aggregations/bucket/filters.md index b3977da7c1..2e9270b30d 100644 --- a/_aggregations/bucket/filters.md +++ b/_aggregations/bucket/filters.md @@ -2,7 +2,6 @@ layout: default title: Filters parent: Bucket aggregations -grand_parent: Aggregations nav_order: 60 redirect_from: - /query-dsl/aggregations/bucket/filters/ diff --git a/_aggregations/bucket/geo-distance.md b/_aggregations/bucket/geo-distance.md index a111015ac1..7b8e660630 100644 --- a/_aggregations/bucket/geo-distance.md +++ b/_aggregations/bucket/geo-distance.md @@ -2,7 +2,6 @@ layout: default title: Geodistance parent: Bucket aggregations -grand_parent: Aggregations nav_order: 70 redirect_from: - /query-dsl/aggregations/bucket/geo-distance/ diff --git a/_aggregations/bucket/geohash-grid.md b/_aggregations/bucket/geohash-grid.md index 13f89799ba..3969ea9a13 100644 --- a/_aggregations/bucket/geohash-grid.md +++ b/_aggregations/bucket/geohash-grid.md @@ -2,7 +2,6 @@ layout: default title: Geohash grid parent: Bucket aggregations -grand_parent: Aggregations nav_order: 80 redirect_from: - /query-dsl/aggregations/bucket/geohash-grid/ diff --git a/_aggregations/bucket/geohex-grid.md b/_aggregations/bucket/geohex-grid.md index 03fd45e369..eef2ed0b21 100644 --- a/_aggregations/bucket/geohex-grid.md +++ b/_aggregations/bucket/geohex-grid.md @@ -2,7 +2,6 @@ layout: default title: Geohex grid parent: Bucket aggregations -grand_parent: Aggregations nav_order: 85 redirect_from: - /opensearch/geohexgrid-agg/ diff --git a/_aggregations/bucket/geotile-grid.md b/_aggregations/bucket/geotile-grid.md index dd0c4f8a1f..e8e80451f5 100644 --- a/_aggregations/bucket/geotile-grid.md +++ b/_aggregations/bucket/geotile-grid.md @@ -2,7 +2,6 @@ layout: default title: Geotile grid parent: Bucket aggregations -grand_parent: Aggregations nav_order: 87 redirect_from: - /query-dsl/aggregations/bucket/geotile-grid/ diff --git a/_aggregations/bucket/global.md b/_aggregations/bucket/global.md index bfd516b8a3..483c28a69e 100644 --- a/_aggregations/bucket/global.md +++ b/_aggregations/bucket/global.md @@ -2,7 +2,6 @@ layout: default title: Global parent: Bucket aggregations -grand_parent: Aggregations nav_order: 90 redirect_from: - /query-dsl/aggregations/bucket/global/ diff --git a/_aggregations/bucket/histogram.md b/_aggregations/bucket/histogram.md index 0d9f2bb964..b97755bcae 100644 --- a/_aggregations/bucket/histogram.md +++ b/_aggregations/bucket/histogram.md @@ -2,7 +2,6 @@ layout: default title: Histogram parent: Bucket aggregations -grand_parent: Aggregations nav_order: 100 redirect_from: - /query-dsl/aggregations/bucket/histogram/ diff --git a/_aggregations/bucket/ip-range.md b/_aggregations/bucket/ip-range.md index 897827d412..bef4eb8f1f 100644 --- a/_aggregations/bucket/ip-range.md +++ b/_aggregations/bucket/ip-range.md @@ -2,7 +2,6 @@ layout: default title: IP range parent: Bucket aggregations -grand_parent: Aggregations nav_order: 110 redirect_from: - /query-dsl/aggregations/bucket/ip-range/ diff --git a/_aggregations/bucket/missing.md b/_aggregations/bucket/missing.md index 547076859d..e9bd5981a1 100644 --- a/_aggregations/bucket/missing.md +++ b/_aggregations/bucket/missing.md @@ -2,7 +2,6 @@ layout: default title: Missing parent: Bucket aggregations -grand_parent: Aggregations nav_order: 120 redirect_from: - /query-dsl/aggregations/bucket/missing/ diff --git a/_aggregations/bucket/multi-terms.md b/_aggregations/bucket/multi-terms.md index 62a4d264e0..8c99a450fe 100644 --- a/_aggregations/bucket/multi-terms.md +++ b/_aggregations/bucket/multi-terms.md @@ -2,7 +2,6 @@ layout: default title: Multi-terms parent: Bucket aggregations -grand_parent: Aggregations nav_order: 130 redirect_from: - /query-dsl/aggregations/bucket/multi-terms/ diff --git a/_aggregations/bucket/nested.md b/_aggregations/bucket/nested.md index 94a0f4416a..89c44c6457 100644 --- a/_aggregations/bucket/nested.md +++ b/_aggregations/bucket/nested.md @@ -2,7 +2,6 @@ layout: default title: Nested parent: Bucket aggregations -grand_parent: Aggregations nav_order: 140 redirect_from: - /query-dsl/aggregations/bucket/nested/ diff --git a/_aggregations/bucket/range.md b/_aggregations/bucket/range.md index f4e19f188d..7b17b64ed4 100644 --- a/_aggregations/bucket/range.md +++ b/_aggregations/bucket/range.md @@ -2,7 +2,6 @@ layout: default title: Range parent: Bucket aggregations -grand_parent: Aggregations nav_order: 150 redirect_from: - /query-dsl/aggregations/bucket/date-range/ diff --git a/_aggregations/bucket/reverse-nested.md b/_aggregations/bucket/reverse-nested.md index bfd04986fa..3757ec7ab7 100644 --- a/_aggregations/bucket/reverse-nested.md +++ b/_aggregations/bucket/reverse-nested.md @@ -2,7 +2,6 @@ layout: default title: Reverse nested parent: Bucket aggregations -grand_parent: Aggregations nav_order: 160 redirect_from: - /query-dsl/aggregations/bucket/reverse-nested/ diff --git a/_aggregations/bucket/sampler.md b/_aggregations/bucket/sampler.md index 5411052d45..7d8d71e04a 100644 --- a/_aggregations/bucket/sampler.md +++ b/_aggregations/bucket/sampler.md @@ -2,7 +2,6 @@ layout: default title: Sampler parent: Bucket aggregations -grand_parent: Aggregations nav_order: 170 redirect_from: - /query-dsl/aggregations/bucket/diversified-sampler/ diff --git a/_aggregations/bucket/significant-terms.md b/_aggregations/bucket/significant-terms.md index 34a4354a73..c255379dd8 100644 --- a/_aggregations/bucket/significant-terms.md +++ b/_aggregations/bucket/significant-terms.md @@ -2,7 +2,6 @@ layout: default title: Significant terms parent: Bucket aggregations -grand_parent: Aggregations nav_order: 180 redirect_from: - /query-dsl/aggregations/bucket/significant-terms/ diff --git a/_aggregations/bucket/significant-text.md b/_aggregations/bucket/significant-text.md index 6f1c7ebeca..b30b3aba95 100644 --- a/_aggregations/bucket/significant-text.md +++ b/_aggregations/bucket/significant-text.md @@ -2,7 +2,6 @@ layout: default title: Significant text parent: Bucket aggregations -grand_parent: Aggregations nav_order: 190 redirect_from: - /query-dsl/aggregations/bucket/significant-text/ diff --git a/_aggregations/bucket/terms.md b/_aggregations/bucket/terms.md index 5d05c328d4..b36214e3f6 100644 --- a/_aggregations/bucket/terms.md +++ b/_aggregations/bucket/terms.md @@ -2,7 +2,6 @@ layout: default title: Terms parent: Bucket aggregations -grand_parent: Aggregations nav_order: 200 redirect_from: - /query-dsl/aggregations/bucket/terms/ diff --git a/_aggregations/index.md b/_aggregations/index.md index 385c7a09d8..a1c457be02 100644 --- a/_aggregations/index.md +++ b/_aggregations/index.md @@ -1,7 +1,7 @@ --- layout: default title: Aggregations -has_children: true +has_children: false nav_order: 5 nav_exclude: true permalink: /aggregations/ diff --git a/_aggregations/metric/average.md b/_aggregations/metric/average.md index 428f1e76b6..9ad0c582fe 100644 --- a/_aggregations/metric/average.md +++ b/_aggregations/metric/average.md @@ -2,7 +2,6 @@ layout: default title: Average parent: Metric aggregations -grand_parent: Aggregations nav_order: 10 redirect_from: - /query-dsl/aggregations/metric/average/ diff --git a/_aggregations/metric/cardinality.md b/_aggregations/metric/cardinality.md index c40dbb4497..e03a561adb 100644 --- a/_aggregations/metric/cardinality.md +++ b/_aggregations/metric/cardinality.md @@ -2,7 +2,6 @@ layout: default title: Cardinality parent: Metric aggregations -grand_parent: Aggregations nav_order: 20 redirect_from: - /query-dsl/aggregations/metric/cardinality/ diff --git a/_aggregations/metric/extended-stats.md b/_aggregations/metric/extended-stats.md index 633407dab0..467fa348b7 100644 --- a/_aggregations/metric/extended-stats.md +++ b/_aggregations/metric/extended-stats.md @@ -2,7 +2,6 @@ layout: default title: Extended stats parent: Metric aggregations -grand_parent: Aggregations nav_order: 30 redirect_from: - /query-dsl/aggregations/metric/extended-stats/ diff --git a/_aggregations/metric/geobounds.md b/_aggregations/metric/geobounds.md index 27b7646ca5..9489c6b18e 100644 --- a/_aggregations/metric/geobounds.md +++ b/_aggregations/metric/geobounds.md @@ -2,7 +2,6 @@ layout: default title: Geobounds parent: Metric aggregations -grand_parent: Aggregations nav_order: 40 redirect_from: - /query-dsl/aggregations/metric/geobounds/ diff --git a/_aggregations/metric/geocentroid.md b/_aggregations/metric/geocentroid.md index 711f49862a..14a2d179bb 100644 --- a/_aggregations/metric/geocentroid.md +++ b/_aggregations/metric/geocentroid.md @@ -2,7 +2,6 @@ layout: default title: Geocentroid parent: Metric aggregations -grand_parent: Aggregations nav_order: 45 --- diff --git a/_aggregations/metric/matrix-stats.md b/_aggregations/metric/matrix-stats.md index 475e0caa24..188f8745fb 100644 --- a/_aggregations/metric/matrix-stats.md +++ b/_aggregations/metric/matrix-stats.md @@ -2,7 +2,6 @@ layout: default title: Matrix stats parent: Metric aggregations -grand_parent: Aggregations nav_order: 50 redirect_from: - /query-dsl/aggregations/metric/matrix-stats/ diff --git a/_aggregations/metric/maximum.md b/_aggregations/metric/maximum.md index 63b4d62a7b..1a1aaff607 100644 --- a/_aggregations/metric/maximum.md +++ b/_aggregations/metric/maximum.md @@ -2,7 +2,6 @@ layout: default title: Maximum parent: Metric aggregations -grand_parent: Aggregations nav_order: 60 redirect_from: - /query-dsl/aggregations/metric/maximum/ diff --git a/_aggregations/metric/median-absolute-deviation.md b/_aggregations/metric/median-absolute-deviation.md index 7332d7eb2f..a882475158 100644 --- a/_aggregations/metric/median-absolute-deviation.md +++ b/_aggregations/metric/median-absolute-deviation.md @@ -2,7 +2,6 @@ layout: default title: Median absolute deviation parent: Metric aggregations -grand_parent: Aggregations nav_order: 65 redirect_from: - /query-dsl/aggregations/metric/median-absolute-deviation/ diff --git a/_aggregations/metric/minimum.md b/_aggregations/metric/minimum.md index dd17c854a9..9455c71fea 100644 --- a/_aggregations/metric/minimum.md +++ b/_aggregations/metric/minimum.md @@ -2,7 +2,6 @@ layout: default title: Minimum parent: Metric aggregations -grand_parent: Aggregations nav_order: 70 redirect_from: - /query-dsl/aggregations/metric/minimum/ diff --git a/_aggregations/metric/percentile-ranks.md b/_aggregations/metric/percentile-ranks.md index 33ccb3d291..660cb01bd1 100644 --- a/_aggregations/metric/percentile-ranks.md +++ b/_aggregations/metric/percentile-ranks.md @@ -2,7 +2,6 @@ layout: default title: Percentile ranks parent: Metric aggregations -grand_parent: Aggregations nav_order: 80 redirect_from: - /query-dsl/aggregations/metric/percentile-ranks/ diff --git a/_aggregations/metric/percentile.md b/_aggregations/metric/percentile.md index c68b0e0ec7..0f241306d1 100644 --- a/_aggregations/metric/percentile.md +++ b/_aggregations/metric/percentile.md @@ -2,7 +2,6 @@ layout: default title: Percentile parent: Metric aggregations -grand_parent: Aggregations nav_order: 90 redirect_from: - /query-dsl/aggregations/metric/percentile/ diff --git a/_aggregations/metric/scripted-metric.md b/_aggregations/metric/scripted-metric.md index d1807efbc0..4247f9aa0e 100644 --- a/_aggregations/metric/scripted-metric.md +++ b/_aggregations/metric/scripted-metric.md @@ -2,7 +2,6 @@ layout: default title: Scripted metric parent: Metric aggregations -grand_parent: Aggregations nav_order: 100 redirect_from: - /query-dsl/aggregations/metric/scripted-metric/ diff --git a/_aggregations/metric/stats.md b/_aggregations/metric/stats.md index 0a54831522..d8ba5963e0 100644 --- a/_aggregations/metric/stats.md +++ b/_aggregations/metric/stats.md @@ -2,7 +2,6 @@ layout: default title: Stats parent: Metric aggregations -grand_parent: Aggregations nav_order: 110 redirect_from: - /query-dsl/aggregations/metric/stats/ diff --git a/_aggregations/metric/sum.md b/_aggregations/metric/sum.md index 0320de63fc..2e0b32cb3d 100644 --- a/_aggregations/metric/sum.md +++ b/_aggregations/metric/sum.md @@ -2,7 +2,6 @@ layout: default title: Sum parent: Metric aggregations -grand_parent: Aggregations nav_order: 120 redirect_from: - /query-dsl/aggregations/metric/sum/ diff --git a/_aggregations/metric/top-hits.md b/_aggregations/metric/top-hits.md index b6752300b2..cead3f77f2 100644 --- a/_aggregations/metric/top-hits.md +++ b/_aggregations/metric/top-hits.md @@ -2,7 +2,6 @@ layout: default title: Top hits parent: Metric aggregations -grand_parent: Aggregations nav_order: 130 redirect_from: - /query-dsl/aggregations/metric/top-hits/ diff --git a/_aggregations/metric/value-count.md b/_aggregations/metric/value-count.md index dfddaf9417..596fb5d806 100644 --- a/_aggregations/metric/value-count.md +++ b/_aggregations/metric/value-count.md @@ -2,7 +2,6 @@ layout: default title: Value count parent: Metric aggregations -grand_parent: Aggregations nav_order: 140 redirect_from: - /query-dsl/aggregations/metric/value-count/ diff --git a/_aggregations/metric/weighted-avg.md b/_aggregations/metric/weighted-avg.md index 268f78bfdc..6f67939d6e 100644 --- a/_aggregations/metric/weighted-avg.md +++ b/_aggregations/metric/weighted-avg.md @@ -2,7 +2,6 @@ layout: default title: Weighted average parent: Metric aggregations -grand_parent: Aggregations nav_order: 150 --- diff --git a/_query-dsl/compound/bool.md b/_query-dsl/compound/bool.md index 12caea2f8d..4479094214 100644 --- a/_query-dsl/compound/bool.md +++ b/_query-dsl/compound/bool.md @@ -2,7 +2,6 @@ layout: default title: Boolean parent: Compound queries -grand_parent: Query DSL nav_order: 10 redirect_from: - /opensearch/query-dsl/compound/bool/ diff --git a/_query-dsl/compound/boosting.md b/_query-dsl/compound/boosting.md index 7aa9c6c035..ede7eb3d51 100644 --- a/_query-dsl/compound/boosting.md +++ b/_query-dsl/compound/boosting.md @@ -2,7 +2,6 @@ layout: default title: Boosting parent: Compound queries -grand_parent: Query DSL nav_order: 30 redirect_from: - /query-dsl/query-dsl/compound/boosting/ diff --git a/_query-dsl/compound/constant-score.md b/_query-dsl/compound/constant-score.md index ed12e33ec0..bb5af3800a 100644 --- a/_query-dsl/compound/constant-score.md +++ b/_query-dsl/compound/constant-score.md @@ -2,7 +2,6 @@ layout: default title: Constant score parent: Compound queries -grand_parent: Query DSL nav_order: 40 redirect_from: - /query-dsl/query-dsl/compound/constant-score/ diff --git a/_query-dsl/compound/disjunction-max.md b/_query-dsl/compound/disjunction-max.md index 8dd9e41d2c..09a7e0e729 100644 --- a/_query-dsl/compound/disjunction-max.md +++ b/_query-dsl/compound/disjunction-max.md @@ -2,7 +2,6 @@ layout: default title: Disjunction max parent: Compound queries -grand_parent: Query DSL nav_order: 50 redirect_from: - /query-dsl/query-dsl/compound/disjunction-max/ diff --git a/_query-dsl/compound/function-score.md b/_query-dsl/compound/function-score.md index 98568e0965..b28a6abed6 100644 --- a/_query-dsl/compound/function-score.md +++ b/_query-dsl/compound/function-score.md @@ -2,7 +2,6 @@ layout: default title: Function score parent: Compound queries -grand_parent: Query DSL nav_order: 60 has_math: true redirect_from: diff --git a/_query-dsl/compound/hybrid.md b/_query-dsl/compound/hybrid.md index 22b3a17fc1..69ce89ce17 100644 --- a/_query-dsl/compound/hybrid.md +++ b/_query-dsl/compound/hybrid.md @@ -2,7 +2,6 @@ layout: default title: Hybrid parent: Compound queries -grand_parent: Query DSL nav_order: 70 --- diff --git a/_query-dsl/full-text/intervals.md b/_query-dsl/full-text/intervals.md index 082f8fbe46..a31f7434b3 100644 --- a/_query-dsl/full-text/intervals.md +++ b/_query-dsl/full-text/intervals.md @@ -3,7 +3,6 @@ layout: default title: Intervals nav_order: 80 parent: Full-text queries -grand_parent: Query DSL --- # Intervals query diff --git a/_query-dsl/full-text/match-bool-prefix.md b/_query-dsl/full-text/match-bool-prefix.md index 3a0d304ce4..3964dc5ee8 100644 --- a/_query-dsl/full-text/match-bool-prefix.md +++ b/_query-dsl/full-text/match-bool-prefix.md @@ -2,7 +2,6 @@ layout: default title: Match Boolean prefix parent: Full-text queries -grand_parent: Query DSL nav_order: 20 --- diff --git a/_query-dsl/full-text/match-phrase-prefix.md b/_query-dsl/full-text/match-phrase-prefix.md index 354dd35c61..9e05f034d7 100644 --- a/_query-dsl/full-text/match-phrase-prefix.md +++ b/_query-dsl/full-text/match-phrase-prefix.md @@ -2,7 +2,6 @@ layout: default title: Match phrase prefix parent: Full-text queries -grand_parent: Query DSL nav_order: 40 --- diff --git a/_query-dsl/full-text/match-phrase.md b/_query-dsl/full-text/match-phrase.md index 18dd6a858c..747c4814d9 100644 --- a/_query-dsl/full-text/match-phrase.md +++ b/_query-dsl/full-text/match-phrase.md @@ -2,7 +2,6 @@ layout: default title: Match phrase parent: Full-text queries -grand_parent: Query DSL nav_order: 30 --- diff --git a/_query-dsl/full-text/match.md b/_query-dsl/full-text/match.md index 746a4cf5b6..b4db30ec1f 100644 --- a/_query-dsl/full-text/match.md +++ b/_query-dsl/full-text/match.md @@ -2,7 +2,6 @@ layout: default title: Match parent: Full-text queries -grand_parent: Query DSL nav_order: 10 --- diff --git a/_query-dsl/full-text/multi-match.md b/_query-dsl/full-text/multi-match.md index 7450b74721..ab1496fdd3 100644 --- a/_query-dsl/full-text/multi-match.md +++ b/_query-dsl/full-text/multi-match.md @@ -2,7 +2,6 @@ layout: default title: Multi-match parent: Full-text queries -grand_parent: Query DSL nav_order: 50 --- diff --git a/_query-dsl/full-text/query-string.md b/_query-dsl/full-text/query-string.md index 12609e29c0..47180e3f6d 100644 --- a/_query-dsl/full-text/query-string.md +++ b/_query-dsl/full-text/query-string.md @@ -2,9 +2,7 @@ layout: default title: Query string parent: Full-text queries -grand_parent: Query DSL nav_order: 60 - redirect_from: - /opensearch/query-dsl/full-text/query-string/ - /query-dsl/query-dsl/full-text/query-string/ diff --git a/_query-dsl/full-text/simple-query-string.md b/_query-dsl/full-text/simple-query-string.md index fbf37f588d..58780cfdb4 100644 --- a/_query-dsl/full-text/simple-query-string.md +++ b/_query-dsl/full-text/simple-query-string.md @@ -2,7 +2,6 @@ layout: default title: Simple query string parent: Full-text queries -grand_parent: Query DSL nav_order: 70 --- diff --git a/_query-dsl/geo-and-xy/geo-bounding-box.md b/_query-dsl/geo-and-xy/geo-bounding-box.md index df697e2ce5..1112a4278e 100644 --- a/_query-dsl/geo-and-xy/geo-bounding-box.md +++ b/_query-dsl/geo-and-xy/geo-bounding-box.md @@ -2,7 +2,6 @@ layout: default title: Geo-bounding box parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 10 redirect_from: - /opensearch/query-dsl/geo-and-xy/geo-bounding-box/ diff --git a/_query-dsl/geo-and-xy/geodistance.md b/_query-dsl/geo-and-xy/geodistance.md index 7a36b0c933..b272cad81e 100644 --- a/_query-dsl/geo-and-xy/geodistance.md +++ b/_query-dsl/geo-and-xy/geodistance.md @@ -2,7 +2,6 @@ layout: default title: Geodistance parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 20 --- diff --git a/_query-dsl/geo-and-xy/geopolygon.md b/_query-dsl/geo-and-xy/geopolygon.md index c53b1379cf..980a0c5a63 100644 --- a/_query-dsl/geo-and-xy/geopolygon.md +++ b/_query-dsl/geo-and-xy/geopolygon.md @@ -2,7 +2,6 @@ layout: default title: Geopolygon parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 30 --- diff --git a/_query-dsl/geo-and-xy/xy.md b/_query-dsl/geo-and-xy/xy.md index 88a22448c3..d0ed61c050 100644 --- a/_query-dsl/geo-and-xy/xy.md +++ b/_query-dsl/geo-and-xy/xy.md @@ -2,9 +2,7 @@ layout: default title: xy parent: Geographic and xy queries -grand_parent: Query DSL nav_order: 50 - redirect_from: - /opensearch/query-dsl/geo-and-xy/xy/ - /query-dsl/query-dsl/geo-and-xy/xy/ diff --git a/_query-dsl/specialized/neural-sparse.md b/_query-dsl/specialized/neural-sparse.md index 47f77fa95d..8de3eaf693 100644 --- a/_query-dsl/specialized/neural-sparse.md +++ b/_query-dsl/specialized/neural-sparse.md @@ -2,7 +2,6 @@ layout: default title: Neural sparse parent: Specialized queries -grand_parent: Query DSL nav_order: 55 --- diff --git a/_query-dsl/specialized/neural.md b/_query-dsl/specialized/neural.md index fea949cf52..14b930cdb6 100644 --- a/_query-dsl/specialized/neural.md +++ b/_query-dsl/specialized/neural.md @@ -2,7 +2,6 @@ layout: default title: Neural parent: Specialized queries -grand_parent: Query DSL nav_order: 50 --- diff --git a/_query-dsl/specialized/script-score.md b/_query-dsl/specialized/script-score.md index d09158f20a..f65c266cc5 100644 --- a/_query-dsl/specialized/script-score.md +++ b/_query-dsl/specialized/script-score.md @@ -2,7 +2,6 @@ layout: default title: Script score parent: Specialized queries -grand_parent: Query DSL nav_order: 60 --- diff --git a/_query-dsl/term/exists.md b/_query-dsl/term/exists.md index 1d52744c91..95573a36f8 100644 --- a/_query-dsl/term/exists.md +++ b/_query-dsl/term/exists.md @@ -2,7 +2,6 @@ layout: default title: Exists parent: Term-level queries -grand_parent: Query DSL nav_order: 10 --- diff --git a/_query-dsl/term/fuzzy.md b/_query-dsl/term/fuzzy.md index 9afa85ea93..bf2bd43bba 100644 --- a/_query-dsl/term/fuzzy.md +++ b/_query-dsl/term/fuzzy.md @@ -2,7 +2,6 @@ layout: default title: Fuzzy parent: Term-level queries -grand_parent: Query DSL nav_order: 20 --- diff --git a/_query-dsl/term/ids.md b/_query-dsl/term/ids.md index 0c3b5393fb..f895745c97 100644 --- a/_query-dsl/term/ids.md +++ b/_query-dsl/term/ids.md @@ -2,7 +2,6 @@ layout: default title: IDs parent: Term-level queries -grand_parent: Query DSL nav_order: 30 --- diff --git a/_query-dsl/term/prefix.md b/_query-dsl/term/prefix.md index eda5307d14..2a429c9f0e 100644 --- a/_query-dsl/term/prefix.md +++ b/_query-dsl/term/prefix.md @@ -2,7 +2,6 @@ layout: default title: Prefix parent: Term-level queries -grand_parent: Query DSL nav_order: 40 --- diff --git a/_query-dsl/term/range.md b/_query-dsl/term/range.md index 8a8f53c480..ceb264db76 100644 --- a/_query-dsl/term/range.md +++ b/_query-dsl/term/range.md @@ -2,7 +2,6 @@ layout: default title: Range parent: Term-level queries -grand_parent: Query DSL nav_order: 50 --- diff --git a/_query-dsl/term/regexp.md b/_query-dsl/term/regexp.md index 65d6953516..4a038729c0 100644 --- a/_query-dsl/term/regexp.md +++ b/_query-dsl/term/regexp.md @@ -2,7 +2,6 @@ layout: default title: Regexp parent: Term-level queries -grand_parent: Query DSL nav_order: 60 --- diff --git a/_query-dsl/term/term.md b/_query-dsl/term/term.md index c1c296b9a0..a33146f6aa 100644 --- a/_query-dsl/term/term.md +++ b/_query-dsl/term/term.md @@ -2,7 +2,6 @@ layout: default title: Term parent: Term-level queries -grand_parent: Query DSL nav_order: 70 --- diff --git a/_query-dsl/term/terms-set.md b/_query-dsl/term/terms-set.md index ea0251ddff..6652d15979 100644 --- a/_query-dsl/term/terms-set.md +++ b/_query-dsl/term/terms-set.md @@ -2,7 +2,6 @@ layout: default title: Terms set parent: Term-level queries -grand_parent: Query DSL nav_order: 90 --- diff --git a/_query-dsl/term/terms.md b/_query-dsl/term/terms.md index fd15126255..42c74c0436 100644 --- a/_query-dsl/term/terms.md +++ b/_query-dsl/term/terms.md @@ -2,7 +2,6 @@ layout: default title: Terms parent: Term-level queries -grand_parent: Query DSL nav_order: 80 --- diff --git a/_query-dsl/term/wildcard.md b/_query-dsl/term/wildcard.md index 0652581941..b2d7238758 100644 --- a/_query-dsl/term/wildcard.md +++ b/_query-dsl/term/wildcard.md @@ -2,7 +2,6 @@ layout: default title: Wildcard parent: Term-level queries -grand_parent: Query DSL nav_order: 100 --- diff --git a/_search-plugins/knn/api.md b/_search-plugins/knn/api.md index 23678063a8..c7314f7ae2 100644 --- a/_search-plugins/knn/api.md +++ b/_search-plugins/knn/api.md @@ -3,7 +3,6 @@ layout: default title: k-NN plugin API nav_order: 30 parent: k-NN search -grand_parent: Search methods has_children: false --- diff --git a/_search-plugins/knn/approximate-knn.md b/_search-plugins/knn/approximate-knn.md index c0a9557728..144365166f 100644 --- a/_search-plugins/knn/approximate-knn.md +++ b/_search-plugins/knn/approximate-knn.md @@ -3,7 +3,6 @@ layout: default title: Approximate k-NN search nav_order: 15 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/filter-search-knn.md b/_search-plugins/knn/filter-search-knn.md index 309cf6850e..2f0c4aa072 100644 --- a/_search-plugins/knn/filter-search-knn.md +++ b/_search-plugins/knn/filter-search-knn.md @@ -3,7 +3,6 @@ layout: default title: k-NN search with filters nav_order: 20 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/jni-libraries.md b/_search-plugins/knn/jni-libraries.md index 59a5b7a1e2..4dbdb2da56 100644 --- a/_search-plugins/knn/jni-libraries.md +++ b/_search-plugins/knn/jni-libraries.md @@ -3,7 +3,6 @@ layout: default title: JNI libraries nav_order: 35 parent: k-NN search -grand_parent: Search methods has_children: false redirect_from: - /search-plugins/knn/jni-library/ diff --git a/_search-plugins/knn/knn-index.md b/_search-plugins/knn/knn-index.md index ab24a0c097..ed8b9217f5 100644 --- a/_search-plugins/knn/knn-index.md +++ b/_search-plugins/knn/knn-index.md @@ -3,7 +3,6 @@ layout: default title: k-NN index nav_order: 5 parent: k-NN search -grand_parent: Search methods has_children: false --- diff --git a/_search-plugins/knn/knn-score-script.md b/_search-plugins/knn/knn-score-script.md index cc79e90850..1696bd4cad 100644 --- a/_search-plugins/knn/knn-score-script.md +++ b/_search-plugins/knn/knn-score-script.md @@ -3,7 +3,6 @@ layout: default title: Exact k-NN with scoring script nav_order: 10 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/knn-vector-quantization.md b/_search-plugins/knn/knn-vector-quantization.md index 549437f346..fe4833ee47 100644 --- a/_search-plugins/knn/knn-vector-quantization.md +++ b/_search-plugins/knn/knn-vector-quantization.md @@ -3,7 +3,6 @@ layout: default title: k-NN vector quantization nav_order: 27 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/nested-search-knn.md b/_search-plugins/knn/nested-search-knn.md index bdc1045387..d947ebc6e6 100644 --- a/_search-plugins/knn/nested-search-knn.md +++ b/_search-plugins/knn/nested-search-knn.md @@ -3,7 +3,6 @@ layout: default title: k-NN search with nested fields nav_order: 21 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/painless-functions.md b/_search-plugins/knn/painless-functions.md index 1f27cc29a6..09eb989702 100644 --- a/_search-plugins/knn/painless-functions.md +++ b/_search-plugins/knn/painless-functions.md @@ -3,7 +3,6 @@ layout: default title: k-NN Painless extensions nav_order: 25 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/performance-tuning.md b/_search-plugins/knn/performance-tuning.md index 24d92bd67d..123b1daef1 100644 --- a/_search-plugins/knn/performance-tuning.md +++ b/_search-plugins/knn/performance-tuning.md @@ -2,7 +2,6 @@ layout: default title: Performance tuning parent: k-NN search -grand_parent: Search methods nav_order: 45 --- diff --git a/_search-plugins/knn/radial-search-knn.md b/_search-plugins/knn/radial-search-knn.md index 48aaac034d..1a4a223294 100644 --- a/_search-plugins/knn/radial-search-knn.md +++ b/_search-plugins/knn/radial-search-knn.md @@ -3,7 +3,6 @@ layout: default title: Radial search nav_order: 28 parent: k-NN search -grand_parent: Search methods has_children: false has_math: true --- diff --git a/_search-plugins/knn/settings.md b/_search-plugins/knn/settings.md index 4d84cc80bb..1b9aa3608c 100644 --- a/_search-plugins/knn/settings.md +++ b/_search-plugins/knn/settings.md @@ -2,7 +2,6 @@ layout: default title: Settings parent: k-NN search -grand_parent: Search methods nav_order: 40 --- From f1cb58fd0c5ad8565e929e162157e857b5d42c06 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Wed, 17 Jul 2024 16:01:56 -0500 Subject: [PATCH 04/13] Remove redundant source (#7755) Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _api-reference/document-apis/reindex.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_api-reference/document-apis/reindex.md b/_api-reference/document-apis/reindex.md index 48f14923f5..c2afa347e1 100644 --- a/_api-reference/document-apis/reindex.md +++ b/_api-reference/document-apis/reindex.md @@ -79,7 +79,6 @@ version_type | The indexing operation's version type. Valid values are `internal op_type | Whether to copy over documents that are missing in the destination index. Valid values are `create` (ignore documents with the same ID from the source index) and `index` (copy everything from the source index). pipeline | Which ingest pipeline to utilize during the reindex. script | A script that OpenSearch uses to apply transformations to the data during the reindex operation. -source | The actual script that OpenSearch runs. lang | The scripting language. Valid options are `painless`, `expression`, `mustache`, and `java`. ## Response From 7bfee4011d0a82644e3e5868717f44eac82ea6b4 Mon Sep 17 00:00:00 2001 From: Michael Oviedo Date: Wed, 17 Jul 2024 15:15:25 -0700 Subject: [PATCH 05/13] Update compare.md (#7765) The `--results-number-align` option for the compare API is actually spelled `--results-numbers-align`. This change updates the spelling for that option. Signed-off-by: Michael Oviedo --- _benchmark/reference/commands/compare.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_benchmark/reference/commands/compare.md b/_benchmark/reference/commands/compare.md index a9eb8bcc10..35bafe0704 100644 --- a/_benchmark/reference/commands/compare.md +++ b/_benchmark/reference/commands/compare.md @@ -130,7 +130,7 @@ You can use the following options to customize the results of your test comparis - `--baseline`: The baseline TestExecution ID used to compare the contender TestExecution. - `--contender`: The TestExecution ID for the contender being compared to the baseline. - `--results-format`: Defines the output format for the command line results, either `markdown` or `csv`. Default is `markdown`. -- `--results-number-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. +- `--results-numbers-align`: Defines the column number alignment for when the `compare` command outputs results. Default is `right`. - `--results-file`: When provided a file path, writes the compare results to the file indicated in the path. - `--show-in-results`: Determines whether or not to include the comparison in the results file. From 3f37837c631d6f99c14f823965e3fdcedfa67dc8 Mon Sep 17 00:00:00 2001 From: gaobinlong Date: Thu, 18 Jul 2024 23:37:15 +0800 Subject: [PATCH 06/13] Document CreateAnomalyDetectorTool (#7742) * Document CreateAnomalyDetectorTool Signed-off-by: gaobinlong * Fix format issue Signed-off-by: gaobinlong * Fix link Signed-off-by: gaobinlong * Update create-anomaly-detector.md Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/index.md Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/index.md Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _ml-commons-plugin/agents-tools/tools/index.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --------- Signed-off-by: gaobinlong Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower --- .../tools/create-anomaly-detector.md | 169 ++++++++++++++++++ .../agents-tools/tools/index.md | 3 +- 2 files changed, 171 insertions(+), 1 deletion(-) create mode 100644 _ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md diff --git a/_ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md b/_ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md new file mode 100644 index 0000000000..b6fae8131c --- /dev/null +++ b/_ml-commons-plugin/agents-tools/tools/create-anomaly-detector.md @@ -0,0 +1,169 @@ +--- +layout: default +title: CreateAnomalyDetectorTool +has_children: false +has_toc: false +nav_order: 70 +parent: Tools +grand_parent: Agents and tools +--- + + +# CreateAnomalyDetectorTool +**Introduced 2.16** +{: .label .label-purple } + + +This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/skills/issues/337). +{: .warning} + +The `CreateAnomalyDetectorTool` helps create anomaly detectors based on your provided index. This tool retrieves index mappings and enables a large language model (LLM) to recommend category fields, aggregation fields, and their corresponding aggregation methods, which are required by the Create Anomaly Detector API. + +For comprehensive information about anomaly detectors, see [Anomaly detection]({{site.url}}{{site.baseurl}}/observing-your-data/ad/index/). +{: .tip} + +## Step 1: Register a flow agent that runs the CreateAnomalyDetectorTool + +A flow agent runs a sequence of tools in order, returning the output of the last tool. To create a flow agent, send the following register agent request: + +```json +POST /_plugins/_ml/agents/_register +{ + "name": "Test_Agent_For_Create_Anomaly_Detector_Tool", + "type": "flow", + "description": "this is a test agent for the CreateAnomalyDetectorTool", + "memory": { + "type": "demo" + }, + "tools": [ + { + "type": "CreateAnomalyDetectorTool", + "name": "DemoCreateAnomalyDetectorTool", + "parameters": { + "model_id": "" + } + } + ] +} +``` +{% include copy-curl.html %} + +OpenSearch responds with an agent ID, for example, as follows: + +```json +{ + "agent_id": "EuJYYo0B9RaBCvhuy1q8" +} +``` +{% include copy-curl.html %} + +## Step 2: Run the agent + +Run the agent by sending the following request: + +```json +POST /_plugins/_ml/agents/EuJYYo0B9RaBCvhuy1q8/_execute +{ + "parameters": { + "index": "sample_weblogs_test" + } +} +``` +{% include copy-curl.html %} + +OpenSearch responds with a JSON string containing all of the recommended parameters for creating an anomaly detector, such as the string shown in the following example repsonse: + +```json +{ + "inference_results": [ + { + "output": [ + { + "name": "response", + "result":"""{"index":"sample_weblogs_test","categoryField":"ip.keyword","aggregationField":"bytes,response,responseLatency","aggregationMethod":"sum,avg,avg","dateFields":"utc_time,timestamp"}""" + } + ] + } + ] +} +``` +{% include copy-curl.html %} + +You can then create an anomaly detector containing the recommended parameters by sending a request similar to the following: + +```json +POST _plugins/_anomaly_detection/detectors +{ + "name": "test-detector", + "description": "Test detector", + "time_field": "timestamp", + "indices": [ + "sample_weblogs_test" + ], + "feature_attributes": [ + { + "feature_name": "feature_bytes", + "feature_enabled": true, + "aggregation_query": { + "agg1": { + "sum": { + "field": "bytes" + } + } + } + }, + { + "feature_name": "feature_response", + "feature_enabled": true, + "aggregation_query": { + "agg2": { + "avg": { + "field": "response" + } + } + } + }, + { + "feature_name": "feature_responseLatency", + "feature_enabled": true, + "aggregation_query": { + "agg3": { + "avg": { + "field": "responseLatency" + } + } + } + } + ], + "detection_interval": { + "period": { + "interval": 1, + "unit": "Minutes" + } + }, + "window_delay": { + "period": { + "interval": 1, + "unit": "Minutes" + } + } +} +``` +{% include copy-curl.html %} + +## Register parameters + +The following table lists the available tool parameters for agent registration. + +Parameter | Type | Required/Optional | Description +:--- | :--- | :--- | :--- +`model_id` | String | Required | The LLM model ID used for suggesting required Create Anomaly Detector API parameters. +`model_type` | String | Optional | The model type. Valid values are `CLAUDE` (Anthropic Claude models) and `OPENAI` (OpenAI models). + +## Execute parameters + +The following table lists the available tool parameters for running the agent. + +Parameter | Type | Required/Optional | Description +:--- | :--- | :--- | :--- +`index` | String | Required | The index name. Supports wildcards (for example, `weblogs-*`). If wildcards are used, then the tool fetches mappings from the first resolved index and sends them to the LLM. diff --git a/_ml-commons-plugin/agents-tools/tools/index.md b/_ml-commons-plugin/agents-tools/tools/index.md index fba47b63da..bc71122949 100644 --- a/_ml-commons-plugin/agents-tools/tools/index.md +++ b/_ml-commons-plugin/agents-tools/tools/index.md @@ -35,6 +35,7 @@ Each tool takes a list of parameters specific to that tool. In the preceding exa |[`CatIndexTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/cat-index-tool/) |Retrieves index information for the OpenSearch cluster. | |[`ConnectorTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/connector-tool/) | Uses a [connector]({{site.url}}{{site.baseurl}}/ml-commons-plugin/remote-models/connectors/) to call any REST API function. | |[`IndexMappingTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/index-mapping-tool/) |Retrieves index mapping and setting information for an index. | +|[`CreateAnomalyDetectorTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/create-anomaly-detector/) | Enables an LLM to suggest required parameters for creating an anomaly detector. | |[`MLModelTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/ml-model-tool/) |Runs machine learning models. | |[`NeuralSparseSearchTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/neural-sparse-tool/) | Performs sparse vector retrieval. | |[`PPLTool`]({{site.url}}{{site.baseurl}}/ml-commons-plugin/agents-tools/tools/ppl-tool/) |Translates natural language into a Piped Processing Language (PPL) query. | @@ -49,4 +50,4 @@ Each tool takes a list of parameters specific to that tool. In the preceding exa ## Developer information -The agents and tools framework is flexible and extensible. You can find the list of tools provided by OpenSearch in the [Tools library](https://github.com/opensearch-project/skills/tree/main/src/main/java/org/opensearch/agent/tools). For a different use case, you can build your own tool by implementing the [_Tool_ interface](https://github.com/opensearch-project/ml-commons/blob/2.x/spi/src/main/java/org/opensearch/ml/common/spi/tools/Tool.java). \ No newline at end of file +The agents and tools framework offers flexibility and extensibility. See the [tools library](https://github.com/opensearch-project/skills/tree/main/src/main/java/org/opensearch/agent/tools) for OpenSearch-provided tools. Implement the [**Tool** interface](https://github.com/opensearch-project/ml-commons/blob/2.x/spi/src/main/java/org/opensearch/ml/common/spi/tools/Tool.java) to build custom tools for different use cases. From 0c3dacee582816b9ab0c7c98f927597b28677a79 Mon Sep 17 00:00:00 2001 From: gaobinlong Date: Thu, 18 Jul 2024 23:55:36 +0800 Subject: [PATCH 07/13] Add strict_allow_templates option for the dynamic mapping parameter (#7745) * Add strict_allow_templates option for the dynamic mapping parameter Signed-off-by: gaobinlong * Fix typo Signed-off-by: gaobinlong * Fix header Signed-off-by: gaobinlong * Update dynamic.md Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/supported-field-types/object.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/supported-field-types/object.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/dynamic.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _field-types/supported-field-types/nested.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update dynamic.md Make changes to address editorial review comments Signed-off-by: Melissa Vagi --------- Signed-off-by: gaobinlong Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower --- _field-types/dynamic.md | 342 +++++++++++++++++++ _field-types/supported-field-types/nested.md | 2 +- _field-types/supported-field-types/object.md | 3 +- 3 files changed, 345 insertions(+), 2 deletions(-) create mode 100644 _field-types/dynamic.md diff --git a/_field-types/dynamic.md b/_field-types/dynamic.md new file mode 100644 index 0000000000..59f59bfe3d --- /dev/null +++ b/_field-types/dynamic.md @@ -0,0 +1,342 @@ +--- +layout: default +title: Dynamic parameter +nav_order: 10 +redirect_from: + - /opensearch/dynamic/ +--- + +# Dynamic parameter + +The `dynamic` parameter specifies whether newly detected fields can be added dynamically to a mapping. It accepts the parameters listed in the following table. + +Parameter | Description +:--- | :--- +`true` | Specfies that new fields can be added dynamically to the mapping. Default is `true`. +`false` | Specifies that new fields cannot be added dynamically to the mapping. If a new field is detected, then it is not indexed or searchable but can be retrieved from the `_source` field. +`strict` | Throws an exception. The indexing operation fails when new fields are detected. +`strict_allow_templates` | Adds new fields if they match predefined dynamic templates in the mapping. + +--- + +## Example: Create an index with `dynamic` set to `true` + +1. Create an index with `dynamic` set to `true` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": true + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + } +} +``` +{% include copy-curl.html %} + +3. Confirm the mapping works as expected by sending the following request: + +```json +GET testindex1/_mapping +``` +{% include copy-curl.html %} + +The object field `patient` and two subfields `name` and `id` are added to the mapping, as shown in the following response: + +```json +{ + "testindex1": { + "mappings": { + "dynamic": "true", + "properties": { + "patient": { + "properties": { + "id": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + }, + "name": { + "type": "text", + "fields": { + "keyword": { + "type": "keyword", + "ignore_above": 256 + } + } + } + } + } + } + } + } +} +``` + +--- + +## Example: Create an index with `dynamic` set to `false` + +1. Create an index with explicit mappings and `dynamic` set to `false` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": false, + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields and additional unmapped fields by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` +{% include copy-curl.html %} + +3. Confirm the mapping works as expected by sending the following request: + +```json +GET testindex1/_mapping +``` +{% include copy-curl.html %} + +The following response shows that the new fields `room` and `floor` were not added to the mapping, which remained unchanged: + +```json +{ + "testindex1": { + "mappings": { + "dynamic": "false", + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } + } +} +``` + +4. Get the unmapped fields `room` and `floor` from the document by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` + +The following request searches for the fields `room` and `floor`: + +```json +POST testindex1/_search +{ + "query": { + "term": { + "room": "room1" + } + } +} +``` + +The response returns no results: + +```json +{ + "took": 3, + "timed_out": false, + "_shards": { + "total": 1, + "successful": 1, + "skipped": 0, + "failed": 0 + }, + "hits": { + "total": { + "value": 0, + "relation": "eq" + }, + "max_score": null, + "hits": [] + } +} +``` + +--- + +## Example: Create an index with `dynamic` set to `strict` + +1. Create an index with explicit mappings and `dynamic` set to `strict` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": strict, + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields and additional unmapped fields by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` +{% include copy-curl.html %} + +Note that an exception is thrown, as shown in the following response: + +```json +{ + "error": { + "root_cause": [ + { + "type": "strict_dynamic_mapping_exception", + "reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed" + } + ], + "type": "strict_dynamic_mapping_exception", + "reason": "mapping set to strict, dynamic introduction of [room] within [_doc] is not allowed" + }, + "status": 400 +} +``` + +--- + +## Example: Create an index with `dynamic` set to `strict_allow_templates` + +1. Create an index with predefined dynamic templates and `dynamic` set to `strict_allow_templates` by sending the following request: + +```json +PUT testindex1 +{ + "mappings": { + "dynamic": "strict_allow_templates", + "dynamic_templates": [ + { + "strings": { + "match": "room*", + "match_mapping_type": "string", + "mapping": { + "type": "keyword" + } + } + } + ], + "properties": { + "patient": { + "properties": { + "id": { + "type": "keyword" + }, + "name": { + "type": "keyword" + } + } + } + } + } +} +``` +{% include copy-curl.html %} + +2. Index a document with an object field `patient` containing two string fields and a new field `room` that matches one of the dynamic templates by sending the following request: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1" +} +``` +{% include copy-curl.html %} + +Indexing succeeds because the new field `room` matches the dynamic templates. However, indexing fails for the new field `floor` because it does not match one of the dynamic templates and is not explicitly mapped, as shown in the following response: + +```json +PUT testindex1/_doc/1 +{ + "patient": { + "name" : "John Doe", + "id" : "123456" + }, + "room": "room1", + "floor": "1" +} +``` diff --git a/_field-types/supported-field-types/nested.md b/_field-types/supported-field-types/nested.md index d61ccd53df..90d09177d1 100644 --- a/_field-types/supported-field-types/nested.md +++ b/_field-types/supported-field-types/nested.md @@ -308,7 +308,7 @@ The following table lists the parameters accepted by object field types. All par Parameter | Description :--- | :--- -[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`. +[`dynamic`]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/object#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to the object. Valid values are `true`, `false`, `strict`, and `strict_allow_templates`. Default is `true`. `include_in_parent` | A Boolean value that specifies whether all fields in the child nested object should also be added to the parent document in flattened form. Default is `false`. `include_in_root` | A Boolean value that specifies whether all fields in the child nested object should also be added to the root document in flattened form. Default is `false`. `properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`. diff --git a/_field-types/supported-field-types/object.md b/_field-types/supported-field-types/object.md index 372a5c46d9..db539a9608 100644 --- a/_field-types/supported-field-types/object.md +++ b/_field-types/supported-field-types/object.md @@ -73,7 +73,7 @@ The following table lists the parameters accepted by object field types. All par Parameter | Description :--- | :--- -[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to this object. Valid values are `true`, `false`, and `strict`. Default is `true`. +[`dynamic`](#the-dynamic-parameter) | Specifies whether new fields can be dynamically added to the object. Valid values are `true`, `false`, `strict`, and `strict_allow_templates`. Default is `true`. `enabled` | A Boolean value that specifies whether the JSON contents of the object should be parsed. If `enabled` is set to `false`, the object's contents are not indexed or searchable, but they are still retrievable from the _source field. Default is `true`. `properties` | Fields of this object, which can be of any supported type. New properties can be dynamically added to this object if `dynamic` is set to `true`. @@ -149,6 +149,7 @@ Value | Description `true` | New fields can be added to the mapping dynamically. This is the default. `false` | New fields cannot be added to the mapping dynamically. If a new field is detected, it is not indexed or searchable. However, it is still retrievable from the _source field. `strict` | When new fields are added to the mapping dynamically, an exception is thrown. To add a new field to an object, you have to add it to the mapping first. +`strict_allow_templates` | If the newly detected fields match any of the predefined dynamic templates in the mapping, then they are added to the mapping; if they do not match any of them, then an exception is thrown. Inner objects inherit the `dynamic` parameter value from their parent unless they declare their own `dynamic` parameter value. {: .note } From 4477614f4fdc980fab0808b210107e6972e057b4 Mon Sep 17 00:00:00 2001 From: AWSHurneyt Date: Thu, 18 Jul 2024 15:30:21 -0700 Subject: [PATCH 08/13] Update per-cluster-metrics-monitors.md (#7769) Fixed typo in example. Signed-off-by: AWSHurneyt --- _observing-your-data/alerting/per-cluster-metrics-monitors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_observing-your-data/alerting/per-cluster-metrics-monitors.md b/_observing-your-data/alerting/per-cluster-metrics-monitors.md index baea9c626b..bcaa03cc0c 100644 --- a/_observing-your-data/alerting/per-cluster-metrics-monitors.md +++ b/_observing-your-data/alerting/per-cluster-metrics-monitors.md @@ -91,7 +91,7 @@ The `script` parameter points the `source` to the Painless script `for (cluster "path": "_cluster/health/", "path_params": "", "url": "http://localhost:9200/_cluster/health/", - "cluster": ["cluster-1", "cluster-2"] + "clusters": ["cluster-1", "cluster-2"] } } ], From 2d55f1cf63203e32e12b3015605b4f0bb7ad7b98 Mon Sep 17 00:00:00 2001 From: Heather Halter Date: Thu, 18 Jul 2024 15:50:25 -0700 Subject: [PATCH 09/13] Update kafka.md (#7774) Fixed capitalization issue. Signed-off-by: Heather Halter --- _data-prepper/pipelines/configuration/sources/kafka.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/sources/kafka.md b/_data-prepper/pipelines/configuration/sources/kafka.md index 4df72cfdd6..e8452a93c3 100644 --- a/_data-prepper/pipelines/configuration/sources/kafka.md +++ b/_data-prepper/pipelines/configuration/sources/kafka.md @@ -120,7 +120,7 @@ Use the following options when setting SSL encryption. Option | Required | Type | Description :--- | :--- | :--- | :--- `type` | No | String | The encryption type. Use `none` to disable encryption. Default is `ssl`. -`Insecure` | No | Boolean | A Boolean flag used to turn off SSL certificate verification. If set to `true`, certificate authority (CA) certificate verification is turned off and insecure HTTP requests are sent. Default is `false`. +`insecure` | No | Boolean | A Boolean flag used to turn off SSL certificate verification. If set to `true`, certificate authority (CA) certificate verification is turned off and insecure HTTP requests are sent. Default is `false`. #### AWS From f63f8b95f6f3dfd5043e8a5152123665fc7e98c0 Mon Sep 17 00:00:00 2001 From: gaobinlong Date: Fri, 19 Jul 2024 21:05:02 +0800 Subject: [PATCH 10/13] Fix ISM error prevention setting key is not correct (#7777) Signed-off-by: gaobinlong --- _im-plugin/ism/api.md | 6 +++--- _im-plugin/ism/error-prevention/api.md | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/_im-plugin/ism/api.md b/_im-plugin/ism/api.md index 441f737e6f..e0fbb904cd 100644 --- a/_im-plugin/ism/api.md +++ b/_im-plugin/ism/api.md @@ -553,13 +553,13 @@ Introduced 2.4 ISM allows you to run an action automatically. However, running an action can fail for a variety of reasons. You can use error prevention validation to test an action in order to rule out failures. -To enable error prevention validation, set the `plugins.index_state_management.validation_service.enabled` setting to `true`: +To enable error prevention validation, set the `plugins.index_state_management.action_validation.enabled` setting to `true`: ```bash PUT _cluster/settings { "persistent":{ - "plugins.index_state_management.validation_action.enabled": true + "plugins.index_state_management.action_validation.enabled": true } } ``` @@ -692,4 +692,4 @@ GET _plugins/_ism/explain/test-000001 }, "total_managed_indices" : 1 } -``` \ No newline at end of file +``` diff --git a/_im-plugin/ism/error-prevention/api.md b/_im-plugin/ism/error-prevention/api.md index a273d25cfb..c03a62d868 100644 --- a/_im-plugin/ism/error-prevention/api.md +++ b/_im-plugin/ism/error-prevention/api.md @@ -12,7 +12,7 @@ The ISM Error Prevention API allows you to enable Index State Management (ISM) e ## Enable error prevention validation -You can configure error prevention validation by setting the `plugins.index_state_management.validation_service.enabled` parameter. +You can configure error prevention validation by setting the `plugins.index_state_management.action_validation.enabled` parameter. #### Example request @@ -20,7 +20,7 @@ You can configure error prevention validation by setting the `plugins.index_stat PUT _cluster/settings { "persistent":{ - "plugins.index_state_management.validation_action.enabled": true + "plugins.index_state_management.action_validation.enabled": true } } ``` From 27c41222955655669469b1319cff9daaa7212956 Mon Sep 17 00:00:00 2001 From: David Venable Date: Fri, 19 Jul 2024 12:16:25 -0500 Subject: [PATCH 11/13] Data Prepper documentation updates: autogeneration campaign (#7707) Updates Data Prepper documentation with some missing fields. Adds support for autogeneration of processors by naming to match the processor and including the autogenerated comment. Signed-off-by: David Venable Signed-off-by: David Venable Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Heather Halter --- .../processors/convert_entry_type.md | 16 ++++++++++++++-- .../processors/{parse-ion.md => parse_ion.md} | 12 +++++++++++- .../processors/{parse-json.md => parse_json.md} | 11 +++++++++++ .../processors/{parse-xml.md => parse_xml.md} | 11 ++++++++++- .../configuration/processors/write_json.md | 11 +++++++++-- .../pipelines/configuration/sources/s3.md | 2 +- 6 files changed, 56 insertions(+), 7 deletions(-) rename _data-prepper/pipelines/configuration/processors/{parse-ion.md => parse_ion.md} (61%) rename _data-prepper/pipelines/configuration/processors/{parse-json.md => parse_json.md} (70%) rename _data-prepper/pipelines/configuration/processors/{parse-xml.md => parse_xml.md} (70%) diff --git a/_data-prepper/pipelines/configuration/processors/convert_entry_type.md b/_data-prepper/pipelines/configuration/processors/convert_entry_type.md index 2fc9fdb9bd..c2c46260ed 100644 --- a/_data-prepper/pipelines/configuration/processors/convert_entry_type.md +++ b/_data-prepper/pipelines/configuration/processors/convert_entry_type.md @@ -14,10 +14,22 @@ The `convert_entry_type` processor converts a value type associated with the spe You can configure the `convert_entry_type` processor with the following options. + + | Option | Required | Description | | :--- | :--- | :--- | -| `key`| Yes | Keys whose value needs to be converted to a different type. | -| `type` | No | Target type for the key-value pair. Possible values are `integer`, `double`, `string`, and `Boolean`. Default value is `integer`. | +| `key`| Yes | Key whose value needs to be converted to a different type. | +| `keys`| Yes | Keys whose value needs to be converted to a different type. | +| `type` | No | Target type for the key-value pair. Possible values are `integer`, `long`, `double`, `big_decimal`, `string`, and `boolean`. Default value is `integer`. | +| `null_values` | No | String representation of what constitutes a `null` value. If the field value equals one of these strings, then the value is considered `null` and is converted to `null`. | +| `scale` | No | Modifies the scale of the `big_decimal` when converting to a `big_decimal`. The default value is `0`. | +| `tags_on_failure` | No | A list of tags to be added to the event metadata when the event fails to convert. | +| `convert_when` | No | Specifies a condition using a [Data Prepper expression]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/) for performing the `convert_entry_type` operation. If specified, the `convert_entry_type` operation runs only when the expression evaluates to `true`. | ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/parse-ion.md b/_data-prepper/pipelines/configuration/processors/parse_ion.md similarity index 61% rename from _data-prepper/pipelines/configuration/processors/parse-ion.md rename to _data-prepper/pipelines/configuration/processors/parse_ion.md index 0edd446c42..8360eaa296 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-ion.md +++ b/_data-prepper/pipelines/configuration/processors/parse_ion.md @@ -14,12 +14,22 @@ The `parse_ion` processor parses [Amazon Ion](https://amazon-ion.github.io/ion-d You can configure the `parse_ion` processor with the following options. + + | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `event` that is parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. | | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. | -| `tags_on_failure` | No | String | A list of strings that specify the tags to be set in the event that the processors fails or an unknown exception occurs while parsing. +| `parse_when` | No | String | Specifies under which conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). | +| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults is `true`. | +| `delete_source` | No | Boolean | If set to `true`, then the source field is deleted. Defaults is `false`. | +| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/parse-json.md b/_data-prepper/pipelines/configuration/processors/parse_json.md similarity index 70% rename from _data-prepper/pipelines/configuration/processors/parse-json.md rename to _data-prepper/pipelines/configuration/processors/parse_json.md index 2cbce4782e..894d5dba42 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-json.md +++ b/_data-prepper/pipelines/configuration/processors/parse_json.md @@ -15,11 +15,22 @@ The `parse_json` processor parses JSON data for an event, including any nested f You can configure the `parse_json` processor with the following options. + + | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | | `source` | No | String | The field in the `event` that will be parsed. Default value is `message`. | | `destination` | No | String | The destination field of the parsed JSON. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only `string` because these are not valid `event` fields. | | `pointer` | No | String | A JSON pointer to the field to be parsed. There is no `pointer` by default, meaning the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid then the entire `source` data is parsed into the outgoing `event`. If the key that is pointed to already exists in the `event` and the `destination` is the root, then the pointer uses the entire path of the key. | +| `parse_when` | No | String | Specifies under which conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Expression syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). | +| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. | +| `delete_source` | No | Boolean | If set to `true` then this will delete the source field. Defaults to `false`. | +| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/parse-xml.md b/_data-prepper/pipelines/configuration/processors/parse_xml.md similarity index 70% rename from _data-prepper/pipelines/configuration/processors/parse-xml.md rename to _data-prepper/pipelines/configuration/processors/parse_xml.md index 861705da2b..c8c9f3eebf 100644 --- a/_data-prepper/pipelines/configuration/processors/parse-xml.md +++ b/_data-prepper/pipelines/configuration/processors/parse_xml.md @@ -14,13 +14,22 @@ The `parse_xml` processor parses XML data for an event. You can configure the `parse_xml` processor with the following options. + + | Option | Required | Type | Description | | :--- | :--- | :--- | :--- | | `source` | No | String | Specifies which `event` field to parse. | | `destination` | No | String | The destination field of the parsed XML. Defaults to the root of the `event`. Cannot be `""`, `/`, or any white-space-only string because these are not valid `event` fields. | | `pointer` | No | String | A JSON pointer to the field to be parsed. The value is null by default, meaning that the entire `source` is parsed. The `pointer` can access JSON array indexes as well. If the JSON pointer is invalid, then the entire `source` data is parsed into the outgoing `event` object. If the key that is pointed to already exists in the `event` object and the `destination` is the root, then the pointer uses the entire path of the key. | | `parse_when` | No | String | Specifies under what conditions the processor should perform parsing. Default is no condition. Accepts a Data Prepper expression string following the [Data Prepper Expression Syntax]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/expression-syntax/). | -| `tags_on_failure` | No | String | A list of strings that specify the tags to be set if the processor fails or an unknown exception occurs while parsing. +| `overwrite_if_destination_exists` | No | Boolean | Overwrites the destination if set to `true`. Set to `false` to prevent changing a destination value that exists. Defaults to `true`. | +| `delete_source` | No | Boolean | If set to `true` then this will delete the source field. Defaults to `false`. | +| `tags_on_failure` | No | String | A list of strings specifying the tags to be set in the event that the processor fails or an unknown exception occurs during parsing. ## Usage diff --git a/_data-prepper/pipelines/configuration/processors/write_json.md b/_data-prepper/pipelines/configuration/processors/write_json.md index 8f1e6851da..20414b4672 100644 --- a/_data-prepper/pipelines/configuration/processors/write_json.md +++ b/_data-prepper/pipelines/configuration/processors/write_json.md @@ -11,8 +11,15 @@ nav_order: 56 The `write_json` processor converts an object in an event into a JSON string. You can customize the processor to choose the source and target field names. -Option | Description | Example -:--- | :--- | :--- + + +Option | Description | Example +:--- | :--- | :--- source | Mandatory field that specifies the name of the field in the event containing the message or object to be parsed. | If `source` is set to `"message"` and the input is `{"message": {"key1":"value1", "key2":{"key3":"value3"}}}`, then the `write_json` processor outputs the event as `"{\"key1\":\"value1\",\"key2\":{\"key3\":\"value3\"}}"`. target | An optional field that specifies the name of the field in which the resulting JSON string should be stored. If `target` is not specified, then the `source` field is used. | `key1` diff --git a/_data-prepper/pipelines/configuration/sources/s3.md b/_data-prepper/pipelines/configuration/sources/s3.md index 5a7d9986e5..7b1599f838 100644 --- a/_data-prepper/pipelines/configuration/sources/s3.md +++ b/_data-prepper/pipelines/configuration/sources/s3.md @@ -138,7 +138,7 @@ The `codec` determines how the `s3` source parses each Amazon S3 object. For inc ### `newline` codec -The `newline` codec parses each single line as a single log event. This is ideal for most application logs because each event parses per single line. It can also be suitable for S3 objects that have individual JSON objects on each line, which matches well when used with the [parse_json]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse-json/) processor to parse each line. +The `newline` codec parses each single line as a single log event. This is ideal for most application logs because each event parses per single line. It can also be suitable for S3 objects that have individual JSON objects on each line, which matches well when used with the [parse_json]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/parse_json/) processor to parse each line. Use the following options to configure the `newline` codec. From 78414ee92fb963dfc7603b78de156e626db35839 Mon Sep 17 00:00:00 2001 From: David Venable Date: Fri, 19 Jul 2024 12:57:24 -0500 Subject: [PATCH 12/13] Adds documentation for the Data Prepper delay processor. (#7708) Adds documentation for the delay processor. Signed-off-by: David Venable Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi --- .../configuration/processors/delay.md | 27 +++++++++++++++++++ .../{delete-entries.md => delete_entries.md} | 2 +- .../configuration/processors/mutate-event.md | 2 +- 3 files changed, 29 insertions(+), 2 deletions(-) create mode 100644 _data-prepper/pipelines/configuration/processors/delay.md rename _data-prepper/pipelines/configuration/processors/{delete-entries.md => delete_entries.md} (99%) diff --git a/_data-prepper/pipelines/configuration/processors/delay.md b/_data-prepper/pipelines/configuration/processors/delay.md new file mode 100644 index 0000000000..c4e9d8e973 --- /dev/null +++ b/_data-prepper/pipelines/configuration/processors/delay.md @@ -0,0 +1,27 @@ +--- +layout: default +title: delay +parent: Processors +grand_parent: Pipelines +nav_order: 41 +--- + +# delay + +This processor will add a delay into the processor chain. Typically, you should use this only for testing, experimenting, and debugging. + +## Configuration + +Option | Required | Type | Description +:--- | :--- | :--- | :--- +`for` | No | Duration | The duration of time to delay. Defaults to `1s`. + +## Usage + +The following example shows using the `delay` processor to delay for 2 seconds. + +```yaml +processor: + - delay: + for: 2s +``` diff --git a/_data-prepper/pipelines/configuration/processors/delete-entries.md b/_data-prepper/pipelines/configuration/processors/delete_entries.md similarity index 99% rename from _data-prepper/pipelines/configuration/processors/delete-entries.md rename to _data-prepper/pipelines/configuration/processors/delete_entries.md index 33c54a0b29..c9a93a1f3e 100644 --- a/_data-prepper/pipelines/configuration/processors/delete-entries.md +++ b/_data-prepper/pipelines/configuration/processors/delete_entries.md @@ -3,7 +3,7 @@ layout: default title: delete_entries parent: Processors grand_parent: Pipelines -nav_order: 41 +nav_order: 43 --- # delete_entries diff --git a/_data-prepper/pipelines/configuration/processors/mutate-event.md b/_data-prepper/pipelines/configuration/processors/mutate-event.md index 9b3b2afb33..1afb34a970 100644 --- a/_data-prepper/pipelines/configuration/processors/mutate-event.md +++ b/_data-prepper/pipelines/configuration/processors/mutate-event.md @@ -13,7 +13,7 @@ Mutate event processors allow you to modify events in Data Prepper. The followin * [add_entries]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/add-entries/) allows you to add entries to an event. * [convert_entry_type]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/convert_entry_type/) allows you to convert value types in an event. * [copy_values]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/copy-values/) allows you to copy values within an event. -* [delete_entries]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/delete-entries/) allows you to delete entries from an event. +* [delete_entries]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/delete_entries/) allows you to delete entries from an event. * [list_to_map]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/list-to-map) allows you to convert list of objects from an event where each object contains a `key` field into a map of target keys. * `map_to_list` allows you to convert a map of objects from an event, where each object contains a `key` field, into a list of target keys. * [rename_keys]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/rename-keys/) allows you to rename keys in an event. From 9dc4d29294550675c1b3a93e9074da07d9455d14 Mon Sep 17 00:00:00 2001 From: Heather Halter Date: Fri, 19 Jul 2024 10:58:35 -0700 Subject: [PATCH 13/13] Update index.md (#7779) Community feedback Signed-off-by: Heather Halter --- _install-and-configure/install-opensearch/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_install-and-configure/install-opensearch/index.md b/_install-and-configure/install-opensearch/index.md index 1afe12f6a5..541321bcdd 100644 --- a/_install-and-configure/install-opensearch/index.md +++ b/_install-and-configure/install-opensearch/index.md @@ -121,5 +121,5 @@ Property | Description `opensearch.xcontent.string.length.max=` | By default, OpenSearch does not impose any limits on the maximum length of the JSON/YAML/CBOR/Smile string fields. To protect your cluster against potential distributed denial-of-service (DDoS) or memory issues, you can set the `opensearch.xcontent.string.length.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.string.length.max=5000000`. | `opensearch.xcontent.fast_double_writer=[true|false]` | By default, OpenSearch serializes floating-point numbers using the default implementation provided by the Java Runtime Environment. Set this value to `true` to use the Schubfach algorithm, which is faster but may lead to small differences in precision. Default is `false`. | `opensearch.xcontent.name.length.max=` | By default, OpenSearch does not impose any limits on the maximum length of the JSON/YAML/CBOR/Smile field names. To protect your cluster against potential DDoS or memory issues, you can set the `opensearch.xcontent.name.length.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.name.length.max=50000`. | -`opensearch.xcontent.depth.max=` | By default, OpenSearch does not impose any limits on the maximum nesting depth for JSON/YAML/CBOR/Smile documents. To protect your cluster against potential DDoS or memory issues, you can set the `opensearch.xcontent.depth.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.name.length.max=1000`. | +`opensearch.xcontent.depth.max=` | By default, OpenSearch does not impose any limits on the maximum nesting depth for JSON/YAML/CBOR/Smile documents. To protect your cluster against potential DDoS or memory issues, you can set the `opensearch.xcontent.depth.max` system property to a reasonable limit (the maximum is 2,147,483,647), for example, `-Dopensearch.xcontent.depth.max=1000`. | `opensearch.xcontent.codepoint.max=` | By default, OpenSearch imposes a limit of `52428800` on the maximum size of the YAML documents (in code points). To protect your cluster against potential DDoS or memory issues, you can change the `opensearch.xcontent.codepoint.max` system property to a reasonable limit (the maximum is 2,147,483,647). For example, `-Dopensearch.xcontent.codepoint.max=5000000`. |