From 2ea5e8070bdb5551e40ad929776c04241bdc3f98 Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 10:32:15 -0700 Subject: [PATCH 1/4] update aggregation support in limitations and add sample req/response. Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 72 ++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index d989c3e4a4..17271d5394 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -28,7 +28,8 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: -- **Aggregation, scoring, and sorting**: Not yet supported. +- **Scoring and sorting**: Not yet supported. +- **Aggregations are supported with limitations**: Unsupported agg types: any Geo based Aggregation, Significant Terms/Text and Scripted Metric - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). @@ -541,6 +542,75 @@ The response specifies highlighting in the `url` field: ``` +## Aggregations + +Most aggregation types are supported with derived fields since 2.17. Geo Aggregations, Significant Terms/Text and Scripted Metric are not supported. +For example, the following request creates a simple terms aggregation on the `status` derived field: + +```json +POST /logs/_search +{ + "size": 0, + "aggs": { + "methods": { + "terms": { + "field": "method" + } + } + } +} +``` +{% include copy-curl.html %} +The response contains the following buckets: + +
+ + Response + + {: .text-delta} + +```json +{ + "took" : 12, + "timed_out" : false, + "_shards" : { + "total" : 1, + "successful" : 1, + "skipped" : 0, + "failed" : 0 + }, + "hits" : { + "total" : { + "value" : 5, + "relation" : "eq" + }, + "max_score" : null, + "hits" : [ ] + }, + "aggregations" : { + "methods" : { + "doc_count_error_upper_bound" : 0, + "sum_other_doc_count" : 0, + "buckets" : [ + { + "key" : "GET", + "doc_count" : 2 + }, + { + "key" : "POST", + "doc_count" : 2 + }, + { + "key" : "DELETE", + "doc_count" : 1 + } + ] + } + } +} +``` +
+ ## Performance Derived fields are not indexed but are computed dynamically by retrieving values from the `_source` field or doc values. Thus, they run more slowly. To improve performance, try the following: From 21ab396a567551d400540dba87ce2d8f8fbed28c Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 10:32:41 -0700 Subject: [PATCH 2/4] remove concurrent search limitation Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 1 - 1 file changed, 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 17271d5394..54bb7bc3e5 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -33,7 +33,6 @@ Currently, derived fields have the following limitations: - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). -- **Concurrent segment search**: Derived fields are not supported for [concurrent segment search]({{site.url}}{{site.baseurl}}/search-plugins/concurrent-segment-search/). We are planning to address these limitations in future versions. From 57d4bf1316bc55c84618e41889ecfbb7a9571a0b Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 10:49:19 -0700 Subject: [PATCH 3/4] minor updates Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index 54bb7bc3e5..d5eacd9c58 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **Aggregations are supported with limitations**: Unsupported agg types: any Geo based Aggregation, Significant Terms/Text and Scripted Metric +- **With 2.17 Aggregations are supported with limitations**: Unsupported agg types: Geo, Significant Terms/Text and Scripted Metric - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/). @@ -544,7 +544,7 @@ The response specifies highlighting in the `url` field: ## Aggregations Most aggregation types are supported with derived fields since 2.17. Geo Aggregations, Significant Terms/Text and Scripted Metric are not supported. -For example, the following request creates a simple terms aggregation on the `status` derived field: +For example, the following request creates a simple terms aggregation on the `method` derived field: ```json POST /logs/_search From a993bb93c1ad52b30a1b52640c6670dd1e128dfc Mon Sep 17 00:00:00 2001 From: Marc Handalian Date: Wed, 11 Sep 2024 11:43:49 -0700 Subject: [PATCH 4/4] fix style job error Signed-off-by: Marc Handalian --- _field-types/supported-field-types/derived.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_field-types/supported-field-types/derived.md b/_field-types/supported-field-types/derived.md index d5eacd9c58..78326ce670 100644 --- a/_field-types/supported-field-types/derived.md +++ b/_field-types/supported-field-types/derived.md @@ -29,7 +29,7 @@ Despite the potential performance impact of query-time computations, the flexibi Currently, derived fields have the following limitations: - **Scoring and sorting**: Not yet supported. -- **With 2.17 Aggregations are supported with limitations**: Unsupported agg types: Geo, Significant Terms/Text and Scripted Metric +- **With 2.17 Aggregations are supported with limitations**: Unsupported aggregation types: Geo, Significant Terms/Text and Scripted Metric - **Dashboard support**: These fields are not displayed in the list of available fields in OpenSearch Dashboards. However, you can still use them for filtering if you know the derived field name. - **Chained derived fields**: One derived field cannot be used to define another derived field. - **Join field type**: Derived fields are not supported for the [join field type]({{site.url}}{{site.baseurl}}/opensearch/supported-field-types/join/).