Add documentation for star tree index feature #8598

bharath-techie · 2024-10-22T15:34:54Z

Description

This PR adds documentation for star tree index feature. OpenSearch RFC / Meta

Issues Resolved

Version

List the OpenSearch version to which this PR applies, e.g. 2.14, 2.12--2.14, or all.

Frontend features

If you're submitting documentation for an OpenSearch Dashboards feature, add a video that shows how a user will interact with the UI step by step. A voiceover is optional.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-10-22T15:35:06Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

Signed-off-by: Bharathwaj G <[email protected]>

_field-types/supported-field-types/index.md

_field-types/supported-field-types/star-tree.md

Signed-off-by: Bharathwaj G <[email protected]>

_search-plugins/star-tree-index.md

_field-types/supported-field-types/star-tree.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

_field-types/supported-field-types/star-tree.md

_search-plugins/star-tree-index.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

natebower

@Naarcha-AWS Final comments/changes. I'd like to read lines 59-62 in star-tree-index.md before approving. Thanks!

_field-types/supported-field-types/star-tree.md

_search-plugins/star-tree-index.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Signed-off-by: Naarcha-AWS <[email protected]>

_field-types/supported-field-types/star-tree.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Signed-off-by: Naarcha-AWS <[email protected]>

natebower

@Naarcha-AWS LGTM!

_search-plugins/star-tree-index.md

bharath-techie · 2024-11-04T12:32:49Z

_field-types/supported-field-types/star-tree.md

+
+Define star-tree index mappings in the `composite` section in `mappings`. 
+
+The following example API request creates a corresponding star-tree index for all `request_aggs`. To compute metric aggregations for `request_size` and `latency` fields with queries on `port` and `status` fields, configure the following mappings:


How about following :

The following example API request creates a corresponding star-tree index configuration under request_aggs

"all request_aggs " for me sounds a bit confusing

bharath-techie · 2024-11-04T12:36:01Z

_field-types/supported-field-types/star-tree.md

+
+| Parameter  | Required/Optional | Description  | 
+| :--- | :--- | :--- |
+| `name` | Required | The name of the field. The field name should be present in the `properties` section as part of the index `mapping`. Ensure that the `doc_values` setting is `enabled` for any associated fields.


Confused on this - config itself doesn't have a name property. Can we remove this ?

Under config , user can specify ordered_dimensions, metrics, max_leaf_docs and skip_star_node_creation_for_dimensions.

I'll remove this. We have the definitions for max_leaf_docs and skip_star_node_creation_for_dimensions on line 193.

bharath-techie · 2024-11-04T12:38:42Z

_field-types/supported-field-types/star-tree.md

+
+| Parameter  | Description   | 
+| :--- | :--- |
+| `max_leaf_docs` | The maximum number of star-tree documents that a leaf node can point to. After the maximum number of documents is reached, the nodes will be split based on the value of the next dimension. Default is `10000`. A lower value will use more storage but result in faster query performance. Inversely, a higher value will use less storage but result in slower query performance. For more information, see [Star-tree indexing structure]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/#star-tree-index-structure).  |


the nodes will be split based on the value of the next dimension.

How about once a node crosses threshold of max_leaf_docs , children nodes will be created based on the unique values or something similar.

bharath-techie · 2024-11-04T12:41:06Z

_search-plugins/star-tree-index.md

+
+<img src="{{site.url}}{{site.baseurl}}/images/star-tree-index.png" alt="A star-tree index containing two dimensions and two metrics" width="700">
+
+Sorted and aggregated star-tree documents are backed by `doc_values` in an index. `doc_values` use the following pattern:


How about

" Values stored in doc_values use the following pattern "

doc_values itself is a singular column and generally end user might not understand it well either , so we need to enhance it a bit similar to above.

bharath-techie · 2024-11-04T12:43:23Z

_search-plugins/star-tree-index.md

+
+Sorted and aggregated star-tree documents are backed by `doc_values` in an index. `doc_values` use the following pattern:
+
+- The values are sorted based on the order of their `ordered_dimension`. In the preceding image, the dimensions are determined by the `status` setting and then by the `port` for each status.


The values across fields are primary sorted based on first field in ordered_dimension, secondary sorted by corresponding fields mentioned in the ordered_dimension

Not exact wording but we need to call out that the sort is based on the fields specified in the ordered dimension.

bharath-techie · 2024-11-04T12:50:14Z

_search-plugins/star-tree-index.md

+
+### Star nodes
+
+Star nodes are children of non-leaf nodes that contain preaggregated records for data split after dimension removal, aggregating metrics for rows containing dimensions with identical values. These aggregated documents are then appended to the end of star-tree documents. If a document does contain a dimension with identical values, it traverses through the star node.


This is inaccurate as star nodes can be leaf or non - leaf nodes. Lets reword as below:

"""
Star nodes are special nodes which has the aggregated data of all the other nodes in the same dimension.

This helps when we need to query the aggregated value of a particular field without traversing through all the nodes of a particular field [ dimension ] in the star tree.

This also helps in skipping the dimension which is not part of the query.
"""

The original wording itself looks good to me

"""
There are special nodes called star nodes (*) which helps in skipping non-competitive nodes and also in fetching aggregated document wherever applicable during query time.
"""

bharath-techie · 2024-11-04T13:21:18Z

_search-plugins/star-tree-index.md

+
+Star nodes are children of non-leaf nodes that contain preaggregated records for data split after dimension removal, aggregating metrics for rows containing dimensions with identical values. These aggregated documents are then appended to the end of star-tree documents. If a document does contain a dimension with identical values, it traverses through the star node.
+
+The star-tree index structure diagram contains the following three examples demonstrating how a document does or does not traverse star-tree nodes (indicated by the `*` symbol in the diagram) during a `Term` query, based on the average request size of the query and whether the document contains matching dimensions:


we must be talking about query algorithm here

"""
The star-tree index structure diagram contains the following three examples demonstrating how query algorithm traverses the star tree to get the results
"""

bharath-techie · 2024-11-04T13:25:29Z

_search-plugins/star-tree-index.md

+
+The star-tree index structure diagram contains the following three examples demonstrating how a document does or does not traverse star-tree nodes (indicated by the `*` symbol in the diagram) during a `Term` query, based on the average request size of the query and whether the document contains matching dimensions:
+
+- When the port equals `8443` and the status equals `200`. Because the status equals `200`, the query does not traverse through a star node, and the aggregated metric is stored at the end of a star-tree document.


Lets not talk about how star node is traversed / not traversed. Lets keep the wording on how query algorithm traverses the star tree as a whole for a particular query. Image clearly explains the path which is taken by the query

Something similar to the original wording

- Compute average request size aggregation with Terms query where port equals 8443 and status equals 200 - the query visits the actual nodes of 8443 and 200 values(Support for Terms query will be added in upcoming release, see https://github.com/opensearch-project/OpenSearch/issues/15257) Compute count of requests aggregation with Term query where status equals 200 (query traverses through * node of port dimension since port is not present as part of query) Compute average request size aggregation with Term query where port equals 5600 (query traverses through * node of status dimension since status is not present as part of query). The second and third examples uses star nodes.

bharath-techie · 2024-11-04T13:25:57Z

_search-plugins/star-tree-index.md

+- When the status equals `200`. The query traverses through a star node in the `port` dimension because `port` is not present as part of the query.
+- When the port equals `5600`. The query traverses through a star node in the `status` dimension because `status` is not present as part of the query.
+
+Support for the `Term` query will be added in a future version. For more information, see [GitHub issue #15257](https://github.com/opensearch-project/OpenSearch/issues/15257).


No this should be Terms query. Term query is already supported.

maybe we can say individual term query is already supported and support for multiple terms query is to be supported to avoid confusion to the user

We can just switch to terms here.

bharath-techie · 2024-11-04T13:27:20Z

_search-plugins/star-tree-index.md

+
+## Example mapping
+
+In the following example, index mappings define the star-tree configuration. This star-tree index precomputes aggregations in the `log` index. The aggregations are calculated using the `size` and `latency` fields for all the combinations of values indexed in the `port` and `status` fields:


nit : logs index

bharath-techie · 2024-11-04T13:28:11Z

_search-plugins/star-tree-index.md

+
+### Aggregation example
+
+The following example gets the sum of the `size` field for all error logs with `status=500`, using the [example mapping](#example-mapping):


Can we reword to something similar :

The following example gets the sum of all the values in the size field - for all error logs with status=500, using the example mapping:

bharath-techie · 2024-11-04T13:28:43Z

_search-plugins/star-tree-index.md

+
+With the star-tree index, the result will be retrieved from a single aggregated document as it traverses to the `status=500` node, as opposed to scanning through all of the matching documents. This results in lower query latency.
+
+## Using queries with a star-tree index


Should this be without star tree index ?

sandeshkr419 · 2024-11-04T18:34:17Z

_search-plugins/star-tree-index.md

+
+Star-tree indexes can be used to optimize queries and aggregations. 
+
+### Supported queries


This section gives the impression that these query shapes are supported independently.
Ideally this should be a sub-section within Supported Aggregations itself.

Basically, with the below supported aggregations, one can also add a term-query to a search request.

* Adding documentation for star tree index feature Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * fixes and addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * fixing json Signed-off-by: Bharathwaj G <[email protected]> * fixing json Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * addressing comments Signed-off-by: Bharathwaj G <[email protected]> * Add edits for star tree field page Signed-off-by: Naarcha-AWS <[email protected]> * Add index edit Signed-off-by: Naarcha-AWS <[email protected]> * Update improving-search-performance.md Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree-index.md Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Update _field-types/supported-field-types/star-tree.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree-index.md Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Bharathwaj G <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Eric Pugh <[email protected]>

bharath-techie requested review from kolchfa-aws, Naarcha-AWS, vagimeli, AMoo-Miki, natebower, dlvenable and epugh as code owners October 22, 2024 15:34

github-actions bot assigned kolchfa-aws Oct 22, 2024

bharath-techie assigned Naarcha-AWS Oct 22, 2024

bharath-techie force-pushed the startree branch from 1a7be5d to fecec7d Compare October 22, 2024 16:44

bharath-techie marked this pull request as draft October 22, 2024 17:11

bharath-techie force-pushed the startree branch from fecec7d to dee4bee Compare October 22, 2024 17:14

Naarcha-AWS added v2.18.0 4 - Doc review PR: Doc review in progress labels Oct 22, 2024

kolchfa-aws added the release-notes PR: Include this PR in the automated release notes label Oct 22, 2024

bharath-techie force-pushed the startree branch 3 times, most recently from 1ac0829 to f3833a0 Compare October 23, 2024 09:10

bharath-techie marked this pull request as ready for review October 23, 2024 09:10

bharath-techie force-pushed the startree branch from f3833a0 to 26ff94f Compare October 23, 2024 09:10

Adding documentation for star tree index feature

95a47ac

Signed-off-by: Bharathwaj G <[email protected]>

bharath-techie force-pushed the startree branch from 26ff94f to 95a47ac Compare October 23, 2024 09:31

expani reviewed Oct 23, 2024

View reviewed changes

_field-types/supported-field-types/index.md Outdated Show resolved Hide resolved

expani reviewed Oct 23, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

expani reviewed Oct 23, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

addressing comments

78b4c41

Signed-off-by: Bharathwaj G <[email protected]>

bharath-techie force-pushed the startree branch from 0369c66 to 78b4c41 Compare October 23, 2024 10:15

expani reviewed Oct 23, 2024

View reviewed changes

_search-plugins/star-tree-index.md Outdated Show resolved Hide resolved

expani reviewed Oct 23, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

Apply suggestions from code review

d249946

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS reviewed Nov 1, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

Naarcha-AWS reviewed Nov 1, 2024

View reviewed changes

_search-plugins/star-tree-index.md Outdated Show resolved Hide resolved

Apply suggestions from code review

6ce9d22

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

natebower reviewed Nov 1, 2024

View reviewed changes

Naarcha-AWS reviewed Nov 1, 2024

View reviewed changes

_search-plugins/star-tree-index.md Outdated Show resolved Hide resolved

Naarcha-AWS and others added 2 commits November 1, 2024 12:25

Apply suggestions from code review

c0c5ec0

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Apply suggestions from code review

e8bdea5

Signed-off-by: Naarcha-AWS <[email protected]>

Naarcha-AWS reviewed Nov 1, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

Naarcha-AWS reviewed Nov 1, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

Naarcha-AWS and others added 2 commits November 1, 2024 14:44

Apply suggestions from code review

3e372f7

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Update star-tree-index.md

19eaad0

Signed-off-by: Naarcha-AWS <[email protected]>

natebower approved these changes Nov 1, 2024

View reviewed changes

Merge branch 'main' into startree

f98e02d

Naarcha-AWS merged commit faa328a into opensearch-project:main Nov 1, 2024
5 checks passed

sandeshkr419 reviewed Nov 4, 2024

View reviewed changes

_search-plugins/star-tree-index.md Show resolved Hide resolved

bharath-techie commented Nov 4, 2024

View reviewed changes

sandeshkr419 reviewed Nov 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for star tree index feature #8598

Add documentation for star tree index feature #8598

bharath-techie commented Oct 22, 2024 •

edited

Loading

github-actions bot commented Oct 22, 2024

natebower left a comment

natebower left a comment

bharath-techie Nov 4, 2024 •

edited

Loading

bharath-techie Nov 4, 2024

Naarcha-AWS Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024 •

edited

Loading

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

Naarcha-AWS Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

bharath-techie Nov 4, 2024

sandeshkr419 Nov 4, 2024


		Define star-tree index mappings in the `composite` section in `mappings`.

		The following example API request creates a corresponding star-tree index for all `request_aggs`. To compute metric aggregations for `request_size` and `latency` fields with queries on `port` and `status` fields, configure the following mappings:


		<img src="{{site.url}}{{site.baseurl}}/images/star-tree-index.png" alt="A star-tree index containing two dimensions and two metrics" width="700">

		Sorted and aggregated star-tree documents are backed by `doc_values` in an index. `doc_values` use the following pattern:


		Sorted and aggregated star-tree documents are backed by `doc_values` in an index. `doc_values` use the following pattern:

		- The values are sorted based on the order of their `ordered_dimension`. In the preceding image, the dimensions are determined by the `status` setting and then by the `port` for each status.


		### Star nodes

		Star nodes are children of non-leaf nodes that contain preaggregated records for data split after dimension removal, aggregating metrics for rows containing dimensions with identical values. These aggregated documents are then appended to the end of star-tree documents. If a document does contain a dimension with identical values, it traverses through the star node.


		Star nodes are children of non-leaf nodes that contain preaggregated records for data split after dimension removal, aggregating metrics for rows containing dimensions with identical values. These aggregated documents are then appended to the end of star-tree documents. If a document does contain a dimension with identical values, it traverses through the star node.

		The star-tree index structure diagram contains the following three examples demonstrating how a document does or does not traverse star-tree nodes (indicated by the `*` symbol in the diagram) during a `Term` query, based on the average request size of the query and whether the document contains matching dimensions:


		The star-tree index structure diagram contains the following three examples demonstrating how a document does or does not traverse star-tree nodes (indicated by the `*` symbol in the diagram) during a `Term` query, based on the average request size of the query and whether the document contains matching dimensions:

		- When the port equals `8443` and the status equals `200`. Because the status equals `200`, the query does not traverse through a star node, and the aggregated metric is stored at the end of a star-tree document.


		## Example mapping

		In the following example, index mappings define the star-tree configuration. This star-tree index precomputes aggregations in the `log` index. The aggregations are calculated using the `size` and `latency` fields for all the combinations of values indexed in the `port` and `status` fields:


		### Aggregation example

		The following example gets the sum of the `size` field for all error logs with `status=500`, using the [example mapping](#example-mapping):


		With the star-tree index, the result will be retrieved from a single aggregated document as it traverses to the `status=500` node, as opposed to scanning through all of the matching documents. This results in lower query latency.

		## Using queries with a star-tree index


		Star-tree indexes can be used to optimize queries and aggregations.

		### Supported queries

Add documentation for star tree index feature #8598

Add documentation for star tree index feature #8598

Conversation

bharath-techie commented Oct 22, 2024 • edited Loading

Description

Issues Resolved

Version

Frontend features

Checklist

github-actions bot commented Oct 22, 2024

natebower left a comment

Choose a reason for hiding this comment

natebower left a comment

Choose a reason for hiding this comment

bharath-techie Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharath-techie Nov 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bharath-techie commented Oct 22, 2024 •

edited

Loading

bharath-techie Nov 4, 2024 •

edited

Loading

bharath-techie Nov 4, 2024 •

edited

Loading