Additional star tree feedback. #8650

Naarcha-AWS · 2024-11-04T15:33:18Z

Adds additional feedback post merging #8131.

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Archer <[email protected]>

github-actions · 2024-11-04T15:33:31Z

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

bharath-techie · 2024-11-04T15:37:12Z

_field-types/supported-field-types/star-tree.md

@@ -109,13 +109,8 @@ PUT logs

 ## Star-tree mapping parameters

-Specify any star-tree configuration mapping options in the `config` section. Parameters cannot be modified without reindexing documents.
+Specify any star-tree configuration mapping options in the `mappings` section. Parameters cannot be modified without reindexing documents.


Should we differentiate from mappings since its present a bit deeper in config section.

under corresponding STIX config section under mappings

or something similar ?

For sure, let me rework the order of this section.

bharath-techie · 2024-11-04T15:41:57Z

_field-types/supported-field-types/star-tree.md


 | Parameter  | Description   | 
 | :--- | :--- |
-| `max_leaf_docs` | The maximum number of star-tree documents that a leaf node can point to. After the maximum number of documents is reached, the nodes will be split based on the value of the next dimension. Default is `10000`. A lower value will use more storage but result in faster query performance. Inversely, a higher value will use less storage but result in slower query performance. For more information, see [Star-tree indexing structure]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/#star-tree-index-structure).  |
-| `skip_star_node_creation_for_dimensions`  | A list of dimensions for which a star-tree index will skip star node creation. When `true`, this reduces storage size at the expense of query performance. Default is `false`. For more information about star nodes, see [Star-tree indexing structure]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/#star-tree-index-structure). |
+| `max_leaf_docs` | The maximum number of STIX documents that a leaf node can point to. After the maximum number of documents is reached, children nodes of the leaf node will be created based on the unique `doc_value`. Default is `10000`. A lower value will use more storage but result in faster query performance. Inversely, a higher value will use less storage but result in slower query performance. For more information, see [Star-tree indexing structure]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/#star-tree-index-structure).  |


The maximum number of STIX documents that a leaf node can point to. Once the node crosses this threshold, children nodes will be created based on the unique values of the next field in `ordered_dimension` (if any).

bharath-techie · 2024-11-04T15:44:43Z

_search-plugins/star-tree-index.md

@@ -39,9 +39,9 @@ The following image illustrates a standard star-tree index structure.

 <img src="{{site.url}}{{site.baseurl}}/images/star-tree-index.png" alt="A star-tree index containing two dimensions and two metrics" width="700">

-Sorted and aggregated star-tree documents are backed by `doc_values` in an index. `doc_values` use the following pattern:
+Sorted and aggregated STIX documents are backed by `doc_values` in an index. `doc_values` use the following pattern:


Sorted and aggregated STIX documents are backed by doc_values in an index. The columnar data part of doc_values are stored with following properties :

bharath-techie · 2024-11-04T15:58:09Z

_search-plugins/star-tree-index.md

- When the port equals `8443` and the status equals `200`. Because the status equals `200`, the query does not traverse through a star node, and the aggregated metric is stored at the end of a star-tree document.
- When the status equals `200`. The query traverses through a star node in the `port` dimension because `port` is not present as part of the query.
- When the port equals `5600`. The query traverses through a star node in the `status` dimension because `status` is not present as part of the query.
+- **Blue**: In a `terms` query searching for the computed average request size aggregation, the port equals `8443` and the status equals `200`. Because the query contains values in both the `status` and `port` dimensions, the query returns the aggregation from a non-star node.


Blue: Example of a terms query searching for the average request size aggregation, with query of port equals 8443 and the status equals 200. The query traverses through 200 node in status dimension and 8443 node in port dimensions, and returns the aggregation result.

bharath-techie · 2024-11-04T15:59:18Z

_search-plugins/star-tree-index.md

- When the status equals `200`. The query traverses through a star node in the `port` dimension because `port` is not present as part of the query.
- When the port equals `5600`. The query traverses through a star node in the `status` dimension because `status` is not present as part of the query.
+- **Blue**: In a `terms` query searching for the computed average request size aggregation, the port equals `8443` and the status equals `200`. Because the query contains values in both the `status` and `port` dimensions, the query returns the aggregation from a non-star node.
+- **Green**: In a `term` query searching for the computed count of requests aggregation, the status equals `200`. Because the query only contains a value from the `status` dimension, the query returns the aggregation from a star node in the `port` dimension.


This is confusing

- When the status equals `200`. The query traverses through the child star node of `200` node as it contains the aggregated value of all other children port nodes.

bharath-techie · 2024-11-04T16:00:13Z

_search-plugins/star-tree-index.md

- When the port equals `5600`. The query traverses through a star node in the `status` dimension because `status` is not present as part of the query.
+- **Blue**: In a `terms` query searching for the computed average request size aggregation, the port equals `8443` and the status equals `200`. Because the query contains values in both the `status` and `port` dimensions, the query returns the aggregation from a non-star node.
+- **Green**: In a `term` query searching for the computed count of requests aggregation, the status equals `200`. Because the query only contains a value from the `status` dimension, the query returns the aggregation from a star node in the `port` dimension.
+- **Red**: In a `term` query searching for the computed average request size aggregation, the port equals `5600`. Because the query does not contain a value from the `status` dimension, the query traverses through a star node in the `status` dimension and returns the aggregation from a star node in the `port` dimension.


"""
When the port equals 5600. The query traverses through a star node in the status dimension because status is not present as part of the query and returns aggregated result from the 5600 child node
"""

bharath-techie · 2024-11-04T16:00:40Z

_search-plugins/star-tree-index.md

@@ -73,7 +73,7 @@ To use a star-tree index, modify the following settings:

 ## Example mapping

-In the following example, index mappings define the star-tree configuration. This star-tree index precomputes aggregations in the `log` index. The aggregations are calculated using the `size` and `latency` fields for all the combinations of values indexed in the `port` and `status` fields:
+In the following example, index mappings define the star-tree configuration. This star-tree index precomputes aggregations in the `logs` index. The aggregations are calculated using the `size` and `latency` fields for all the combinations of values indexed in the `port` and `status` fields:


Nit:
In the following example, index mappings define the star-tree configuration. This star-tree index precomputes aggregations in the logs index. The aggregations are calculated for the size and latency fields for all the combinations of values indexed in the port and status fields:

Signed-off-by: Archer <[email protected]>

Naarcha-AWS · 2024-11-04T16:51:02Z

@bharath-techie: Ready for a second review.

Signed-off-by: Archer <[email protected]>

bharath-techie · 2024-11-05T03:09:49Z

_field-types/supported-field-types/star-tree.md

-## Star-tree mapping parameters
+| Parameter  | Description   | 
+| :--- | :--- |
+| `mappings` | A list of [mapping](#mapping-parameters) parameters. Required.


This is inaccurate, the config is part of mapping. And config contains 4 parameters , 2 simple ones max leaf docs and skip star node. And 2 complex ones, ordered dimensions and metrics.

If needed we can give a brief about ordered dimensions and metrics in the table here. And then we can continue with separate sections for ordered dimensions and metrics as already present

_field-types/supported-field-types/star-tree.md

_search-plugins/star-tree-index.md

_field-types/supported-field-types/star-tree.md

Signed-off-by: Naarcha-AWS <[email protected]>

_search-plugins/star-tree-index.md

Signed-off-by: Naarcha-AWS <[email protected]>

_search-plugins/star-tree-index.md

Signed-off-by: Naarcha-AWS <[email protected]>

bharath-techie · 2024-11-05T15:15:22Z

_field-types/supported-field-types/star-tree.md

@@ -23,10 +23,10 @@ To use a star-tree index, follow the instructions in [Enabling a star-tree index

 ## Limitations

-The star-tree index feature has the following limitations:
+The STIX feature has the following limitations:


We can either have all the limitations here or we can link up the limitations section in the other page.

Let's move all the limitations here.

_field-types/supported-field-types/star-tree.md

natebower

@Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Globally, for consistency, please define STIX where it first appears in a file and then use only the acronym thereafter (most often, "a STIX"). Additionally, please ensure that any remaining instances of "traverses to" or "traverses through" are changed to just "traverses". Tag me for approval when addressed. Thanks!

_field-types/supported-field-types/star-tree.md

_search-plugins/star-tree-index.md

natebower · 2024-11-05T15:20:43Z

@Naarcha-AWS @bharath-techie We need to wrap up and implement any remaining tech feedback ASAP so that I can read again and approve. Thank you.

bharath-techie · 2024-11-05T15:23:38Z

@Naarcha-AWS @bharath-techie We need to wrap up and implement any remaining tech feedback ASAP so that I can read again and approve. Thank you.

No more comments from my end, just the last two.

_field-types/supported-field-types/star-tree.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Signed-off-by: Naarcha-AWS <[email protected]>

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

Signed-off-by: Naarcha-AWS <[email protected]>

natebower

@Naarcha-AWS LGTM with the final noted changes. Thanks!

natebower · 2024-11-05T16:24:56Z

_field-types/supported-field-types/star-tree.md

+| :--- | :--- |
+| `ordered_dimensions` | A [list of fields](#ordered-dimensions) based on which metrics will be aggregated in a star-tree index. Required. | 
+| `metrics` | A [list of metric](#metrics) fields required in order to perform aggregations. Required. |
+| `max_leaf_docs` | The maximum number of star-tree documents that a leaf node can point to. After the maximum number of documents is reached, child nodes will be created based on the unique values of the next field in the `ordered_dimension` (if any). Default is `10000`. A lower value will use more storage but result in faster query performance. Inversely, a higher value will use less storage but result in slower query performance. For more information, see [Star-tree indexing structure]({{site.url}}{{site.baseurl}}/search-plugins/star-tree-index/#star-tree-index-structure).  |


"unique values" => "unique value"?

_search-plugins/star-tree-index.md

natebower · 2024-11-05T16:29:18Z

_search-plugins/star-tree-index.md

- When the status equals `200`. The query traverses through a star node in the `port` dimension because `port` is not present as part of the query.
- When the port equals `5600`. The query traverses through a star node in the `status` dimension because `status` is not present as part of the query.
+- **Blue**: In a `terms` query that searches for the average request size aggregation, the `port` equals `8443` and the status equals `200`. Because the query contains values in both the `status` and `port` dimensions, the query traverses status node `200` and returns the aggregations from child node `8443`.
+- **Green**: In a `term` query that searches for the number of requests aggregation, the `status` equals `200`. Because the query only contains a value from the `status` dimension, the query traverses the `200` node's child star node, which contains the aggregated value of all the `port` child nodes.


"requests aggregation" => "request aggregations" or "aggregation requests"?

_search-plugins/star-tree-index.md

_field-types/supported-field-types/star-tree.md

_search-plugins/star-tree-index.md

Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]>

* Additional star tree feedback. Signed-off-by: Archer <[email protected]> * Add tech feedback. Signed-off-by: Archer <[email protected]> * Fix broken link Signed-off-by: Archer <[email protected]> * Update star-tree.md Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree-index.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree.md Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> * Update star-tree-index.md Signed-off-by: Naarcha-AWS <[email protected]> * Apply suggestions from code review Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> --------- Signed-off-by: Archer <[email protected]> Signed-off-by: Naarcha-AWS <[email protected]> Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Eric Pugh <[email protected]>

Additional star tree feedback.

72ac245

Signed-off-by: Archer <[email protected]>

Naarcha-AWS added 3 - Tech review PR: Tech review in progress v2.18.0 labels Nov 4, 2024

Naarcha-AWS self-assigned this Nov 4, 2024

Naarcha-AWS requested review from kolchfa-aws, vagimeli, AMoo-Miki, natebower, dlvenable and epugh as code owners November 4, 2024 15:33

bharath-techie reviewed Nov 4, 2024

View reviewed changes

Add tech feedback.

5cbcf99

Signed-off-by: Archer <[email protected]>

Fix broken link

e2ee028

Signed-off-by: Archer <[email protected]>

bharath-techie reviewed Nov 5, 2024

View reviewed changes

_field-types/supported-field-types/star-tree.md Outdated Show resolved Hide resolved

bharath-techie reviewed Nov 5, 2024

View reviewed changes

_search-plugins/star-tree-index.md Outdated Show resolved Hide resolved