Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add document for top n queries improvements in 2.15 #7326

Merged
86 changes: 69 additions & 17 deletions _observing-your-data/query-insights/top-n-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,23 +9,28 @@ nav_order: 65

Monitoring the top N queries in query insights features can help you gain real-time insights into the top queries with high latency within a certain time frame (for example, the last hour).

## Getting started
## Configuring top N query monitoring

To enable monitoring of the top N queries, configure the following [dynamic settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#dynamic-settings):
You can configure monitoring top N queries by the following metric types:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

- `search.insights.top_queries.latency.enabled`: Set to `true` to [enable monitoring of the top N queries](#enabling-the-top-n-queries-feature).
- `search.insights.top_queries.latency.window_size`: [Configure the window size](#configuring-window-size).
- `search.insights.top_queries.latency.top_n_size`: [Specify the value of n](#configuring-the-value-of-n).
- `latency`
- `cpu`
- `memory`

It's important to exercise caution when enabling this feature because it can consume system resources.
{: .important}
Each metric has a set of corresponding settings:

- `search.insights.top_queries.<metric>.enabled`: Set to `true` to [enable top N query monitoring by the metric](#enabling-top-n-query-monitoring).
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- `search.insights.top_queries.<metric>.window_size`: [Configure the window size the top N queries by the metric](#configuring-the-window-size).
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
- `search.insights.top_queries.<metric>.top_n_size`: [Specify the value of N for the top N queries by the metric](#configuring-the-value-of-n).
Copy link
Collaborator

@natebower natebower Jun 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `search.insights.top_queries.<metric>.top_n_size`: [Specify the value of N for the top N queries by the metric](#configuring-the-value-of-n).
- `search.insights.top_queries.<metric>.top_n_size`: [Specify the value of N for the top N queries](#configuring-the-value-of-n) by the metric.


For example, to enable top N query monitoring by CPU usage, set `search.insights.top_queries.cpu.enabled` to `true`. For more information about ways to specify dynamic settings, see [Dynamic settings]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/index/#dynamic-settings).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"by CPU usage" => "of CPU usage"?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, you monitor queries that use the most CPU ("by")


For detailed information about enabling and configuring this feature, see the following sections.
It's important to exercise caution when enabling this feature because it can consume system resources.
{: .important}

## Enabling the top N queries feature
## Enabling top N query monitoring

After installing the `query-insights` plugin, you can enable the top N queries feature (which is disabled by default) by using the following dynamic setting. This setting enables the corresponding collectors and aggregators in the running cluster:
When you install the `query-insights` plugin, top N query monitoring is disabled by default. To enable top N query monitoring, update the dynamic settings for the desired metric types. These settings enable the corresponding collectors and aggregators in the running cluster. For example, to enable monitoring top N queries by latency, update the `search.insights.top_queries.latency.enabled` setting:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

```json
PUT _cluster/settings
Expand All @@ -37,9 +42,9 @@ PUT _cluster/settings
```
{% include copy-curl.html %}

## Configuring window size
## Configuring the window size

You can configure the window size for the top N queries by latency with `search.insights.top_queries.latency.window_size`. For example, a cluster with the following configuration will collect top N queries in a 60-minute window:
To configure the monitoring window size, update the `window_size` setting for the desired metric type. For example, to collect the top N queries by latency in a 60-minute window, update the `search.insights.top_queries.latency.window_size` setting:

```json
PUT _cluster/settings
Expand All @@ -53,9 +58,9 @@ PUT _cluster/settings

## Configuring the value of N

You can configure the value of N in the `search.insights.top_queries.latency.top_n_size` parameter. For example, a cluster with the following configuration will collect the top 10 queries in the specified window size:
To configure the value of N, update the `top_n_size` setting for the desired metric type. For example, to collect the top 10 queries by latency, update the `insights.top_queries.latency.top_n_size` setting:

```
```json
PUT _cluster/settings
{
"persistent" : {
Expand All @@ -67,16 +72,63 @@ PUT _cluster/settings

## Monitoring the top N queries

You can use the Insights API endpoint to obtain top N queries by latency:
You can use the Insights API endpoint to obtain top N queries for all metric types:
kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved

```json
GET /_insights/top_queries
```
{% include copy-curl.html %}

Specify a metric type to filter the response by metric type (latency is the only supported type as of 2.12):
Specify a metric type to filter the response:

```json
GET /_insights/top_queries?type=latency
```
{% include copy-curl.html %}
{% include copy-curl.html %}

```json
GET /_insights/top_queries?type=cpu
```
{% include copy-curl.html %}

```json
GET /_insights/top_queries?type=memory
```
{% include copy-curl.html %}

## Exporting top N query data

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
You can configure your desired exporter to export top N query data to different sinks, allowing for better monitoring and analysis of your OpenSearch queries. Currently, the supported exporters are:
- [Debug exporter](#configuring-a-debug-exporter)
- [Local index exporter](#configuring-a-local-index-exporter)

### Configuring a debug exporter

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exporter setting?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a full setting name so I'll leave without code font.

To configure a debug exporter, update the exporter setting for the desired metric type. For example, to export the top N queries by latency using the debug exporter, send the following request:

```json
PUT _cluster/settings
{
"persistent" : {
"search.insights.top_queries.latency.exporter.type" : "debug"
}
}
```
{% include copy-curl.html %}

### Configuring a local index exporter

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
A local index exporter allows you to export the top N queries to local OpenSearch indexes. The default index pattern for top N query indexes is `top_queries-YYYY.MM.dd`. All top queries for the same day are saved to the same index, and a new index is created every day. You can change the default index pattern to use other date formats. For more information about supported formats, see [DateTimeFormat](https://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html).

kolchfa-aws marked this conversation as resolved.
Show resolved Hide resolved
To configure the local index exporter for the top N queiries by latency, send the following request:

```json
PUT _cluster/settings
{
"persistent" : {
"search.insights.top_queries.latency.exporter.type" : "local_index",
"search.insights.top_queries.latency.exporter.config.index" : "YYYY.MM.dd"
}
}
```
{% include copy-curl.html %}
Loading