Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add query assist documentation to Observability #6419

Merged
merged 29 commits into from
Feb 19, 2024
Merged

Add query assist documentation to Observability #6419

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
8344898
Add query assist documentation to Observability
lezzago Feb 16, 2024
aba121a
Add summary detail
lezzago Feb 16, 2024
989028a
Doc review with comments and edits
vagimeli Feb 16, 2024
dc09f31
Update _observing-your-data/event-analytics.md
vagimeli Feb 16, 2024
4b9f314
Update event-analytics.md
vagimeli Feb 16, 2024
c73538d
Update event-analytics.md
vagimeli Feb 16, 2024
b05775e
Update event-analytics.md
vagimeli Feb 16, 2024
b337adc
Update event-analytics.md
vagimeli Feb 16, 2024
d999a9e
Copy edits
vagimeli Feb 16, 2024
c423514
Update event-analytics.md
vagimeli Feb 16, 2024
8bc17e5
Update event-analytics.md
vagimeli Feb 19, 2024
c6e2dcf
Update event-analytics.md
vagimeli Feb 19, 2024
856adb6
Update event-analytics.md
vagimeli Feb 19, 2024
acf7b1f
Update event-analytics.md
vagimeli Feb 19, 2024
d456738
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
c91b284
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
3ebbb70
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
6339ed4
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
5eb833e
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
a2bd1b4
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
32a60f4
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
c678781
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
ba8b45e
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
ad5735e
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
c738d05
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
5e58566
Update _observing-your-data/event-analytics.md
vagimeli Feb 19, 2024
2e6ea7a
Update event-analytics.md
vagimeli Feb 19, 2024
bc02e48
Merge branch 'main' into main
vagimeli Feb 19, 2024
b27995c
Update event-analytics.md
vagimeli Feb 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 60 additions & 24 deletions _observing-your-data/event-analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,57 +8,87 @@

# Event analytics

Event analytics in Observability is where you can use [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index) (PPL) queries to build and view different visualizations of your data.
Event analytics in OpenSearch Observability provides for creating data visualizations using [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/) (PPL) queries.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Getting started with event analytics

To get started, choose **Observability** in OpenSearch Dashboards and then choose **Event analytics**. If you want to start exploring without adding any of your own data, choose **Add samples**, and Dashboards adds sample visualizations you can interact with.
To get started, choose **Observability** in OpenSearch Dashboards and then choose **Logs**. If you want to start exploring without adding your own data, choose **Add samples**. Dashboards adds sample visualizations you can interact with. You can also try out pre-configured analytics in the [OpenSearch Playground](https://playground.opensearch.org/app/observability-logs#/).

Check failure on line 15 in _observing-your-data/event-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.SubstitutionsError] Use 'preconfigured' instead of 'pre-configured'.

Raw Output:
{"message": "[OpenSearch.SubstitutionsError] Use 'preconfigured' instead of 'pre-configured'.", "location": {"path": "_observing-your-data/event-analytics.md", "range": {"start": {"line": 15, "column": 260}}}, "severity": "ERROR"}
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Building a query

To generate custom visualizations, you must first specify a PPL query. OpenSearch Dashboards then automatically creates a visualization based on the results of your query.
To generate custom visualizations, you must first specify a PPL query. OpenSearch Dashboards then automatically creates a visualization based on your query results.

For example, the following PPL query returns a count of how many host addresses are currently in your data.

```
source = opensearch_dashboards_sample_data_logs | fields host | stats count()
```

By default, Dashboards shows results from the last 15 minutes of your data. To see data from a different time frame, use the date and time selector.
By default, Dashboards shows results from the last 15 minutes of your data. To see data from a different time frame, use the date and time selector to choose the desired settings.

For more information about building PPL queries, see [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index).
For more information about building PPL queries, see [Piped Processing Language]({{site.url}}{{site.baseurl}}/search-plugins/sql/ppl/index/).

## Saving a visualization
### OpenSearch Dashboards Query Assistant

Check failure on line 31 in _observing-your-data/event-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'OpenSearch Dashboards Query Assistant' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'OpenSearch Dashboards Query Assistant' is a heading and should be in sentence case.", "location": {"path": "_observing-your-data/event-analytics.md", "range": {"start": {"line": 31, "column": 5}}}, "severity": "ERROR"}

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [OpenSearch forum thread](https://forum.opensearch.org/t/feedback-opensearch-assistant/16741).
{: .warning}

To simplify query building, the **OpenSearch Assistant** toolkit offers an assistant to convert natural language queries into PPL. A screenshot is shown in the following image.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

![Sample OpenSearch Query Assist screen view]({{site.url}}{{site.baseurl}}/images/log-explorer-query-assist.png)

#### Enabling Query Assistant

Check failure on line 40 in _observing-your-data/event-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'Enabling Query Assistant' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'Enabling Query Assistant' is a heading and should be in sentence case.", "location": {"path": "_observing-your-data/event-analytics.md", "range": {"start": {"line": 40, "column": 6}}}, "severity": "ERROR"}

After Dashboards generates a visualization, you must save it if you want to return to it at a later time or if you want to add it to an [operational panel]({{site.url}}{{site.baseurl}}/observing-your-data/operational-panels).
To enable **Query Assistant** in OpenSearch Dashboards, locate your copy of the `opensearch_dashboards.yml` file and set the following option:

To save a visualization, expand the save dropdown menu next to **Refresh**, enter a name for your visualization, then choose **Save**. You can reopen any saved visualizations on the event analytics page.
```
observability.query_assist.enabled: true
observability.query_assist.ppl_agent_name: "PPL agent"
lezzago marked this conversation as resolved.
Show resolved Hide resolved
```

To enable the option to summarize the responses, locate your copy of the `opensearch_dashboards.yml` file and set the following option:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"To enable summarization of responses"?


```
observability.summarize.enabled: true
observability.summarize.response_summary_agent_name: "Response summary agent"
observability.summarize.error_summary_agent_name: "Error summary agent"
```

#### Setting up Query Assistant

Check failure on line 57 in _observing-your-data/event-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.HeadingCapitalization] 'Setting up Query Assistant' is a heading and should be in sentence case.

Raw Output:
{"message": "[OpenSearch.HeadingCapitalization] 'Setting up Query Assistant' is a heading and should be in sentence case.", "location": {"path": "_observing-your-data/event-analytics.md", "range": {"start": {"line": 57, "column": 6}}}, "severity": "ERROR"}

To set up **Query Assistant**, follow the steps in the [Getting started guide](https://github.com/opensearch-project/dashboards-assistant/blob/main/GETTING_STARTED_GUIDE.md) on GitHub. You'll receive step-by-step setup instructions for **OpenSearch Assistant** and **Query Assistant**. To set up **Query Assistant** only, use the `query-assist-agent` template included in the guide.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Saving a visualization

After Dashboards generates a visualization, save it if you wish to revisit it or include it in an [operational panel]({{site.url}}{{site.baseurl}}/observing-your-data/operational-panels). To save a visualization, expand the save dropdown menu to the right of the **Refresh** button, enter a name for your visualization, and then select **Save**. You can reopen saved visualizations from the event analytics page.

Check failure on line 63 in _observing-your-data/event-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.LinksEndSlash] Add a trailing slash to the link '({{site.url}}{{site.baseurl}}/observing-your-data/operational-panels)'.

Raw Output:
{"message": "[OpenSearch.LinksEndSlash] Add a trailing slash to the link '({{site.url}}{{site.baseurl}}/observing-your-data/operational-panels)'.", "location": {"path": "_observing-your-data/event-analytics.md", "range": {"start": {"line": 63, "column": 118}}}, "severity": "ERROR"}
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should "save" before "dropdown menu" be formatted as the name of a UI element?


## Creating event analytics visualizations and adding them to dashboards

This feature is available in OpenSearch Dashboards version 2.7 and later. It works with new visualizations created in version 2.7 or later that use PPL to query data from OpenSearch or federated data sources such as Prometheus.
This feature is available in OpenSearch Dashboards 2.7 and later. It works with new visualizations that use PPL to query data from OpenSearch or federated data sources such as Prometheus.
{: .note}

Presenting your visualizations on a dashboard, instead of the event analytics page, makes it easier for users to understand and interpret the data at a glance.
Presenting your visualizations on a dashboard, instead of from the event analytics page, simplifies understanding and interpreting data at a glance.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sentence reads as a bit unnecessary. Could it be safely removed?


To create a PPL visualization, follow these steps:

1. On the main menu, choose **Visualize** > **PPL**.
2. In the **Observability** > **Logs** > **Explorer** window, enter the index source in the **PPL query** field, for example, `source = opensearch_dashboards_sample_data_flights | stats count() by DestCountry`. You must enter the query using PPL syntax.
2. From the **Observability** > **Logs** > **Explorer** window, enter the index source in the **PPL query** field, for example, `source = opensearch_dashboards_sample_data_flights | stats count() by DestCountry`. You must enter the query using PPL syntax.

Check failure on line 75 in _observing-your-data/event-analytics.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: _flights. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: _flights. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_observing-your-data/event-analytics.md", "range": {"start": {"line": 75, "column": 173}}}, "severity": "ERROR"}
3. Set the time filter, for example, **This week**, and then select **Refresh**.
4. Choose the visualization type, for example, **Pie**, from the right sidebar dropdown menu.
5. Select **Save** and enter a name for the visualization.
4. Choose the visualization type, for example, **Pie**, from the sidebar dropdown menu to the right.
5. Select **Save**, and enter a name for the visualization.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

You've created a new visualization that can be added to a new or existing dashboard. To add a PPL query to a dashboard, follow these steps:
You've created a new visualization that can be added to a new or an existing dashboard. To add a PPL query to a dashboard, follow these steps:
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"You've now created"?


1. Select **Dashboard** from the main menu.
2. In the **Dashboards** window, select **Create > Dashboard**.
1. Select **Dashboards** from the main menu.
2. In the **Dashboards** window, select **Create** > **Dashboard**.
3. In the **Editing New Dashboard** window, choose **Add an existing**.
4. In the **Add panels** window, choose **PPL** and select the visualization. It is now displayed on your dashboard.
5. Select **Save** and enter a name for the dashboard.
6. To add more visualizations to the dashboard, choose **Select existing visualization** and follow the steps above. Alternatively, choose **Create new** and then select **PPL** in the **New Visualization** window. You'll return to the event analytics page and follow steps 1--6 in the preceding instructions.
4. In the **Add panels** window, choose **PPL** from the **Types** dropdown menu, and then select the visualization. It is now displayed on your dashboard.
5. Select **Save**, and enter a name for the dashboard.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
6. To add more visualizations to the dashboard, choose **Select existing visualization** and follow steps 1--5. Alternatively, choose **Create new** and then select **PPL** in the **New Visualization** window. You'll return to the event analytics page and follow steps 1--5 in the preceding instructions.

The following demo provides an overview of creating event analytics visualizations and adding them to a dashboard.

![Demo of how to create event analytics visualizations and add them to a dashboard]({{site.url}}{{site.baseurl}}/images/dashboards/event-analytics-dashboard.gif)
![Demo of creating event analytics visualizations and adding them to a dashboard]({{site.url}}{{site.baseurl}}/images/dashboards/event-analytics-dashboard.gif)

### Limitations of event analytics visualizations

Expand All @@ -73,20 +103,26 @@

### Correlating logs and traces

If you regularly track events across applications, you can correlate logs and traces. To view the correlation, you have to index the traces according to Open Telemetry standards (similar to trace analytics). Once you add a `TraceId` field to your logs, you can view the correlated trace information in the event explorer log details. This method lets you correlate logs and traces that correspond to the same execution context.
If you regularly track events across applications, you can correlate logs and traces. To view the correlation, you have to index the traces according to Open Telemetry standards, similarly to [trace analytics]({{site.url}}{{site.baseurl}}/observing-your-data/trace/index/). Once you add a `TraceId` field to your logs, you can view the correlated trace information in the event explorer log details. This method correlates logs and traces that correspond to the same execution context. The following demo shows this feature in action.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

![Trace Log Correlation]({{site.url}}{{site.baseurl}}/images/trace_log_correlation.gif)

### Viewing surrounding events

If you want to know more about a log event you're looking at, you can select **View surrounding events** to get a bigger picture of what was happening around the time of interest.
If you need more information about a log event you're analyzing, you can select **View surrounding events** to gain a more comprehensive understanding of the context around the time of interest. The following demo shows this feature in action.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "you're analyzing" necessary here, or could it be safely removed?


![Surrounding Events]({{site.url}}{{site.baseurl}}/images/surrounding_events.gif)

### Livestreaming logs

If you prefer watching events happen live, you can configure an interval so event analytics automatically refreshes the content. Live tail lets you stream logs live to OpenSearch observability event analytics based on the provided PPL query, as well as provide rich functionality such as filters. Doing so improves your debugging experience and lets you monitor your logs in real-time without having to manually refresh.
If you prefer real-time monitoring, you can set up an interval for automatic content refresh in event analytics. Live Tail enables you to stream logs directly to OpenSearch Observability event analytics using the specified PPL query, while offering robust features like filters. This enhances your debugging process and enables seamless real-time monitoring of logs without the need for manual refreshing.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First sentence: "an interval at which event analytics content will be automatically refreshed"? Last sentence: "without the need to manually refresh content"?


You can also choose intervals and switch between them to dictate how often live tail should stream live logs. This feature is similar to the CLI's `tail -f` command in that it only retrieves the most recent live logs by possibly eliminating a large portion of live logs. Live tail also provides you with the total count of live logs received by OpenSearch during the live stream, which you can use to better understand the incoming traffic.
You can select intervals and seamlessly switch between them to control the frequency of live log streaming with Live Tail. This functionality is similar to the CLI command `tail -f` command, as it retrieves only the most recent live logs, potentially eliminating a significant portion of live logs. Live Tail displays the total count of live logs received by OpenSearch during the live stream, offering insights into incoming traffic patterns. The following demo shows this feature in action.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be another - to mark the next bullet point?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Formatting done.

vagimeli marked this conversation as resolved.
Show resolved Hide resolved

![Live Tail]({{site.url}}{{site.baseurl}}/images/live_tail.gif)

## Related articles

- [Demonstrating OpenSearch Assistant toolkit for Analysis](https://www.youtube.com/watch?v=VTiJtGI2Sr4&t=152s)
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed to this to reflect the exact title on YouTube.

- [Getting started guide for OpenSearch Assistant in OpenSearch Dashboards](https://github.com/opensearch-project/dashboards-assistant/blob/main/GETTING_STARTED_GUIDE.md)
- [OpenSearch Assistant configuration through the REST API]({site.url}}{{site.baseurl}}/ml-commons-plugin/opensearch-assistant/)
Binary file added images/log-explorer-query-assist.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading