-
Notifications
You must be signed in to change notification settings - Fork 507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update the integration page to reflect new integration catalog features #7324
Changes from 115 commits
93516cf
2615a67
49c42c0
f311e7d
cfcec50
1cfd6ff
272d5ee
c78a878
569da98
8879cd1
6ad6c33
569fc94
785ad27
b14f430
0f1edd9
fe96327
454eaf6
d6b7578
063300e
da691ae
ad584de
546df12
08a6f65
52d44a5
a02b9ce
299e34e
217d3a9
8857415
7f867d5
ad8da8c
7ac1d84
fb62391
da4a7ba
dd09883
3487063
00963f3
6be61b4
bc51f08
0a84371
0968065
aec912b
7e15223
91d7322
2ccbb68
04f416b
2d5340e
bfcd535
fa3239f
b3d2e4f
d040f9e
d90a7c7
00638a6
11de5f6
d8dab6b
7852e3a
1d89a46
65e60c0
846eac9
aa6b6a3
22dac2b
e69848c
ad9e2dd
adb1d47
d96441c
37a6034
47e652f
c813e75
29d5d37
f765189
e29380e
5fb500e
4ad8159
48ea732
1645420
6b11b59
fda846d
4fae073
784b476
91b4106
9a7bed6
e496e9d
f602b5b
9871ac7
5989e6f
0d67707
39c6a4b
d10a9c2
c321bf3
4116cc5
92bea17
1869cb2
0d4ae92
ab91f66
8bbabbe
f745177
a2f4963
0ec77b8
cd6ae7a
9882b16
6559ff0
e36429a
e428009
d013595
9808726
f96293e
b06545a
210c2df
83f04fc
be2b6cb
64cb7ac
58eee30
86879ef
7f84cc0
594ddd2
d82658c
523074b
e7e7af8
e457b27
eff1acd
736143c
1d08fa5
723a285
7d8c583
55ffab8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
--- | ||
layout: default | ||
title: OpenSearch Integrations | ||
title: Integrations in OpenSearch Dashboards | ||
nav_order: 1 | ||
has_children: false | ||
nav_exclude: true | ||
|
@@ -9,60 +9,102 @@ | |
- /integrations/index/ | ||
--- | ||
|
||
# OpenSearch Integrations | ||
# Integrations in OpenSearch Dashboards | ||
Introduced 2.9 | ||
{: .label .label-purple } | ||
|
||
OpenSearch Integrations is a straightforward starting point that OpenSearch and OpenSearch Dashboards users can use to visualize and understand log and metric data for a particular resource, such as NGINX. An _integration_ contains a bundle of metadata, data mappings, and visualizations that make it simple to monitor data from the Integrations resource. | ||
The **Integrations** application in OpenSearch Dashboards provides a user-friendly platform for data visualization, querying, and projection of your resource data, such as flow logs. An _integration asset_, such NGINX or Amazon VPC, contains a bundle of metadata, data mappings, and visualizations, streamlining the monitoring of resource data without redundant configuration steps. | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
The OpenSearch Project seeks your feedback on this feature. Let us know on the [OpenSearch forum](https://forum.opensearch.org/) how OpenSearch Integrations works for you or how it can be improved. | ||
{: .label-yellow} | ||
Available integration assets in OpenSearch Dashboards are shown in the following image. | ||
YANG-DB marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Setting up OpenSearch Integrations | ||
![Integrations assets available in OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/images/dashboards/integrations-assets.png) | ||
|
||
For the latest developer information, including sample code, articles, tutorials, and an API reference, see the following resources: | ||
--- | ||
|
||
- [Integrations repository](https://github.com/opensearch-project/observability/tree/e18cf354fd7720a6d5df6a6de5d53e51a9d43127/integrations) on GitHub | ||
- [Integration Creation Guide](https://github.com/opensearch-project/dashboards-observability/wiki/Integration-Creation-Guide) | ||
- [Integration Documentation Reference](https://github.com/opensearch-project/dashboards-observability/wiki/Integration-Documentation-Reference) | ||
- [Observability plugin for OpenSearch Dashboards](https://github.com/opensearch-project/dashboards-observability/wiki) | ||
|
||
## Integrations schema | ||
## Use cases | ||
|
||
**Integrations** considers established data schemas across multiple domains to give you seamless data mapping and integration for various use cases, such as e-commerce product search, observability monitoring (for example, tracing and metrics analysis), and security monitoring and threat analysis. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "tracing" => "trace"?
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## OpenTelemetry protocol for observability | ||
|
||
A consistent telemetry data schema is crucial for effective observability, enabling data correlation and analysis across applications, services, and infrastructure components to provide a holistic view of system behavior and performance. | ||
|
||
OpenSearch adopted the [OpenTelemetry (OTel)](https://opentelemetry.io/) protocol as the foundation for its observability solution. OTel is a community-driven standard that defines a consistent schema and data collection approach for metrics, logs, and traces. It is widely supported by APIs, SDKs, and telemetry collectors, enabling features like auto-instrumentation for seamless observability integration. | ||
|
||
This shared schema allows cross-correlation and analysis across different data sources. To this end, OpenSearch derived the [Simple Schema for Observability](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/observability), which encodes the OTel standard as OpenSearch mappings. OpenSearch also supports the [Piped Processing Language (PPL)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/), which is designed for high-dimensionality querying in observability use cases. | ||
|
||
--- | ||
|
||
## Ingesting data | ||
|
||
Data ingested into OpenSearch must conform to the supported schemas for data integrations and their associated dashboards. Compatible data pipelines are required, such as the following: | ||
|
||
The OpenSearch Integrations schema outlines how to capture, analyze, and visualize data. It includes the selection and configuration of monitoring tools, data collection methods, data storage and retention strategies, and visualization and alerting mechanisms. It follows the [OpenTelemetry Protocol convention](https://github.com/open-telemetry), with OpenSearch [Simple Schema for Observability](https://opensearch.org/docs/latest/observing-your-data/ssfo/) handling the translation from the OpenTelemetry (OTel) schema to the physical index mapping templates. | ||
- [Data Prepper](https://github.com/opensearch-project/data-prepper) | ||
- [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) | ||
- [Fluent Bit for OpenSearch](https://docs.fluentbit.io/manual/pipeline/outputs/opensearch) | ||
|
||
Find detailed information about the schema, including schema examples, in the [OpenSearch Observability READ.me file](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/README.md) and on the [OpenSearch Observability wiki](https://github.com/opensearch-project/dashboards-observability/wiki/OpenSearch-Observability--Home#observability-schema). | ||
These pipelines use the OTel schema (or a simple schema) to index signal documents into the correct index representing the observed resource signals. See [Naming Convention](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/Naming-convention.md) for index naming conventions. | ||
|
||
## Get started | ||
### Ingestion structure | ||
|
||
Using the OpenSearch Dashboards interface, you can connect your data, applications, and processes so that you can centrally manage the services you use. All integrations are available in a single view, and OpenSearch Dashboards guides you there from the home page and main menu. | ||
Each integration asset contains the following metadata and assets: | ||
|
||
Learn how to do the following using the OpenSearch Dashboards interface: | ||
* Name and description | ||
* Source URL and license | ||
* Schema specification, for example, mapping or component mapping | ||
* Sample data for testing the feature | ||
* Assets such as dashboards, index patterns, queries, or alerts | ||
|
||
- Access integrations | ||
- View integrations | ||
- Add integrations | ||
--- | ||
|
||
## Installing an integration asset | ||
|
||
Integration assets can be installed directly from the [default catalog](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/integrations/Release.md) shipped with every OpenSearch release. | ||
|
||
The following image gives you a snapshot of the Integrations interface: | ||
To install an asset, follow these steps: | ||
|
||
![Get started with Integrations demo]({{site.url}}{{site.baseurl}}/images/integrations/nginx-integration.gif) | ||
1. Go to **Integrations** > **Available** to view the available options. | ||
2. Select a tool, such as **Nginx** or **Amazon VPC**. You can choose **Add** to add or configure a new data integration using a prepackaged integration asset. You can choose **Try it** to test or explore the integration before fully adding it. | ||
Check failure on line 67 in _integrations/index.md GitHub Actions / style-job
|
||
3. On the **Available** page, select the **Categories** dropdown menu to filter the list of integrations. | ||
|
||
### Access integrations | ||
### Try out demo | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Try it" demo?
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
To access integrations, open OpenSearch Dashboards and select **Integrations** from the **Management** menu. The interface displays installed and available integrations. | ||
To try out a prepackaged integration asset, follow these steps: | ||
YANG-DB marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## View integrations | ||
1. On the **Integrations** page, select **Nginx**. | ||
Check failure on line 74 in _integrations/index.md GitHub Actions / style-job
|
||
2. Select the **Try it** button. The **Try it** option automatically creates a sample index template, adds sample data to the template, and then creates the integration based on that data. | ||
3. Select an asset from the **Asset List**. Assets include dashboards, index patterns, and visualizations. | ||
4. Preview the data visualizations and sample data details. An example is shown in the following image. | ||
|
||
To view an integration, view the dashboard associated with the integration. If the integration doesn't have an associated dashboard, select the desired integration listed under the **Installed** window. View the integration details, such as assets and fields. | ||
![Integrations dashboard with visualizations]({{site.url}}{{site.baseurl}}/images/integrations/nginx-integration-dashboard.png) | ||
|
||
## Add integrations | ||
## Loading custom integration assets | ||
|
||
If you have not installed any integrations, you'll be prompted to install them from the Integrations interface. Supported integrations are listed in the **Available** window. | ||
To load a custom integration asset, follow these steps: | ||
|
||
1. Download an integration artifact from the [catalog repository](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/integrations/Release.md). | ||
2. Go to **Dashboards Management** > **Saved objects**. | ||
3. Select **Import** on the upper-right toolbar menu and navigate to the folder where you saved the integration artifact and then choose the file (a file with an .ndjson extension). An example of this step is shown in the following image. | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
![Import folder window]({{site.url}}{{site.baseurl}}/images/integrations/integration-import-file.png) | ||
4. Select the saved object you uploaded to confirm it is uploaded to **Saved objects**. An example of this step is shown in the following image. | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
![List of integrations saved objects]({{site.url}}{{site.baseurl}}/images/integrations/select-uploaded-integration.png) | ||
|
||
--- | ||
|
||
To add an integration, select the desired prepackaged assets. Currently, OpenSearch Integrations has two flows: Add or Try it. The following example uses the Try it flow: | ||
## Developer resources | ||
|
||
1. On the **Integrations** page, select **NginX Dashboard**. | ||
2. Select the **Try it** button. _The Try it flow automatically creates a sample index template, adds sample data to the template, and then creates the integration based on that data._ | ||
3. View the asset list and select a dashboard asset. | ||
4. Preview the data visualizations and sample data details. | ||
See the following developer resources for sample code, articles, tutorials, and an API reference: | ||
|
||
- [OpenSearch Integrations repository](https://github.com/opensearch-project/opensearch-catalog) | ||
- [OpenSearch Integrations reference documentation](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/integrations) | ||
- [OpenSearch Observability Catalog](https://htmlpreview.github.io/?https://github.com/opensearch-project/opensearch-catalog/blob/main/integrations/observability/catalog.html) | ||
- [OpenSearch Observability Catalog release page](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/integrations/Release.md) | ||
- [Simple Schema for Observability](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/observability) | ||
|
||
--- | ||
|
||
## Community contribution | ||
|
||
The OpenSearch Project seeks your feedback on this feature. Post on the [OpenSearch forum](https://forum.opensearch.org/) to let us know how OpenSearch Integrations works for you or how it can be improved. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we want to bold "OpenSearch Integrations" here as well? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. yes There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Contribute to the project by submitting an [integration request](https://github.com/opensearch-project/dashboards-observability/issues/new?assignees=&labels=integration%2C+untriaged&projects=&template=integration_request.md&title=%5BIntegration%5D). |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should advise users to use "Check for existing objects" + "Auto overwrite", because we ship the integrations with a specific UUID based on the integration+version to help keep duplicates down. Uploading the same integration template multiple times isn't that useful. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we define Amazon VPC here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think its well a known service - no need to define all the terms
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Defined. We need to follow AWS naming conventions when we mention a service on first mention.