-
Notifications
You must be signed in to change notification settings - Fork 507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update the integration page to reflect new integration catalog features #7324
Changes from 109 commits
93516cf
2615a67
49c42c0
f311e7d
cfcec50
1cfd6ff
272d5ee
c78a878
569da98
8879cd1
6ad6c33
569fc94
785ad27
b14f430
0f1edd9
fe96327
454eaf6
d6b7578
063300e
da691ae
ad584de
546df12
08a6f65
52d44a5
a02b9ce
299e34e
217d3a9
8857415
7f867d5
ad8da8c
7ac1d84
fb62391
da4a7ba
dd09883
3487063
00963f3
6be61b4
bc51f08
0a84371
0968065
aec912b
7e15223
91d7322
2ccbb68
04f416b
2d5340e
bfcd535
fa3239f
b3d2e4f
d040f9e
d90a7c7
00638a6
11de5f6
d8dab6b
7852e3a
1d89a46
65e60c0
846eac9
aa6b6a3
22dac2b
e69848c
ad9e2dd
adb1d47
d96441c
37a6034
47e652f
c813e75
29d5d37
f765189
e29380e
5fb500e
4ad8159
48ea732
1645420
6b11b59
fda846d
4fae073
784b476
91b4106
9a7bed6
e496e9d
f602b5b
9871ac7
5989e6f
0d67707
39c6a4b
d10a9c2
c321bf3
4116cc5
92bea17
1869cb2
0d4ae92
ab91f66
8bbabbe
f745177
a2f4963
0ec77b8
cd6ae7a
9882b16
6559ff0
e36429a
e428009
d013595
9808726
f96293e
b06545a
210c2df
83f04fc
be2b6cb
64cb7ac
58eee30
86879ef
7f84cc0
594ddd2
d82658c
523074b
e7e7af8
e457b27
eff1acd
736143c
1d08fa5
723a285
7d8c583
55ffab8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
--- | ||
layout: default | ||
title: OpenSearch Integrations | ||
title: Integrations | ||
nav_order: 1 | ||
has_children: false | ||
nav_exclude: true | ||
|
@@ -9,60 +9,116 @@ | |
- /integrations/index/ | ||
--- | ||
|
||
# OpenSearch Integrations | ||
# Integrations | ||
Introduced 2.9 | ||
{: .label .label-purple } | ||
|
||
OpenSearch Integrations is a straightforward starting point that OpenSearch and OpenSearch Dashboards users can use to visualize and understand log and metric data for a particular resource, such as NGINX. An _integration_ contains a bundle of metadata, data mappings, and visualizations that make it simple to monitor data from the Integrations resource. | ||
You can use **Integrations** to visualize and understand log and metric data for a particular resource in OpenSearch or OpenSearch Dashboards, such as NGINX or Amazon Virtual Private Cloud (Amazon VPC) flow logs. An _integration_ contains a bundle of metadata, data mappings, and visualizations that make it simple to monitor data from the resource. | ||
|
||
The OpenSearch Project seeks your feedback on this feature. Let us know on the [OpenSearch forum](https://forum.opensearch.org/) how OpenSearch Integrations works for you or how it can be improved. | ||
The OpenSearch Project seeks your feedback on this feature. Post on the [OpenSearch forum](https://forum.opensearch.org/) to let us know how OpenSearch Integrations works for you or how it can be improved. | ||
{: .label-yellow} | ||
|
||
## Setting up OpenSearch Integrations | ||
## Integrations for OpenSearch Dashboards | ||
|
||
For the latest developer information, including sample code, articles, tutorials, and an API reference, see the following resources: | ||
**Integrations** provides a user-friendly and robust platform for data visualization, querying, and projection, eliminating the need for the multiple redundant steps previously required to configure dashboards. | ||
|
||
- [Integrations repository](https://github.com/opensearch-project/observability/tree/e18cf354fd7720a6d5df6a6de5d53e51a9d43127/integrations) on GitHub | ||
- [Integration Creation Guide](https://github.com/opensearch-project/dashboards-observability/wiki/Integration-Creation-Guide) | ||
- [Integration Documentation Reference](https://github.com/opensearch-project/dashboards-observability/wiki/Integration-Documentation-Reference) | ||
- [Observability plugin for OpenSearch Dashboards](https://github.com/opensearch-project/dashboards-observability/wiki) | ||
|
||
## Integrations schema | ||
### Use cases | ||
|
||
**Integrations** aligns with established data schemas across multiple domains, enabling seamless data mapping and integration for various use cases, such as e-commerce product search, observability monitoring (for example, tracing and metrics analysis), and security monitoring and threat analysis. | ||
|
||
### OpenTelemetry protocol for observability | ||
|
||
A consistent telemetry data schema is crucial for effective observability, enabling data correlation and analysis across applications, services, and infrastructure components to provide a holistic view of system behavior and performance. | ||
|
||
OpenSearch adopted the [OpenTelemetry (OTel)](https://opentelemetry.io/) protocol as the foundation for its observability solution. OTel is a community-driven standard that defines a consistent schema and data collection approach for metrics, logs, and traces. It is widely supported by APIs, SDKs, and telemetry collectors, enabling features like auto-instrumentation for seamless observability integration. | ||
|
||
Adopting the OTel schema and leveraging its ecosystem enables **Integrations** to take a standardized approach to observability, providing you with comprehensive system insights. This shared schema allows cross-correlation and analysis across different data sources, enabling deeper insight into application and infrastructure performance. To this end, OpenSearch derived the [Simple Schema for Observability](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/observability), which encodes the OTel standard as OpenSearch mappings. | ||
|
||
Additionally, OpenSearch supports the [Piped Processing Language (PPL)](https://opensearch.org/docs/latest/search-plugins/sql/ppl/index/), designed for high-dimensionality querying in observability contexts, empowering you with advanced analysis and troubleshooting capabilities beyond traditional dashboarding. | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
The OpenSearch Integrations schema outlines how to capture, analyze, and visualize data. It includes the selection and configuration of monitoring tools, data collection methods, data storage and retention strategies, and visualization and alerting mechanisms. It follows the [OpenTelemetry Protocol convention](https://github.com/open-telemetry), with OpenSearch [Simple Schema for Observability](https://opensearch.org/docs/latest/observing-your-data/ssfo/) handling the translation from the OpenTelemetry (OTel) schema to the physical index mapping templates. | ||
### Ingesting data | ||
|
||
Find detailed information about the schema, including schema examples, in the [OpenSearch Observability READ.me file](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/README.md) and on the [OpenSearch Observability wiki](https://github.com/opensearch-project/dashboards-observability/wiki/OpenSearch-Observability--Home#observability-schema). | ||
Data ingested into OpenSearch must conform to the supported schemas for data integrations and associated dashboards. Compatible data pipelines are required, such as the following: | ||
|
||
## Get started | ||
- [Data Prepper](https://github.com/opensearch-project/data-prepper) | ||
- [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) | ||
- [Fluent Bit for OpenSearch](https://docs.fluentbit.io/manual/pipeline/outputs/opensearch) | ||
|
||
Using the OpenSearch Dashboards interface, you can connect your data, applications, and processes so that you can centrally manage the services you use. All integrations are available in a single view, and OpenSearch Dashboards guides you there from the home page and main menu. | ||
These pipelines use the OTel schema (or a simple schema) to index signal documents into the correct index representing the observed resource signals. See [Naming Convention](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/schema/observability/Naming-convention.md) for index naming conventions. | ||
|
||
Learn how to do the following using the OpenSearch Dashboards interface: | ||
### Applying the ingestion structure | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- Access integrations | ||
- View integrations | ||
- Add integrations | ||
Each integration contains the following metadata and assets: | ||
|
||
The following image gives you a snapshot of the Integrations interface: | ||
* Name and description | ||
* Source URL and license | ||
* Schema specification, for example, mapping or component mapping | ||
* Sample data for testing the feature | ||
* Assets such as dashboards, index patterns, queries, or alerts | ||
|
||
![Get started with Integrations demo]({{site.url}}{{site.baseurl}}/images/integrations/nginx-integration.gif) | ||
--- | ||
|
||
## Using Integrations | ||
Check failure on line 61 in _integrations/index.md GitHub Actions / style-job
|
||
|
||
### Access integrations | ||
**Integrations** can be installed directly from the [default catalog](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/integrations/Release.md) shipped with every OpenSearch release. Additionally, **Integrations** can be manually loaded and updated using the **Dashboards Management** console. | ||
|
||
To access integrations, open OpenSearch Dashboards and select **Integrations** from the **Management** menu. The interface displays installed and available integrations. | ||
### Installing integrations | ||
|
||
If no integration is installed, then you are prompted to install one. Go to **Integrations** > **Available** to view the supported integrations available from the default catalog. This feature is shown in the following image. | ||
|
||
![]({{site.url}}{{site.baseurl}}/images/integrations/empty-installed-integrations.png) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Since we ship with a default catalog, this state will never be visible There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. the integrations are not "installed" by default but rather the integration template are - we need to agree on the terminology here... There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This image should be referenced in the preceding text. |
||
|
||
## View integrations | ||
To add an integration, select the desired prepackaged asset. **Integrations** offers the following options: **Add** or **Try it**. You can choose **Add** to add or configure a new data integration by selecting a prepackaged integration asset. You can choose **Try it** to to test or explore the integration before fully adding it. This feature is shown in the following image. | ||
Check failure on line 71 in _integrations/index.md GitHub Actions / style-job
|
||
|
||
To view an integration, view the dashboard associated with the integration. If the integration doesn't have an associated dashboard, select the desired integration listed under the **Installed** window. View the integration details, such as assets and fields. | ||
![integrations-observability-catalog.png](/images/integrations/integrations-observability-catalog.png) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @YANG-DB This image differs from what I see in the nightly playground. Please let me know if I should update this image. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. keep the image - its up to date |
||
|
||
## Add integrations | ||
On the **Available** page, select the **Categories** dropdown menu to filter the list of integrations. This feature is shown in the following image. | ||
|
||
If you have not installed any integrations, you'll be prompted to install them from the Integrations interface. Supported integrations are listed in the **Available** window. | ||
![cloud-integrations-filter.png]({{site.url}}{{site.baseurl}}/images/integrations/cloud-integrations-filter.png) | ||
|
||
To add an integration, select the desired prepackaged assets. Currently, OpenSearch Integrations has two flows: Add or Try it. The following example uses the Try it flow: | ||
To try out the prepackaged integration assets, follow these steps: | ||
|
||
1. On the **Integrations** page, select **Nginx**. | ||
Check failure on line 81 in _integrations/index.md GitHub Actions / style-job
|
||
![]({{site.url}}{{site.baseurl}}/images/integrations/nginx-integration.png) | ||
|
||
1. On the **Integrations** page, select **NginX Dashboard**. | ||
2. Select the **Try it** button. _The Try it flow automatically creates a sample index template, adds sample data to the template, and then creates the integration based on that data._ | ||
3. View the asset list and select a dashboard asset. | ||
|
||
3. Select an asset from the **Asset List**. Assets include dashboards, index patterns, and visualizations. | ||
![]({{site.url}}{{site.baseurl}}/images/integrations/nginx-installed-integration-assets.png) | ||
|
||
4. Preview the data visualizations and sample data details. | ||
![]({{site.url}}{{site.baseurl}}/images/integrations/nginx-integration-dashboard.png) | ||
|
||
### Loading custom integrations | ||
|
||
To load a custom integration, follow these steps: | ||
|
||
1. Download an integration artifact from the [catalog repository](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/integrations/Release.md). An example of this step is shown in the following image. | ||
![]({{site.url}}{{site.baseurl}}/images/integrations/integration-catalog-release-page.png) | ||
|
||
2. Go to **Dashboards Management** > **Saved objects**. An example of this step is shown in the following image. | ||
![]({{site.url}}{{site.baseurl}}/images/integrations/import-saved-objects.png) | ||
|
||
3. Select **Import** on the upper-right toolbar menu and navigate to the folder where you saved the integration artifact. The file explorer window should open automatically. Choose the file (a file with an .ndjson extension). An example of this step is shown in the following image. | ||
![]({{site.url}}{{site.baseurl}}/images/integrations/integration-import-file.png) | ||
|
||
4. Select the object you uploaded. An example of this step is shown in the following image. | ||
![]({{site.url}}{{site.baseurl}}/images/integrations/select-uploaded-integration.png) | ||
|
||
--- | ||
|
||
## Developer resources | ||
|
||
See the following developer resources for sample code, articles, tutorials, and an API reference: | ||
|
||
- [OpenSearch Integrations repository](https://github.com/opensearch-project/opensearch-catalog) | ||
- [OpenSearch Integrations reference documentation](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/integrations) | ||
- [OpenSearch Observability Catalog](https://htmlpreview.github.io/?https://github.com/opensearch-project/opensearch-catalog/blob/main/integrations/observability/catalog.html) | ||
- [OpenSearch Observability Catalog release page](https://github.com/opensearch-project/opensearch-catalog/blob/main/docs/integrations/Release.md) | ||
- [Simple Schema for Observability](https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema/observability) | ||
|
||
--- | ||
|
||
## Community contribution | ||
|
||
Contribute to the project by submitting an [integration request](https://github.com/opensearch-project/dashboards-observability/issues/new?assignees=&labels=integration%2C+untriaged&projects=&template=integration_request.md&title=%5BIntegration%5D). |
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should advise users to use "Check for existing objects" + "Auto overwrite", because we ship the integrations with a specific UUID based on the integration+version to help keep duplicates down. Uploading the same integration template multiple times isn't that useful. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: A lot of this doc reads more like marketing copy and less like technical documentation (unlocking, powerful, empowering, etc.). Keep in mind that we aren't selling anything here---just describing for the user what something does and how to use it 😄.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. I tried to remove as much as of that tone as possible. I'll give it a re-review.