Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the OTel quickstart #4615

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions docs/en/observability/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ include::quickstarts/monitor-hosts-with-elastic-agent.asciidoc[leveloffset=+2]

include::quickstarts/monitor-k8s-logs-metrics.asciidoc[leveloffset=+2]

include::quickstarts/monitor-hosts-with-otel.asciidoc[leveloffset=+2]
mdbirnstiehl marked this conversation as resolved.
Show resolved Hide resolved

include::quickstarts/monitor-k8s-otel.asciidoc[leveloffset=+2]

include::quickstarts/collect-data-with-aws-firehose.asciidoc[leveloffset=+2]
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
[[quickstart-monitor-hosts-with-otel]]
= Quickstart: Monitor hosts with OpenTelemetry

preview::[]

In this quickstart guide, you'll learn how to monitor your hosts using the Elastic Distribution of OpenTelemetry (EDOT) Collector.
You'll also learn how to use {observability} features to gain deeper insight into your observability data after collecting it.

[discrete]
== Prerequisites

* An {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data. This quickstart is available for all Elastic deployment models. The quickest way to get started with this quickstart is using a trial project on {serverless-docs}/quickstart-monitor-hosts-with-otel.html[Elastic serverless].
* This quickstart is only available for Linux and MacOS systems.
* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to {cloud}/ec-user-privileges.html[User roles and privileges].
* Root privileges on the host—required to run the OpenTelemetry collector because of these components:
** `hostmetrics` receiver to read all system metrics (all processes, memory, etc.).
** `filelog` to allow the collector to read any user or application log files.


[discrete]
== Limitations
Refer to https://github.com/elastic/opentelemetry/blob/main/docs/collector-limitations.md[Elastic OpenTelemetry Collector limitations] for known limitations when using the EDOT Collector.

[discrete]
== Collect your data

Follow these steps to collect logs and metrics using the EDOT Collector:

. In {kib}, go to the **Observability** UI and click **Add Data**.
. Under **What do you want to monitor?** select **Host**, and then select **OpenTelemetry: Logs & Metrics**.
+
[role="screenshot"]
image::images/quickstart-monitor-hosts-otel-entry-point.png[Host monitoring entry point]
. Select the appropriate platform.
. Copy the command under step 1, open a terminal on your host, and run the command.
+
This command downloads the {agent} package, extracts it in a EDOT directory. For example, `elastic-distro-8.16.0-linux-x86_64`.
It also adds a sample `otel.yml` configuration file to the directory and updates the storage directory, Elastic endpoint, and API key in the file.
+
The default log path is `/var/log/*.log`. To update the path, modify the `otel.yml` in the EDOT directory.
+
Find additional sample `otel.yml` configuration files in the EDOT directory in the `otel_samples` folder.
. Copy the command under Step 2 and run it in your terminal to start the EDOT Collector.

NOTE: Logs are collected from setup onward, so you won't see logs that occurred before starting the EDOT Collector.

Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.

[discrete]
== Gain deeper insight into your host data

After using the Hosts page and Discover to confirm you've ingested all the host logs and metrics you want to monitor,
use Elastic {observability} to gain deeper insight into your host data with the following capabilities and features:

* In the <<monitor-infrastructure-and-hosts,Infrastructure UI>>, analyze and compare data collected from your hosts.
You can also:
** <<inspect-metric-anomalies,Detect anomalies>> for memory usage and network traffic on hosts.
** <<create-alerts,Create alerts>> that notify you when an anomaly is detected or a metric exceeds a given value.
* In the <<explore-logs,Logs Explorer>>, search and filter your log data,
get information about the structure of log fields, and display your findings in a visualization.
You can also:
** <<monitor-datasets,Monitor log data set quality>> to find degraded documents.
** {kibana-ref}/xpack-ml-aiops.html#log-pattern-analysis[Run a pattern analysis] to find patterns in unstructured log messages.
** <<create-alerts,Create alerts>> that notify you when an Observability data type reaches or exceeds a given value.
* Use {kibana-ref}/xpack-ml.html[machine learning] to apply predictive analytics to your data:
** {kibana-ref}/xpack-ml-anomalies.html[Detect anomalies] by comparing real-time and historical data from different sources to look for unusual, problematic patterns.
** {kibana-ref}/xpack-ml-aiops.html#log-rate-analysis[Analyze log spikes and drops].
** {kibana-ref}/xpack-ml-aiops.html#change-point-detection[Detect change points] in your time series data.

Refer to the <<observability-introduction>> for a description of other useful features.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions docs/en/serverless/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ include::./projects/create-an-observability-project.asciidoc[leveloffset=+3]
// Quickstarts
include::./quickstarts/monitor-hosts-with-elastic-agent.asciidoc[leveloffset=+3]
include::./quickstarts/k8s-logs-metrics.asciidoc[leveloffset=+3]
include::./quickstarts/monitor-hosts-with-otel.asciidoc[leveloffset=+3]
include::./quickstarts/monitor-k8s-otel.asciidoc[leveloffset=+3]
include::./quickstarts/collect-data-with-aws-firehose.asciidoc[leveloffset=+3]

Expand Down
68 changes: 68 additions & 0 deletions docs/en/serverless/quickstarts/monitor-hosts-with-otel.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
[[quickstart-monitor-hosts-with-otel]]
= Quickstart: Monitor hosts with OpenTelemetry

preview::[]

In this quickstart guide, you'll learn how to monitor your hosts using the Elastic Distribution of OpenTelemetry (EDOT) Collector.
You'll also learn how to use {observability} features to gain deeper insight into your observability data after collecting it.

[discrete]
== Prerequisites

* An {observability} project. To learn more, refer to <<observability-create-an-observability-project>>.
* This quickstart is only available for Linux and MacOS systems.
* A user with the **Admin** role or higher—required to onboard system logs and metrics. To learn more, refer to <<general-assign-user-roles>>.
* Root privileges on the host—required to run the OpenTelemetry collector because of these components:
** `hostmetrics` receiver to read all system metrics (all processes, memory, etc.).
** `filelog` to allow the collector to read any user or application log files.

[discrete]
== Limitations
Refer to https://github.com/elastic/opentelemetry/blob/main/docs/collector-limitations.md[Elastic OpenTelemetry Collector limitations] for known limitations when using the EDOT Collector.

[discrete]
== Collect your data

Follow these steps to collect logs and metrics using the EDOT Collector:

. <<observability-create-an-observability-project,Create a new {obs-serverless} project>>, or open an existing one.
. To open the quickstart, go to **Add Data**.
. Select **Collect and analyze logs**, and then select **OpenTelemetry**.
. Under **What do you want to monitor?** select **Host**, and then select **Elastic Agent: Logs & Metrics**.
+
[role="screenshot"]
image::images/quickstart-monitor-hosts-otel-entry-point.png[Host monitoring entry point]
. Select the appropriate platform, and complete the following:
. For **MacOS and Linux**, copy the command, open a terminal on your host, and run the command to download and configure the OpenTelemetry collector.
. For **Kubernetes**, download the manifest.
. Copy the command under Step 2:
. For **MacOS and Linux**, run the command in your terminal to start the EDOT Collector.
. For **Kubernetes**, run the command from the directory where you downloaded the manifest to install the EDOT Collector on every node of your cluster.

Logs are collected from setup onward, so you won't see logs that occurred before starting the EDOT Collector.
The default log path is `/var/log/*`. To update the path, modify `otel.yml`.

Under **Visualize your data**, you'll see links to **Logs Explorer** to view your logs and **Hosts** to view your host metrics.

[discrete]
== Get value out of your data

After using the Hosts page and Discover to confirm you've ingested all the host logs and metrics you want to monitor,
use Elastic {observability} to gain deeper insight into your host data with the following capabilities and features:

* In the <<observability-infrastructure-monitoring,Infrastructure UI>>, analyze and compare data collected from your hosts.
You can also:
** <<observability-detect-metric-anomalies,Detect anomalies>> for memory usage and network traffic on hosts.
** <<observability-create-manage-rules,Create alerts>> that notify you when an anomaly is detected or a metric exceeds a given value.
* In the <<observability-discover-and-explore-logs,Logs Explorer>>, search and filter your log data,
get information about the structure of log fields, and display your findings in a visualization.
You can also:
** <<observability-monitor-datasets,Monitor log data set quality>> to find degraded documents.
** {kibana-ref}/xpack-ml-aiops.html#log-pattern-analysis[Run a pattern analysis] to find patterns in unstructured log messages.
** <<observability-create-manage-rules,Create alerts>> that notify you when an Observability data type reaches or exceeds a given value.
* Use {kibana-ref}/xpack-ml.html[machine learning] to apply predictive analytics to your data:
** {kibana-ref}/xpack-ml-anomalies.html[Detect anomalies] by comparing real-time and historical data from different sources to look for unusual, problematic patterns.
** {kibana-ref}/xpack-ml-aiops.html#log-rate-analysis[Analyze log spikes and drops].
** {kibana-ref}/xpack-ml-aiops.html#change-point-detection[Detect change points] in your time series data.

Refer to the <<what-is-observability-serverless>> for a description of other useful features.