Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Observability Onboarding] Add AWS Firehose quickstart #4433

Merged
merged 11 commits into from
Nov 12, 2024
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions docs/en/observability/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ include::quickstarts/monitor-hosts-with-elastic-agent.asciidoc[leveloffset=+2]

include::quickstarts/monitor-k8s-logs-metrics.asciidoc[leveloffset=+2]

include::quickstarts/collect-data-with-aws-firehose.asciidoc[leveloffset=+2]

include::splunk-get-started.asciidoc[leveloffset=+2]

// Group: Application and service monitoring
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
[[collect-data-with-aws-firehose]]
= Quickstart: Collect data with AWS Firehose

preview::[]

In this quickstart guide, you'll learn how to send logs and metrics to Elastic using AWS Firehose.
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

The AWS Firehose receiver is based on a CloudFormation template that can collect all available CloudWatch logs and metrics for your AWS account.
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

This approach requires minimal configuration as the CloudFormation template creates a Firehose stream, enables CloudWatch metrics collection, and sets up account level subscription filter for CloudWatch log groups to send logs to Elastic via Firehose.
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

NOTE: Optionally, you can download the CloudFormation template to review and modify the default settings. For more information, check the
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved
link:https://www.elastic.co/docs/current/integrations/aws[AWS integration] documentation.
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

Data collection with AWS Firehose is supported on ESS deployments in AWS, Azure and GCP.

[discrete]
== Prerequisites
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

* A deployment using our hosted {ess} on {ess-trial}[{ecloud}]. The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data.
* A user with the `superuser` {ref}/built-in-roles.html[built-in role] or the privileges required to onboard data.
+
[%collapsible]
.Expand to view required privileges
====
* {ref}/security-privileges.html#privileges-list-cluster[**Cluster**]: `['monitor', 'manage_own_api_key']`
* {ref}/security-privileges.html#privileges-list-indices[**Index**]: `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }`
* {kibana-ref}/kibana-privileges.html[**Kibana**]: `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }`
====
* An active AWS account and the necessary permissions to create delivery streams.

[discrete]
== Limitations

The AWS Firehose receiver has the following limitations:

* It does not support AWS PrivateLink.
* It is not available for on-premise Elastic Stack deployments.

[discrete]
== Collect your data

. In {kib}, go to **Observability** and click **Add Data**.

. Select **Cloud**, **AWS**, and then select **AWS Firehose**.
+
[role="screenshot"]
image::images/quickstart-aws-firehose-entry-point.png[AWS Firehose entry point]

. Click **Create Firehose Stream in AWS** to create a CloudFormation stack from the CloudFormation template.

. Go back to the **Add Observability Data** page.

[discrete]
== Visualize your data

After installation is complete and all relevant data is flowing into Elastic,
the **Visualize your data** section allows you to access the different dashboards for the various services.

[role="screenshot"]
image::images/quickstart-aws-firehose-dashboards.png[AWS Firehose dashboards]
alaudazzi marked this conversation as resolved.
Show resolved Hide resolved

Refer to <<observability-introduction>> for a description of other useful features.
bmorelli25 marked this conversation as resolved.
Show resolved Hide resolved