[Observability Onboarding] Add AWS Firehose quickstart (#4433)

* Add AWS Firehose quickstart

* More edits

* Add link to AWS integration docs

* Integrate reviewers feedback

* Update docs/en/observability/quickstarts/collect-data-with-aws-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/quickstarts/collect-data-with-aws-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Update docs/en/observability/quickstarts/collect-data-with-aws-firehose.asciidoc

Co-authored-by: Brandon Morelli <[email protected]>

* Integrate Mykola's feedback

* Add list of parameters

* Integrate reviewer feedback

* Add table

---------

Co-authored-by: Brandon Morelli <[email protected]>

docs/en/observability/index.asciidoc

@@ -22,6 +22,8 @@ include::quickstarts/monitor-hosts-with-elastic-agent.asciidoc[leveloffset=+2]
 
 include::quickstarts/monitor-k8s-logs-metrics.asciidoc[leveloffset=+2]
 
+include::quickstarts/collect-data-with-aws-firehose.asciidoc[leveloffset=+2]
+
 include::splunk-get-started.asciidoc[leveloffset=+2]
 
 // Group: Application and service monitoring

docs/en/observability/quickstarts/collect-data-with-aws-firehose.asciidoc

@@ -0,0 +1,141 @@
+[[collect-data-with-aws-firehose]]
+= Quickstart: Collect data with AWS Firehose
+
+preview::[]
+
+In this quickstart guide, you'll learn how to use AWS Firehose to send logs and metrics to Elastic.
+
+The AWS Firehose streams are created using a CloudFormation template, which can collect all available CloudWatch logs and metrics for your AWS account.
+
+This approach requires minimal configuration as the CloudFormation template creates a Firehose stream, enables CloudWatch metrics collection across all namespaces, and sets up an account-level subscription filter for CloudWatch log groups to send logs to Elastic via Firehose.
+You can use an AWS CLI command or upload the template to the AWS CloudFormation portal to customize the following parameter values:
+
+[%collapsible]
+.Required Input Parameters
+====
+* `ElasticEndpointURL`: Elastic endpoint URL.
+* `ElasticAPIKey`: Elastic API Key.
+====
+
+[%collapsible]
+.Optional Input Parameters
+====
+* `HttpBufferInterval`: The Kinesis Firehose HTTP buffer interval, in seconds. Default is `60`.
+* `HttpBufferSize`: The Kinesis Firehose HTTP buffer size, in MiB. Default is `1`.
+* `S3BackupMode`: Source record backup in Amazon S3, failed data only or all data. Default is `FailedDataOnly`.
+* `S3BufferInterval`: The Kinesis Firehose S3 buffer interval, in seconds. Default is `300`.
+* `S3BufferSize`: The Kinesis Firehose S3 buffer size, in MiB. Default is `5`.
+* `S3BackupBucketARN`: By default, an S3 bucket for backup will be created. You can override this behaviour by providing an ARN of an existing S3 bucket that ensures the data can be recovered if record processing transformation does not produce the desired results.
+* `Attributes`: List of attribute name-value pairs for HTTP endpoint separated by commas. For example "name1=value1,name2=value2".
+====
+
+[%collapsible]
+.Optional Input Parameters Specific for Metrics
+====
+* `EnableCloudWatchMetrics`: Enable CloudWatch Metrics collection. Default is `true`. When CloudWatch metrics collection is enabled, by default a metric stream will be created with metrics from all namespaces.
+* `FirehoseStreamNameForMetrics`: Name for Amazon Data Firehose Stream for collecting CloudWatch metrics. Default is `elastic-firehose-metrics`.
+* `IncludeOrExclude`: Select the metrics you want to stream. You can include or exclude specific namespaces and metrics. If no filter namespace is given, then default to all namespaces. Default is `Include`.
+* `MetricNameFilters`: Comma-delimited list of namespace-metric names pairs to use for filtering metrics from the stream. If no metric name filter is given, then default to all namespaces and all metrics. For example "AWS/EC2:CPUUtilization|NetworkIn|NetworkOut,AWS/RDS,AWS/S3:AllRequests".
+* `IncludeLinkedAccountsMetrics`: If you are creating a metric stream in a monitoring account, specify `true` to include metrics from source accounts that are linked to this monitoring account, in the metric stream. Default is `false`.
+* `Tags`: Comma-delimited list of tags to apply to the metric stream. For example "org:eng,project:firehose".
+====
+
+[%collapsible]
+.Optional Input Parameters Specific for Logs
+====
+* `EnableCloudWatchLogs`: Enable CloudWatch Logs collection. Default is `true`. When CloudWatch logs collection is enabled, an account-level subscription filter policy is created for all CloudWatch log groups (except the log groups created for Firehose logs).
+* `FirehoseStreamNameForLogs`: Name for Amazon Data Firehose Stream for collecting CloudWatch logs. Default is `elastic-firehose-logs`.
+====
+
+IMPORTANT: Some AWS services need additional manual configuration to properly ingest logs and metrics. For more information, check the 
+link:https://www.elastic.co/docs/current/integrations/aws[AWS integration] documentation.
+
+Data collection with AWS Firehose is supported on ESS deployments in AWS, Azure and GCP. 
+
+[discrete]
+== Prerequisites
+
+* A deployment using our hosted {ess} on {ess-trial}[{ecloud}]. The deployment includes an {es} cluster for storing and searching your data, and {kib} for visualizing and managing your data.
+* A user with the `superuser` {ref}/built-in-roles.html[built-in role] or the privileges required to onboard data.
++ 
+[%collapsible]
+.Expand to view required privileges
+====
+* {ref}/security-privileges.html#privileges-list-cluster[**Cluster**]: `['monitor', 'manage_own_api_key']`
+* {ref}/security-privileges.html#privileges-list-indices[**Index**]: `{ names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }`
+* {kibana-ref}/kibana-privileges.html[**Kibana**]: `{ spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }`
+====
+* An active AWS account and the necessary permissions to create delivery streams.
+
+NOTE: The default CloudFormation stack is created in the AWS region selected for the user's account. This region can be modified either through the AWS Console interface or by specifying a `--region` parameter in the AWS CLI command when creating the stack.
+
+[discrete]
+== Limitations
+
+The AWS Firehose receiver has the following limitations:
+
+* It does not support AWS PrivateLink.
+* It is not available for on-premise Elastic Stack deployments. 
+* The CloudFormation template detects and ingests logs and metrics within a single AWS region only. 
+
+The following table shows the type of data ingested by the supported AWS services:
+
+|===
+| AWS Service | Data type 
+
+| VPC Flow Logs |Logs 
+| API Gateway|Logs, Metrics 
+| CloudTrail | Logs
+| Network Firewall | Logs, Metrics
+| Route53 | Logs
+| WAF | Logs
+| DynamoDB | Metrics
+| EBS | Metrics
+| EC2 | Metrics
+| ECS | Metrics
+| ELB | Metrics
+| EMR | Metrics
+| MSK | Metrics
+| Kinesis Data Stream | Metrics
+| Lambda | Metrics
+| NAT Gateway | Metrics
+| RDS | Metrics
+| S3 | Metrics
+| SNS | Metrics
+| SQS | Metrics
+| Transit Gateway | Metrics
+| AWS Usage | Metrics
+| VPN | Metrics
+| Uncategorized Firehose Logs | Logs
+
+|===
+
+[discrete]
+== Collect your data
+
+. In {kib}, go to **Observability** and click **Add Data**.
+
+. Select **Cloud**, **AWS**, and then select **AWS Firehose**.
++
+[role="screenshot"]
+image::images/quickstart-aws-firehose-entry-point.png[AWS Firehose entry point]
+
+. Click **Create Firehose Stream in AWS** to create a CloudFormation stack from the CloudFormation template.
+
+. Go back to the **Add Observability Data** page.
+
+[discrete]
+== Visualize your data
+
+After installation is complete and all relevant data is flowing into Elastic,
+the **Visualize your data** section allows you to access the different dashboards for the various services.
+
+[role="screenshot"]
+image::images/quickstart-aws-firehose-dashboards.png[AWS Firehose dashboards]
+
+Here is an example of the VPC Flow logs dashboard:
+
+[role="screenshot"]
+image::images/quickstart-aws-firehose-vpc-flow.png[AWS Firehose VPC flow]
+
+Refer to <<observability-introduction>> for a description of other useful features.