[REQUEST]: Add "Elastic Agent to Kafka to Kafka ES Sink to Elasticsearch" architecture

[a03nikki]

Description

We have Elastic Docs ›Elastic Ingest Reference Architectures [8.16] ›Elastic Agent to Elasticsearch with Kafka as middleware message queue Elastic Agent to Logstash to Kafka to Kafka ES Sink to Elasticsearch: Kafka as middleware message queue at https://www.elastic.co/guide/en/ingest/current/agent-kafka-essink.html. However, Logstash is not necessary between Elastic Agent as it can ship directly to Kafka. Please add this to this to the documentation.

Also please add the configuration for Kafka ES Sink that will pass through the collected data types (e.g logs, metrics, etc.) to the correct data stream and ingest pipelines. This is a critical piece of information for understanding how to set this all up.

Also, could add the link to the Sink documentation on https://www.elastic.co/guide/en/ingest/current/agent-kafka-essink.html page as well.

Resources

• https://www.elastic.co/guide/en/ingest/current/agent-kafka-essink.html
• https://www.elastic.co/blog/index-data-elasticsearch-confluent-cloud
• https://www.elastic.co/blog/elastic-agent-output-kafka-data-collection-streaming
• https://www.elastic.co/blog/generative-ai-applications-confluent-cloud-apache-flink-elasticsearch
• https://docs.confluent.io/cloud/current/connectors/cc-elasticsearch-service-sink.html
• https://docs.confluent.io/cloud/current/connectors/limits.html#elasticsearch-service-sink-limits
• https://www.elastic.co/blog/elastic-agent-send-process-any-data-anywhere
• https://www.elastic.co/guide/en/fleet/current/kafka-output-settings.html

Collaboration

Other (please describe below in the Point of contact section).

Point of contact.

Main contact: @

Stakeholders:

• @a03nikki (I can help review the documentation as I have a user who is looking for this specifically)

[kilfoyle]

@karenzone Just putting this on your radar since this affects the ingest architectures.