From 87299c3c534a3b0cfa2db10513bed498d2c0e25d Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Mon, 7 Oct 2024 19:42:01 -0400 Subject: [PATCH 1/9] First take on a comprehensive ingest guide --- docs/en/ingest-guide/index.asciidoc | 18 ++++ docs/en/ingest-guide/ingest-faq.asciidoc | 72 ++++++++++++++ docs/en/ingest-guide/ingest-intro.asciidoc | 42 +++++++++ .../en/ingest-guide/ingest-solutions.asciidoc | 44 +++++++++ docs/en/ingest-guide/ingest-static.asciidoc | 32 +++++++ .../ingest-guide/ingest-timestamped.asciidoc | 33 +++++++ docs/en/ingest-guide/ingest-tools.asciidoc | 93 +++++++++++++++++++ 7 files changed, 334 insertions(+) create mode 100644 docs/en/ingest-guide/index.asciidoc create mode 100644 docs/en/ingest-guide/ingest-faq.asciidoc create mode 100644 docs/en/ingest-guide/ingest-intro.asciidoc create mode 100644 docs/en/ingest-guide/ingest-solutions.asciidoc create mode 100644 docs/en/ingest-guide/ingest-static.asciidoc create mode 100644 docs/en/ingest-guide/ingest-timestamped.asciidoc create mode 100644 docs/en/ingest-guide/ingest-tools.asciidoc diff --git a/docs/en/ingest-guide/index.asciidoc b/docs/en/ingest-guide/index.asciidoc new file mode 100644 index 000000000..cf66302b2 --- /dev/null +++ b/docs/en/ingest-guide/index.asciidoc @@ -0,0 +1,18 @@ +include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[] +include::{docs-root}/shared/attributes.asciidoc[] + +:doctype: book + +[[ingest-guide]] += Elastic Ingest Guide + +include::ingest-intro.asciidoc[] +include::ingest-tools.asciidoc[] +include::ingest-static.asciidoc[] +include::ingest-timestamped.asciidoc[] +include::ingest-solutions.asciidoc[] +include::ingest-faq.asciidoc[] + +//include:: Prereqs (for using data after ingest) +//include:: Migration for ingest +//include:: Troubleshooting diff --git a/docs/en/ingest-guide/ingest-faq.asciidoc b/docs/en/ingest-guide/ingest-faq.asciidoc new file mode 100644 index 000000000..df515e626 --- /dev/null +++ b/docs/en/ingest-guide/ingest-faq.asciidoc @@ -0,0 +1,72 @@ +[[ingest-faq]] +== Frequently Asked Questions + +Q: What Elastic products and tools are available for ingesting data into Elasticsearch. + +Q: What's the best option for ingesting data? + +Q: What's the role of Logstash `filter-elastic-integration`? + + +// Temporary parking lot to capture outstanding questions and notes. + + +Also cover (here or in general outline): + +- https://www.elastic.co/guide/en/kibana/master/connect-to-elasticsearch.html#_add_sample_data[Sample data] +- OTel +- Beats +- Use case: GeoIP +- Airgapped +- Place for table, also adding use case + products (Exp: Logstash for multi-tenant) +- Role of LS in general content use cases + + + +[discrete] +=== Questions to answer: + +* Messaging for data sources that don't have an integration + - We're deemphasizing beats in preparation for deprecation + - We're not quite there with OTel yet + * How should we handle this in the near term? + Probably doesn't make sense to either ignore or jump them straight to Logstash + +* Should we mention Fleet and Stand-alone agent? +** If so, when, where, and how? +* How does this relate to Ingest Architectures +* Enrichment for general content + +* How to message current vs. desired state. + Especially Beats and OTel. +* HOW TO MESSAGE OTel - Current state. Future state. +* Consistent use of terminology vs. matching users' vocabulary (keywords) + +[discrete] +==== Random + +* DocsV3 - need for a sheltered space to develop new content +** Related: https://github.com/elastic/docsmobile/issues/708 +** Need a place to incubate a new doc (previews, links, etc.) +** Refine messaging in private + + +[discrete] +=== Other resources to use, reference, reconcile + +* Timeseries decision tree (needs updates) +* PM's video +** Needs an update. (We might relocate content before updating.) +* PM's product table +** Needs an update.(We might relocate content before updating.) +** Focuses on Agent over integrations. +** Same link text resolves to different locations. +** Proposal: Harvest the good and possibly repurpose the table format. +* Ingest Reference architectures +* Linkable content such as beats? Solutions ingest resources? + +* https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-guides.html[Starting with the Elastic Platform and Solutions] +* https://www.elastic.co/guide/en/observability/current/observability-get-started.html[Get started with Elastic Observability] +* https://www.elastic.co/guide/en/security/current/ingest-data.html[Ingest data into Elastic Security] +* + diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc new file mode 100644 index 000000000..9d4d2615a --- /dev/null +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -0,0 +1,42 @@ +[discrete] +[[ingest-intro]] +== Ingesting data into {es} + +Bring your data! +Whether you call it _adding_, _indexing_, or _ingesting_ data, you have to get +the data into {es} before you can search it, visualize it, and use it for insights. + +Our ingest tools are flexible, and support a wide range of scenarios. +We can help you with everything from popular and straightforward use cases, all +the way to advanced use cases that require additional processing in order to modify or +reshape your data before sending it to {es}. + +You can ingest: + +* **General content** (data without timestamps), such as HTML pages, catalogs, and files +* **Timestamped (time series) data**, such as logs, metrics and traces for Search, Security, Observability, or your own solution + +[ingest-best-approach] +.What's the best approach for ingesting data? +**** +The best choice for ingesting data is the _simplest option_ that _meets your needs_ and _satisfies your use case_. + +**General content**. Choose the ingest tool that aligns with your data source. + +* To index **documents** directly into {es}, use the {es} document APIs. +* To send **application data** directly to {es}, use an Elastic language client. +* To index **web page content**, use the Elastic web crawler. +* To sync **data from third-party sources**, use connectors. +* To index **single files** for testing, use the Kibana file uploader. + +If you would like to play around before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. + +**Timestamped data**. Start with {fleet-guide}[Elastic Agent] and one of the hundreds of {integrations-docs}[Elastic integrations] that are available. +Integrations are available for many popular platforms and services, and are a good place to start. +Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations. +If you don't find an integration for your data source, or if you need additional processing, we still have you covered. +**** + + + + diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc new file mode 100644 index 000000000..3fc3a7ced --- /dev/null +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -0,0 +1,44 @@ +[[ingest-for-solutions]] +== Ingesting data for Elastic solutions + + + +[discrete] +[[ingest-for-obs]] +=== Observability +Monitor and gain insights + +* Logs +* Metrics +* Application traces + +Add links to targeted ingest resources in Observability docs + +[discrete] +[[ingest-for-security]] +=== Security +Detect and respond to threats + +* Logs +* Metrics +* SIEM +* Endpoint +* Files + + + +https://www.elastic.co/guide/en/security/current/ingest-data.html[Ingest data into Elastic Security]: + +* Elastic Agent + integrations (spotlight Defend integration) +* Beats +* Elastic Agent from Splunk +* Third-party collectors + ECS + + +[discrete] +[[ingest-for-search]] +== Search + +* https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html[Adding data with APIs] +* https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html#additional-capabilities-beats-and-agent[Elasticsearch ingest pipelines] + diff --git a/docs/en/ingest-guide/ingest-static.asciidoc b/docs/en/ingest-guide/ingest-static.asciidoc new file mode 100644 index 000000000..b25c313d0 --- /dev/null +++ b/docs/en/ingest-guide/ingest-static.asciidoc @@ -0,0 +1,32 @@ +[[intro-general]] +== Ingesting general content + +Describe general content (non-timestamped)and give examples. + +[discrete] +=== Basic use cases + +* {es} document APIs for documents. +* Elastic language clients for application data. +* Elastic web crawler for web page content. +* Connectors for data from third-party sources, such as Slack, etc. +* Kibana file uploader for individual files. +* LOGSTASH??? +** ToDO: Check out Logstash enterprisesearch-integration + +* To index **documents** directly into {es}, use the {es} document APIs. +* To send **application data** directly to {es}, use an Elastic language client. +* To index **web page content**, use the Elastic web crawler. +* To sync **data from third-party sources**, use connectors. +* To index **single files** for testing, use the Kibana file uploader. + +[discrete] +=== Advanced use cases: Data enrichment and transformation + +Tools for enriching ingested data: + +- Logstash - GEOIP enrichment. Other examples? +** Use enterprisesearch input -> Filter(s) -> ES or enterprisesearch output +- What else? + + diff --git a/docs/en/ingest-guide/ingest-timestamped.asciidoc b/docs/en/ingest-guide/ingest-timestamped.asciidoc new file mode 100644 index 000000000..8d7ccbd86 --- /dev/null +++ b/docs/en/ingest-guide/ingest-timestamped.asciidoc @@ -0,0 +1,33 @@ +[[intro-timeseries]] +== Ingesting timeseries data + +Timestamped data: +The preferred way to index timestamped data is to use Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, and forward data from remote services or hardware. Each Elastic Agent based integration includes default ingestion rules, dashboards, and visualizations to start analyzing your data right away. Fleet Management enables you to centrally manage all of your deployed Elastic Agents from Kibana. + +If no Elastic Agent integration is available for your data source, use Beats to collect your data. Beats are data shippers designed to collect and ship a particular type of data from a server. You install a separate Beat for each type of data to collect. Modules that provide default configurations, Elasticsearch ingest pipeline definitions, and Kibana dashboards are available for some Beats, such as Filebeat and Metricbeat. No Fleet management capabilities are provided for Beats. + +If neither Elastic Agent or Beats supports your data source, use Logstash. Logstash is an open source data collection engine with real-time pipelining capabilities that supports a wide variety of data sources. You might also use Logstash to persist incoming data to ensure data is not lost if there’s an ingestion spike, or if you need to send the data to multiple destinations. + +---> Basic diagram + +[discrete] +=== Basic use case: Integrations to ES + +Reiterate Integrations as basic ingest use case + +ToDo: evaluate terminology (basic???) + + +[discrete] +=== Advanced use case: Integration to Logstash to ES + +Highlight logstash-filter-elastic_agent capabilities + + +[discrete] +=== Other advanced use cases (from decision tree) + +* Agent + Agent processors??? +* Agent + Runtime fields??? + + diff --git a/docs/en/ingest-guide/ingest-tools.asciidoc b/docs/en/ingest-guide/ingest-tools.asciidoc new file mode 100644 index 000000000..dd1c13a78 --- /dev/null +++ b/docs/en/ingest-guide/ingest-tools.asciidoc @@ -0,0 +1,93 @@ +[[ingest-tools]] +== Tools for ingesting data + +// Iterative messaging as our recommended strategy morphs. +// This section is the summary. "Here's the story _now_." +// Hint at upcoming changes, but do it cautiously and responsibly. +// Modular and co-located to make additions/updates/deprecations easier as our story matures. + +Elastic and others offer tools to help you get your data from the original data source into {es}. +Some tools are best suited for certain data sources, and others are multi-purpose. + +Elastic agent and Elastic integrations:: +You can install a single Elastic Agent and collect a variety of data types from a single host computer. +You can manage all of your agents and policies with the Fleet UI in {kib}. ++ +Use Elastic Agent with one of hundreds of Elastic integrations to simplify collecting, transforming, and visualizing data. +Integrations include default ingestion rules, dashboards, and visualizations to start analyzing your data right away. +Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations that can reduce your time to value. ++ +Elastic Agent is the best approach for collecting timestamped data for most data sources and use cases. +If you want to use the features of Elastic Agent but need additional processing, consider Agent processors or Logstash. +//ToDo: Add info on agent processors, Logstash inputs/filters/output, and Logstash integration filter. ++ +**Beats.** Beats are Elastic's original lightweight data shippers, and their capabilities live on in Elastic Agent. +When you use Elastic Agent, you're getting core Beats functionality and more added features. +Beats requires that you install a separate Beat for each type of data you want to collect. +A single Elastic Agent installed on a host can collect multiple types of data. ++ +Best practice: Use Elastic Agent whenever possible. +If your data source is not yet supported by Elastic Agent, use Beats +Check out Beats and Agent https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html#additional-capabilities-beats-and-agent[capabilities comparison] for more info. + +OpenTelemetry (OTel) collectors:: +link:https://opentelemetry.io/docs[OpenTelemetry] is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data. +Elastic is a member of the Cloud Native Computing Foundation (CNCF) and active contributor to the OpenTelemetry project. ++ +In addition to supporting upstream OTel development, Elastic provides link:https://github.com/elastic/opentelemetry[Elastic Distributions of OpenTelemetry], specifically designed to work with Elastic Observability. + + +Logstash:: +{ls} is an open source data collection engine that you can use to extend Elastic integrations. +It supports a wide variety of data sources, and can dynamically unify data from various sources and normalize the data into destinations of your choice. ++ +{ls} can collect data using a variety of {ls} input plugins, enrich and transform the data with {ls} filter plugins, and output the data to {es} and other destinations using the {ls} output plugins. + +You can use Logstash to extend Elastic Agent for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent. + +* {ls} input for when no integration is available +* {ls} integrations filter for advanced processing + +TIP: + +If an integration is available for your datasource, start with Elastic Agent + integration. + +Use Logstash if there's no integration for your data source or for advanced processing: + +Use {ls} when: + +* no integration (use Logstash input) +* an Elastic integration exists, but you need advanced processing between the Elastic integration and {es}: + +Advanced use cases solved by {ls}: + +* {ls} for [data enrichment]https://www.elastic.co/guide/en/ingest/current/ls-enrich.html before sending data to {es} +* [{ls} Persistent Queue (PQ) for buffering]https://www.elastic.co/guide/en/ingest/current/lspq.html +* [{ls} as a proxy]https://www.elastic.co/guide/en/ingest/current/ls-networkbridge.html when there are network restrictions that prevent connections between Elastic Agent and {es} +* [{ls} for routing data to multiple {es} clusters and additional destinations]https://www.elastic.co/guide/en/ingest/current/ls-multi.html +* data persistence + + +Language clients:: +Use an Elastic language client to send **application data**, such as from NodeJS or Python, directly to {es}. +//ToDo: Figure out trademark considerations. + +APIs:: +Use the {es} document APIs to index **documents** directly into {es}. + +File uploader:: +Use the {kib} file uploader to index **single files** into {es}. +This tool can be helpful for testing with small numbers of files. + +Web crawler:: +Use the Elastic web crawler to index **web page content**. + +Connectors:: +Use connectors to index **data from third-party sources**, such as Amazon S3, GMail, Outlook, and Salesforce. +//ToDo: Figure out trademark considerations. + +Elasticsearch ingest pipelines:: +Should we discuss native ingest pipelines? + +Elastic serverless forwarder:: +The Elastic Serverless Forwarder is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to {es}. From aa263e9f2efa0cb29011b4e3e4d7e79a1929b10e Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Tue, 8 Oct 2024 17:04:22 -0400 Subject: [PATCH 2/9] More stuff --- docs/en/ingest-arch/index.asciidoc | 2 + docs/en/ingest-guide/ingest-faq.asciidoc | 7 +- .../en/ingest-guide/ingest-solutions.asciidoc | 5 +- docs/en/ingest-guide/ingest-static.asciidoc | 7 ++ .../ingest-guide/ingest-timestamped.asciidoc | 71 +++++++++++++++++ docs/en/ingest-guide/ingest-tools.asciidoc | 76 ++++++++++--------- 6 files changed, 131 insertions(+), 37 deletions(-) diff --git a/docs/en/ingest-arch/index.asciidoc b/docs/en/ingest-arch/index.asciidoc index 97ee6acb8..118f5dfc0 100644 --- a/docs/en/ingest-arch/index.asciidoc +++ b/docs/en/ingest-arch/index.asciidoc @@ -18,6 +18,8 @@ include::8-ls-input.asciidoc[] include::99-airgapped.asciidoc[] +include::../ingest-guide/index.asciidoc[] + // === Next set of architectures // include::3-schemamod.asciidoc[] // include::6b-filebeat-es.asciidoc[] diff --git a/docs/en/ingest-guide/ingest-faq.asciidoc b/docs/en/ingest-guide/ingest-faq.asciidoc index df515e626..dea6534b2 100644 --- a/docs/en/ingest-guide/ingest-faq.asciidoc +++ b/docs/en/ingest-guide/ingest-faq.asciidoc @@ -8,7 +8,12 @@ Q: What's the best option for ingesting data? Q: What's the role of Logstash `filter-elastic-integration`? -// Temporary parking lot to capture outstanding questions and notes. + +.WORK IN PROGRESS +**** +Temporary parking lot to capture outstanding questions and notes. +**** + Also cover (here or in general outline): diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index 3fc3a7ced..c51f44781 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -1,7 +1,10 @@ [[ingest-for-solutions]] == Ingesting data for Elastic solutions - +.WORK IN PROGRESS +**** +For MVP: Add descriptions and links to existing solutions content. +**** [discrete] [[ingest-for-obs]] diff --git a/docs/en/ingest-guide/ingest-static.asciidoc b/docs/en/ingest-guide/ingest-static.asciidoc index b25c313d0..162bd243c 100644 --- a/docs/en/ingest-guide/ingest-static.asciidoc +++ b/docs/en/ingest-guide/ingest-static.asciidoc @@ -3,6 +3,13 @@ Describe general content (non-timestamped)and give examples. +.WORK IN PROGRESS +**** +Progressive disclosure: Start with basic use cases and work up to advanced processing + +Possibly repurpose and use ingest decision tree with Beats removed? +**** + [discrete] === Basic use cases diff --git a/docs/en/ingest-guide/ingest-timestamped.asciidoc b/docs/en/ingest-guide/ingest-timestamped.asciidoc index 8d7ccbd86..a73fe30c9 100644 --- a/docs/en/ingest-guide/ingest-timestamped.asciidoc +++ b/docs/en/ingest-guide/ingest-timestamped.asciidoc @@ -1,6 +1,13 @@ [[intro-timeseries]] == Ingesting timeseries data +.WORK IN PROGRESS +**** +Progressive disclosure: Start with basic use cases and work up to advanced processing + +Possibly repurpose and use ingest decision tree with Beats removed? +**** + Timestamped data: The preferred way to index timestamped data is to use Elastic Agent. Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, and forward data from remote services or hardware. Each Elastic Agent based integration includes default ingestion rules, dashboards, and visualizations to start analyzing your data right away. Fleet Management enables you to centrally manage all of your deployed Elastic Agents from Kibana. @@ -31,3 +38,67 @@ Highlight logstash-filter-elastic_agent capabilities * Agent + Runtime fields??? + +// CONTENT LIFTED FROM former `TOOLS` topic + + +[discrete] +=== Elastic agent and Elastic integrations +The best choice for ingesting data is the _simplest option_ that _meets your needs_ and _satisfies your use case_. +For many popular ingest scenarios, the best option is Elastic agent and Elastic integrations. + +* Elastic agent installed on the endpoints where you want to collect data. +Elastic Agent collects the data from one or more endpoints, and forwards the data to the service or location where is used. +* An Elastic integration to receive that data from agents + +TIP: Start here! +Elastic Agent for data collection paired with Elastic integrations is the best ingest option for most use cases. + + +[discrete] +=== OTel +Coming on strong. Where are we now, and cautiously explain where we're going in the near term. + +Open Telemetry is a leader for collecting Observability data + +Elastic is a supporting member. +We're contributing to the OTel project, and are using elastic/opentelemetry for specialized development not applicable to upstream. + +* https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html + +Contributing to upstream and doing our on for work specific to Elastic +* https://github.com/open-telemetry/opentelemetry-collector-contrib +* https://github.com/elastic/opentelemetry + +[discrete] +=== Logstash + +{ls} is an open source data collection engine with real-time pipelining capabilities. +It supports a wide variety of data sources, and can dynamically unify data from disparate sources and normalize the data into destinations of your choice. + +{ls} can collect data using a variety of {ls} input plugins, enrich and transform the data with {ls} filter plugins, and output the data to {es} and other destinations using the {ls} output plugins. + +You can use Logstash to extend Beats for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent. + +* {ls} input for when no integration is available +* {ls} integrations filter for advanced processing + +TIP: + +If an integration is available for your datasource, start with Elastic Agent + integration. + +Use Logstash if there's no integration for your data source or for advanced processing: + +Use {ls} when: + +* no integration (use Logstash input) +* an Elastic integration exists, but you need advanced processing between the Elastic integration and {es}: + +Advanced use cases solved by {ls}: + +* {ls} for https://www.elastic.co/guide/en/ingest/current/ls-enrich.html[data enrichment] before sending data to {es} +* https://www.elastic.co/guide/en/ingest/current/lspq.html[{ls} Persistent Queue (PQ) for buffering] +* https://www.elastic.co/guide/en/ingest/current/ls-networkbridge.html[{ls} as a proxy] when there are network restrictions that prevent connections between Elastic Agent and {es} +* https://www.elastic.co/guide/en/ingest/current/ls-multi.html[{ls} for routing data to multiple {es} clusters and additional destinations] +* https://www.elastic.co/guide/en/ingest/current/agent-proxy.html[{ls} as a proxy] + diff --git a/docs/en/ingest-guide/ingest-tools.asciidoc b/docs/en/ingest-guide/ingest-tools.asciidoc index dd1c13a78..f3d6d8c5a 100644 --- a/docs/en/ingest-guide/ingest-tools.asciidoc +++ b/docs/en/ingest-guide/ingest-tools.asciidoc @@ -18,10 +18,10 @@ Integrations include default ingestion rules, dashboards, and visualizations to Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations that can reduce your time to value. + Elastic Agent is the best approach for collecting timestamped data for most data sources and use cases. -If you want to use the features of Elastic Agent but need additional processing, consider Agent processors or Logstash. +If you want to use the features of Elastic Agent but need additional processing, consider adding Agent processors or Logstash. //ToDo: Add info on agent processors, Logstash inputs/filters/output, and Logstash integration filter. + -**Beats.** Beats are Elastic's original lightweight data shippers, and their capabilities live on in Elastic Agent. +**Beats.** Beats are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. When you use Elastic Agent, you're getting core Beats functionality and more added features. Beats requires that you install a separate Beat for each type of data you want to collect. A single Elastic Agent installed on a host can collect multiple types of data. @@ -36,40 +36,35 @@ Elastic is a member of the Cloud Native Computing Foundation (CNCF) and active c + In addition to supporting upstream OTel development, Elastic provides link:https://github.com/elastic/opentelemetry[Elastic Distributions of OpenTelemetry], specifically designed to work with Elastic Observability. - -Logstash:: -{ls} is an open source data collection engine that you can use to extend Elastic integrations. -It supports a wide variety of data sources, and can dynamically unify data from various sources and normalize the data into destinations of your choice. +Logstash:: +{ls} is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. +{ls} can _collect data_ from a wide variety of data sources with {ls} link:{logstash-ref}/input-plugins.html[input +plugins], _enrich and transform_ the data with {ls} link:{logstash-ref}/filter-plugins.html[filter plugins], and _output_ the +data to {es} and other destinations with the {ls} link:{logstash-ref}/output-plugins.html[output plugins]. +Many users never need to use {ls}, but it's there if you need it. + -{ls} can collect data using a variety of {ls} input plugins, enrich and transform the data with {ls} filter plugins, and output the data to {es} and other destinations using the {ls} output plugins. - -You can use Logstash to extend Elastic Agent for advanced use cases, such as data routed to multiple destinations or when you need to make your data persistent. - -* {ls} input for when no integration is available -* {ls} integrations filter for advanced processing - -TIP: - -If an integration is available for your datasource, start with Elastic Agent + integration. - -Use Logstash if there's no integration for your data source or for advanced processing: - -Use {ls} when: - -* no integration (use Logstash input) -* an Elastic integration exists, but you need advanced processing between the Elastic integration and {es}: - -Advanced use cases solved by {ls}: - -* {ls} for [data enrichment]https://www.elastic.co/guide/en/ingest/current/ls-enrich.html before sending data to {es} -* [{ls} Persistent Queue (PQ) for buffering]https://www.elastic.co/guide/en/ingest/current/lspq.html -* [{ls} as a proxy]https://www.elastic.co/guide/en/ingest/current/ls-networkbridge.html when there are network restrictions that prevent connections between Elastic Agent and {es} -* [{ls} for routing data to multiple {es} clusters and additional destinations]https://www.elastic.co/guide/en/ingest/current/ls-multi.html -* data persistence +One of the most common {ls} use cases is link:{logstash-ref}/ea-integrations.html[extending Elastic integrations]. +You can take advantage of the extensive, built-in capabilities of Elastic Integrations, and +then use {ls} for additional data processing before sending the data on to {es}. ++ +{ls} can help with advanced use cases such as the need for additional +link:https://www.elastic.co/guide/en/ingest/current/ls-enrich.html[data +enrichment], +link:https://www.elastic.co/guide/en/ingest/current/lspq.html[persistence or +buffering], +link:https://www.elastic.co/guide/en/ingest/current/ls-networkbridge.html[proxying] +to bridge network connections, or the ability to route data to +link:https://www.elastic.co/guide/en/ingest/current/ls-multi.html[multiple +destinations]. +Elasticsearch ingest pipelines:: +{es} link:{ref}/ingest.html[ingest pipelines] let you complete common transformations on your data before the data is indexed. +//ToDo: Explain when to use Language clients:: -Use an Elastic language client to send **application data**, such as from NodeJS or Python, directly to {es}. +link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[Elastic +language clients] help you send **application data**, such as from NodeJS or Python, +directly to {es} for search and analysis. //ToDo: Figure out trademark considerations. APIs:: @@ -86,8 +81,19 @@ Connectors:: Use connectors to index **data from third-party sources**, such as Amazon S3, GMail, Outlook, and Salesforce. //ToDo: Figure out trademark considerations. -Elasticsearch ingest pipelines:: -Should we discuss native ingest pipelines? - Elastic serverless forwarder:: The Elastic Serverless Forwarder is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to {es}. + +[discrete] +[[ingest-addl-proc]] +== Tools for additional processing + +* link:[Agent processors] for sanitizing or enriching raw data at the source +* {es} link:{ref}/[ingest pipelines] for enriching incoming data +// ToDo: Decide if ingest pipelines should be included for additional processing. +// Decision tree lists ECS-ifying or normalizing field data, but that should be handled by Agent. +// Does that leave enriching data as the only enrichment use case? +// Does the Logstash elastic_integration filter remove that one, also? +* {es} link:{ref}/runtime.html[runtime fields] +* {ls} link:[elastic_integration filter] +* {ls} link:[filter plugins] From f9e41253ca51c24fe43207d41085e2b289537dd7 Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Mon, 21 Oct 2024 19:21:55 -0400 Subject: [PATCH 3/9] Proposal for MVP content --- docs/en/ingest-guide/index.asciidoc | 8 ++++---- docs/en/ingest-guide/ingest-solutions.asciidoc | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/ingest-guide/index.asciidoc b/docs/en/ingest-guide/index.asciidoc index cf66302b2..9618e8cc8 100644 --- a/docs/en/ingest-guide/index.asciidoc +++ b/docs/en/ingest-guide/index.asciidoc @@ -4,14 +4,14 @@ include::{docs-root}/shared/attributes.asciidoc[] :doctype: book [[ingest-guide]] -= Elastic Ingest Guide += Elastic Ingest Overview include::ingest-intro.asciidoc[] include::ingest-tools.asciidoc[] -include::ingest-static.asciidoc[] -include::ingest-timestamped.asciidoc[] +//include::ingest-static.asciidoc[] +//include::ingest-timestamped.asciidoc[] include::ingest-solutions.asciidoc[] -include::ingest-faq.asciidoc[] +//include::ingest-faq.asciidoc[] //include:: Prereqs (for using data after ingest) //include:: Migration for ingest diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index c51f44781..0cde6fa3b 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -3,7 +3,7 @@ .WORK IN PROGRESS **** -For MVP: Add descriptions and links to existing solutions content. +For MVP: Add descriptions and links to existing solutions content with focus on tutorials where appropriate. **** [discrete] From c5832a92aecc291e345490a0d9db77150cfbd38d Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Fri, 25 Oct 2024 21:16:34 -0400 Subject: [PATCH 4/9] More content and fixes --- docs/en/ingest-guide/ingest-intro.asciidoc | 19 ++-- .../en/ingest-guide/ingest-solutions.asciidoc | 7 +- docs/en/ingest-guide/ingest-tools.asciidoc | 104 ++++++++++-------- 3 files changed, 70 insertions(+), 60 deletions(-) diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc index 9d4d2615a..f3e0de53d 100644 --- a/docs/en/ingest-guide/ingest-intro.asciidoc +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -23,20 +23,17 @@ The best choice for ingesting data is the _simplest option_ that _meets your nee **General content**. Choose the ingest tool that aligns with your data source. -* To index **documents** directly into {es}, use the {es} document APIs. -* To send **application data** directly to {es}, use an Elastic language client. -* To index **web page content**, use the Elastic web crawler. -* To sync **data from third-party sources**, use connectors. -* To index **single files** for testing, use the Kibana file uploader. +* To index **documents** directly into {es}, use the {es} link:{ref}/docs.html[document APIs]. +* To send **application data** directly to {es}, use an link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} +language clients]. +* To index **web page content**, use the Elastic link:https://www.elastic.co/web-crawler[web crawler]. +* To sync **data from third-party sources**, use link:{ref}/es-connectors.html[connectors]. +* To index **single files** for testing, use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader]. -If you would like to play around before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. +If you would like to test things out before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. **Timestamped data**. Start with {fleet-guide}[Elastic Agent] and one of the hundreds of {integrations-docs}[Elastic integrations] that are available. Integrations are available for many popular platforms and services, and are a good place to start. Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations. -If you don't find an integration for your data source, or if you need additional processing, we still have you covered. +If you don't find an integration for your data source, or if you need <>, we still have you covered. **** - - - - diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index 0cde6fa3b..7210507ea 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -15,7 +15,12 @@ Monitor and gain insights * Metrics * Application traces -Add links to targeted ingest resources in Observability docs +Add links to targeted ingest resources in Observability docs. + +* Logs and metrics: Use system integration: https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html#add-system-integration +* Traces and APM: https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html#add-system-integration + * Is this integration listed in https://www.elastic.co/docs/current/integrations + [discrete] [[ingest-for-security]] diff --git a/docs/en/ingest-guide/ingest-tools.asciidoc b/docs/en/ingest-guide/ingest-tools.asciidoc index f3d6d8c5a..1668a67c0 100644 --- a/docs/en/ingest-guide/ingest-tools.asciidoc +++ b/docs/en/ingest-guide/ingest-tools.asciidoc @@ -7,59 +7,57 @@ // Modular and co-located to make additions/updates/deprecations easier as our story matures. Elastic and others offer tools to help you get your data from the original data source into {es}. -Some tools are best suited for certain data sources, and others are multi-purpose. +Some tools are designed for particular data sources, and others are multi-purpose. -Elastic agent and Elastic integrations:: -You can install a single Elastic Agent and collect a variety of data types from a single host computer. -You can manage all of your agents and policies with the Fleet UI in {kib}. +{agent} and Elastic integrations:: +A single link:{fleet-guide}[{agent}] can collect multiple types of data when it is link:{fleet-guide}/elastic-agent-installation.html[installed] on a host computer. +You can use standalone {agent}s and manage them locally on the systems where they are installed, or you can manage all of your agents and policies with the link:{fleet-guide}/manage-agents-in-fleet.html[Fleet UI in {kib}]. + -Use Elastic Agent with one of hundreds of Elastic integrations to simplify collecting, transforming, and visualizing data. +Use {agent} with one of hundreds of link:{integrations-docs}[Elastic integrations] to simplify collecting, transforming, and visualizing data. Integrations include default ingestion rules, dashboards, and visualizations to start analyzing your data right away. Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations that can reduce your time to value. + -Elastic Agent is the best approach for collecting timestamped data for most data sources and use cases. -If you want to use the features of Elastic Agent but need additional processing, consider adding Agent processors or Logstash. -//ToDo: Add info on agent processors, Logstash inputs/filters/output, and Logstash integration filter. +{agent} is the best option for collecting timestamped data for most data sources and use cases. +If your data requires additional processing before going to {es}, you can use {agent} with link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] or link:{logstash-ref}[{ls}] + -**Beats.** Beats are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. +Ready to try link:{fleet-guide}[Elastic Agent]? Check out the link:{fleet-guide}/elastic-agent-installation.html[installation instructions]. ++ +**Beats.** link:{beats-ref}/beats-reference.html}[Beats] are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. When you use Elastic Agent, you're getting core Beats functionality and more added features. -Beats requires that you install a separate Beat for each type of data you want to collect. +Beats require that you install a separate Beat for each type of data you want to collect. A single Elastic Agent installed on a host can collect multiple types of data. + -Best practice: Use Elastic Agent whenever possible. -If your data source is not yet supported by Elastic Agent, use Beats -Check out Beats and Agent https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html#additional-capabilities-beats-and-agent[capabilities comparison] for more info. +**Best practice:** Use link:{fleet-guide}[Elastic Agent] whenever possible. +If your data source is not yet supported by Elastic Agent, use Beats. +Check out {beats} and {agent} link:{fleet-guide}/beats-agent-comparison.html#additional-capabilities-beats-and-agent[comparison] for more info. +When you are ready to upgrade, check out link:{fleet-guide}/migrate-beats-to-agent.html[Migrate from {beats} to {agent}]. OpenTelemetry (OTel) collectors:: link:https://opentelemetry.io/docs[OpenTelemetry] is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data. Elastic is a member of the Cloud Native Computing Foundation (CNCF) and active contributor to the OpenTelemetry project. + -In addition to supporting upstream OTel development, Elastic provides link:https://github.com/elastic/opentelemetry[Elastic Distributions of OpenTelemetry], specifically designed to work with Elastic Observability. +In addition to supporting upstream OTel development, Elastic provides link:https://github.com/elastic/opentelemetry[Elastic Distributions of OpenTelemetry], specifically designed to work with Elastic Observability. +We're also expanding link:{fleet-guide}[{agent}] to use OTel collection. Logstash:: -{ls} is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. +link:{logstash-ref}[{ls}] is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. {ls} can _collect data_ from a wide variety of data sources with {ls} link:{logstash-ref}/input-plugins.html[input plugins], _enrich and transform_ the data with {ls} link:{logstash-ref}/filter-plugins.html[filter plugins], and _output_ the data to {es} and other destinations with the {ls} link:{logstash-ref}/output-plugins.html[output plugins]. -Many users never need to use {ls}, but it's there if you need it. + -One of the most common {ls} use cases is link:{logstash-ref}/ea-integrations.html[extending Elastic integrations]. -You can take advantage of the extensive, built-in capabilities of Elastic Integrations, and -then use {ls} for additional data processing before sending the data on to {es}. +Most users never need to use {ls}, but it's available if you need it for: + -{ls} can help with advanced use cases such as the need for additional -link:https://www.elastic.co/guide/en/ingest/current/ls-enrich.html[data -enrichment], -link:https://www.elastic.co/guide/en/ingest/current/lspq.html[persistence or -buffering], -link:https://www.elastic.co/guide/en/ingest/current/ls-networkbridge.html[proxying] -to bridge network connections, or the ability to route data to -link:https://www.elastic.co/guide/en/ingest/current/ls-multi.html[multiple -destinations]. - -Elasticsearch ingest pipelines:: -{es} link:{ref}/ingest.html[ingest pipelines] let you complete common transformations on your data before the data is indexed. -//ToDo: Explain when to use +* **Data collection** (if an Elastic integration isn't available). +{agent} and Elastic {integrations-docs}/all_integrations[integrations] provide many features out-of-the-box, so be sure to search or browse integrations for your data source. +If you don't find an Elastic integration for your data source, check {ls} for an {logstash-ref}/input-plugins.html[input plugin]. +* **Additional processing.** One of the most common {ls} use cases is link:{logstash-ref}/ea-integrations.html[extending Elastic integrations]. +You can take advantage of the extensive, built-in capabilities of Elastic Agent and Elastic Integrations, and +then use {ls} for additional data processing before sending the data on to {es}. +* **Advanced use cases.** {ls} can help with advanced use cases, such as when you need +link:{ingest-guide}/lspq.html[persistence or buffering], +additional link:{ingest-guide}/ls-enrich.html[data enrichment], +link:{ingest-guide}/ls-networkbridge.html[proxying] to bridge network connections, or the ability to route data to +link:{ingest-guide}/ls-multi.html[multiple destinations]. Language clients:: link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[Elastic @@ -68,32 +66,42 @@ directly to {es} for search and analysis. //ToDo: Figure out trademark considerations. APIs:: -Use the {es} document APIs to index **documents** directly into {es}. +Use the {es} link:{ref}/docs.html[document APIs] to index **documents** directly into {es}. File uploader:: -Use the {kib} file uploader to index **single files** into {es}. +Use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader] to index **single files** into {es}. This tool can be helpful for testing with small numbers of files. Web crawler:: -Use the Elastic web crawler to index **web page content**. +Use the Elastic link:https://www.elastic.co/web-crawler[web crawler] to index **web page content**. Connectors:: -Use connectors to index **data from third-party sources**, such as Amazon S3, GMail, Outlook, and Salesforce. +Use link:{ref}/es-connectors.html[connectors] to index **data from third-party sources**, such as Amazon S3, GMail, Outlook, and Salesforce. //ToDo: Figure out trademark considerations. Elastic serverless forwarder:: -The Elastic Serverless Forwarder is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to {es}. +The link:https://www.elastic.co/guide/en/esf/current/aws-elastic-serverless-forwarder.html[Elastic Serverless Forwarder] is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to {es}. [discrete] [[ingest-addl-proc]] -== Tools for additional processing - -* link:[Agent processors] for sanitizing or enriching raw data at the source -* {es} link:{ref}/[ingest pipelines] for enriching incoming data -// ToDo: Decide if ingest pipelines should be included for additional processing. -// Decision tree lists ECS-ifying or normalizing field data, but that should be handled by Agent. -// Does that leave enriching data as the only enrichment use case? -// Does the Logstash elastic_integration filter remove that one, also? -* {es} link:{ref}/runtime.html[runtime fields] -* {ls} link:[elastic_integration filter] -* {ls} link:[filter plugins] +== Tools and features for additional processing +You can start with {agent} and Elastic {integrations-docs}[integrations], and still +take advantage of additional processing options if you need them: + +* link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] for sanitizing or enriching raw data at the source. + Use {agent} processors if you need to control what data is sent across the wire, or need to enrich the raw data with information available on the host. +* {es} link:{ref}/[ingest pipelines] for enriching incoming data or normalizing field data before the data is indexed. + {es} ingest pipelines enable you to manipulate the data as it comes in. + This approach helps you avoid adding processing overhead to the hosts from which you're collecting data. + +* {es} link:{ref}/runtime.html[runtime fields] for defining or altering the schema at query time. + You can use runtime fields at query time to start working with your data without needing to understand how it is structured, + add fields to existing documents without reindexing your data, + override the value returned from an indexed field, and/or + define fields for a specific use without modifying the underlying schema. + +* {ls} `elastic_integration filter` for link:{logstash-ref}/ea-integrations.html[extending Elastic integrations], and other {ls} link:[filter plugins] for transforming data before it goes to {es}. + + + + From 066b32736a8f669e83757410b01529f2c9c0d468 Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Mon, 28 Oct 2024 20:34:35 -0400 Subject: [PATCH 5/9] Flesh out solutions sections --- docs/en/ingest-guide/ingest-intro.asciidoc | 18 ++- .../en/ingest-guide/ingest-solutions.asciidoc | 110 +++++++++++++----- docs/en/ingest-guide/ingest-tools.asciidoc | 6 +- 3 files changed, 97 insertions(+), 37 deletions(-) diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc index f3e0de53d..4abdd2db6 100644 --- a/docs/en/ingest-guide/ingest-intro.asciidoc +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -30,10 +30,16 @@ language clients]. * To sync **data from third-party sources**, use link:{ref}/es-connectors.html[connectors]. * To index **single files** for testing, use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader]. -If you would like to test things out before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. - -**Timestamped data**. Start with {fleet-guide}[Elastic Agent] and one of the hundreds of {integrations-docs}[Elastic integrations] that are available. -Integrations are available for many popular platforms and services, and are a good place to start. -Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations. -If you don't find an integration for your data source, or if you need <>, we still have you covered. +If you would like to try things out before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. + +**Timestamped data**. Start with {fleet-guide}[Elastic Agent] and one of the +hundreds of {integrations-docs}[Elastic integrations] that are available. +Integrations are available for many popular platforms and services, and are a +good place to start for ingesting data into Elastic solutions--Observability, +Security, and Search--or your own search application. + +Check out the {integrations-docs}/all_integrations[Integration quick reference] +to search for available integrations. +If you don't find an integration for your data source or if you need +<> to extend the integration, we still have you covered. **** diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index 7210507ea..9f02b704b 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -1,52 +1,106 @@ [[ingest-for-solutions]] == Ingesting data for Elastic solutions -.WORK IN PROGRESS -**** -For MVP: Add descriptions and links to existing solutions content with focus on tutorials where appropriate. +Elastic solutions--Security, Observability, and Search--are loaded with features +and functionality to help you get value and insights from your data. +{fleet-guide}[Elastic Agent] and {integrations-docs}[Elastic integrations] can help. + +When you use integrations with solutions, you have an integrated experience that offers +easier implementation and decreases the time it takes to get insights and value from your data. + +[ingest-process-overview] +.High-level overview +**** +To use {fleet-guide}[Elastic Agent] and {integrations-docs}[Elastic integrations] +with Elastic solutions: + +1. Create an link:https://www.elastic.co/cloud[{ecloud}] deployment for your solution. + If you don't have a {ecloud} account, you can sign up for a link:https://cloud.elastic.co/registration[free trial] get started. +2. Add the {integrations-docs}[Elastic integration] for your data source to the deployment. +3. link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] on the systems whose data you want to collect. **** +NOTE: {serverless-docs}[Elastic serverless] makes using solutions even easier. +Sign up for a link:{serverless-docs}/general/sign-up-trial[free trial], and check it out. + + + +[discrete] +[[ingest-for-search]] +=== Ingesting data for Search + +{es} is the magic behind Search and our other solutions. +The solution gives you more pre-built components to get you up and running quickly for common use cases. + +**Resources** + +* link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] +* link:https://www.elastic.co/integrations/data-integrations?solution=search[Elastic Search for integrations] +* link:{ref}[{es} Guide] +** link:{ref}/docs.html[{es} document APIs] +** link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} language clients] +** link:https://www.elastic.co/web-crawler[Elastic web crawler] +** link:{ref}/es-connectors.html[Elastic connectors] + + [discrete] [[ingest-for-obs]] -=== Observability -Monitor and gain insights +=== Ingesting data for Observability + +With link:https://www.elastic.co/observability[Elastic Observability], you can monitor and gain insights into logs, metrics, and application traces. + +**Resources** -* Logs -* Metrics -* Application traces +* link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] +* link:https://www.elastic.co/integrations/data-integrations?solution=observability[Elastic Observability integrations] -Add links to targeted ingest resources in Observability docs. +**Guides for popular Observability use case** -* Logs and metrics: Use system integration: https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html#add-system-integration -* Traces and APM: https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html#add-system-integration - * Is this integration listed in https://www.elastic.co/docs/current/integrations +* link:{estc-welcome}/getting-started-observability.html[Monitor applications and systems with Elastic Observability] +* link:https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html[Get started with logs and metrics] +** link:https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html#add-system-integration[Step 1: Add the {agent} System integration] +** link:https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html#add-agent-to-fleet[Step 2: Install and run {agent}] + +* link:{serverless-docs}/observability/what-is-observability-serverless[Observability] on link:{serverless-docs}[{serverless-full}]: +** link:{serverless-docs}/observability/quickstarts/monitor-hosts-with-elastic-agent[Monitor hosts with {agent} ({serverless-short})] +** link:{serverless-docs}/observability/quickstarts/k8s-logs-metrics[Monitor your K8s cluster with {agent} ({serverless-short})] [discrete] [[ingest-for-security]] -=== Security -Detect and respond to threats +=== Ingesting data for Security -* Logs -* Metrics -* SIEM -* Endpoint -* Files +You can detect and respond to threats when you use link:https://www.elastic.co/security[Elastic Security] to analyze and take action on your data. +**Resources** +* link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] +* link:https://www.elastic.co/integrations/data-integrations?solution=search[Elastic Security integrations] +* link:{security-guide}/es-overview.html[Elastic Security documentation] -https://www.elastic.co/guide/en/security/current/ingest-data.html[Ingest data into Elastic Security]: +**Guides for popular Security use case** -* Elastic Agent + integrations (spotlight Defend integration) -* Beats -* Elastic Agent from Splunk -* Third-party collectors + ECS +* link:https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-siem-security.html[Use Elastic Security for SIEM] +* link:https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-endpoint-security.html[Protect hosts with endpoint threat intelligence from Elastic Security] [discrete] -[[ingest-for-search]] -== Search +[[ingest-for-custom]] +=== Ingesting data for your own custom search solution + +Elastic solutions can give you a head start for common use cases, but you are not limited. +You can still do your own thing with a custom solution designed by you. + +Bring your ideas and use {es} and the {stack} to store, search, and visualize your data. + +**Resources** + +* link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] +* link:{ref}[{es} Guide] +** link:{ref}/docs.html[{es} document APIs] +** link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} language clients] +** link:https://www.elastic.co/web-crawler[Elastic web crawler] +** link:{ref}/es-connectors.html[Elastic connectors] -* https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html[Adding data with APIs] -* https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html#additional-capabilities-beats-and-agent[Elasticsearch ingest pipelines] +* link:{estc-welcome}/getting-started-general-purpose.html[Tutorial: Get started with vector search and generative AI] diff --git a/docs/en/ingest-guide/ingest-tools.asciidoc b/docs/en/ingest-guide/ingest-tools.asciidoc index 1668a67c0..f6599b909 100644 --- a/docs/en/ingest-guide/ingest-tools.asciidoc +++ b/docs/en/ingest-guide/ingest-tools.asciidoc @@ -20,12 +20,12 @@ Check out the {integrations-docs}/all_integrations[Integration quick reference] {agent} is the best option for collecting timestamped data for most data sources and use cases. If your data requires additional processing before going to {es}, you can use {agent} with link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] or link:{logstash-ref}[{ls}] + -Ready to try link:{fleet-guide}[Elastic Agent]? Check out the link:{fleet-guide}/elastic-agent-installation.html[installation instructions]. +Ready to try link:{fleet-guide}[{agent}]? Check out the link:{fleet-guide}/elastic-agent-installation.html[installation instructions]. + -**Beats.** link:{beats-ref}/beats-reference.html}[Beats] are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. +**Beats.** link:{beats-ref}/beats-reference.html[Beats] are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. When you use Elastic Agent, you're getting core Beats functionality and more added features. Beats require that you install a separate Beat for each type of data you want to collect. -A single Elastic Agent installed on a host can collect multiple types of data. +A single Elastic Agent installed on a host can collect and transport multiple types of data. + **Best practice:** Use link:{fleet-guide}[Elastic Agent] whenever possible. If your data source is not yet supported by Elastic Agent, use Beats. From b7ff238b479e982cb225281a445d425a69f96d32 Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Tue, 29 Oct 2024 17:12:51 -0400 Subject: [PATCH 6/9] More fixes and cleanup --- docs/en/ingest-guide/ingest-intro.asciidoc | 22 +++++++++---- .../en/ingest-guide/ingest-solutions.asciidoc | 17 ++++++---- docs/en/ingest-guide/ingest-tools.asciidoc | 32 +++++++++++-------- 3 files changed, 44 insertions(+), 27 deletions(-) diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc index 4abdd2db6..6b799d1d3 100644 --- a/docs/en/ingest-guide/ingest-intro.asciidoc +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -9,31 +9,37 @@ the data into {es} before you can search it, visualize it, and use it for insigh Our ingest tools are flexible, and support a wide range of scenarios. We can help you with everything from popular and straightforward use cases, all the way to advanced use cases that require additional processing in order to modify or -reshape your data before sending it to {es}. +reshape your data before it goes to {es}. You can ingest: * **General content** (data without timestamps), such as HTML pages, catalogs, and files -* **Timestamped (time series) data**, such as logs, metrics and traces for Search, Security, Observability, or your own solution +* **Timestamped (time series) data**, such as logs, metrics, and traces for Search, Security, Observability, or your own solution [ingest-best-approach] .What's the best approach for ingesting data? **** The best choice for ingesting data is the _simplest option_ that _meets your needs_ and _satisfies your use case_. -**General content**. Choose the ingest tool that aligns with your data source. +**Best practice for general content**. Choose the ingest tool that aligns with your data source. * To index **documents** directly into {es}, use the {es} link:{ref}/docs.html[document APIs]. * To send **application data** directly to {es}, use an link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} -language clients]. +language client]. * To index **web page content**, use the Elastic link:https://www.elastic.co/web-crawler[web crawler]. * To sync **data from third-party sources**, use link:{ref}/es-connectors.html[connectors]. * To index **single files** for testing, use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader]. If you would like to try things out before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. -**Timestamped data**. Start with {fleet-guide}[Elastic Agent] and one of the -hundreds of {integrations-docs}[Elastic integrations] that are available. +**Best practice for timestamped data**. Start with Elastic Agent and an Elastic integration. + + + Usually, the _simplest option_ for ingesting timestamped data is {agent} paired with an Elastic integration. + +* Install {fleet-guide}[Elastic Agent] on the computer(s) from which you want to collect data. +* Add the {integrations-docs}[Elastic integration] for the data source to your deployment. + Integrations are available for many popular platforms and services, and are a good place to start for ingesting data into Elastic solutions--Observability, Security, and Search--or your own search application. @@ -41,5 +47,7 @@ Security, and Search--or your own search application. Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations. If you don't find an integration for your data source or if you need -<> to extend the integration, we still have you covered. +additional processing to extend the integration, we still have you covered. +Check out <> for a sneak peek. + **** diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index 9f02b704b..8a1d0306a 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -3,7 +3,7 @@ Elastic solutions--Security, Observability, and Search--are loaded with features and functionality to help you get value and insights from your data. -{fleet-guide}[Elastic Agent] and {integrations-docs}[Elastic integrations] can help. +{fleet-guide}[Elastic Agent] and {integrations-docs}[Elastic integrations] can help, and are the best place to start. When you use integrations with solutions, you have an integrated experience that offers easier implementation and decreases the time it takes to get insights and value from your data. @@ -24,7 +24,6 @@ NOTE: {serverless-docs}[Elastic serverless] makes using solutions even easier. Sign up for a link:{serverless-docs}/general/sign-up-trial[free trial], and check it out. - [discrete] [[ingest-for-search]] === Ingesting data for Search @@ -47,7 +46,10 @@ The solution gives you more pre-built components to get you up and running quick [[ingest-for-obs]] === Ingesting data for Observability -With link:https://www.elastic.co/observability[Elastic Observability], you can monitor and gain insights into logs, metrics, and application traces. +With link:https://www.elastic.co/observability[Elastic Observability], you can +monitor and gain insights into logs, metrics, and application traces. +The resources and guides in this section illustrate how to ingest data and use +it with the Observability solution. **Resources** @@ -70,7 +72,10 @@ With link:https://www.elastic.co/observability[Elastic Observability], you can m [[ingest-for-security]] === Ingesting data for Security -You can detect and respond to threats when you use link:https://www.elastic.co/security[Elastic Security] to analyze and take action on your data. +You can detect and respond to threats when you use +link:https://www.elastic.co/security[Elastic Security] to analyze and take +action on your data. +The resources and guides in this section illustrate how to ingest data and use it with the Security solution. **Resources** @@ -88,8 +93,8 @@ You can detect and respond to threats when you use link:https://www.elastic.co/s [[ingest-for-custom]] === Ingesting data for your own custom search solution -Elastic solutions can give you a head start for common use cases, but you are not limited. -You can still do your own thing with a custom solution designed by you. +Elastic solutions can give you a head start for common use cases, but you are not at all limited. +You can still do your own thing with a custom solution designed by _you_. Bring your ideas and use {es} and the {stack} to store, search, and visualize your data. diff --git a/docs/en/ingest-guide/ingest-tools.asciidoc b/docs/en/ingest-guide/ingest-tools.asciidoc index f6599b909..632bfab7a 100644 --- a/docs/en/ingest-guide/ingest-tools.asciidoc +++ b/docs/en/ingest-guide/ingest-tools.asciidoc @@ -14,11 +14,17 @@ A single link:{fleet-guide}[{agent}] can collect multiple types of data when it You can use standalone {agent}s and manage them locally on the systems where they are installed, or you can manage all of your agents and policies with the link:{fleet-guide}/manage-agents-in-fleet.html[Fleet UI in {kib}]. + Use {agent} with one of hundreds of link:{integrations-docs}[Elastic integrations] to simplify collecting, transforming, and visualizing data. -Integrations include default ingestion rules, dashboards, and visualizations to start analyzing your data right away. +Integrations include default ingestion rules, dashboards, and visualizations to help you start analyzing your data right away. Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations that can reduce your time to value. + -{agent} is the best option for collecting timestamped data for most data sources and use cases. -If your data requires additional processing before going to {es}, you can use {agent} with link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] or link:{logstash-ref}[{ls}] + +{agent} is the best option for collecting timestamped data for most data sources +and use cases. +If your data requires additional processing before going to {es}, you can use +link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} +processors], link:{logstash-ref}[{ls}], or additional processing features in +{es}. +Check out <> to see options. + Ready to try link:{fleet-guide}[{agent}]? Check out the link:{fleet-guide}/elastic-agent-installation.html[installation instructions]. + @@ -49,14 +55,14 @@ Most users never need to use {ls}, but it's available if you need it for: + * **Data collection** (if an Elastic integration isn't available). {agent} and Elastic {integrations-docs}/all_integrations[integrations] provide many features out-of-the-box, so be sure to search or browse integrations for your data source. -If you don't find an Elastic integration for your data source, check {ls} for an {logstash-ref}/input-plugins.html[input plugin]. +If you don't find an Elastic integration for your data source, check {ls} for an {logstash-ref}/input-plugins.html[input plugin] for your data source. * **Additional processing.** One of the most common {ls} use cases is link:{logstash-ref}/ea-integrations.html[extending Elastic integrations]. You can take advantage of the extensive, built-in capabilities of Elastic Agent and Elastic Integrations, and then use {ls} for additional data processing before sending the data on to {es}. * **Advanced use cases.** {ls} can help with advanced use cases, such as when you need link:{ingest-guide}/lspq.html[persistence or buffering], additional link:{ingest-guide}/ls-enrich.html[data enrichment], -link:{ingest-guide}/ls-networkbridge.html[proxying] to bridge network connections, or the ability to route data to +link:{ingest-guide}/ls-networkbridge.html[proxying] as a way to bridge network connections, or the ability to route data to link:{ingest-guide}/ls-multi.html[multiple destinations]. Language clients:: @@ -86,22 +92,20 @@ The link:https://www.elastic.co/guide/en/esf/current/aws-elastic-serverless-forw [[ingest-addl-proc]] == Tools and features for additional processing You can start with {agent} and Elastic {integrations-docs}[integrations], and still -take advantage of additional processing options if you need them: +take advantage of additional processing options if you need them. +You can use: -* link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] for sanitizing or enriching raw data at the source. - Use {agent} processors if you need to control what data is sent across the wire, or need to enrich the raw data with information available on the host. -* {es} link:{ref}/[ingest pipelines] for enriching incoming data or normalizing field data before the data is indexed. +* link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] to sanitize or enrich raw data at the source. + Use {agent} processors if you need to control what data is sent across the wire, or if you need to enrich the raw data with information available on the host. +* {es} link:{ref}/[ingest pipelines] to enrich incoming data or normalize field data before the data is indexed. {es} ingest pipelines enable you to manipulate the data as it comes in. This approach helps you avoid adding processing overhead to the hosts from which you're collecting data. -* {es} link:{ref}/runtime.html[runtime fields] for defining or altering the schema at query time. +* {es} link:{ref}/runtime.html[runtime fields] to define or alter the schema at query time. You can use runtime fields at query time to start working with your data without needing to understand how it is structured, add fields to existing documents without reindexing your data, override the value returned from an indexed field, and/or define fields for a specific use without modifying the underlying schema. -* {ls} `elastic_integration filter` for link:{logstash-ref}/ea-integrations.html[extending Elastic integrations], and other {ls} link:[filter plugins] for transforming data before it goes to {es}. +* {ls} `elastic_integration filter` to link:{logstash-ref}/ea-integrations.html[extend Elastic integrations], and other {ls} link:{logstash-ref}/filter-plugins.html[filter plugins] to transform data before it goes to {es}. - - - From fa049fcb99ce355585d0afb6b1b560d681d7b5c4 Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Fri, 6 Dec 2024 18:28:54 -0500 Subject: [PATCH 7/9] Incorporate review feedback --- docs/en/ingest-guide/index.asciidoc | 1 + .../ingest-additional-proc.asciidoc | 27 +++++ docs/en/ingest-guide/ingest-intro.asciidoc | 25 ++-- .../en/ingest-guide/ingest-solutions.asciidoc | 23 ++-- docs/en/ingest-guide/ingest-tools.asciidoc | 108 +++++++----------- 5 files changed, 98 insertions(+), 86 deletions(-) create mode 100644 docs/en/ingest-guide/ingest-additional-proc.asciidoc diff --git a/docs/en/ingest-guide/index.asciidoc b/docs/en/ingest-guide/index.asciidoc index 9618e8cc8..a5404cc15 100644 --- a/docs/en/ingest-guide/index.asciidoc +++ b/docs/en/ingest-guide/index.asciidoc @@ -8,6 +8,7 @@ include::{docs-root}/shared/attributes.asciidoc[] include::ingest-intro.asciidoc[] include::ingest-tools.asciidoc[] +include::ingest-additional-proc.asciidoc[] //include::ingest-static.asciidoc[] //include::ingest-timestamped.asciidoc[] include::ingest-solutions.asciidoc[] diff --git a/docs/en/ingest-guide/ingest-additional-proc.asciidoc b/docs/en/ingest-guide/ingest-additional-proc.asciidoc new file mode 100644 index 000000000..23d8bf54b --- /dev/null +++ b/docs/en/ingest-guide/ingest-additional-proc.asciidoc @@ -0,0 +1,27 @@ +[[ingest-addl-proc]] +== Additional ingest processing + +You can start with {agent} and Elastic {integrations-docs}[integrations], and still +take advantage of additional processing options if you need them. + +{agent} processors:: +You can use link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] to sanitize or enrich raw data at the source. +Use {agent} processors if you need to control what data is sent across the wire, or if you need to enrich the raw data with information available on the host. + +{es} ingest pipelines:: +You can use {es} link:{ref}/[ingest pipelines] to enrich incoming data or normalize field data before the data is indexed. +{es} ingest pipelines enable you to manipulate the data as it comes in. +This approach helps you avoid adding processing overhead to the hosts from which you're collecting data. + +{es} runtime fields:: +You can use {es} link:{ref}/runtime.html[runtime fields] to define or alter the schema at query time. +You can start working with your data without needing to understand how it is +structured, add fields to existing documents without reindexing your data, +override the value returned from an indexed field, and/or define fields for a +specific use without modifying the underlying schema. + +{ls} `elastic_integration filter`:: +You can use the {ls} link:{logstash-ref}/[`elastic_integration filter`] and +other link:{logstash-ref}/filter-plugins.html[{ls} filters] to +link:{logstash-ref}/ea-integrations.html[extend Elastic integrations] by +transforming data before it goes to {es}. diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc index 6b799d1d3..cbed25e62 100644 --- a/docs/en/ingest-guide/ingest-intro.asciidoc +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -14,14 +14,14 @@ reshape your data before it goes to {es}. You can ingest: * **General content** (data without timestamps), such as HTML pages, catalogs, and files -* **Timestamped (time series) data**, such as logs, metrics, and traces for Search, Security, Observability, or your own solution +* **Timestamped (time series) data**, such as logs, metrics, and traces for Elastic Security, Observability, Search solutions, or for your own custom solutions -[ingest-best-approach] -.What's the best approach for ingesting data? -**** -The best choice for ingesting data is the _simplest option_ that _meets your needs_ and _satisfies your use case_. +[discrete] +[[ingest-general]] +=== Ingesting general content -**Best practice for general content**. Choose the ingest tool that aligns with your data source. +Elastic offer tools designed to ingest specific types of general content. +The content type determines the best ingest option. * To index **documents** directly into {es}, use the {es} link:{ref}/docs.html[document APIs]. * To send **application data** directly to {es}, use an link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} @@ -32,10 +32,18 @@ language client]. If you would like to try things out before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data]. -**Best practice for timestamped data**. Start with Elastic Agent and an Elastic integration. +[discrete] +[[ingest-timestamped]] +=== Ingesting time-stamped data + + +[ingest-best-timestamped] +.What's the best approach for ingesting time-stamped data? +**** +The best approach for ingesting data is the _simplest option_ that _meets your needs_ and _satisfies your use case_. - Usually, the _simplest option_ for ingesting timestamped data is {agent} paired with an Elastic integration. +In most cases, the _simplest option_ for ingesting timestamped data is using {agent} paired with an Elastic integration. * Install {fleet-guide}[Elastic Agent] on the computer(s) from which you want to collect data. * Add the {integrations-docs}[Elastic integration] for the data source to your deployment. @@ -49,5 +57,4 @@ to search for available integrations. If you don't find an integration for your data source or if you need additional processing to extend the integration, we still have you covered. Check out <> for a sneak peek. - **** diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index 8a1d0306a..d89bda215 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -15,7 +15,7 @@ To use {fleet-guide}[Elastic Agent] and {integrations-docs}[Elastic integrations with Elastic solutions: 1. Create an link:https://www.elastic.co/cloud[{ecloud}] deployment for your solution. - If you don't have a {ecloud} account, you can sign up for a link:https://cloud.elastic.co/registration[free trial] get started. + If you don't have an {ecloud} account, you can sign up for a link:https://cloud.elastic.co/registration[free trial] to get started. 2. Add the {integrations-docs}[Elastic integration] for your data source to the deployment. 3. link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] on the systems whose data you want to collect. **** @@ -51,12 +51,8 @@ monitor and gain insights into logs, metrics, and application traces. The resources and guides in this section illustrate how to ingest data and use it with the Observability solution. -**Resources** -* link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] -* link:https://www.elastic.co/integrations/data-integrations?solution=observability[Elastic Observability integrations] - -**Guides for popular Observability use case** +**Guides for popular Observability use cases** * link:{estc-welcome}/getting-started-observability.html[Monitor applications and systems with Elastic Observability] * link:https://www.elastic.co/guide/en/observability/current/logs-metrics-get-started.html[Get started with logs and metrics] @@ -67,6 +63,10 @@ it with the Observability solution. ** link:{serverless-docs}/observability/quickstarts/monitor-hosts-with-elastic-agent[Monitor hosts with {agent} ({serverless-short})] ** link:{serverless-docs}/observability/quickstarts/k8s-logs-metrics[Monitor your K8s cluster with {agent} ({serverless-short})] +**Resources** + +* link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] +* link:https://www.elastic.co/integrations/data-integrations?solution=observability[Elastic Observability integrations] [discrete] [[ingest-for-security]] @@ -77,17 +77,17 @@ link:https://www.elastic.co/security[Elastic Security] to analyze and take action on your data. The resources and guides in this section illustrate how to ingest data and use it with the Security solution. +**Guides for popular Security use cases** + +* link:https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-siem-security.html[Use Elastic Security for SIEM] +* link:https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-endpoint-security.html[Protect hosts with endpoint threat intelligence from Elastic Security] + **Resources** * link:{fleet-guide}/elastic-agent-installation.html[Install {agent}] * link:https://www.elastic.co/integrations/data-integrations?solution=search[Elastic Security integrations] * link:{security-guide}/es-overview.html[Elastic Security documentation] -**Guides for popular Security use case** - -* link:https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-siem-security.html[Use Elastic Security for SIEM] -* link:https://www.elastic.co/guide/en/starting-with-the-elasticsearch-platform-and-its-solutions/current/getting-started-endpoint-security.html[Protect hosts with endpoint threat intelligence from Elastic Security] - [discrete] [[ingest-for-custom]] @@ -106,6 +106,5 @@ Bring your ideas and use {es} and the {stack} to store, search, and visualize yo ** link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[{es} language clients] ** link:https://www.elastic.co/web-crawler[Elastic web crawler] ** link:{ref}/es-connectors.html[Elastic connectors] - * link:{estc-welcome}/getting-started-general-purpose.html[Tutorial: Get started with vector search and generative AI] diff --git a/docs/en/ingest-guide/ingest-tools.asciidoc b/docs/en/ingest-guide/ingest-tools.asciidoc index 632bfab7a..28a876338 100644 --- a/docs/en/ingest-guide/ingest-tools.asciidoc +++ b/docs/en/ingest-guide/ingest-tools.asciidoc @@ -1,22 +1,33 @@ [[ingest-tools]] -== Tools for ingesting data +== Tools for ingesting time-series data + + +Elastic and others offer tools to help you get your data from the original data source into {es}. +Some tools are designed for particular data sources, and others are multi-purpose. // Iterative messaging as our recommended strategy morphs. // This section is the summary. "Here's the story _now_." // Hint at upcoming changes, but do it cautiously and responsibly. // Modular and co-located to make additions/updates/deprecations easier as our story matures. -Elastic and others offer tools to help you get your data from the original data source into {es}. -Some tools are designed for particular data sources, and others are multi-purpose. -{agent} and Elastic integrations:: +In this section, we'll help you determine which option is best for you. + +* <> +* <> +* <> +* <> + +[discrete] +[[ingest-ea]] +=== {agent} and Elastic integrations + A single link:{fleet-guide}[{agent}] can collect multiple types of data when it is link:{fleet-guide}/elastic-agent-installation.html[installed] on a host computer. You can use standalone {agent}s and manage them locally on the systems where they are installed, or you can manage all of your agents and policies with the link:{fleet-guide}/manage-agents-in-fleet.html[Fleet UI in {kib}]. -+ + Use {agent} with one of hundreds of link:{integrations-docs}[Elastic integrations] to simplify collecting, transforming, and visualizing data. Integrations include default ingestion rules, dashboards, and visualizations to help you start analyzing your data right away. Check out the {integrations-docs}/all_integrations[Integration quick reference] to search for available integrations that can reduce your time to value. -+ {agent} is the best option for collecting timestamped data for most data sources and use cases. @@ -25,34 +36,46 @@ link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors], link:{logstash-ref}[{ls}], or additional processing features in {es}. Check out <> to see options. -+ + Ready to try link:{fleet-guide}[{agent}]? Check out the link:{fleet-guide}/elastic-agent-installation.html[installation instructions]. -+ -**Beats.** link:{beats-ref}/beats-reference.html[Beats] are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. -When you use Elastic Agent, you're getting core Beats functionality and more added features. + +[discrete] +[[ingest-beats]] +=== {beats} + +link:{beats-ref}/beats-reference.html[Beats] are the original Elastic lightweight data shippers, and their capabilities live on in Elastic Agent. +When you use Elastic Agent, you're getting core Beats functionality, but with more added features. + + Beats require that you install a separate Beat for each type of data you want to collect. A single Elastic Agent installed on a host can collect and transport multiple types of data. -+ -**Best practice:** Use link:{fleet-guide}[Elastic Agent] whenever possible. -If your data source is not yet supported by Elastic Agent, use Beats. -Check out {beats} and {agent} link:{fleet-guide}/beats-agent-comparison.html#additional-capabilities-beats-and-agent[comparison] for more info. + +**Best practice:** Use link:{fleet-guide}[{agent}] whenever possible. +If your data source is not yet supported by {agent}, use {beats}. +Check out the {beats} and {agent} link:{fleet-guide}/beats-agent-comparison.html#additional-capabilities-beats-and-agent[comparison] for more info. When you are ready to upgrade, check out link:{fleet-guide}/migrate-beats-to-agent.html[Migrate from {beats} to {agent}]. -OpenTelemetry (OTel) collectors:: +[discrete] +[[ingest-otel]] +=== OpenTelemetry (OTel) collectors + link:https://opentelemetry.io/docs[OpenTelemetry] is a vendor-neutral observability framework for collecting, processing, and exporting telemetry data. Elastic is a member of the Cloud Native Computing Foundation (CNCF) and active contributor to the OpenTelemetry project. -+ + In addition to supporting upstream OTel development, Elastic provides link:https://github.com/elastic/opentelemetry[Elastic Distributions of OpenTelemetry], specifically designed to work with Elastic Observability. We're also expanding link:{fleet-guide}[{agent}] to use OTel collection. -Logstash:: +[discrete] +[[ingest-logstash]] +=== Logstash + link:{logstash-ref}[{ls}] is a versatile open source data ETL (extract, transform, load) engine that can expand your ingest capabilities. {ls} can _collect data_ from a wide variety of data sources with {ls} link:{logstash-ref}/input-plugins.html[input plugins], _enrich and transform_ the data with {ls} link:{logstash-ref}/filter-plugins.html[filter plugins], and _output_ the data to {es} and other destinations with the {ls} link:{logstash-ref}/output-plugins.html[output plugins]. -+ -Most users never need to use {ls}, but it's available if you need it for: -+ + +Many users never need to use {ls}, but it's available if you need it for: + * **Data collection** (if an Elastic integration isn't available). {agent} and Elastic {integrations-docs}/all_integrations[integrations] provide many features out-of-the-box, so be sure to search or browse integrations for your data source. If you don't find an Elastic integration for your data source, check {ls} for an {logstash-ref}/input-plugins.html[input plugin] for your data source. @@ -64,48 +87,3 @@ link:{ingest-guide}/lspq.html[persistence or buffering], additional link:{ingest-guide}/ls-enrich.html[data enrichment], link:{ingest-guide}/ls-networkbridge.html[proxying] as a way to bridge network connections, or the ability to route data to link:{ingest-guide}/ls-multi.html[multiple destinations]. - -Language clients:: -link:https://www.elastic.co/guide/en/elasticsearch/client/index.html[Elastic -language clients] help you send **application data**, such as from NodeJS or Python, -directly to {es} for search and analysis. -//ToDo: Figure out trademark considerations. - -APIs:: -Use the {es} link:{ref}/docs.html[document APIs] to index **documents** directly into {es}. - -File uploader:: -Use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader] to index **single files** into {es}. -This tool can be helpful for testing with small numbers of files. - -Web crawler:: -Use the Elastic link:https://www.elastic.co/web-crawler[web crawler] to index **web page content**. - -Connectors:: -Use link:{ref}/es-connectors.html[connectors] to index **data from third-party sources**, such as Amazon S3, GMail, Outlook, and Salesforce. -//ToDo: Figure out trademark considerations. - -Elastic serverless forwarder:: -The link:https://www.elastic.co/guide/en/esf/current/aws-elastic-serverless-forwarder.html[Elastic Serverless Forwarder] is an Amazon Web Services (AWS) Lambda function that ships logs from your AWS environment to {es}. - -[discrete] -[[ingest-addl-proc]] -== Tools and features for additional processing -You can start with {agent} and Elastic {integrations-docs}[integrations], and still -take advantage of additional processing options if you need them. -You can use: - -* link:{fleet-guide}/elastic-agent-processor-configuration.html[{agent} processors] to sanitize or enrich raw data at the source. - Use {agent} processors if you need to control what data is sent across the wire, or if you need to enrich the raw data with information available on the host. -* {es} link:{ref}/[ingest pipelines] to enrich incoming data or normalize field data before the data is indexed. - {es} ingest pipelines enable you to manipulate the data as it comes in. - This approach helps you avoid adding processing overhead to the hosts from which you're collecting data. - -* {es} link:{ref}/runtime.html[runtime fields] to define or alter the schema at query time. - You can use runtime fields at query time to start working with your data without needing to understand how it is structured, - add fields to existing documents without reindexing your data, - override the value returned from an indexed field, and/or - define fields for a specific use without modifying the underlying schema. - -* {ls} `elastic_integration filter` to link:{logstash-ref}/ea-integrations.html[extend Elastic integrations], and other {ls} link:{logstash-ref}/filter-plugins.html[filter plugins] to transform data before it goes to {es}. - From 8a2c881b6150e4620067d284e2bf91f138ce09e8 Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Mon, 9 Dec 2024 12:59:41 -0500 Subject: [PATCH 8/9] More cleanup --- docs/en/ingest-guide/ingest-intro.asciidoc | 2 -- docs/en/ingest-guide/ingest-solutions.asciidoc | 4 ++-- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc index cbed25e62..b3509fa1a 100644 --- a/docs/en/ingest-guide/ingest-intro.asciidoc +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -36,8 +36,6 @@ If you would like to try things out before you add your own data, try using our [[ingest-timestamped]] === Ingesting time-stamped data - - [ingest-best-timestamped] .What's the best approach for ingesting time-stamped data? **** diff --git a/docs/en/ingest-guide/ingest-solutions.asciidoc b/docs/en/ingest-guide/ingest-solutions.asciidoc index d89bda215..b76f3dd5c 100644 --- a/docs/en/ingest-guide/ingest-solutions.asciidoc +++ b/docs/en/ingest-guide/ingest-solutions.asciidoc @@ -48,7 +48,7 @@ The solution gives you more pre-built components to get you up and running quick With link:https://www.elastic.co/observability[Elastic Observability], you can monitor and gain insights into logs, metrics, and application traces. -The resources and guides in this section illustrate how to ingest data and use +The guides and resources in this section illustrate how to ingest data and use it with the Observability solution. @@ -75,7 +75,7 @@ it with the Observability solution. You can detect and respond to threats when you use link:https://www.elastic.co/security[Elastic Security] to analyze and take action on your data. -The resources and guides in this section illustrate how to ingest data and use it with the Security solution. +The guides and resources in this section illustrate how to ingest data and use it with the Security solution. **Guides for popular Security use cases** From 99280a10b3b80477cc2777d7f24b5c4b957c5efd Mon Sep 17 00:00:00 2001 From: Karen Metts Date: Tue, 21 Jan 2025 10:49:20 -0500 Subject: [PATCH 9/9] Refine connector content --- docs/en/ingest-guide/ingest-intro.asciidoc | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/en/ingest-guide/ingest-intro.asciidoc b/docs/en/ingest-guide/ingest-intro.asciidoc index b3509fa1a..1b3965018 100644 --- a/docs/en/ingest-guide/ingest-intro.asciidoc +++ b/docs/en/ingest-guide/ingest-intro.asciidoc @@ -28,7 +28,9 @@ The content type determines the best ingest option. language client]. * To index **web page content**, use the Elastic link:https://www.elastic.co/web-crawler[web crawler]. * To sync **data from third-party sources**, use link:{ref}/es-connectors.html[connectors]. -* To index **single files** for testing, use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader]. + A connector syncs content from an original data source to an {es} index. + Using connectors you can create _searchable_, read-only replicas of your data sources. +* To index **single files** for testing in a non-production environment, use the {kib} link:{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[file uploader]. If you would like to try things out before you add your own data, try using our {kibana-ref}/connect-to-elasticsearch.html#_add_sample_data[sample data].