Skip to content

Commit

Permalink
Add docs to configure remote ES output (#850)
Browse files Browse the repository at this point in the history
(cherry picked from commit 7734028)
  • Loading branch information
kilfoyle authored and mergify[bot] committed Jan 24, 2024
1 parent e3fb3fe commit 8b0a7b9
Show file tree
Hide file tree
Showing 4 changed files with 66 additions and 47 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
:type: output-elasticsearch-fleet-settings

[[remote-elasticsearch-output]]
= Remote {es} output

Beginning in version 8.12.0, you can send {agent} data to a remote {es} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {fleet} to manage the agents.

A remote {es} cluster supports the same <<es-output-settings,output settings>> as your main {es} cluster.

To configure a remote {es} cluster for your {agent} data:

. In {kib}, go to **Management -> {fleet} -> Settings**.

. In the **Outputs** section, select **Add output**.

. In the **Add new output** flyout, provide a name for the output and select **Remote Elasticsearch** as the output type.

. In the **Hosts** field, add the URL that agents should use to access the remote {es} cluster.

.. To find the remote host address, in the remote cluster open {kib} and go to **Management -> {fleet} -> Settings**.

.. Copy the **Hosts** value for the default output.

.. Back in your main cluster, paste the value you copied into the output **Hosts** field.

. Create a service token to access the remote cluster.

.. Below the **Service Token** field, copy the API request.

.. In the remote cluster, open the {kib} menu and go to **Management -> Dev Tools**.

.. Run the API request.

.. Copy the value for the generated token.

.. Back in your main cluster, paste the value you copied into the output **Service Token** field.
+
NOTE: To prevent unauthorized access the {es} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {fleet-server} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See {kibana-ref}/fleet-settings-kb.html#_preconfiguration_settings_for_advanced_use_cases[Preconfiguration settings] in the {kib} Guide to learn more.

. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {agent}s use this output to send data if no other output is set in the <<agent-policy,agent policy>>.

. Select which <<es-output-settings-performance-tuning-settings,performance tuning settings>> you'd prefer in order to optimize {agent} for throughput, scale, or latency, or leave the default `balanced` setting.

. Add any <<es-output-settings-yaml-config,advanced YAML configuration settings>> that you'd like for the output.

. Click **Save and apply settings**.

After the output is created, you can update an {agent} policy to use the new remote {es} cluster:

. In {kib}, go to **Management -> {fleet} -> Agent policies**.

. Click the agent policy to edit it, then click **Settings**.

. To send integrations data, set the **Output for integrations** option to use the output that you configured in the previous steps.

. To send {agent} monitoring data, set the **Output for agent monitoring** option to use the output that you configured in the previous steps.

. Click **Save changes**.

The remote {es} cluster is now configured.

As a final step before using the remote {es} output, you need to make sure that for any integrations that have been <<add-integration-to-policy,added to your {agent} policy>>, the integration assets have been installed on the remote {es} cluster. Refer to <<install-uninstall-integration-assets,Install and uninstall {agent} integration assets>> for the steps.
1 change: 1 addition & 0 deletions docs/en/ingest-management/fleet/fleet-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ The **Add new output** UI opens.
* <<es-output-settings>>
* <<ls-output-settings>>
* <<kafka-output-settings>>
* <<remote-elasticsearch-output>>

. Click **Save and apply settings**.

Expand Down
48 changes: 1 addition & 47 deletions docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -238,53 +238,7 @@ To turn off agent monitoring when creating a new agent policy:

You may want to store all of the health and status data about your {agents} in a remote {es} cluster, so that it's separate and independent from the deployment where you use {fleet} to manage the agents.

To configure a remote {es} cluster for your {agent} monitoring data:

. In {kib}, go to **Management -> {fleet} -> Settings**.

. In the **Outputs** section, select **Add output**.

. In the **Add new output** flyout, provide a name for the output and select **Remote Elasticsearch** as the output type.

. In the **Hosts** field, add the URL that agents should use to access the remote {es} cluster.

.. To find the remote host address, in the remote cluster open {kib} and go to **Management -> {fleet} -> Settings**.

.. Copy the **Hosts** value for the default output.

.. Back in your main cluster, paste the value you copied into the output **Hosts** field.

. Create a service token to access the remote cluster.

.. Below the **Service Token** field, copy the API request.

.. In the remote cluster, open the {kib} menu and go to **Management -> Dev Tools**.

.. Run the API request.

.. Copy the value for the generated token.

.. Back in your main cluster, paste the value you copied into the output **Service Token** field.
+
NOTE: To prevent unauthorized access the {es} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {fleet-server} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See {kibana-ref}/fleet-settings-kb.html#_preconfiguration_settings_for_advanced_use_cases[Preconfiguration settings] in the {kib} Guide to learn more.

. Choose whether or not the remote output should be the default for agent monitoring. When set, {agent}s use this output to send data if no other output is set in the <<agent-policy,agent policy>>.

. Add any <<es-output-settings-yaml-config,advanced YAML configuration settings>> that you'd like for the output.

. Click **Save and apply settings**.

After the output is created, you can update an {agent} policy to use the new remote {es} cluster:

. In {kib}, go to **Management -> {fleet} -> Agent policies**.

. Click the agent policy to edit it, then click **Settings**.

. Set the **Output for agent monitoring** option to use the output that you configured in the previous steps.

. Click **Save changes**.

The remote {es} cluster is now configured.
To do so, follow the steps in <<remote-elasticsearch-output>>. After the new output is configured, follow the steps to update the {agent} policy and make sure that the **Output for agent monitoring** setting is enabled. {agent} monitoring data will use the remote {es} output that you configured.

[discrete]
[[fleet-alerting]]
Expand Down
2 changes: 2 additions & 0 deletions docs/en/ingest-management/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ include::fleet/fleet-settings-output-logstash.asciidoc[leveloffset=+3]

include::fleet/fleet-settings-output-kafka.asciidoc[leveloffset=+3]

include::fleet/fleet-settings-remote-elasticsearch.asciidoc[leveloffset=+3]

include::fleet/fleet-manage-agents.asciidoc[leveloffset=+2]

include::fleet/unenroll-elastic-agent.asciidoc[leveloffset=+3]
Expand Down

0 comments on commit 8b0a7b9

Please sign in to comment.