Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.12] Add docs to configure remote ES output (backport #850) #852

Merged
merged 1 commit into from
Jan 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
:type: output-elasticsearch-fleet-settings

[[remote-elasticsearch-output]]
= Remote {es} output

Beginning in version 8.12.0, you can send {agent} data to a remote {es} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {fleet} to manage the agents.

A remote {es} cluster supports the same <<es-output-settings,output settings>> as your main {es} cluster.

To configure a remote {es} cluster for your {agent} data:

. In {kib}, go to **Management -> {fleet} -> Settings**.

. In the **Outputs** section, select **Add output**.

. In the **Add new output** flyout, provide a name for the output and select **Remote Elasticsearch** as the output type.

. In the **Hosts** field, add the URL that agents should use to access the remote {es} cluster.

.. To find the remote host address, in the remote cluster open {kib} and go to **Management -> {fleet} -> Settings**.

.. Copy the **Hosts** value for the default output.

.. Back in your main cluster, paste the value you copied into the output **Hosts** field.

. Create a service token to access the remote cluster.

.. Below the **Service Token** field, copy the API request.

.. In the remote cluster, open the {kib} menu and go to **Management -> Dev Tools**.

.. Run the API request.

.. Copy the value for the generated token.

.. Back in your main cluster, paste the value you copied into the output **Service Token** field.
+
NOTE: To prevent unauthorized access the {es} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {fleet-server} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See {kibana-ref}/fleet-settings-kb.html#_preconfiguration_settings_for_advanced_use_cases[Preconfiguration settings] in the {kib} Guide to learn more.

. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {agent}s use this output to send data if no other output is set in the <<agent-policy,agent policy>>.

. Select which <<es-output-settings-performance-tuning-settings,performance tuning settings>> you'd prefer in order to optimize {agent} for throughput, scale, or latency, or leave the default `balanced` setting.

. Add any <<es-output-settings-yaml-config,advanced YAML configuration settings>> that you'd like for the output.

. Click **Save and apply settings**.

After the output is created, you can update an {agent} policy to use the new remote {es} cluster:

. In {kib}, go to **Management -> {fleet} -> Agent policies**.

. Click the agent policy to edit it, then click **Settings**.

. To send integrations data, set the **Output for integrations** option to use the output that you configured in the previous steps.

. To send {agent} monitoring data, set the **Output for agent monitoring** option to use the output that you configured in the previous steps.

. Click **Save changes**.

The remote {es} cluster is now configured.

As a final step before using the remote {es} output, you need to make sure that for any integrations that have been <<add-integration-to-policy,added to your {agent} policy>>, the integration assets have been installed on the remote {es} cluster. Refer to <<install-uninstall-integration-assets,Install and uninstall {agent} integration assets>> for the steps.
1 change: 1 addition & 0 deletions docs/en/ingest-management/fleet/fleet-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,7 @@ The **Add new output** UI opens.
* <<es-output-settings>>
* <<ls-output-settings>>
* <<kafka-output-settings>>
* <<remote-elasticsearch-output>>

. Click **Save and apply settings**.

Expand Down
48 changes: 1 addition & 47 deletions docs/en/ingest-management/fleet/monitor-elastic-agent.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -238,53 +238,7 @@ To turn off agent monitoring when creating a new agent policy:

You may want to store all of the health and status data about your {agents} in a remote {es} cluster, so that it's separate and independent from the deployment where you use {fleet} to manage the agents.

To configure a remote {es} cluster for your {agent} monitoring data:

. In {kib}, go to **Management -> {fleet} -> Settings**.

. In the **Outputs** section, select **Add output**.

. In the **Add new output** flyout, provide a name for the output and select **Remote Elasticsearch** as the output type.

. In the **Hosts** field, add the URL that agents should use to access the remote {es} cluster.

.. To find the remote host address, in the remote cluster open {kib} and go to **Management -> {fleet} -> Settings**.

.. Copy the **Hosts** value for the default output.

.. Back in your main cluster, paste the value you copied into the output **Hosts** field.

. Create a service token to access the remote cluster.

.. Below the **Service Token** field, copy the API request.

.. In the remote cluster, open the {kib} menu and go to **Management -> Dev Tools**.

.. Run the API request.

.. Copy the value for the generated token.

.. Back in your main cluster, paste the value you copied into the output **Service Token** field.
+
NOTE: To prevent unauthorized access the {es} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {fleet-server} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See {kibana-ref}/fleet-settings-kb.html#_preconfiguration_settings_for_advanced_use_cases[Preconfiguration settings] in the {kib} Guide to learn more.

. Choose whether or not the remote output should be the default for agent monitoring. When set, {agent}s use this output to send data if no other output is set in the <<agent-policy,agent policy>>.

. Add any <<es-output-settings-yaml-config,advanced YAML configuration settings>> that you'd like for the output.

. Click **Save and apply settings**.

After the output is created, you can update an {agent} policy to use the new remote {es} cluster:

. In {kib}, go to **Management -> {fleet} -> Agent policies**.

. Click the agent policy to edit it, then click **Settings**.

. Set the **Output for agent monitoring** option to use the output that you configured in the previous steps.

. Click **Save changes**.

The remote {es} cluster is now configured.
To do so, follow the steps in <<remote-elasticsearch-output>>. After the new output is configured, follow the steps to update the {agent} policy and make sure that the **Output for agent monitoring** setting is enabled. {agent} monitoring data will use the remote {es} output that you configured.

[discrete]
[[fleet-alerting]]
Expand Down
2 changes: 2 additions & 0 deletions docs/en/ingest-management/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ include::fleet/fleet-settings-output-logstash.asciidoc[leveloffset=+3]

include::fleet/fleet-settings-output-kafka.asciidoc[leveloffset=+3]

include::fleet/fleet-settings-remote-elasticsearch.asciidoc[leveloffset=+3]

include::fleet/fleet-manage-agents.asciidoc[leveloffset=+2]

include::fleet/unenroll-elastic-agent.asciidoc[leveloffset=+3]
Expand Down