Merge branch 'main' into obs-docs-issue-3354

docs/apm-data-security.asciidoc

@@ -483,18 +483,6 @@ TIP: If you prefer using a GUI, you can instead open {kib} and navigate to
 **Stack Management** -> **Ingest Pipelines** -> **Create pipeline**.
 Use the same naming convention explained previously to ensure your new pipeline matches the correct APM data stream.
 
-**Roll over the data stream (optional)**
-
-Pipeline changes are not applied retroactively to existing indices.
-For changes to take effect immediately, you must create a new write index for the data stream.
-This can be done with the {es} {ref}/indices-rollover-index.html[Rollover API].
-For example, to roll over the default application traces data stream, with a `namespace` of "default", run:
-
-[source,console]
-----
-POST /traces-apm-default/_rollover/
-----
-
 That's it! Passwords will now be redacted from your APM HTTP body data.
 
 To learn more about ingest pipelines, see <<custom-index-template>>.

docs/command-reference.asciidoc

@@ -34,6 +34,7 @@ ifdef::serverless[]
 endif::serverless[]
 
 :help-command-short-desc: Shows help for any command
+:keystore-command-short-desc: Manages the <<keystore,secrets keystore>>
 :modules-command-short-desc: Manages configured modules
 :package-command-short-desc: Packages the configuration and executable into a zip file
 :remove-command-short-desc: Removes the specified function from your serverless environment
@@ -106,6 +107,9 @@ ifdef::apm-server[]
 endif::[]
 |<<export-command,`export`>> |{export-command-short-desc}.
 |<<help-command,`help`>> |{help-command-short-desc}.
+ifndef::serverless[]
+|<<keystore-command,`keystore`>> |{keystore-command-short-desc}.
+endif::[]
 ifeval::["{beatname_lc}"=="functionbeat"]
 |<<package-command,`package`>> |{package-command-short-desc}.
 |<<remove-command,`remove`>> |{remove-command-short-desc}.
@@ -430,6 +434,65 @@ Specifies the name of the command to show help for.
 {beatname_lc} help export
 -----
 
+ifndef::serverless[]
+[float]
+[[keystore-command]]
+==== `keystore` command
+
+{keystore-command-short-desc}.
+
+*SYNOPSIS*
+
+["source","sh",subs="attributes"]
+----
+{beatname_lc} keystore SUBCOMMAND [FLAGS]
+----
+
+*`SUBCOMMAND`*
+
+*`add KEY`*::
+Adds the specified key to the keystore. Use the `--force` flag to overwrite an
+existing key. Use the `--stdin` flag to pass the value through `stdin`.
+
+*`create`*::
+Creates a keystore to hold secrets. Use the `--force` flag to overwrite the
+existing keystore.
+
+*`list`*::
+Lists the keys in the keystore.
+
+*`remove KEY`*::
+Removes the specified key from the keystore.
+
+*FLAGS*
+
+*`--force`*::
+Valid with the `add` and `create` subcommands. When used with `add`, overwrites
+the specified key. When used with `create`, overwrites the keystore.
+
+*`--stdin`*::
+When used with `add`, uses the stdin as the source of the key's value.
+
+*`-h, --help`*::
+Shows help for the `keystore` command.
+
+
+{global-flags}
+
+*EXAMPLES*
+
+["source","sh",subs="attributes"]
+-----
+{beatname_lc} keystore create
+{beatname_lc} keystore add ES_PWD
+{beatname_lc} keystore remove ES_PWD
+{beatname_lc} keystore list
+-----
+
+See <<keystore>> for more examples.
+
+endif::[]
+
 ifeval::["{beatname_lc}"=="functionbeat"]
 [float]
 [[package-command]]

docs/custom-index-template.asciidoc

@@ -15,7 +15,7 @@ These index templates are composed of multiple component templates--reusable bui
 that configure index mappings, settings, and aliases.
 
 The default APM index templates can be viewed in {kib}.
-Navigate to **{stack-manage-app}** > **Index Management** > **Index Templates**, and search for `apm`.
+Navigate to **{stack-manage-app}** → **Index Management** → **Index Templates**, and search for `apm`.
 Select any of the APM index templates to view their relevant component templates.
 
 [discrete]
@@ -30,21 +30,27 @@ When you install the APM integration, {fleet} creates a default `@custom` compon
 You can edit this `@custom` component template to customize your {es} indices.
 
 First, determine which <<apm-data-streams,data stream>> you'd like to edit.
-Then, open {kib} and navigate to **{stack-manage-app}** > **Index Management** > **Component Templates**.
+Then, open {kib} and navigate to **{stack-manage-app}** → **Index Management** → **Component Templates**.
 
 Custom component templates are named following this pattern: `<name_of_data_stream>@custom`.
 Search for the name of the data stream, like `traces-apm`, and select its custom component template.
 In this example, that'd be, `traces-apm@custom`.
-Then click **Manage** > **Edit**.
+Then click **Manage** → **Edit**.
 
-Add any custom index settings, metadata, or mappings.
-For example, you may want to...
+Add any custom metadata, index settings, or mappings.
+
+[discrete]
+[[custom-index-template-index-settings]]
+==== Index settings
+
+In the **Index settings** step, you can specify custom {ref}/index-modules.html#index-modules-settings[index settings].
+For example, you could:
 
 * Customize the index lifecycle policy applied to a data stream.
 See <<data-streams-custom-policy,custom index lifecycle policies>> for a walk-through.
 
 * Change the number of {ref}/scalability.html[shards] per index.
-Specify the number of primary shards in the **index settings**:
+Specify the number of primary shards:
 +
 [source,json]
 ----
@@ -56,7 +62,7 @@ Specify the number of primary shards in the **index settings**:
 ----
 
 * Change the number of {ref}/docs-replication.html[replicas] per index.
-Specify the number of replica shards in the **index settings**:
+Specify the number of replica shards:
 +
 [source,json]
 ----
@@ -67,6 +73,29 @@ Specify the number of replica shards in the **index settings**:
 }
 ----
 
+[discrete]
+[[custom-index-template-mappings]]
+==== Mappings
+
+{ref}/mapping.html[Mapping] is the process of defining how a document, and the fields it contains, are stored and indexed.
+In the *Mappings* step, you can add custom field mappings.
+For example, you could:
+
+* Add custom field mappings that you can index on and search.
+In the *Mapped fields* tab, add a new field including the {ref}/mapping-types.html[field type]:
++
+image::images/custom-index-template-mapped-fields.png[Editing a component template to add a new mapped field]
+
+* Add a {ref}/runtime.html[runtime field] that is evaluated at query time.
+In the *Runtime fields* tab, click *Create runtime field* and provide a field name,
+type, and optionally a script:
++
+image::images/custom-index-template-runtime-fields.png[Editing a component template to add a new runtime field]
+
+[discrete]
+[[custom-index-template-rollover]]
+=== Roll over the data stream
+
 Changes to component templates are not applied retroactively to existing indices.
 For changes to take effect, you must create a new write index for the data stream.
 This can be done with the {es} {ref}/indices-rollover-index.html[Rollover API].

docs/ingest-pipelines.asciidoc

@@ -60,7 +60,6 @@ The process for creating a custom ingest pipeline is as follows:
 
 * Create a pipeline with processors specific to your use case
 * Add the newly created pipeline to an `@custom` pipeline that matches an APM data stream
-* Roll over your data stream
 
 If you prefer more guidance, see one of these tutorials:
 

docs/keystore.asciidoc

@@ -0,0 +1,109 @@
+[[keystore]]
+=== Secrets keystore for secure settings
+
+++++
+<titleabbrev>Secrets keystore</titleabbrev>
+++++
+
+IMPORTANT: The APM Server keystore only applies to the APM Server binary installation method.
+
+When you configure APM Server, you might need to specify sensitive settings,
+such as passwords. Rather than relying on file system permissions to protect
+these values, you can use the APM Server keystore to securely store secret
+values for use in configuration settings.
+
+After adding a key and its secret value to the keystore, you can use the key in
+place of the secret value when you configure sensitive settings.
+
+The syntax for referencing keys is identical to the syntax for environment
+variables:
+
+`${KEY}`
+
+Where KEY is the name of the key.
+
+For example, imagine that the keystore contains a key called `ES_PWD` with the
+value `yourelasticsearchpassword`:
+
+* In the configuration file, use `output.elasticsearch.password: "${ES_PWD}"`
+* On the command line, use: `-E "output.elasticsearch.password=\${ES_PWD}"`
+
+When APM Server unpacks the configuration, it resolves keys before resolving
+environment variables and other variables.
+
+Notice that the APM Server keystore differs from the {es} keystore.
+Whereas the {es} keystore lets you store `elasticsearch.yml` values by
+name, the APM Server keystore lets you specify arbitrary names that you can
+reference in the APM Server configuration.
+
+To create and manage keys, use the `keystore` command.
+See the <<keystore-command,command reference>> for the full command syntax,
+including optional flags.
+
+NOTE: The `keystore` command must be run by the same user who will run
+APM Server.
+
+[discrete]
+[[creating-keystore]]
+=== Create a keystore
+
+To create a secrets keystore, use:
+
+[source,sh]
+-----
+apm-server keystore create
+-----
+
+APM Server creates the keystore in the directory defined by the `path.data`
+configuration setting.
+
+[discrete]
+[[add-keys-to-keystore]]
+=== Add keys
+
+To store sensitive values, such as authentication credentials for {es},
+use the `keystore add` command:
+
+[source,sh]
+-----
+apm-server keystore add ES_PWD
+-----
+
+When prompted, enter a value for the key.
+
+To overwrite an existing key's value, use the `--force` flag:
+
+[source,sh]
+-----
+apm-server keystore add ES_PWD --force
+-----
+
+To pass the value through stdin, use the `--stdin` flag. You can also use
+`--force`:
+
+[source,sh]
+-----
+cat /file/containing/setting/value | apm-server keystore add ES_PWD --stdin --force
+-----
+
+[discrete]
+[[list-settings]]
+=== List keys
+
+To list the keys defined in the keystore, use:
+
+[source,sh]
+-----
+apm-server keystore list
+-----
+
+[discrete]
+[[remove-settings]]
+=== Remove keys
+
+To remove a key from the keystore, use:
+
+[source,sh]
+-----
+apm-server keystore remove ES_PWD
+-----

docs/setting-up-and-running.asciidoc

@@ -12,12 +12,16 @@ for basic installation and running instructions.
 This section includes additional information on how to set up and run APM Server, including:
 
 * <<directory-layout>>
+* <<keystore>>
 * <<command-line-options>>
+* <<tune-data-ingestion>>
 * <<high-availability>>
 * <<running-on-docker>>
 
 include::{docdir}/shared-directory-layout.asciidoc[]
 
+include::{docdir}/keystore.asciidoc[]
+
 include::{docdir}/command-reference.asciidoc[]
 
 include::{docdir}/data-ingestion.asciidoc[]