Add memory queue settings (#859)

docs/en/ingest-management/elastic-agent/configuration/outputs/output-elasticsearch.asciidoc

@@ -55,6 +55,8 @@ category. Many of these settings have sensible defaults that allow you to run
 
 * <<output-elasticsearch-http-settings>>
 
+* <<output-elasticsearch-memory-queue-settings>>
+
 * <<output-elasticsearch-performance-tuning-settings>>
 
 [[output-elasticsearch-commonly-used-settings]]
@@ -477,6 +479,56 @@ exports the API under a custom prefix.
 
 |===
 
+[[output-elasticsearch-memory-queue-settings]]
+== Memory queue settings
+
+The memory queue keeps all events in memory.
+
+The memory queue waits for the output to acknowledge or drop events. If
+the queue is full, no new events can be inserted into the memory queue. Only
+after the signal from the output will the queue free up space for more events to be accepted.
+
+The memory queue is controlled by the parameters `flush.min_events` and `flush.timeout`. If
+`flush.timeout` is `0s` or `flush.min_events` is `0` or `1` then events can be sent by the output as
+soon as they are available. If the output supports a `bulk_max_size` parameter it controls the
+maximum batch size that can be sent.
+
+If `flush.min_events` is greater than `1` and `flush.timeout` is greater than `0s`, events will only
+be sent to the output when the queue contains at least `flush.min_events` events or the
+`flush.timeout` period has expired. In this mode the maximum size batch that that can be sent by the
+output is `flush.min_events`. If the output supports a `bulk_max_size` parameter, values of
+`bulk_max_size` greater than `flush.min_events` have no effect. The value of `flush.min_events`
+should be evenly divisible by `bulk_max_size` to avoid sending partial batches to the output.
+
+This sample configuration forwards events to the output if 512 events are available or the oldest
+available event has been waiting for 5s in the queue:
+
+[source,yaml]
+------------------------------------------------------------------------------
+queue.mem:
+  events: 4096
+  flush.min_events: 512
+  flush.timeout: 5s
+------------------------------------------------------------------------------
+
+[cols="2*<a"]
+|===
+| Setting | Description
+
+include::output-shared-settings.asciidoc[tag=events-setting]
+
+// =============================================================================
+
+include::output-shared-settings.asciidoc[tag=flush.min_events-setting]
+
+// =============================================================================
+
+include::output-shared-settings.asciidoc[tag=flush.timeout-setting]
+
+// =============================================================================
+
+|===
+
 [[output-elasticsearch-performance-tuning-settings]]
 == Performance tuning settings
 

docs/en/ingest-management/elastic-agent/configuration/outputs/output-logstash.asciidoc

@@ -69,6 +69,8 @@ minimal configuration.
 
 * <<output-logstash-authentication-settings>>
 
+* <<output-logstash-memory-queue-settings>>
+
 * <<output-logstash-performance-tuning-settings>>
 
 [[output-logstash-commonly-used-settings]]
@@ -162,6 +164,56 @@ use SSL/TLS.
 
 For more information, refer to <<secure-logstash-connections>>.
 
+[[output-logstash-memory-queue-settings]]
+== Memory queue settings
+
+The memory queue keeps all events in memory.
+
+The memory queue waits for the output to acknowledge or drop events. If
+the queue is full, no new events can be inserted into the memory queue. Only
+after the signal from the output will the queue free up space for more events to be accepted.
+
+The memory queue is controlled by the parameters `flush.min_events` and `flush.timeout`. If
+`flush.timeout` is `0s` or `flush.min_events` is `0` or `1` then events can be sent by the output as
+soon as they are available. If the output supports a `bulk_max_size` parameter it controls the
+maximum batch size that can be sent.
+
+If `flush.min_events` is greater than `1` and `flush.timeout` is greater than `0s`, events will only
+be sent to the output when the queue contains at least `flush.min_events` events or the
+`flush.timeout` period has expired. In this mode the maximum size batch that that can be sent by the
+output is `flush.min_events`. If the output supports a `bulk_max_size` parameter, values of
+`bulk_max_size` greater than `flush.min_events` have no effect. The value of `flush.min_events`
+should be evenly divisible by `bulk_max_size` to avoid sending partial batches to the output.
+
+This sample configuration forwards events to the output if 512 events are available or the oldest
+available event has been waiting for 5s in the queue:
+
+[source,yaml]
+------------------------------------------------------------------------------
+queue.mem:
+  events: 4096
+  flush.min_events: 512
+  flush.timeout: 5s
+------------------------------------------------------------------------------
+
+[cols="2*<a"]
+|===
+| Setting | Description
+
+include::output-shared-settings.asciidoc[tag=events-setting]
+
+// =============================================================================
+
+include::output-shared-settings.asciidoc[tag=flush.min_events-setting]
+
+// =============================================================================
+
+include::output-shared-settings.asciidoc[tag=flush.timeout-setting]
+
+// =============================================================================
+
+|===
+
 [[output-logstash-performance-tuning-settings]]
 == Performance tuning settings
 

docs/en/ingest-management/elastic-agent/configuration/outputs/output-shared-settings.asciidoc

@@ -43,6 +43,42 @@ escaping.
 
 // =============================================================================
 
+// tag::events-setting[]
+|
+[id="{type}-events-setting"]
+`events`
+
+| The number of events the queue can store. This value should be evenly divisible by `flush.min_events` to avoid sending partial batches to the output.
+
+*Default:* `3200 events`
+// end::events-setting[]
+
+// =============================================================================
+
+// tag::flush.min_events-setting[]
+|
+[id="{type}-flush.min_events-setting"]
+`flush.min_events`
+
+| The minimum number of events required for publishing. If this value is set to 0 or 1, events are available to the output immediately. If this value is greater than 1 the output must wait for the queue to accumulate this minimum number of events or for `flush.timeout`` to expire before publishing. When greater than 1 this value also defines the maximum possible batch that can be sent by the output.
+
+*Default:* `1600 events`
+// end::flush.min_events-setting[]
+
+// =============================================================================
+
+// tag::flush.timeout-setting[]
+|
+[id="{type}-flush.timeout-setting"]
+`flush.timeout`
+
+| (int) The maximum wait time for `flush.min_events` to be fulfilled. If set to 0s, events are available to the output immediately.
+
+*Default:* `10s`
+// end::flush.timeout-setting[]
+
+// =============================================================================
+
 // tag::worker-setting[]
 |
 [id="{type}-worker-setting"]

docs/en/ingest-management/fleet/fleet-settings-output-elasticsearch.asciidoc

@@ -114,8 +114,21 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[
 
 // =============================================================================
 
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.min_events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.timeout-setting]
+
+// =============================================================================
+
 include::../elastic-agent/configuration/outputs/output-elasticsearch.asciidoc[tag=max_retries-setting]
 
+
 // =============================================================================
 
 include::../elastic-agent/configuration/outputs/output-elasticsearch.asciidoc[tag=timeout-setting]

docs/en/ingest-management/fleet/fleet-settings-output-kafka.asciidoc

@@ -511,6 +511,18 @@ output.console:
 
 // =============================================================================
 
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.min_events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.timeout-setting]
+
+// =============================================================================
+
 // tag::keep_alive-setting[]
 |
 [id="{type}-keep_alive-setting"]
@@ -555,6 +567,8 @@ is false.
 *`retry.backoff`*:: Waiting time between retries. The default is 250ms.
 // end::metadata-setting[]
 
+// =============================================================================
+
 |===
 
 :type!:

docs/en/ingest-management/fleet/fleet-settings-output-logstash.asciidoc

@@ -142,6 +142,18 @@ include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[
 
 // =============================================================================
 
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.min_events-setting]
+
+// =============================================================================
+
+include::../elastic-agent/configuration/outputs/output-shared-settings.asciidoc[tag=flush.timeout-setting]
+
+// =============================================================================
+
 // tag::index-setting[]
 |
 [id="{type}-index-setting"]