Scale Preset should prevent Thundering Herd issue with many Agents

[StephanErb]

Describe the enhancement:

The scale preset introduced in elastic/kibana#166870 and #3797 is a great start to make agent ingestion more scalable. However, assuming one wants to ingest using thousand of agents, there is a high risk of a thundering herd problem. If Elastic starts choking the the agents reconnect and retry logic further increases the load on the cluster. This might kick the entire system over the edge.

This should be addressed with further options being tuned by the preset.

Describe a specific use case for the enhancement or feature:

The scale preset should adjust the following additional options:

• backoff.init and backoff.max should use a higher default and leverage a jitter. There is currently no jitter in the backoff which leads to a synchronized retry across all agents.
• max_retries should be reduced for metric data sets (but not logs). It is better to focus on delivering recent data then retrying the delivery of outdated scrapes. This will act as a form of back pressure to ensure a single failure does not lead to further increased load on the server side.

What is the definition of done?

Better defaults with no actions needed by users.

Implementation tasks

Preview Give feedback

1. Add a random exponential backoff to backoff.init and backoff.max so that when agents reconnect en-mass (or wait to reconnect) the connection requests are somewhat spread out. This should be the default behavior regardless of the preset.
2. For the "optimized for Scale" preset: change the default for backoff.init to 5s
3. For the "optimized for Scale" preset: change the default forbackoff.max default to 300s

[cmacknz]

backoff.init and backoff.max should use a higher default and leverage a jitter. There is currently no jitter in the backoff which leads to a synchronized retry across all agents.

+1 thanks for pointing this out, we should be using pseudo-random exponential backoff, not just exponential backoff.

max_retries should be reduced for metric data sets (but not logs).

Metricbeat should already default to 3 retries, while Filebeat defaults to infinite retries. Agent doesn't modify this part of the Metricbeat configuration. https://www.elastic.co/guide/en/beats/metricbeat/current/elasticsearch-output.html#_max_retries

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ has the modifications to the backoff algorithm we should make along with their relative performance.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)