Merge branch 'main' into systemtest_disable_ignore_malformed

NOTICE.txt

@@ -573,11 +573,11 @@ Contents of probable licence file $GOMODCACHE/github.com/elastic/[email protected].
 
 --------------------------------------------------------------------------------
 Dependency : github.com/elastic/beats/v7
-Version: v7.0.0-alpha2.0.20241105110521-38dfc52434ed
+Version: v7.0.0-alpha2.0.20241106142216-cf13781a6fca
 Licence type (autodetected): Elastic
 --------------------------------------------------------------------------------
 
-Contents of probable licence file $GOMODCACHE/github.com/elastic/beats/[email protected].20241105110521-38dfc52434ed/LICENSE.txt:
+Contents of probable licence file $GOMODCACHE/github.com/elastic/beats/[email protected].20241106142216-cf13781a6fca/LICENSE.txt:
 
 Source code in this repository is variously licensed under the Apache License
 Version 2.0, an Apache compatible license, or the Elastic License. Outside of

go.mod

@@ -8,7 +8,7 @@ require (
 	github.com/dustin/go-humanize v1.0.1
 	github.com/elastic/apm-aggregation v1.1.0
 	github.com/elastic/apm-data v1.14.1
-	github.com/elastic/beats/v7 v7.0.0-alpha2.0.20241105110521-38dfc52434ed
+	github.com/elastic/beats/v7 v7.0.0-alpha2.0.20241106142216-cf13781a6fca
 	github.com/elastic/elastic-agent-client/v7 v7.16.0
 	github.com/elastic/elastic-agent-libs v0.17.1
 	github.com/elastic/elastic-agent-system-metrics v0.11.3

go.sum

@@ -132,8 +132,8 @@ github.com/elastic/apm-aggregation v1.1.0 h1:eTHGd5w99JoRFJ763MJP6nUX4hkkeSil3KW
 github.com/elastic/apm-aggregation v1.1.0/go.mod h1:YBQ77Jt7vOy2/7w4q5SsEifhc0V02dJxRcyyeC6HsvQ=
 github.com/elastic/apm-data v1.14.1 h1:3G8gJ4wjdd5O6JnxtdiEzVrKuqPT9mVqo03d5i5sRJ4=
 github.com/elastic/apm-data v1.14.1/go.mod h1:AlwPwWh15EWP5yLDm/X40mdw1z/pHdlcTgMUkS9tKmY=
-github.com/elastic/beats/v7 v7.0.0-alpha2.0.20241105110521-38dfc52434ed h1:S8dxSZfUpOusaNmgALz/TJEqCRVgkRlLbGtMIEvn9Sw=
-github.com/elastic/beats/v7 v7.0.0-alpha2.0.20241105110521-38dfc52434ed/go.mod h1:eCxJTNQhr9Uqd8AxN5ZmoEmwP3siol1GZ6cwu0CemVI=
+github.com/elastic/beats/v7 v7.0.0-alpha2.0.20241106142216-cf13781a6fca h1:187f3h510qko1D8eFEXFDYtRKdiTCovEVWWkwj4r+w8=
+github.com/elastic/beats/v7 v7.0.0-alpha2.0.20241106142216-cf13781a6fca/go.mod h1:AM/24BOEwZhMVAAY1d85Bd1CU6dAZr4XiwWrbIjojn4=
 github.com/elastic/elastic-agent-autodiscover v0.9.0 h1:+iWIKh0u3e8I+CJa3FfWe9h0JojNasPgYIA47gpuuns=
 github.com/elastic/elastic-agent-autodiscover v0.9.0/go.mod h1:5iUxLHhVdaGSWYTveSwfJEY4RqPXTG13LPiFoxcpFd4=
 github.com/elastic/elastic-agent-client/v7 v7.16.0 h1:yKGq2+CxAuW8Kh0EoNl202tqAyQKfBcPRawVKs2Jve0=

testing/benchmark/README.md

@@ -76,43 +76,34 @@ The main commands are:
 Helper commands
 
 - `~/.ssh/id_rsa_terraform`: Generates a new SSH key without passphrase for the worker VMs.
-- `terraform.tfvars`: Copies the examples tfvars and sets the `user_name` var with to `$USER`.
+- `terraform.tfvars`:
+    - Set `TFVARS_SOURCE` to use an explicit profile
+    - Otherwise the `terraform.tfvars.example` file is used
+    - Sets the `USER_NAME` var to `$USER`.
 - `apmbench`: Compiles the `apmbench` binary from the provided location (`APMBENCH_PATH`).
 
-### Override the docker image  and image tag
+### Override Docker Image
 
-Running `make docker-override-committed-version` will create new docker images for `kibana` and `elastic-agent`
-with local `apm` package and `apm-server` and a Terraform variable file. The file named 
-`docker_image.auto.tfvars` contains Terraform Docker image Terraform variables overrides. This file is not 
-overridden automatically, you need to remove it manually if present.
+There are two ways to override the docker images (or tags) that Terraform will provision:
 
-#### Override docker image tag
+1. **Manual**: To use an already existing container, change the values of the `docker_image_override` (or `docker_image_tag_override`) field in the profile defined by `TFVARS_SOURCE` (default is `terraform.tfvars`).
 
-It is possible to override the tag of the docker image that is run in the remote ESS deployment. You can
-specify any of the available tags (such as `8.3.0-SNAPSHOT` or a more specific tag `8.3.0-c655cda8-SNAPSHOT`).
-Alternatively, you can run `make docker-override-committed-version` in your shell, to have use the committed
-tags in the `docker-compose.yml` file in the repository root.
-
-#### Override the docker image
-
-It is also possible to override the docker image to one that is allowed to run in ESS. For more information
-on which repositories can be used, please refer to our internal docs. To override the docker image, you'll need
-to specify the full object of images that is defined in `variables.tf`: `docker_image_override`.
-Alternatively, you can run `make docker-override-committed-version` in your shell, to have use the committed
-tags in the `docker-compose.yml` file in the repository root.
+2. **Automatic**: To encode local changes in `apm-server`, run `make docker-override-committed-version`.
+    - This will build and push a new set of docker images to the internal docker registery.
+    - If `IMAGE_TAG` is not set, the committed tags will be used as defined in `docker-compose.yml` in the repository root.
+    - The file `docker_image.auto.tfvars` contains the Terraform variable overrides for the defined docker images. No need to override them manualy.
+    - This file is not overridden automatically, you need to remove it manually if present.
 
 ### Set APM index shards
 
-By default, the APM indices ship with `number_of_shards` set to `1`. To override this behavior, you can modify the
-`apm_shards` variable and individually set the setting for each of the component templates. See an example of how to
-do that in `terraform.tfvars.example`.
+By default, the APM indices ship with `number_of_shards` set to `1`. To override this behavior, you can modify the `apm_shards` variable and individually set the setting for each of the component templates. See an example of how to do that in `terraform.tfvars.example`.
 
 ### Delete all the APM data streams
 
-`make cleanup-elasticsearch` will delete all the APM data streams. This may be useful in case you'd like to re-run
-the benchmarks without destroying the deployment.
+`make cleanup-elasticsearch` will delete all the APM data streams. This may be useful in case you'd like to re-run the benchmarks without destroying the deployment.
 
 ### Slack reporting
+
 Reporting data is taken from the https://`<replace-with-kibana-benchmark-url>`/app/dashboards#/view/a5bc8390-2f8e-11ed-a369-052d8245fa04?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))
 It's possible to add or modify any metric.
 

testing/infra/terraform/modules/tags/output.tf

@@ -2,7 +2,7 @@ locals {
   tags = {
     "division" : "engineering"
     "org" : "obs"
-    "team" : "apm-server"
+    "team" : "obs-ds-intake-services"
     "project" : var.project
     "build" : var.build
     "ephemeral" : "true"