Filestream returns error when an input with duplicated ID is created

[AndersonQ]

Proposed commit message

The Filestream input now enforces uniqueness of it ID when starting. When the manager is creating an input it checks whether another input with the same ID has already been created, if this is true, then it will log the duplicated input configuration and return an error.

An empty Filestream input ID is a valid ID and treated as any other, so if there are two or more Filestream inputs without an ID, only the first one to be created will run and the others will error.

Filestream inputs now accept allow_deprecated_id_duplication: true to keep the previous behaviour of running multiple inputs with the same ID.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

This is a breaking change, if there are two or more Filestream inputs with the same ID, only the first to be created will run, all others will error on creation.

Author's Checklist

• Test with Elastic-Agent
• Add allow_deprecated_id_duplication and tests for it.

How to test this PR locally

Testing input reload

1. Create the following filebeat.yml and inputs.d/filestream.yml

   filebeat.yml

   filebeat.config.inputs:
     enabled: true
     path: inputs.d/*.yml
     reload.enabled: true
     reload.period: 1s
   
   output.discard.enabled: true
   
   logging:
     level: info
     metrics:
       enabled: false

   inputs.d/filestream.yml

   - type: filestream
     id: duplicated-id
     enabled: true
     paths:
       - /tmp/duplicated-id-1/*.log
   - type: filestream
     id: duplicated-id
     enabled: true
     paths:
       - /tmp/duplicated-id-2/*.log

2. Build filebeat from this PR

3. Start Filebeat

4. Look the logs, you should see the following errors

   error logs

   {
     "log.level": "error",
     "@timestamp": "2024-12-16T18:23:03.539-0500",
     "log.logger": "input",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*InputManager).Create",
       "file.name": "input-logfile/manager.go",
       "file.line": 180
     },
     "message": "filestream input 'duplicated-id' is duplicated: input will NOT start",
     "service.name": "filebeat",
     "input.cfg": "{\n  \"enabled\": true,\n  \"id\": \"duplicated-id\",\n  \"paths\": [\n    \"/tmp/duplicated-id-2/*.log\"\n  ],\n  \"type\": \"filestream\"\n}",
     "ecs.version": "1.6.0"
   }
   

   {
     "log.level": "error",
     "@timestamp": "2024-12-16T18:23:03.539-0500",
     "log.logger": "reload",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload",
       "file.name": "cfgfile/list.go",
       "file.line": 138
     },
     "message": "Error creating runner from config: filestream input with ID 'duplicated-id' already exists, this will lead to data duplication, please use a different ID",
     "service.name": "filebeat",
     "ecs.version": "1.6.0"
   }
   

5. You can also check the IDs and number of running inputs with:

   tiago@millennium-falcon ~ % curl -s localhost:5066/inputs/|jq '.[].id'
   "duplicated-id"
   
   tiago@millennium-falcon ~ % curl -s localhost:5066/inputs/|jq '.[].id'|wc -l
   1
   

Testing Autodiscover

Any autodiscover will do, so we'll use docker on this example for the sake of simplicity

1. Start Filebeat with the following configuration

   filebeat.yml

   filebeat.autodiscover.providers:
     - type: docker
       hints.enabled: true
       hints.default_config:
         type: filestream
         id: docker-id
         prospector:
           scanner:
             fingerprint.enabled: true
             symlinks: true
         file_identity.fingerprint: ~
         paths:
           - /var/lib/docker/containers/${data.container.id}/*.log
   
   output.discard.enabled: true
   
   http.enabled: true
   http.port: 5066

2. Make sure you have at least two docker container running

3. Look at the logs and make sure the following entry appear

   autodiscover log error

   {
     "log.level": "error",
     "@timestamp": "2024-12-18T16:43:14.602-0500",
     "log.logger": "autodiscover",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker",
       "file.name": "autodiscover/autodiscover.go",
       "file.line": 175
     },
     "message": "all new inputs failed to start with a non-retriable error",
     "service.name": "filebeat",
     "error": {
       "message": "Error creating runner from config: ErrNonReloadable: filestream input with ID 'docker-id' already exists, this will lead to data duplication, please use a different ID\nError creating runner from config: ErrNonReloadable: filestream input with ID 'docker-id' already exists, this will lead to data duplication, please use a different ID\nError creating runner from config: ErrNonReloadable: filestream input with ID 'docker-id' already exists, this will lead to data duplication, please use a different ID"
     },
     "ecs.version": "1.6.0"
   }

Testing allow_deprecated_id_duplication

1. Follow any of the above testing procedures, but add allow_deprecated_id_duplication: true to the filestream configuration

2. Look at the logs and make sure the following entry appear

   running duplicated ID error

   {
     "log.level": "error",
     "@timestamp": "2024-12-18T17:09:08.819-0500",
     "log.logger": "input",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*InputManager).Create",
       "file.name": "input-logfile/manager.go",
       "file.line": 184
     },
     "message": "filestream input with ID 'duplicated-id' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input. The input will start only because 'allow_deprecated_id_duplication' is set to true",
     "service.name": "filebeat",
     "ecs.version": "1.6.0"
   }

Related issues

• Closes filestream should not run inputs with duplicated IDs #41938
• Closes Investigate if it is possible to prevent inputs with non-unique IDs from being templated by autodiscovery #41881

## Use cases
## Screenshots

Logs

Warning logged during config validation when 'allow_deprecated_id_duplication' is enabled

{
  "log.level": "warn",
  "@timestamp": "2024-12-18T17:09:08.819-0500",
  "log.logger": "input.filestream",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*config).Validate",
    "file.name": "filestream/config.go",
    "file.line": 150
  },
  "message": "setting `allow_deprecated_id_duplication` will lead to data duplication and incomplete input metrics, it's use is highly discouraged.",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

Error logged by Filestream's input manager when an input has got a duplicated ID and 'allow_deprecated_id_duplication' is enabled

{
  "log.level": "error",
  "@timestamp": "2024-12-18T17:09:08.819-0500",
  "log.logger": "input",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*InputManager).Create",
    "file.name": "input-logfile/manager.go",
    "file.line": 184
  },
  "message": "filestream input with ID 'duplicated-id' already exists, this will lead to data duplication, please use a different ID. Metrics collection has been disabled on this input. The  input will start only because 'allow_deprecated_id_duplication' is set to true",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

Error logged by input reloader (inputs.d directory) when a Filestream input has got duplicated IDs

{
  "log.level": "error",
  "@timestamp": "2024-12-16T18:23:03.539-0500",
  "log.logger": "reload",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload",
    "file.name": "cfgfile/list.go",
    "file.line": 138
  },
  "message": "Error creating runner from config: filestream input with ID 'duplicated-id' already exists, this will lead to data duplication, please use a different ID",
  "service.name": "filebeat",
  "ecs.version": "1.6.0"
}

Error logged by Filestream's input manager when an input has got a duplicated ID

{
  "log.level": "error",
  "@timestamp": "2024-12-16T18:23:03.539-0500",
  "log.logger": "input",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*InputManager).Create",
    "file.name": "input-logfile/manager.go",
    "file.line": 180
  },
  "message": "filestream input 'duplicated-id' is duplicated: input will NOT start",
  "service.name": "filebeat",
  "input.cfg": "{\n  \"enabled\": true,\n  \"id\": \"duplicated-id\",\n  \"paths\": [\n    \"/tmp/duplicated-id-2/*.log\"\n  ],\n  \"type\": \"filestream\"\n}",
  "ecs.version": "1.6.0"
}

Error logged by autodiscover when Filestream inputs with duplicated IDs cannot be created

{
  "log.level": "error",
  "@timestamp": "2024-12-18T16:43:14.602-0500",
  "log.logger": "autodiscover",
  "log.origin": {
    "function": "github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).worker",
    "file.name": "autodiscover/autodiscover.go",
    "file.line": 175
  },
  "message": "all new inputs failed to start with a non-retriable error",
  "service.name": "filebeat",
  "error": {
    "message": "Error creating runner from config: ErrNonReloadable: filestream input with ID 'docker-id' already exists, this will lead to data duplication, please use a different ID\nError creating runner from config: ErrNonReloadable: filestream input with ID 'docker-id' already exists, this will lead to data duplication, please use a different ID\nError creating runner from config: ErrNonReloadable: filestream input with ID 'docker-id' already exists, this will lead to data duplication, please use a different ID"
  },
  "ecs.version": "1.6.0"
}

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @AndersonQ? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[jlind23]

@belimawr assigning this to you as discussed over slack.

[belimawr]

This file was actually renamed from reload_test.go. However because a test is added to reload_test.go, Git got confused and thinks this file is new and the other has been edited.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @AndersonQ? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 41938-filestream-do-not-run-duplicated-id upstream/41938-filestream-do-not-run-duplicated-id
git merge upstream/main
git push upstream 41938-filestream-do-not-run-duplicated-id

[mauri870]

LGTM. I tested the first two cases manually (input reload and autodiscover) and confirmed the log messages appear in the output.

[belimawr]

The failing tests seem to be unrelated, I created a standalone PR (#42213) to fix them, however I also included the fix commit on this PR to unblock it.