[Fleet] Update Fleet's custom ingest pipeline names to avoid collisions + add descriptions to each pipeline

[kpollich]

Summary

Closes #175254
Ref #168019
Ref #170270

In 8.12.0, Fleet unintentionally shipped a breaking change in #170270 for APM users who make use of a custom traces-apm data stream. If a user had previously defined this ingest pipeline to customize documents ingested for the traces-apm data stream (defined here, then they would unexpectedly see that pipeline called when documents were ingested to the traces-apm.rum and traces-apm.sampled datastreams as well.

This PR addresses this collision by adding a .integration suffix to the "package level" ingest pipeline introduced in 8.12.0.

So, in 8.12.0 a processor would be defined as such on the traces-apm.rum or traces-apm.sampled ingest pipeline

{
  "pipeline": {
    "name": "traces-apm@custom",
    "ignore_missing_pipeline": true,
  }
},

This PR replaces the pipeline with one that looks as follows:

{
  "pipeline": {
    "name": "traces-apm.integration@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration"
  }
},

To be clear: this is a breaking change if you have defined the traces-apm@custom integration on 8.12. In 8.12.1, it will no longer be called for documents ingested to the traces-apm, traces-apm.rum, or traces-apm.sampled data streams. You will need to rename your pipeline to traces-apm.integration@custom to preserve this behavior.

This change also applies to logs-elastic_agent.* ingest pipelines. See this comment for more information.

There is still technically room for a collision, though it's unlikely, if the data stream name is package. This will be handled by a package spec validation proposed in elastic/package-spec#699.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[juliaElastic]

LGTM

[nchaulet]

Looks like there is some integration tests failing otherwise 🚀, did we validated there is not dataset with {packagename}.package currently existing?

[kpollich]

@nchaulet There is a logs-system_audit.package data stream here 😞:

https://github.com/elastic/integrations/blob/main/packages/system_audit/data_stream/package/manifest.yml

This means we do wind up with a collision, e.g.

{
  "pipeline": {
    "name": "logs-system_audit.package@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `logs` defined by the `system_audit` integration"
  }
},
{
  "pipeline": {
    "name": "logs-system_audit.package@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for the `system_audit.package` dataset"
  }
}

Not sure what to do about this. We could document the known collision and reach out to the SEI team about renaming this data stream 🤷

I'll bring it up in elastic/package-spec#699.

[kpollich]

I've pushed f878ae9 which replaces the .package suffix with .integration instead to avoid the conflict.

[kpollich]

Following that commit here's what some relevant ingest pipelines look like

// traces-apm
{
  "pipeline": {
    "name": "global@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Global pipeline for all data streams"
  }
},
{
  "pipeline": {
    "name": "traces@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces`"
  }
},
{
  "pipeline": {
    "name": "traces-apm.integration@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration"
  }
},
{
  "pipeline": {
    "name": "traces-apm@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for the `apm` dataset"
  }
}

// traces-apm.rum
{
  "pipeline": {
    "name": "global@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Global pipeline for all data streams"
  }
},
{
  "pipeline": {
    "name": "traces@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces`"
  }
},
{
  "pipeline": {
    "name": "traces-apm.integration@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration"
  }
},
{
  "pipeline": {
    "name": "traces-apm.rum@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for the `apm.rum` dataset"
  }
}

// traces-apm.sampled
{
  "pipeline": {
    "name": "global@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Global pipeline for all data streams"
  }
},
{
  "pipeline": {
    "name": "traces@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces`"
  }
},
{
  "pipeline": {
    "name": "traces-apm.integration@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `traces` defined by the `apm` integration"
  }
},
{
  "pipeline": {
    "name": "traces-apm.sampled@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for the `apm.sampled` dataset"
  }
}

// logs-system_audit.package
{
  "pipeline": {
    "name": "global@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Global pipeline for all data streams"
  }
},
{
  "pipeline": {
    "name": "logs@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `logs`"
  }
},
{
  "pipeline": {
    "name": "logs-system_audit.integration@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for all data streams of type `logs` defined by the `system_audit` integration"
  }
},
{
  "pipeline": {
    "name": "logs-system_audit.package@custom",
    "ignore_missing_pipeline": true,
    "description": "[Fleet] Pipeline for the `system_audit.package` dataset"
  }
}

[kpollich]

@elasticmachine merge upstream

[kpollich]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 2e89a27

Failed CI Steps

• FTR Configs #1

Test Failures

• [job] [logs] FTR Configs #1 / Observability Rules Rules Endpoints Custom Threshold rule - CUSTOM_EQ - AVG - BYTES - FIRED Rule creation should set correct information in the alert document

Metrics [docs]

✅ unchanged

History

• 💔 Build #189105 failed 4a1a671
• 💔 Build #189093 failed f878ae9
• 💔 Build #189020 failed 479265c

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kpollich

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.12

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation