v0.40 Reduce transform breaking changes

[matt-simons]

Hi,

I'm not sure if this is an issue or intended breaking change to the reduce transform but I would really appreciate some guidance on how to continue supporting the following:

transforms:
  reduce_scalyr:
    type: reduce
    inputs:
    - my_input
    group_by:
    - session
    expire_after_ms: 2000
    max_events: 500
    merge_strategies:
      events: array
tests:
- name: map_reduce_scalyr.no_sev
  inputs:
  - type: vrl
    insert_at: reduce_scalyr
    source: |
      . = {
        "events": {
          "attrs": {
            "msg": "foo"
          },
          "sev": 3,
          "ts": "1602349656452332000"
        },
        "session": "session",
        "sessionInfo": {
          "kubernetes.cluster": "cluster",
          "kubernetes.container.name": "container",
          "kubernetes.namespace.name": "namespace",
          "kubernetes.pod.name": "pod",
          "logfile": "p",
          "serverHost": "n"
        },
        "token": "token"
      }
  - type: vrl
    insert_at: reduce_scalyr
    source: |
      . = {
        "events": {
          "attrs": {
            "msg": "bar"
          },
          "sev": 3,
          "ts": "1602349656460000000"
        },
        "session": "session",
        "sessionInfo": {
          "kubernetes.cluster": "cluster",
          "kubernetes.container.name": "container",
          "kubernetes.namespace.name": "namespace",
          "kubernetes.pod.name": "pod",
          "logfile": "p",
          "serverHost": "n"
        },
        "token": "token"
      }
  outputs:
  - extract_from: reduce_scalyr
    conditions:
    - type: vrl
      source: |
        assert_eq!(., {
          "events": [
            {
              "ts": "1602349656452332000",
              "sev": 3,
              "attrs": {
                "msg": "foo",
              },
            },
            {
              "ts": "1602349656460000000",
              "sev": 3,
              "attrs": {
                "msg": "bar",
              },
            },
          ],
          "session": "session",
          "sessionInfo": {
            "kubernetes.cluster": "cluster",
            "kubernetes.container.name": "container",
            "kubernetes.namespace.name": "namespace",
            "kubernetes.pod.name": "pod",
            "logfile": "p",
            "serverHost": "n"
          },
          "token": "token",
        })

In Vector 0.39 and previous versions this test passed and meant we could aggregate events before sending them to a http sink (https://app.scalyr.com/help/api#addEvents).

However in 0.40 I think #20800 introduced breaking changes to this behaviour. The aggregated event now looks like the following:

{
  "events": {
    "attrs": {
      "msg": "foo"
    },
    "sev": 6,
    "ts": "1602349656452332000"
  },
  "session": "session",
  "sessionInfo": {
    "logfile": "p",
    "serverHost": "n"
  },
  "token": "token"
}

instead of

{
  "events": [
    {
      "attrs": {
        "msg": "foo"
      },
      "sev": 3,
      "ts": "1602349656452332000"
    },
    {
      "attrs": {
        "msg": "bar"
      },
      "sev": 3,
      "ts": "1602349656460000000"
    }
  ],
  "session": "session",
  "sessionInfo": {
    "kubernetes.cluster": "cluster",
    "kubernetes.container.name": "container",
    "kubernetes.namespace.name": "namespace",
    "kubernetes.pod.name": "pod",
    "logfile": "p",
    "serverHost": "n"
  },
  "token": "token"
}

• The events objects are no longer merged into an array
• The sev field is summed
• The fields which contain a "." are discarded

Is this the new intended behaviour or a bug?

If this is intended how can I retain the previous behaviour?

[jszwedko]

Thanks for this report @matt-simons . I think this is an unintended regression. We are looking into it.

[matt-simons]

Thanks @jszwedko for the v0.40.1 fix, much appreciated.

My tests are passing however I'm now seeing the following in our logs:

2024-08-27T13:36:51.993121Z ERROR transform{component_kind="transform" component_id=kubernetes_logs_multiline_reduce component_type=reduce}: vector::internal_events::reduce: Event field could not be reduced. path=KeyString("kubernetes.pod.labels.controller-revision-hash") error=InvalidPathSyntax { path: "kubernetes.pod.labels.controller-revision-hash" } error_type="condition_failed" stage="processing" internal_log_rate_limit=true

We don't specify any merge strategies for the kubernetes object in our logs. I'm wondering if it may be due to the dashes? Adding a discard strategy to kubernetes at the top level does not seem to fix this either.

[jszwedko]

Bah, thanks @matt-simons . We'll look into that.

[pront]

Thank you for sharing the above @matt-simons. I suppose we already hit this TODO 😓:

vector/src/transforms/reduce/transform.rs

Line 82 in 06ad674

// Technically we need to quote any special characters (like `-` or `*` or ` `).

I will take a look shortly.

[pront]

Spawned an issue and fixed by #21067.