reduce transform v0.40.0 regression with reducing structured fields

[pront]

Discussed in #21061

^{Originally posted by matt-simons August 13, 2024}
Hi,

I'm not sure if this is an issue or intended breaking change to the reduce transform but I would really appreciate some guidance on how to continue supporting the following:

transforms:
  reduce_scalyr:
    type: reduce
    inputs:
    - my_input
    group_by:
    - session
    expire_after_ms: 2000
    max_events: 500
    merge_strategies:
      events: array
tests:
- name: map_reduce_scalyr.no_sev
  inputs:
  - type: vrl
    insert_at: reduce_scalyr
    source: |
      . = {
        "events": {
          "attrs": {
            "msg": "foo"
          },
          "sev": 3,
          "ts": "1602349656452332000"
        },
        "session": "session",
        "sessionInfo": {
          "kubernetes.cluster": "cluster",
          "kubernetes.container.name": "container",
          "kubernetes.namespace.name": "namespace",
          "kubernetes.pod.name": "pod",
          "logfile": "p",
          "serverHost": "n"
        },
        "token": "token"
      }
  - type: vrl
    insert_at: reduce_scalyr
    source: |
      . = {
        "events": {
          "attrs": {
            "msg": "bar"
          },
          "sev": 3,
          "ts": "1602349656460000000"
        },
        "session": "session",
        "sessionInfo": {
          "kubernetes.cluster": "cluster",
          "kubernetes.container.name": "container",
          "kubernetes.namespace.name": "namespace",
          "kubernetes.pod.name": "pod",
          "logfile": "p",
          "serverHost": "n"
        },
        "token": "token"
      }
  outputs:
  - extract_from: reduce_scalyr
    conditions:
    - type: vrl
      source: |
        assert_eq!(., {
          "events": [
            {
              "ts": "1602349656452332000",
              "sev": 3,
              "attrs": {
                "msg": "foo",
              },
            },
            {
              "ts": "1602349656460000000",
              "sev": 3,
              "attrs": {
                "msg": "bar",
              },
            },
          ],
          "session": "session",
          "sessionInfo": {
            "kubernetes.cluster": "cluster",
            "kubernetes.container.name": "container",
            "kubernetes.namespace.name": "namespace",
            "kubernetes.pod.name": "pod",
            "logfile": "p",
            "serverHost": "n"
          },
          "token": "token",
        })

In Vector 0.39 and previous versions this test passed and meant we could aggregate events before sending them to a http sink (https://app.scalyr.com/help/api#addEvents).

However in 0.40 I think #20800 introduced breaking changes to this behaviour. The aggregated event now looks like the following:

{
  "events": {
    "attrs": {
      "msg": "foo"
    },
    "sev": 6,
    "ts": "1602349656452332000"
  },
  "session": "session",
  "sessionInfo": {
    "logfile": "p",
    "serverHost": "n"
  },
  "token": "token"
}

instead of

{
  "events": [
    {
      "attrs": {
        "msg": "foo"
      },
      "sev": 3,
      "ts": "1602349656452332000"
    },
    {
      "attrs": {
        "msg": "bar"
      },
      "sev": 3,
      "ts": "1602349656460000000"
    }
  ],
  "session": "session",
  "sessionInfo": {
    "kubernetes.cluster": "cluster",
    "kubernetes.container.name": "container",
    "kubernetes.namespace.name": "namespace",
    "kubernetes.pod.name": "pod",
    "logfile": "p",
    "serverHost": "n"
  },
  "token": "token"
}

• The events objects are no longer merged into an array
• The sev field is summed
• The fields which contain a "." are discarded

Is this the new intended behaviour or a bug?

If this is intended how can I retain the previous behaviour?

[Ilmarii]

I also encountered this problem. The discarding of fields containing "." was very unexpected.

[jszwedko]

I also encountered this problem. The discarding of fields containing "." was very unexpected.

Can you explain a bit more what you were running into? I want to make sure it is the same issue. Supplying before/after examples with config (like in the original issue report) would help us verify if it is the same.