Issue troubleshooting an attribute not inserting

[arnaudvre]

The real issue I'm facing is that I can't find a way to troubleshoot this further, I've enabled detailed logging but it's not giving me any information on how my journald logs are being processed. I'm expecting to be able to access the value for one of the key/value pairs(_HOSTNAME: server1) in the json data for a given log and use it to add a custom attribute (hostname: server1) This is what my configuration looks like currently:

receivers:
  journald:

processors:
  resource:
    attributes:
      - action: insert
        key: loki.format
        value: json
      - action: insert
        key: environment
        value: dev
      - action: insert
        key: loki.attribute.labels
        value: hostname
  attributes:
    actions:
      - action: insert
        key: hostname
        from_attribute: body._HOSTNAME

exporters:
  loki:
    endpoint: "https://loki.net/loki/api/v1/push"
    sending_queue:
      num_consumers: 10
      queue_size: 50000
    headers:
      "X-Scope-OrgID": "xxx"
    tls:
      ca_file: "xxx"
      cert_file: "xxx"
      key_file: "xxx"
  file:
    path: /home/otelcol-contrib/temp.log
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [resource,attributes]
      exporters: [debug]
  telemetry:
    logs:
      encoding: json  

Any tips on how to troubleshoot the processing of my logs would be appreciated, in this case being able to see what elements are available and under what namespace would be helpful, is body._HOSTNAME even being looked up and just missed because I don't have the correct namespace?

When written out to file the logs look something like the below, logs viewed in Grafana using Loki as the datasource are also not how I'd like them to appear, all of my fields like _HOSTNAME and MESSAGE appear nested under a body element but this is secondary to my main question. Below this I'll include a screenshot of the messages in Grafana to show what I mean just in case someone has any ideas.

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "loki.format",
            "value": {
              "stringValue": "json"
            }
          },
          {
            "key": "environment",
            "value": {
              "stringValue": "dev"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1721831731222344000",
              "observedTimeUnixNano": "1721831731422059659",
              "body": {
                "kvlistValue": {
                  "values": [
                    {
                      "key": "_SELINUX_CONTEXT",
                      "value": {
                        "stringValue": "unconfined\n"
                      }
                    },
                    {
                      "key": "_SYSTEMD_SLICE",
                      "value": {
                        "stringValue": "system.slice"
                      }
                    },
                    {
                      "key": "_HOSTNAME",
                      "value": {
                        "stringValue": "hostname-dev"
                      }
                    },
                    {
                      "key": "_SYSTEMD_CGROUP",
                      "value": {
                        "stringValue": "/system.slice/otelcol-contrib.service"
                      }
                    },
                    {
                      "key": "_CAP_EFFECTIVE",
                      "value": {
                        "stringValue": "0"
                      }
                    },
                    {
                      "key": "_PID",
                      "value": {
                        "stringValue": "2262330"
                      }
                    },
                    {
                      "key": "SYSLOG_FACILITY",
                      "value": {
                        "stringValue": "3"
                      }
                    },
                    {
                      "key": "_EXE",
                      "value": {
                        "stringValue": "/usr/bin/otelcol-contrib"
                      }
                    },
                    {
                      "key": "PRIORITY",
                      "value": {
                        "stringValue": "6"
                      }
                    },
                    {
                      "key": "SYSLOG_IDENTIFIER",
                      "value": {
                        "stringValue": "otelcol-contrib"
                      }
                    },
                    {
                      "key": "_TRANSPORT",
                      "value": {
                        "stringValue": "stdout"
                      }
                    },
                    {
                      "key": "_SYSTEMD_UNIT",
                      "value": {
                        "stringValue": "otelcol-contrib.service"
                      }
                    },
                    {
                      "key": "_UID",
                      "value": {
                        "stringValue": "997"
                      }
                    },
                    {
                      "key": "_SYSTEMD_INVOCATION_ID",
                      "value": {
                        "stringValue": "231cd2139b2f4664b755d43b4690aef0"
                      }
                    },
                    {
                      "key": "_GID",
                      "value": {
                        "stringValue": "997"
                      }
                    },
                    {
                      "key": "_STREAM_ID",
                      "value": {
                        "stringValue": "8a194d71567d4ae4b4752a9f132eb336"
                      }
                    },
                    {
                      "key": "_MACHINE_ID",
                      "value": {
                        "stringValue": "5fca68f59b9583a46efa7b9f6697ef74"
                      }
                    },
                    {
                      "key": "_BOOT_ID",
                      "value": {
                        "stringValue": "f0d1b58e375844c3961c527ffc365e6c"
                      }
                    },
                    {
                      "key": "_CMDLINE",
                      "value": {
                        "stringValue": "/usr/bin/otelcol-contrib --config=/etc/otelcol- contrib/config.yaml"
                      }
                    },
                    {
                      "key": "__MONOTONIC_TIMESTAMP",
                      "value": {
                        "stringValue": "598265666336"
                      }
                    },
                    {
                      "key": "_COMM",
                      "value": {
                        "stringValue": "otelcol-contrib"
                      }
                    },
                    {
                      "key": "MESSAGE",
                      "value": {
                        "stringValue": "2024-07-24T14:35:31.222Z\tinfo\[email protected]/service.go:270\tShutdown complete."
                      }
                    },
                    {
                      "key": "__CURSOR",
                      "value": {
                        "stringValue": "s=92ed14a0f4d04f8f9b315bb73d09e814;i=20f19;b=f0d1b58e375844c3961c527ffc365e6c;m=8b4b699b20;t=61dff30b3d748;x=3a5de1616420ef73"
                      }
                    }
                  ]
                }
              },
              "traceId": "",
              "spanId": ""
            },
            {
              "timeUnixNano": "1721831731241174000",
              "observedTimeUnixNano": "1721831731422135940",
              "body": {
                "kvlistValue": {
                  "values": [
                    {
                      "key": "UNIT",
                      "value": {
                        "stringValue": "otelcol-contrib.service"
                      }
                    },
                    {
                      "key": "CODE_FUNC",
                      "value": {
                        "stringValue": "unit_log_success"
                      }
                    },
                    {
                      "key": "_MACHINE_ID",
                      "value": {
                        "stringValue": "5fca68f59b9583a46efa7b9f6697ef74"
                      }
                    },
                    {
                      "key": "_SELINUX_CONTEXT",
                      "value": {
                        "stringValue": "unconfined\n"
                      }
                    },
                    {
                      "key": "_SYSTEMD_SLICE",
                      "value": {
                        "stringValue": "-.slice"
                      }
                    },
                    {
                      "key": "_UID",
                      "value": {
                        "stringValue": "0"
                      }
                    },
                    {
                      "key": "TID",
                      "value": {
                        "stringValue": "1"
                      }
                    },
                    {
                      "key": "MESSAGE_ID",
                      "value": {
                        "stringValue": "7ad2d189f7e94e70a38c781354912448"
                      }
                    },
                    {
                      "key": "_TRANSPORT",
                      "value": {
                        "stringValue": "journal"
                      }
                    },
                    {
                      "key": "SYSLOG_FACILITY",
                      "value": {
                        "stringValue": "3"
                      }
                    },
                    {
                      "key": "_CAP_EFFECTIVE",
                      "value": {
                        "stringValue": "1ffffffffff"
                      }
                    },
                    {
                      "key": "_CMDLINE",
                      "value": {
                        "stringValue": "/lib/systemd/systemd autoinstall --system --deserialize 41"
                      }
                    },
                    {
                      "key": "SYSLOG_IDENTIFIER",
                      "value": {
                        "stringValue": "systemd"
                      }
                    },
                    {
                      "key": "__MONOTONIC_TIMESTAMP",
                      "value": {
                        "stringValue": "598265685166"
                      }
                    },
                    {
                      "key": "_BOOT_ID",
                      "value": {
                        "stringValue": "f0d1b58e375844c3961c527ffc365e6c"
                      }
                    },
                    {
                      "key": "_SOURCE_REALTIME_TIMESTAMP",
                      "value": {
                        "stringValue": "1721831731241074"
                      }
                    },
                    {
                      "key": "__CURSOR",
                      "value": {
                        "stringValue": "s=92ed14a0f4d04f8f9b315bb73d09e814;i=20f1a;b=f0d1b58e375844c3961c527ffc365e6c;m=8b4b69e4ae;t=61dff30b420d6;x=d6baded09f1b919b"
                      }
                    },
                    {
                      "key": "_EXE",
                      "value": {
                        "stringValue": "/usr/lib/systemd/systemd"
                      }
                    },
                    {
                      "key": "_PID",
                      "value": {
                        "stringValue": "1"
                      }
                    },
                    {
                      "key": "_SYSTEMD_CGROUP",
                      "value": {
                        "stringValue": "/init.scope"
                      }
                    },
                    {
                      "key": "PRIORITY",
                      "value": {
                        "stringValue": "6"
                      }
                    },
                    {
                      "key": "_SYSTEMD_UNIT",
                      "value": {
                        "stringValue": "init.scope"
                      }
                    },
                    {
                      "key": "_GID",
                      "value": {
                        "stringValue": "0"
                      }
                    },
                    {
                      "key": "INVOCATION_ID",
                      "value": {
                        "stringValue": "231cd2139b2f4664b755d43b4690aef0"
                      }
                    },
                    {
                      "key": "_HOSTNAME",
                      "value": {
                        "stringValue": "hostname-dev"
                          }
                    },
                    {
                      "key": "_COMM",
                      "value": {
                        "stringValue": "systemd"
                      }
                    },
                    {
                      "key": "CODE_LINE",
                      "value": {
                        "stringValue": "5553"
                      }
                    },
                    {
                      "key": "MESSAGE",
                      "value": {
                        "stringValue": "otelcol-contrib.service: Deactivated successfully."
                      }
                    },
                    {
                      "key": "CODE_FILE",
                      "value": {
                        "stringValue": "src/core/unit.c"
                      }
                    }
                  ]
                }
              }
              "traceId": "",
              "spanId": ""
            }
          ]
        }
      ]
    }
  ]
}

Logs as they appear in Grafana

[arnaudvre]

See open-telemetry/opentelemetry-collector-contrib#34808