Parse Json with null values

[alex-dengx]

A note for the community

While my inputs from kafka is like this:

{
    "bytes_read": 4096,
    "company_id": "65b8b9cb881d8861bc00bf82",
    "ctc_version": "2.13.8.4",
    "datetime": 1719967862303,
    "entropy": 0,
    "event_name": "file_read",
    "event_version": 1,
    "extra_keys": null,
    "extra_values": null,
    "hardware_id": "9ADDD4CC-217A-11B2-A85C-F4CA8A22C9AE-9C2DCD0F83BA",
    "is_remote": 0,
    "machine_name": "CN-PC2JD8BZ",
    "os_family": "windows",
    "os_platform": "x64",
    "os_type": "client",
    "os_version": "Windows 11",
    "path": "c:\\users\\vickyyang\\appdata\\local\\microsoft\\office\\16.0\\wef\\resources\\ol7jgfcq1kr2jzrs1fo0hw==",
    "pid": 15328,
    "process_md5": "e360fbe011e6d4a5a304f32b50ad66dd",
    "process_path": "c:\\program files\\microsoft office\\root\\office16\\outlook.exe",
    "process_sha": "951fae7f8bd49a63a76f18954ea2ae72f548de01113c0df1adbcbc23cb419702",
    "product_version": "7.9.9.370",
    "type": 999,
    "type_sz": "Unknown",
    "user_name": ""
}

Problem

the key extra_keys and extra_values are all null , if I don't filter these keys. I will get an error :

2024-07-03T01:12:39.359246Z ERROR transform{component_kind="transform" component_id=json_parser component_type=remap}: vector::internal_events::remap: Mapping failed with event. error="function call error for "parse_json" at (4:25): expected string, got null" error_type="conversion_failed" stage="processing" internal_log_rate_limit=true

Configuration

sources:
   kafka_consumer:
    type: "kafka"
    bootstrap_servers: "10.173.67.103:9092,10.173.67.104:9092,10.173.67.105:9092"
    group_id: "deloitte_logstash"
    auto_offset_reset: "earliest"
    topics:
      - test
    decoding:
      codec: "json"
    key_field: "kafka_key"

# Parse Syslog logs
# See the Vector Remap Language reference for more info: https://vrl.dev

transforms:
  remove_null_filed:
    type: "remap"
    inputs : ["kafka_consumer"]
    source: |
        if exists(.extra_keys) && .extra_keys == null{
          del(.extra_keys)
        }

  json_parser:
    type: "remap"
    inputs: ["remove_null_filed"]
    drop_on_error: false
    source : |
      . = parse_json!(.message)


# Print parsed logs to stdout
sinks:
  elastic_sink:
    type: "elasticsearch"
    inputs: ["json_parser"]
    api_version: "v8"
    auth:
      strategy: "basic"
      user: "elastic"
      password: "mypassword"
    bulk:
      index: "{{ event_name }}_logs"
    endpoints: ["https://10.173.67.100:9200","https://10.173.67.101:9200", "https://10.173.67.102:9200"]
    tls:
      ca_file: "/opt/logstash/config/ca.pem"
      verify_certificate: false

#  print:
#    type: "console"
#    inputs: ["json_parser"]
#    encoding:
#      codec: "json"
#      json:
#        pretty: true

# Vector's GraphQL API (disabled by default)
# Uncomment to try it out with the `vector top` command or
# in your browser at http://localhost:8686
api:
  enabled: true
  address: "127.0.0.1:8686"

Version

vector 0.39.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[alex-dengx]

another inputs is like this:

 {
        "ctc_version": "2.13.4.3",
        "process_sha": "9785001b0dcf755eddb8af294a373c0b87b2498660f724e76c4d53f9c217c7a3",
        "extra_keys": [],
        "type_sz": "Unknown",
        "machine_name": "1022H2X64-N74",
        "event_version": 1,
        "entropy": 0,
        "event_name": "file_read",
        "pid": 8756,
        "process_md5": "2e5a8590cf6848968fc23de3fa1e25f1",
        "user_name": "",
        "process_path": "c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe",
        "is_remote": 0,
        "datetime": 1715024308353,
        "path": "c:\\programdata\\readfileaf8f7981-30ed-4661-8ced-a34ad484c872.txt",
        "bytes_read": 4096,
        "extra_values": [],
        "os_version": "Windows 10",
        "type": 999,
        "company_id": "5b9bc2701da197f07a8b4567",
        "hardware_id": "4D8E0E42-22F0-2D94-6823-C3863931877A-0050568E1657",
        "os_family": "windows",
        "os_platform": "x64",
        "os_type": "client",
        "product_version": "7.9.11.406"
    }

so you can see that extra_keys are set to [] .

[jszwedko]

Discussing this in #20770