[Auto Import] Fix cases where LLM generates incorrect array field access

[ilyannn]

Release Note

Fixes cases where LLM was likely to generate invalid processors containing array access in Automatic Import.

Context

Previously, it happened from time to time that the LLM attempts to add related fields or apply categorization conditions that use a field, path to which goes through an array. Here's an example output of the review step in the related chain:

output:
  - field: related.ip
    value_field: source.ip
  - field: related.user
    value_field: user.id
  - field: related.user
    value_field: user.email
  - field: related.hosts
    value_field: host.hostname
  - field: related.user
    value_field: ai_falcon_202410141910.audit.event.AuditKeyValues.[?Key=='assigned_to_uid'].ValueString | [0] 

(check the original event)

{
    "metadata": {
        "customerIDString": "8f69fe9e-b995-4204-95ad-44f9bcf75b6b",
        "offset": 10,
        "eventType": "UserActivityAuditEvent",
        "eventCreationTime": 1581603262000,
        "version": "1.0"
    },
    "event": {
        "UserId": "[email protected]",
        "UserIp": "192.168.6.8",
        "OperationName": "detection_update",
        "ServiceName": "detections",
        "AuditKeyValues": [
            {
                "Key": "detection_id",
                "ValueString": "ldt:5a6fd0b7347440cd74cb84855a8aee18:17180539745"
            },
            {
                "Key": "new_state",
                "ValueString": "in_progress"
            },
            {
                "Key": "assigned_to",
                "ValueString": "First Last"
            },
            {
                "Key": "assigned_to_uid",
                "ValueString": "[email protected]"
            }
        ],
        "UTCTimestamp": 1581603262
    }
}

(check the event as seen by the LLM)

 {
    "@timestamp": "2020-02-13T14:14:22.000Z",
    "ecs": {
      "version": "8.11.0"
    },
    "related": {
      "user": [
        "[email protected]"
      ],
      "ip": [
        "192.168.6.8"
      ]
    },
    "organization": {
      "id": "8f69fe9e-b995-4204-95ad-44f9bcf75b6b"
    },
    "ai_falcon_202410141910": {
      "audit": {
        "event": {
          "UTCTimestamp": 1581603262,
          "ServiceName": "detections",
          "AuditKeyValues": [
            {
              "ValueString": "ldt:5a6fd0b7347440cd74cb84855a8aee18:17180539745",
              "Key": "detection_id"
            },
            {
              "ValueString": "in_progress",
              "Key": "new_state"
            },
            {
              "ValueString": "First Last",
              "Key": "assigned_to"
            },
            {
              "ValueString": "[email protected]",
              "Key": "assigned_to_uid"
            }
          ]
        },
        "metadata": {
          "eventType": "UserActivityAuditEvent",
          "offset": 10,
          "version": "1.0",
          "eventCreationTime": 1581603262000
        }
      }
    },
    "source": {
      "ip": "192.168.6.8"
    },
    "event": {
      "action": "detection_update",
      "category": [
        "intrusion_detection"
      ],
      "type": [
        "info"
      ]
    },
    "user": {
      "id": "[email protected]"
    },
    "tags": [
      "_geoip_database_unavailable_GeoLite2-City.mmdb",
      "_geoip_database_unavailable_GeoLite2-ASN.mmdb",
      "_geoip_database_unavailable_GeoLite2-City.mmdb",
      "_geoip_database_unavailable_GeoLite2-ASN.mmdb"
    ]
  },

The problem is that such an access is invalid and leads to an immediate error (key part highlighted):

Even including explicit instructions to avoid brackets or an array access did not seem enough, as the LLM would try to use a different syntax, owing to the aggressiveness of our review instructions:

output:
  - field: related.ip
    value_field: source.ip
  - field: related.user
    value_field: user.id
  - field: related.user
    value_field: user.email
  - field: related.hosts
    value_field: host.hostname
  - field: related.user
    value_field: ai_falcon_202410141910.audit.event.AuditKeyValues.ValueString 

The suggested solution is to remove all arrays from the information shown to the LLM in the related chain. This guarantees that no illegal access will ever be attempted.

Summary

• Introduces a utility function to remove all arrays from a JSON object.
• Applies this function for all LLM calls in the related chain.
• Modifies the prompts of related and categorization chain to skip the arrays as well.

Testing

• Regular main with these changes: ai_falcon_202410142221-1.0.0.zip
• Modified main where 100 samples are given to the chains with these changes: ai_falcon_202410141910-1.0.0.zip

(check the event as seen by the LLM now)

{
    "@timestamp": "2020-02-13T14:14:22.000Z",
    "ecs": {
      "version": "8.11.0"
    },
    "related": {},
    "ai_falcon_202410141910": {
      "audit": {
        "event": {
          "UTCTimestamp": 1581603262,
          "OperationName": "detection_update",
          "ServiceName": "detections"
        },
        "metadata": {
          "customerIDString": "8f69fe9e-b995-4204-95ad-44f9bcf75b6b",
          "offset": 10,
          "version": "1.0",
          "eventCreationTime": 1581603262000
        }
      }
    },
    "source": {
      "ip": "192.168.6.8"
    },
    "event": {
      "action": "UserActivityAuditEvent"
    },
    "user": {
      "id": "[email protected]"
    }
  },

For maintainers

• This will appear in the Release Notes and follow the guidelines

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[bhapas]

How are the custom fields with an array of ip or hosts or users getting into related fields.
We will never be able to put them into the related fields , right?
For example:

{
some_user: [ 'john' , 'smith' ],
some_host: [ 'machine1' , 'machine2']
}

[ilyannn]

How are the custom fields with an array of ip or hosts or users getting into related fields. We will never be able to put them into the related fields , right? For example:

{
some_user: [ 'john' , 'smith' ],
some_host: [ 'machine1' , 'machine2']
}

I assume they get there with some custom Painless code, which we can also add in the future. I actually don't know – maybe it works in this example; we'll need to check.

But if there is a dictionary inside some_user then I don't think it's simple.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4e4cdc9

Metrics [docs]

✅ unchanged

History

• 💚 Build #242461 succeeded bbcf82b
• 💚 Build #242460 succeeded e95896d

[bhapas]

Can merge this now. Will need to work on the fields inside arrays for related graph in a later PR.

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11348150567

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation