[Security Solution][DQD] add nested field items for incompatible and …

[kapral18]

…same family fields

Addresses #184037

• Add incompatibleFieldItems and sameFamilyFieldItems as nested fields with required attributes.

Steps to verify the change:

1. Bootup PR branch with local es + kibana
2. Open Kibana DevTools
3. Call GET .kibana-data-quality-dashboard-results-default/_mapping
4. Verify existence of properly nested incompatibleFieldItems and sameFamilyFieldItems new fields

[elasticmachine]

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[angorayc]

/ci

[angorayc]

@elasticmachine merge upstream

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 4d47d27

Failed CI Steps

• :pipeline:🪄 Run Dynamic Pipeline: .buildkite/scripts/pipelines/pull_request/pipeline.sh

Metrics [docs]

✅ unchanged

History

• 💔 Build #213629 failed b43912a
• 💔 Build #213609 failed b43912a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kapral18

[angorayc]

From this mapping I can see we have expectedValue and actualValue

Please correct me I am wrong, it seemed that we miss the information for expectedType and actualType

In the incompatible field, it could be incompatible mapping or incompatible values

To create an index contains incompatible field types:

PUT auditbeat-custom-index-1

PUT auditbeat-custom-index-1/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "event.category": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
}

POST auditbeat-custom-index-1/_doc
{
  "@timestamp": "2024-06-03T09:41:49.668Z",
  "host": {
    "name": "foo"
  },
  "event": {
    "category": "an_invalid_category"
  },
  "some.field": "this",
  "source": {
    "port": 90210,
    "ip": "10.1.2.3"
  }
}

POST auditbeat-custom-index-1/_doc
{
  "@timestamp": "2024-06-03T09:42:22.123Z",
  "host": {
    "name": "bar"
  },
  "event": {
    "category": "an_invalid_category"
  },
  "some.field": "space",
  "source": {
    "port": 867,
    "ip": "10.9.8.7"
  }
}

POST auditbeat-custom-index-1/_doc
{
  "@timestamp": "2024-06-03T09:43:35.456Z",
  "host": {
    "name": "baz"
  },
  "event": {
    "category": "theory"
  },
  "some.field": "for",
  "source": {
    "port": 5,
    "ip": "10.4.6.6"
  }
}

POST auditbeat-custom-index-1/_doc
{
  "@timestamp": "2024-06-03T09:44:36.700Z",
  "host": {
    "name": "@baz"
  },
  "event": {
    "category": "malware"
  },
  "some.field": "rent",
  "source": {
    "port": 309,
    "ip": "10.1.1.1"
  }
}

To create an index with incompatible value:

PUT auditbeat-custom-index-2

PUT auditbeat-custom-index-2/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "event.category": {
      "type": "keyword",
      "ignore_above": 1024
    }
  }
}

POST auditbeat-custom-index-2/_doc
{
  "@timestamp": "2024-06-03T09:41:49.668Z",
  "host": {
    "name": "foo"
  },
  "event": {
    "category": "an_invalid_category"
  },
  "some.field": "this",
  "source": {
    "port": 90210,
    "ip": "10.1.2.3"
  }
}

[angorayc]

Here is a PR explains the basic functionality of data quality dashboard: #150063
Test plan: https://docs.google.com/document/d/14oTLA85lkpHdjOTm0FG03Vzpsdi5iMRHQMhBOlc6eTI/edit

[semd]

@angorayc The plan was to use the reason field to know if the incompatibility comes from the value or the mapping, and use the expectedValue and actualValue accordingly.

incompatibleFields: [
  {
    fieldName: 'agent.type'
    expectedValue: 'keyword',
    actualValue: 'text'
    description: 'Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.'
    reason: 'mapping'
  },
  {
    fieldName: 'event.category'
    expectedValue: 'api,authentication,configuration,database,driver,email,file,host,iam,intrusion_detection,library,malware,network,package,process,registry,session,threat,vulnerability,web',
    actualValue: 'behavior'
    description: 'This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories.'
    reason: 'value'
  },
]

This way we don't store undefined values, but I agree that we can also have expectedType and actualType and drop the reason field. I don't have a strong preference.

[angorayc]

@angorayc The plan was to use the reason field to know if the incompatibility comes from the value or the mapping, and use the expectedValue and actualValue accordingly.

incompatibleFields: [
  {
    fieldName: 'agent.type'
    expectedValue: 'keyword',
    actualValue: 'text'
    description: 'Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.'
    reason: 'mapping'
  },
  {
    fieldName: 'event.category'
    expectedValue: 'api,authentication,configuration,database,driver,email,file,host,iam,intrusion_detection,library,malware,network,package,process,registry,session,threat,vulnerability,web',
    actualValue: 'behavior'
    description: 'This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories.'
    reason: 'value'
  },
]

This way we don't store undefined values, but I agree that we can also have expectedType and actualType and drop the reason field. I don't have a strong preference.

Thanks for the clarification @semd. I didn't understand properly of how the data was stored, but I am happy with the existing mapping after understanding how it works.