[Security Solution][DQD] Add new fields to results field map (Phase 1)

[semd]

Summary

Improve the way we store the incompatible/same family details in the results storage. So we have the flexibility to render that information using react components instead of always relying on the markdownComments static text stored.

Proposal

Add 2 new fields to the mapping, the incompatibleFields and sameFamilyFields, for example:

incompatibleFields: [
  {
    fieldName: 'agent.type'
    expectedValue: 'keyword',
    actualValue: 'text'
    description: 'Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.'
    reason: 'mapping'
  },
  {
    fieldName: 'event.category'
    expectedValue: 'api,authentication,configuration,database,driver,email,file,host,iam,intrusion_detection,library,malware,network,package,process,registry,session,threat,vulnerability,web',
    actualValue: 'behavior'
    description: 'This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories.'
    reason: 'value'
  },
]

sameFamilyFields: [
  {
    fieldName: 'agent.type'
    expectedValue: 'keyword',
    actualValue: 'constant_keyword'
    description: 'This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories.'
  },
]

To achieve this, we have add this (untested) to the results data stream mapping (code) :

  'incompatibleFields.fieldName': { type: 'keyword', required: true },
  'incompatibleFields.expectedValue': { type: 'keyword', required: true },
  'incompatibleFields.actualValue': { type: 'keyword', required: true },
  'incompatibleFields.description': { type: 'keyword', required: true },
  'incompatibleFields.reason': { type: 'keyword', required: true }, // mapping or value

And remove:

  unallowedMappingFields: { type: 'keyword', required: true, array: true },
  unallowedValueFields: { type: 'keyword', required: true, array: true },

We can keep the markdownComments information in case we want to use it for something else in the future.

[angorayc]

Some thought on this task:

1. Removing fields in the mapping might not be ideal, as it affects users existing data.
2. https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

example:

PUT my-index-000001


PUT my-index-000001/_doc/1
{
  "group" : "fans",
  "user" : [ 
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}

PUT my-index-000001/_doc/2
{
  "group" : "fans",
  "user" : [ 
    {
      "first" : "Alice",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}


GET my-index-000001/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "user.first": "Alice" }},
        { "match": { "user.last":  "Smith" }}
      ]
    }
  }
}



return:


{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 0.43301374,
    "hits": [
      {
        "_index": "my-index-000001",
        "_id": "2",
        "_score": 0.43301374,
        "_source": {
          "group": "fans",
          "user": [
            {
              "first": "Alice",
              "last": "Smith"
            },
            {
              "first": "Alice",
              "last": "White"
            }
          ]
        }
      },
      {
        "_index": "my-index-000001",
        "_id": "1",
        "_score": 0.36464313,
        "_source": {
          "group": "fans",
          "user": [
            {
              "first": "John",
              "last": "Smith"
            },
            {
              "first": "Alice",
              "last": "White"
            }
          ]
        }
      }
    ]
  }
}


==================


PUT test-index-2
{
  "mappings": {
    "properties": {
      "user": {
        "type": "nested" 
      }
    }
  }
}


PUT test-index-2/_doc/1
{
  "group" : "fans",
  "user" : [ 
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}

PUT test-index-2/_doc/2
{
  "group" : "fans",
  "user" : [ 
    {
      "first" : "Alice",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}

GET test-index-2/_search
{
  "query": {
    "nested": {
      "path": "user",
      "query": {
        "bool": {
          "must": [
            { "match": { "user.first": "Alice" }},
            { "match": { "user.last":  "Smith" }} 
          ]
        }
      }
    }
  }
}


returns:


{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1.0498221,
    "hits": [
      {
        "_index": "test-index-2",
        "_id": "2",
        "_score": 1.0498221,
        "_source": {
          "group": "fans",
          "user": [
            {
              "first": "Alice",
              "last": "Smith"
            },
            {
              "first": "Alice",
              "last": "White"
            }
          ]
        }
      }
    ]
  }
}

[kapral18]

per our discussion with @angorayc created this sibling ticket #184427

We decided to implement this as 2 tickets 2 PR into branch in my fork with subsequent merging into 1 final combining PR into main elastic/kibana for ease of review

cc: @semd