[Security Solution][DQD] Persist new fields in results storage

[kapral18]

Addresses #184751

Summary

This PR addresses couple of issues:

Main:

Persist revamped resultsFieldMap schema fields, namely incompatibleFieldMappingItems, incompatibleFieldValueItems and sameFamilyFieldItems in the StorageResult after index check, so that after release user can start accumulating data in these fields, while we prepare main UI changes.

Additional:

Improve and narrow down existing in-house EcsFlat override type that originally comes from @elastic/ecs npm package, because currently it is too generic and too loose, resulting in an unnecessary conditional checks and leads to perception of impossible states most of which are refactored, cleaned and fixed in this PR.

Screenshots

How to test

1. Prepare index with invalid mapping and value fields + 1 same family field

DELETE test-field-items

PUT test-field-items
{
  "mappings": {
    "properties": {
      "event.category": { "type": "keyword"},
      "agent.type": {"type": "constant_keyword" },
      "source.ip": {"type": "text"}
    }
  }
}

PUT test-field-items/_doc/1
{
  "@timestamp": "2016-05-23T08:05:34.853Z",
  "event.category": "behavior"
}

PUT test-field-items/_doc/2
{
  "@timestamp": "2016-05-23T08:05:34.853Z",
  "event.category": "shmehavior"
}

2. Open DQD dashboard in kibana
3. Create test-* data-view with test-* index pattern
4. Select it in the sourcerer
5. Click expand button near test-field-items index
6. Verify that you have 1 mapping + 1 value incompatible field + 1 same family field
7. Open kibana devtools
8. Run

GET .kibana-data-quality-dashboard-results-default/_search
{
  "size": 0,
  "query": { 
    "term": {
      "indexName": {
        "value": "test-field-items"
      }
    } 
  },
  "aggs": {
    "latest": {
      "terms": { "field": "indexName", "size": 10000 },
      "aggs": { 
        "latest_doc": { 
          "top_hits": { 
            "size": 1, 
            "sort": [{ "@timestamp": { "order": "desc" } }] 
          } 
        } 
      }
    }
  }
}

9. Verify that latest result contains incompatibleFieldItems and sameFamilyFieldItems of expected shape:

//...
                     "incompatibleFieldValueItems": [
                      {
                        "fieldName": "event.category",
                        "expectedValues": [
                          "api",
                          "authentication",
                          "configuration",
                          "database",
                          "driver",
                          "email",
                          "file",
                          "host",
                          "iam",
                          "intrusion_detection",
                          "library",
                          "malware",
                          "network",
                          "package",
                          "process",
                          "registry",
                          "session",
                          "threat",
                          "vulnerability",
                          "web"
                        ],
                        "actualValues": [
                          { "name": "behavior",  count: 2 },
                          { "name": "shmehavior", count: 1}
                        ],
                        "description": """This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy.
`event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory.
This field is an array. This will allow proper categorization of some events that fall in multiple categories."""
                      }
                     ],
                     "incompatibleFieldMappingItems": [
                      {
                        "fieldName": "source.ip",
                        "expectedValue": "ip",
                        "actualValue": "text",
                        "description": "IP address of the source (IPv4 or IPv6)."
                      }
                    ]
//...
"sameFamilyFieldItems": [
                      {
                        "fieldName": "agent.type",
                        "expectedValue": "keyword",
                        "actualValue": "constant_keyword",
                        "description": """Type of the agent.
The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine."""
                      }
                    ]

[elasticmachine]

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[angorayc]

Tested locally, all worked as expected.

[andrew-goldstein]

Thanks @kapral18 for the enhancements to persistence, and the improved types! 🙏
✅ Desk tested locally
LGTM 🚀

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 00aa6c7

Failed CI Steps

• FTR Configs #38
• FTR Configs #46

Test Failures

• [job] [logs] FTR Configs #46 / GenAI - NLP @serverless NLP Cleanup Task in Essentials Tier New Essentials Deployment Model Loading "before all" hook for "executes NLP Cleanup Task and successfully cleans up only pytorch models"
• [job] [logs] FTR Configs #38 / X-Pack Accessibility Tests - Group 3 Security Solution Accessibility Detections Create Rule Flow Custom Query Rule Define Step preview section meets a11y requirements

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5507	5508	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	13.6MB	13.6MB	+394.0B

History

• 💔 Build #215404 failed 7347cc5b2c799b563f9df8445568e82665281be6
• 💛 Build #215376 was flaky 69966f6e697fc89bbb5e9c2a38a65788fceb4d34
• 💛 Build #215264 was flaky f8e62c8937a760500a5816f51826df5a5770faa1
• 💚 Build #215066 succeeded c907d302fec4d54d267f44e06bd627cd5cdcdfb2
• 💛 Build #214959 was flaky eb873c2c6aceccb39e1ac175acb0740a690d1006

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kapral18