Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution][DQD] Persist new fields in results storage (#185025)
Addresses #184751 ## Summary This PR addresses couple of issues: ### Main: Persist revamped `resultsFieldMap` schema fields, namely `incompatibleFieldMappingItems`, `incompatibleFieldValueItems` and `sameFamilyFieldItems` in the `StorageResult` after index check, so that after release user can start accumulating data in these fields, while we prepare main UI changes. ### Additional: Improve and narrow down existing in-house `EcsFlat` override type that originally comes from `@elastic/ecs` npm package, because currently it is too generic and too loose, resulting in an unnecessary conditional checks and leads to perception of impossible states most of which are refactored, cleaned and fixed in this PR. ### Screenshots    ### How to test 1. Prepare index with invalid mapping and value fields + 1 same family field ```graphql DELETE test-field-items PUT test-field-items { "mappings": { "properties": { "event.category": { "type": "keyword"}, "agent.type": {"type": "constant_keyword" }, "source.ip": {"type": "text"} } } } PUT test-field-items/_doc/1 { "@timestamp": "2016-05-23T08:05:34.853Z", "event.category": "behavior" } PUT test-field-items/_doc/2 { "@timestamp": "2016-05-23T08:05:34.853Z", "event.category": "shmehavior" } ``` 2. Open DQD dashboard in kibana 3. Create `test-*` data-view with `test-*` index pattern 4. Select it in the sourcerer 5. Click expand button near test-field-items index 6. Verify that you have 1 mapping + 1 value incompatible field + 1 same family field 7. Open kibana devtools 8. Run ```graphql GET .kibana-data-quality-dashboard-results-default/_search { "size": 0, "query": { "term": { "indexName": { "value": "test-field-items" } } }, "aggs": { "latest": { "terms": { "field": "indexName", "size": 10000 }, "aggs": { "latest_doc": { "top_hits": { "size": 1, "sort": [{ "@timestamp": { "order": "desc" } }] } } } } } } ``` 9. Verify that latest result contains `incompatibleFieldItems` and `sameFamilyFieldItems` of expected shape: ```json5 //... "incompatibleFieldValueItems": [ { "fieldName": "event.category", "expectedValues": [ "api", "authentication", "configuration", "database", "driver", "email", "file", "host", "iam", "intrusion_detection", "library", "malware", "network", "package", "process", "registry", "session", "threat", "vulnerability", "web" ], "actualValues": [ { "name": "behavior", count: 2 }, { "name": "shmehavior", count: 1} ], "description": """This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories.""" } ], "incompatibleFieldMappingItems": [ { "fieldName": "source.ip", "expectedValue": "ip", "actualValue": "text", "description": "IP address of the source (IPv4 or IPv6)." } ] //... "sameFamilyFieldItems": [ { "fieldName": "agent.type", "expectedValue": "keyword", "actualValue": "constant_keyword", "description": """Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.""" } ] ```
- Loading branch information
