[Security Solution][DQD] Persist new fields in results storage (Phase 1) #184751
Assignees
Labels
8.15 candidate
Feature:Data Health Quality
Data health quality dashboard and related features
Team:Threat Hunting:Explore
Team:Threat Hunting
Security Solution Threat Hunting Team
Comments
kapral18
added
Team:Threat Hunting
|
Pinging @elastic/security-threat-hunting (Team:Threat Hunting) |
|
Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore) |
kapral18
added a commit
to kapral18/kibana
that referenced
this issue
Jun 12, 2024
… field items - Added `incompatibleFieldItems` and `sameFamilyFieldItems` to `StorageResult` for detailed field information. - Updated tests to validate the new functionality. - Adjusted type definitions and mock data to support the changes. Addresses elastic#184751
kapral18
added a commit
that referenced
this issue
Jun 13, 2024
Addresses #184751 ## Summary This PR addresses couple of issues: ### Main: Persist revamped `resultsFieldMap` schema fields, namely `incompatibleFieldMappingItems`, `incompatibleFieldValueItems` and `sameFamilyFieldItems` in the `StorageResult` after index check, so that after release user can start accumulating data in these fields, while we prepare main UI changes. ### Additional: Improve and narrow down existing in-house `EcsFlat` override type that originally comes from `@elastic/ecs` npm package, because currently it is too generic and too loose, resulting in an unnecessary conditional checks and leads to perception of impossible states most of which are refactored, cleaned and fixed in this PR. ### Screenshots    ### How to test 1. Prepare index with invalid mapping and value fields + 1 same family field ```graphql DELETE test-field-items PUT test-field-items { "mappings": { "properties": { "event.category": { "type": "keyword"}, "agent.type": {"type": "constant_keyword" }, "source.ip": {"type": "text"} } } } PUT test-field-items/_doc/1 { "@timestamp": "2016-05-23T08:05:34.853Z", "event.category": "behavior" } PUT test-field-items/_doc/2 { "@timestamp": "2016-05-23T08:05:34.853Z", "event.category": "shmehavior" } ``` 2. Open DQD dashboard in kibana 3. Create `test-*` data-view with `test-*` index pattern 4. Select it in the sourcerer 5. Click expand button near test-field-items index 6. Verify that you have 1 mapping + 1 value incompatible field + 1 same family field 7. Open kibana devtools 8. Run ```graphql GET .kibana-data-quality-dashboard-results-default/_search { "size": 0, "query": { "term": { "indexName": { "value": "test-field-items" } } }, "aggs": { "latest": { "terms": { "field": "indexName", "size": 10000 }, "aggs": { "latest_doc": { "top_hits": { "size": 1, "sort": [{ "@timestamp": { "order": "desc" } }] } } } } } } ``` 9. Verify that latest result contains `incompatibleFieldItems` and `sameFamilyFieldItems` of expected shape: ```json5 //... "incompatibleFieldValueItems": [ { "fieldName": "event.category", "expectedValues": [ "api", "authentication", "configuration", "database", "driver", "email", "file", "host", "iam", "intrusion_detection", "library", "malware", "network", "package", "process", "registry", "session", "threat", "vulnerability", "web" ], "actualValues": [ { "name": "behavior", count: 2 }, { "name": "shmehavior", count: 1} ], "description": """This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories.""" } ], "incompatibleFieldMappingItems": [ { "fieldName": "source.ip", "expectedValue": "ip", "actualValue": "text", "description": "IP address of the source (IPv4 or IPv6)." } ] //... "sameFamilyFieldItems": [ { "fieldName": "agent.type", "expectedValue": "keyword", "actualValue": "constant_keyword", "description": """Type of the agent. The agent type always stays the same and should be given by the agent used. In case of Filebeat the agent would always be Filebeat also if two Filebeat instances are run on the same machine.""" } ] ```
kapral18
added
the
Feature:Data Health Quality
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue is a follow up from #184037
In previous issue we added 2 new results datastream mapping fields
sameFamilyFieldItemsandincompatibleFielditems.But since they are not used they are just a schema definition.
In this ticket we want to introduce persistence of those fields in the actual index, so that by the time we ship the main feature #184158 there will be enough results data accumulated on consumer side to be presented in the new historical flyout view with proper UI instead of classic markdown UI
The text was updated successfully, but these errors were encountered: