[Security Solution][DQD] Historical results (Phase 1)

[kapral18]

Changes we are introducing in scope of Phase 1 DQD Historical results implementation agreed with @dhru42 :

UI Changes

DQD Checks ListView UI overhaul:

changes:

• Check All Panel ui redesign (behavior of checkbox for "deprecated indices" will be clarified later)
• Collapsible panels for index patterns
• New action buttons for checking individual indexes (expand flyout + check)
• Summary field tab is completely removed
• Remaining field tabs are moved into Latest Check View Flyout Tab (see below)
• Expand action icon opens Latest Check View Flyout Tab (see below) instead of opening nested panel like it is happening now
• Refresh (check) icon triggers in-place checking of latest index result
• The rest is the same (including pagination after 10 indexes per pattern)

(NEW) Latest Check View Flyout Tab:

use cases:

• View latest check of a given index
• Latest automatic check happening upon entering the tab
• Manually check the index again via Check now button at the bottom
• List field tabs the same way as before (uncluding count labels that are currently missing from the new design) (table features like sorting, pagination and search are still discussed in the new design)
• Add telemetry tracking for check button

(NEW) Historical Checks View Flyout Tab:

use cases:

• View list of historical results for given index
• Trigger manual check with button at the bottom
• Select range of historical results by creation date
• Filter results by outcome status
• No automatic check upon transition into the view
• Paginate per 10 items (not in design yet).
• Sort historical results in descending order ( latest - oldest )
• tab specific actions are the same (copy to clipboard, add to new case)
• add telemetry tracking for check button
• add telemetry tracking click on historical view tab

Server / API Changes (Internal)

api changes:

• GET /internal/ecs_data_quality_dashboard/results?pattern -> GET /internal/ecs_data_quality_dashboard/results_latest/:pattern

new api:

• GET /internal/ecs_data_quality_dashboard/results/:pattern - fetches list of authorized indices results without aggregation.
  Supports new query params
  • from for pagination (ex. from=5)
  • size for doc size limiting (ex. size=2)
  • startDate and endDate for historical range selection (ex. startDate=now-1w/w&endDate=now-1d/d (inclusive))
  • outcome for filtering by presence of incompatibleFieldCount field (ex. outcome=pass or outcome=fail)

P. S.

All relevant tickets are or will be also added to the linked epic.

Tasks

Preview Give feedback

1. [Security Solution][DQD] Add new fields to results field map (Phase 1) #184037
   8.15 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore release_note:skip +5
   
2. [Security Solution][DQD][API] Change /results route to support new api design (Phase 1) #182868
   8.15 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore +4
   
3. [Security Solution][DQD][API] Add new /internal/ecs_data_quality_dashboard/results/:pattern route (Phase 1) #183698
   8.15 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore +4
   
4. [Security Solution][DQD] Add migration for outdated mapping (Phase 1) #184427
   8.15 candidate Team:Threat Hunting Team:Threat Hunting:Explore +3
   
5. [Security Solution][DQD] Persist new fields in results storage (Phase 1) #184751
   8.15 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore +4
   
6. [Security Solution][DQD] Add New UI For List View + Latest Flyout (Phase 1) #185881
   8.16 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore +4
   
7. [Security Solution][DQD] Add New Historical Flyout Tab UI (Phase 1) #185882
   8.16 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore +4
   
8. [Security Solution][DQD] Add historical results tour guide #195971
   8.16 candidate Feature:Data Health Quality Team:Threat Hunting Team:Threat Hunting:Explore +4
   

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-threat-hunting-explore (Team:Threat Hunting:Explore)

[kapral18]

The internal api interface has been simplified, description has been updated