[Alerts] Use unmapped_type: keyword for unmapped fields in alerts query

[kqualters-elastic]

Summary

Currently if a user tries to sort the alerts table by an umapped field, for instance data_stream.dataset, the table will show 0 results, with no easy way to remove this sort. Adds unmapped_type: keyword to the alerts query to help mitigate this, but there should be a ux way to clear filters/sort as well. Related: #170167 and #171104

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[elasticmachine]

Pinging @elastic/security-threat-hunting-investigations (Team:Threat Hunting:Investigations)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 2396f33

Failed CI Steps

• x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts
• x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts
• x-pack/test_serverless/functional/test_suites/security/common_configs/config.group6.ts
• x-pack/test_serverless/functional/test_suites/security/common_configs/config.group6.ts
• FTR Configs #21
• FTR Configs #21
• FTR Configs #45
• FTR Configs #45
• FTR Configs #46
• FTR Configs #46
• FTR Configs #51
• FTR Configs #51
• FTR Configs #54
• FTR Configs #54
• FTR Configs #80
• FTR Configs #80
• FTR Configs #85
• FTR Configs #85
• FTR Configs #91
• FTR Configs #91
• Investigations - Security Solution Cypress Tests #1
• Investigations - Security Solution Cypress Tests #2
• Investigations - Security Solution Cypress Tests #3
• Investigations - Security Solution Cypress Tests #4
• Investigations - Security Solution Cypress Tests #5
• Investigations - Security Solution Cypress Tests #6
• Investigations - Security Solution Cypress Tests #7
• Serverless Investigations - Security Solution Cypress Tests #1
• Serverless Investigations - Security Solution Cypress Tests #2
• Serverless Investigations - Security Solution Cypress Tests #3
• Serverless Investigations - Security Solution Cypress Tests #4
• Serverless Investigations - Security Solution Cypress Tests #5
• Serverless Investigations - Security Solution Cypress Tests #6
• Serverless Investigations - Security Solution Cypress Tests #7
• Serverless Investigations - Security Solution Cypress Tests #8

Test Failures

• [job] [logs] Investigations - Security Solution Cypress Tests #4 / Alert details expandable flyout "before each" hook for "should allow user to switch between push and overlay modes for flyout opened from alerts page" "before each" hook for "should allow user to switch between push and overlay modes for flyout opened from alerts page"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Alert details expandable flyout "before each" hook for "should allow user to switch between push and overlay modes for flyout opened from alerts page" "before each" hook for "should allow user to switch between push and overlay modes for flyout opened from alerts page"
• [job] [logs] Investigations - Security Solution Cypress Tests #6 / Alert details expandable flyout left panel entities "before each" hook for "should display host details and user details under Insights Entities" "before each" hook for "should display host details and user details under Insights Entities"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Alert details expandable flyout left panel entities "before each" hook for "should display host details and user details under Insights Entities" "before each" hook for "should display host details and user details under Insights Entities"
• [job] [logs] Investigations - Security Solution Cypress Tests #2 / Alert details expandable flyout left panel investigation "before each" hook for "should display empty response message" "before each" hook for "should display empty response message"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Alert details expandable flyout left panel investigation "before each" hook for "should display empty response message" "before each" hook for "should display empty response message"
• [job] [logs] Investigations - Security Solution Cypress Tests #7 / Alert details expandable flyout left panel investigation "before each" hook for "should display investigation guide" "before each" hook for "should display investigation guide"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Alert details expandable flyout left panel investigation "before each" hook for "should display investigation guide" "before each" hook for "should display investigation guide"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Alert details expandable flyout left panel prevalence "before each" hook for "should display prevalence tab" "before each" hook for "should display prevalence tab"
• [job] [logs] Investigations - Security Solution Cypress Tests #7 / Alert details expandable flyout right panel "before each" hook for "should display header and footer basics" "before each" hook for "should display header and footer basics"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Alert details expandable flyout right panel "before each" hook for "should display header and footer basics" "before each" hook for "should display header and footer basics"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Alert details expandable flyout right panel json tab "before each" hook for "should display the json component" "before each" hook for "should display the json component"
• [job] [logs] Investigations - Security Solution Cypress Tests #2 / Alert details expandable flyout right panel overview tab about section "before each" hook for "should display about section" "before each" hook for "should display about section"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Alert details expandable flyout right panel overview tab about section "before each" hook for "should display about section" "before each" hook for "should display about section"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Alert details expandable flyout right panel table tab "before each" hook for "should display and filter the table" "before each" hook for "should display and filter the table"
• [job] [logs] Investigations - Security Solution Cypress Tests #5 / Alert details expandable flyout rule preview panel alert reason preview "before each" hook for "should display alert reason preview" "before each" hook for "should display alert reason preview"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Alert details expandable flyout rule preview panel alert reason preview "before each" hook for "should display alert reason preview" "before each" hook for "should display alert reason preview"
• [job] [logs] Investigations - Security Solution Cypress Tests #6 / Alert details expandable flyout rule preview panel rule preview "before each" hook for "should display rule preview and its sub sections" "before each" hook for "should display rule preview and its sub sections"
• [job] [logs] Investigations - Security Solution Cypress Tests #2 / Alert Table API calls "before each" hook for "should call api/lists/index only once" "before each" hook for "should call api/lists/index only once"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Alert Table API calls "before each" hook for "should call api/lists/index only once" "before each" hook for "should call api/lists/index only once"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #3 / Alerts generated by building block rules Alerts should be visible on the Rule Detail page and not visible on the Overview page Alerts should be visible on the Rule Detail page and not visible on the Overview page
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #2 / Bulk Investigate in Timeline Alerts "before each" hook for "Adding multiple alerts to the timeline should be successful" "before each" hook for "Adding multiple alerts to the timeline should be successful"
• [job] [logs] FTR Configs #21 / Cloud Security Posture Security Alerts Page - Graph visualization "before all" hook for "should render graph visualization"
• [job] [logs] FTR Configs #21 / Cloud Security Posture Security Alerts Page - Graph visualization "before all" hook for "should render graph visualization"
• [job] [logs] Investigations - Security Solution Cypress Tests #7 / Create DataView runtime field adds field to alert table adds field to alert table
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Create DataView runtime field adds field to alert table adds field to alert table
• [job] [logs] x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] x-pack/test_serverless/functional/test_suites/security/common_configs/config.group6.ts / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] FTR Configs #45 / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] FTR Configs #91 / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] FTR Configs #45 / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] x-pack/test_serverless/functional/test_suites/search/common_configs/config.group6.ts / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] x-pack/test_serverless/functional/test_suites/security/common_configs/config.group6.ts / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] FTR Configs #91 / Discover alerting Search source Alert should navigate to alert results via view in app link
• [job] [logs] Investigations - Security Solution Cypress Tests #5 / Expandable flyout left panel correlations "before each" hook for "should render correlations details correctly" "before each" hook for "should render correlations details correctly"
• [job] [logs] Investigations - Security Solution Cypress Tests #4 / Expandable flyout state sync "before each" hook for "should test flyout url sync" "before each" hook for "should test flyout url sync"
• [job] [logs] FTR Configs #46 / InfraOps App Metrics UI Node Details #Asset Type: host Overview Tab Alerts Section with alerts should show alert summary
• [job] [logs] FTR Configs #46 / InfraOps App Metrics UI Node Details #Asset Type: host Overview Tab Alerts Section with alerts should show alert summary
• [job] [logs] Investigations - Security Solution Cypress Tests #4 / Investigate in timeline From alerts table "before each" hook for "should open new timeline from alerts table" "before each" hook for "should open new timeline from alerts table"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #4 / Investigate in timeline From alerts table "before each" hook for "should open new timeline from alerts table" "before each" hook for "should open new timeline from alerts table"
• [job] [logs] FTR Configs #85 / ObservabilityApp Observability alerts > Alerts table Renders the table
• [job] [logs] FTR Configs #85 / ObservabilityApp Observability alerts > Alerts table Renders the table
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Open timeline "before each" hook for "should open a timeline via url alone without a saved object id" "before each" hook for "should open a timeline via url alone without a saved object id"
• [job] [logs] Investigations - Security Solution Cypress Tests #4 / Opening alert previews from alert details flyout "before each" hook for "should render alert preview" "before each" hook for "should render alert preview"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #7 / Opening alert previews from alert details flyout "before each" hook for "should render alert preview" "before each" hook for "should render alert preview"
• [job] [logs] Investigations - Security Solution Cypress Tests #5 / Ransomware Detection Alerts Ransomware in Alerts Page "before each" hook for "should show ransomware alerts on alerts page" "before each" hook for "should show ransomware alerts on alerts page"
• [job] [logs] Serverless Investigations - Security Solution Cypress Tests #5 / Ransomware Detection Alerts Ransomware in Alerts Page "before each" hook for "should show ransomware alerts on alerts page" "before each" hook for "should show ransomware alerts on alerts page"
• [job] [logs] Investigations - Security Solution Cypress Tests #6 / Ransomware Prevention Alerts Ransomware in Alerts Page should show ransomware alerts on alerts page should show ransomware alerts on alerts page
• [job] [logs] FTR Configs #51 / serverless observability UI Rules list should disable all selection
• [job] [logs] FTR Configs #51 / serverless observability UI Rules list should disable all selection

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
triggersActionsUi	1.7MB	1.7MB	+24.0B

History

[pmuellr]

I can see the following from some of the failed tests:

warn browser[SEVERE] http://localhost:5620/internal/search/privateRuleRegistryAlertsSearchStrategy - Failed to load resource: the server responded with a status of 400 (Bad Request)

Guessing the new unmapped_type param is not expected, and so a validation is failing, or some code is throwing an exception, but I'm kinda lost in this code ...

[kqualters-elastic]

Guessing the new unmapped_type param is not expected, and so a validation is failing, or some code is throwing an exception, but I'm kinda lost in this code ...

Ya that sounds about right, I will dig a little deeper on this, probably should have opened this as a draft pr first. Mostly just wanted to show @kapral18 that this is "all" (outside of validation/any tightly coupled to the es request body tests) we have to do to fix this in most cases, and it's a pretty common source of SDHs.

[pmuellr]

For reference, and to decrease the # of searches I need to make for this, the unmapped_type property is doc'd here: https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html#_ignoring_unmapped_fields