[Security Solution] Filter in does not work properly for non-ECS fields

[MadameSheema]

Describe the bug:

• Filter in does not work properly for non-ECS fields

Kibana/Elasticsearch Stack version:

• 8.3.0 (main 58bc0f7)

Initial Setup:

• To have a document with a non-ECS field with a map i.e.
• Mapping:

{
  "properties" : {
    "@timestamp" : {
    	"type" : "date"
  },
   "dummy_field_1" : { 
    "type" : "keyword", 
    "ignore_above" : 1024
   },
   "dummy_field_2" : { 
    "type" : "keyword", 
    "ignore_above" : 1024
   }

• Document

{
   "@timestamp":"2022-05-05T09:38:19.579Z",
   "dummy_field_1":"field_1",
   "dummy_field_2":"field_2"
}

• Make sure that your index is part of the Security Solution data view
• To generate an alert using the above data (make sure you are using a big look-back time for the rule in order to fit the @timestamp)

Steps to reproduce:

1. Navigate to the alerts page
2. Click on the fields browser
3. Search for dummy
4. Select dummy_field_1
5. Click on Close
6. Filter in by the field_1 value on the dummy_field_1 column

Current behavior:

Expected behavior:

• The alert is displayed since it has the field_1 value for the dummy_field_1 column

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[MadameSheema]

@peluja1012 @michaelolo24 can you please confirm if the following statement is still valid? Also if there is any plan to work on this?

cc @toby-sutor

[frconil]

FWIW I think it also applies for ECS fields, I've seen the same issue with data_stream.namespace

[lgestc]

Problem has been reproduced, even without having custom index created

[lgestc]

Root cause, right now we dont know which fields are mapped or not #154714 (comment)

[lgestc]

on hold until the we refactor data view selection