[Security Solution] All the fields for all indices show when trying to add the a filters in a rule #179468

jsanz · 2024-03-26T16:59:54Z

Kibana version:
8.12.2

Original install method:
Cloud

Describe the bug:
When editing a security rule, the UI in the filter pill does not honor the index(es) that define the rule.

Steps to reproduce: (Original issue with index patterns, fixed in 8.13)

Create a minimal new index
Create a threshold security rule for that index
Edit the rule and try to add a filter
Fields for all indices show up instead of the fields from the rule index

Re-adding the index forces makes the UI to refresh and get just the fields from the defined index.

video with the index pattern bug

vokoscreenNG-2024-03-26_17-52-47.mp4

Steps to reproduce: (Issue with data views, present in 8.15)

Create a minimal new index and corresponding data view

PUT fields_index
PUT fields_index/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "field-1": {
      "type": "keyword"
    },
    "field-2": {
      "type": "keyword"
    },
    "field-3": {
      "type": "keyword"
    }
  }
}

POST fields_index/_doc
{
 "@timestamp": "2024-10-01T09:26:30.425Z",
 "field-1": "test-0"
}

Create a security rule with that data view
Edit the rule and try to add a filter
Fields for all indices show up instead of the fields from the rule index
Switching to indices and back to data view on rule form fixes issue

video with the Data view bug

Screen.Recording.2024-10-03.at.10.57.05.mov

Expected behavior:

Only the index/data view defined in the rule drives the fields showing up in the filter UI.

Screenshots (if relevant):

Errors in browser console (if relevant):
Nothing shows up

elasticmachine · 2024-03-26T16:59:56Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-03-27T08:41:31Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-03-27T08:41:31Z

Pinging @elastic/security-detection-engine (Team:Detection Engine)

jsanz · 2024-03-27T10:14:34Z

Seems to be fixed by #178207, but keeping it open for the team to confirm.

shayfeld · 2024-03-27T11:24:46Z

Hi @jsanz ,

Is there any chance this problem can be resolved with index pattern and not with data view?

jsanz · 2024-03-27T17:23:13Z

Is there any chance this problem can be resolved with index pattern and not with data view?

@shayfeld, I'm not sure what you mean here. Are you referring to the fix implemented at #178207 or something else?

shayfeld · 2024-03-28T16:02:46Z

@jsanz ,
Yes , Because of the bug (#178207) in version 8.12, my rules work with index patterns as sources instead of data views. Is the bug in this ticket fixed with index patterns or only with data views as sources?

nkhristinin · 2024-03-31T12:15:50Z

Hey @shayfeld, can you provide more details/examples about ?

my rules work with index patterns as sources instead of data views

shayfeld · 2024-03-31T14:38:36Z

Hey @shayfeld, can you provide more details/examples about ?

my rules work with index patterns as sources instead of data views

As you can see in the link of create new rule, using Index patterns: winlogbeat-* as source of the rule. In 178207 they are using Data View as a source.

shayfeld · 2024-06-18T10:59:39Z

Hi @jsanz @nkhristinin @yctercero ,

In v8.13.4, the bug still exists. I sent a video to the support team.

yctercero · 2024-06-19T00:47:52Z

Thanks so much @shayfeld ! This is on our radar. We'll update the ticket as soon as we are able to take it up.

vitaliidm · 2024-10-01T09:29:37Z

@shayfeld , @jsanz

I tried to reproduce the issue in 8.13.4, following the instructions in the ticket, but could not.
But issue is easily can be easily reproduced in 8.12.2 as reported in the issue

Issue reproduced in 8.12.2

Screen.Recording.2024-10-01.at.10.17.19.mov

Issue NOT reproduced in 8.13.4

Screen.Recording.2024-10-01.at.10.19.18.mov

What are the steps to reproduce it in 8.13.4? Is something missing in my approach?
Actions are identical in both videos, but for newer version, not issue found. It works as expected

shayfeld · 2024-10-01T13:24:39Z

@vitaliidm ,

If you'd like, I can schedule a Zoom meeting for tomorrow to show you the steps I took.
Looking at the video again:

Create rule with the dataview .kibana-event-logs* (contain index pattern .kibana-event-logs*)
give it make 1 run (maybe it will help)
Edit the rule and check if the filter showing other fields. (when I switch between dataview and index pattern, it works without the bug)

vitaliidm · 2024-10-01T13:57:05Z

@shayfeld

So, I was able to reproduce the issue.

It does, in fact, fixed for indices, as in original ticket's description.
But your case is different, your rule is configured with Data view, not index.
When I reconfigured my rule, to use Data view, I was able to see the issue.

@timestamp

…r all indices when trying to edit filters in a rule (#194678) ## Summary - addresses #179468 - fixes issue when rule configured with Data view **Steps to reproduce:** 1. Create a minimal new index and corresponding data view ```JSON PUT fields_index PUT fields_index/_mapping { "properties": { "@timestamp": { "type": "date" }, "field-1": { "type": "keyword" }, "field-2": { "type": "keyword" }, "field-3": { "type": "keyword" } } } POST fields_index/_doc { "@timestamp": "2024-10-01T09:26:30.425Z", "field-1": "test-0" } ``` 2. Create a security rule with that data view 3. Edit the rule and try to add a filter 4. Fields for all indices show up instead of the fields from the rule index 5. Switching to indices and back to data view on rule form fixes issue <details> <summary>video with the bug</summary> https://github.com/user-attachments/assets/fc83356d-d727-4662-856e-a4f0b386b71f </details> ### Additional benefit of fixing the issue. Previously, there would be 2 additional field_caps requests, querying ALL indices in ES, when rule edit page loads and rule configured with data view. ``` http://localhost:5601/kbn/internal/data_views/fields?pattern=&meta_fields=_source&meta_fields=_id&meta_fields=_index&meta_fields=_score&meta_fields=_ignored&allow_no_index=true&apiVersion=1 ``` Notice, there is `pattern=` query value, which results in querying all existing indices Now, these requests eliminated. #### Before <img width="2551" alt="Screenshot 2024-10-02 at 18 21 04" src="https://github.com/user-attachments/assets/aa2b6acb-897d-488f-9ddd-409379c6b54a"> #### After <img width="2557" alt="Screenshot 2024-10-02 at 18 22 41" src="https://github.com/user-attachments/assets/baeeecda-bf16-4d37-ae07-3cdc136d18b4">

@timestamp

…r all indices when trying to edit filters in a rule (elastic#194678) ## Summary - addresses elastic#179468 - fixes issue when rule configured with Data view **Steps to reproduce:** 1. Create a minimal new index and corresponding data view ```JSON PUT fields_index PUT fields_index/_mapping { "properties": { "@timestamp": { "type": "date" }, "field-1": { "type": "keyword" }, "field-2": { "type": "keyword" }, "field-3": { "type": "keyword" } } } POST fields_index/_doc { "@timestamp": "2024-10-01T09:26:30.425Z", "field-1": "test-0" } ``` 2. Create a security rule with that data view 3. Edit the rule and try to add a filter 4. Fields for all indices show up instead of the fields from the rule index 5. Switching to indices and back to data view on rule form fixes issue <details> <summary>video with the bug</summary> https://github.com/user-attachments/assets/fc83356d-d727-4662-856e-a4f0b386b71f </details> ### Additional benefit of fixing the issue. Previously, there would be 2 additional field_caps requests, querying ALL indices in ES, when rule edit page loads and rule configured with data view. ``` http://localhost:5601/kbn/internal/data_views/fields?pattern=&meta_fields=_source&meta_fields=_id&meta_fields=_index&meta_fields=_score&meta_fields=_ignored&allow_no_index=true&apiVersion=1 ``` Notice, there is `pattern=` query value, which results in querying all existing indices Now, these requests eliminated. #### Before <img width="2551" alt="Screenshot 2024-10-02 at 18 21 04" src="https://github.com/user-attachments/assets/aa2b6acb-897d-488f-9ddd-409379c6b54a"> #### After <img width="2557" alt="Screenshot 2024-10-02 at 18 22 41" src="https://github.com/user-attachments/assets/baeeecda-bf16-4d37-ae07-3cdc136d18b4"> (cherry picked from commit 5a71d84)

vitaliidm · 2024-10-09T16:50:37Z

fixed in #194678

@timestamp

…r all indices when trying to edit filters in a rule (elastic#194678) ## Summary - addresses elastic#179468 - fixes issue when rule configured with Data view **Steps to reproduce:** 1. Create a minimal new index and corresponding data view ```JSON PUT fields_index PUT fields_index/_mapping { "properties": { "@timestamp": { "type": "date" }, "field-1": { "type": "keyword" }, "field-2": { "type": "keyword" }, "field-3": { "type": "keyword" } } } POST fields_index/_doc { "@timestamp": "2024-10-01T09:26:30.425Z", "field-1": "test-0" } ``` 2. Create a security rule with that data view 3. Edit the rule and try to add a filter 4. Fields for all indices show up instead of the fields from the rule index 5. Switching to indices and back to data view on rule form fixes issue <details> <summary>video with the bug</summary> https://github.com/user-attachments/assets/fc83356d-d727-4662-856e-a4f0b386b71f </details> ### Additional benefit of fixing the issue. Previously, there would be 2 additional field_caps requests, querying ALL indices in ES, when rule edit page loads and rule configured with data view. ``` http://localhost:5601/kbn/internal/data_views/fields?pattern=&meta_fields=_source&meta_fields=_id&meta_fields=_index&meta_fields=_score&meta_fields=_ignored&allow_no_index=true&apiVersion=1 ``` Notice, there is `pattern=` query value, which results in querying all existing indices Now, these requests eliminated. #### Before <img width="2551" alt="Screenshot 2024-10-02 at 18 21 04" src="https://github.com/user-attachments/assets/aa2b6acb-897d-488f-9ddd-409379c6b54a"> #### After <img width="2557" alt="Screenshot 2024-10-02 at 18 22 41" src="https://github.com/user-attachments/assets/baeeecda-bf16-4d37-ae07-3cdc136d18b4"> (cherry picked from commit 5a71d84)

MadameSheema · 2024-10-11T07:44:59Z

@pborgonovi 8.15.3 BC has been built already, please validate the fix and keep the ticket open until is validated in 8.16.0 BC, thanks!

pborgonovi · 2024-10-15T15:19:27Z

Validated latest 8.15.3 BC and bug fix looks good:

Screen.Recording.2024-10-11.at.11.37.06.AM.mov

Waiting to validate with 8.16 BC

vitaliidm · 2024-10-16T16:23:32Z

looks like this fix did not make to 8.15.3, despite PR being labeled as 8.15.3 by Kibana bot.
According to https://staging.elastic.co/8.15.3-82db83de/summary-8.15.3.html -
https://github.com/elastic/kibana/commits/3933429968aafb1ba31319fc38649d0f974044bf, 8.15.3 does not contain that fix.
I have relabelled it as 8.15.4. Not sure why it did not reproduce in your video. I as able consistently reproduce it

pborgonovi · 2024-10-16T16:37:59Z

Thanks for the analysis @vitaliidm I'll make sure to validate 8.15.4 and 8.16 BC

cc @MadameSheema

pborgonovi · 2024-10-18T16:37:43Z

Issue is fixed in 8.16.

Screen.Recording.2024-10-18.at.9.31.10.AM.mov

Waiting to validate with 8.15.4

pborgonovi · 2024-11-05T22:41:15Z

Validated with 8.15.4 BC and fix looks good.

Creating new rule:

Screen.Recording.2024-11-05.at.2.38.33.PM.mov

Updating a rule:

Screen.Recording.2024-11-05.at.2.39.49.PM.mov

jsanz added bug Fixes for quality problems that affect the customer experience Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Mar 26, 2024

MadameSheema added triage_needed Team:Detections and Resp Security Detection Response Team Team:Detection Engine Security Solution Detection Engine Area labels Mar 27, 2024

MadameSheema assigned yctercero Mar 27, 2024

yctercero removed the triage_needed label Apr 1, 2024

yctercero removed their assignment Apr 1, 2024

yctercero added the Feature:Rule Creation Security Solution Detection Rule Creation workflow label Jun 5, 2024

yctercero added the 8.15 candidate label Jun 19, 2024

yctercero removed the 8.15 candidate label Aug 27, 2024

yctercero added impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. 8.16 candidate labels Sep 28, 2024

vitaliidm self-assigned this Sep 30, 2024

vitaliidm mentioned this issue Oct 2, 2024

[Security Solution][Detection Engine] fixes showing all the fields for all indices when trying to edit filters in a rule #194678

Merged

vitaliidm added the fixed label Oct 9, 2024

vitaliidm added v9.0.0 v8.16.0 labels Oct 9, 2024

vitaliidm added the v8.15.3 label Oct 10, 2024

vitaliidm added v8.15.4 and removed v8.15.3 labels Oct 16, 2024

pborgonovi closed this as completed Nov 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] All the fields for all indices show when trying to add the a filters in a rule #179468

[Security Solution] All the fields for all indices show when trying to add the a filters in a rule #179468

jsanz commented Mar 26, 2024 •

edited by vitaliidm

Loading

elasticmachine commented Mar 26, 2024

elasticmachine commented Mar 27, 2024

elasticmachine commented Mar 27, 2024

jsanz commented Mar 27, 2024

shayfeld commented Mar 27, 2024 •

edited

Loading

jsanz commented Mar 27, 2024

shayfeld commented Mar 28, 2024 •

edited

Loading

nkhristinin commented Mar 31, 2024

shayfeld commented Mar 31, 2024 •

edited

Loading

shayfeld commented Jun 18, 2024

yctercero commented Jun 19, 2024

vitaliidm commented Oct 1, 2024

shayfeld commented Oct 1, 2024

vitaliidm commented Oct 1, 2024

vitaliidm commented Oct 9, 2024

MadameSheema commented Oct 11, 2024

pborgonovi commented Oct 15, 2024 •

edited

Loading

vitaliidm commented Oct 16, 2024

pborgonovi commented Oct 16, 2024

pborgonovi commented Oct 18, 2024

pborgonovi commented Nov 5, 2024

[Security Solution] All the fields for all indices show when trying to add the a filters in a rule #179468

[Security Solution] All the fields for all indices show when trying to add the a filters in a rule #179468

Comments

jsanz commented Mar 26, 2024 • edited by vitaliidm Loading

elasticmachine commented Mar 26, 2024

elasticmachine commented Mar 27, 2024

elasticmachine commented Mar 27, 2024

jsanz commented Mar 27, 2024

shayfeld commented Mar 27, 2024 • edited Loading

jsanz commented Mar 27, 2024

shayfeld commented Mar 28, 2024 • edited Loading

nkhristinin commented Mar 31, 2024

shayfeld commented Mar 31, 2024 • edited Loading

shayfeld commented Jun 18, 2024

yctercero commented Jun 19, 2024

vitaliidm commented Oct 1, 2024

shayfeld commented Oct 1, 2024

vitaliidm commented Oct 1, 2024

vitaliidm commented Oct 9, 2024

MadameSheema commented Oct 11, 2024

pborgonovi commented Oct 15, 2024 • edited Loading

vitaliidm commented Oct 16, 2024

pborgonovi commented Oct 16, 2024

pborgonovi commented Oct 18, 2024

pborgonovi commented Nov 5, 2024

jsanz commented Mar 26, 2024 •

edited by vitaliidm

Loading

shayfeld commented Mar 27, 2024 •

edited

Loading

shayfeld commented Mar 28, 2024 •

edited

Loading

shayfeld commented Mar 31, 2024 •

edited

Loading

pborgonovi commented Oct 15, 2024 •

edited

Loading