Custom filter warnings incorrectly shown on the rule editing page

[nikitaindik]

We've recently received SDH #945 (internal). The issue there seems to be caused by a bug on the rule editing page. When editing a rule with custom filters, QueryBar displays warnings for filters, even though there's nothing wrong with filters' configuration.

It's a UI-only issue. It doesn't affect submitted data or rule execution.

The issue reproduces only if you go into edit mode by clicking the "Edit rule settings" button. It works as expected if you navigate to the edit page directly or if you reload the edit page.

Kibana version:
8.11.x

Steps to reproduce:
(also see video below)

1. Start creating a new custom query rule
2. Choose "Data view" as source and select a data view (for example: logs-*)
3. Fill in anything as a Custom query (for example: *)
4. Add a new custom filter (for example: {"regexp":{"file.path":{"value":".*abc"}}})
5. Notice that the newly added filter is shown as green
6. Fill in other required fields and save the rule
7. Click "Edit rule settings" to go into edit screen for this rule
8. Notice that the filter is now yellow (has a warning) and if you hover over it you see a tooltip: "Field query does not exist in current view"
9. Reload the edit page
10. Notice that the filter is green again

filters_warning.mp4

Expected behavior:

• Filters should not have a warning
• When opening the "edit filter" modal after clicking on the filter, user should be able to see the selected data view (not an empty dropdown)

Any additional context:
There seems to be a race condition.

1. EditRulePageComponent: We fetch indexPattern using the useRuleIndexPattern hook. Initially, while the hook is still fetching, it returns an "empty" object
   { fields: [], title: "" }
   This "empty" object is passed down the components tree as indexPattern prop until it reaches the QueryBar component.
2. QueryBar has a useEffect that listens for indexPattern changes.
   QueryBar's useEffect checks whether the indexPattern prop value is a data view.

If it's a data view: it sets it into state (and uses it later to render filters).
If it's not a data view: the code inside useEffect creates a data view from it using an async call like this:
dv = await data.dataViews.create({ id: indexPattern.title, title: indexPattern.title });

3. In our case this "empty" object is not a data view, so useEffect tries to create a data view from it. The "await" begins.
   ...
4. EditRulePageComponent: A moment later useRuleIndexPattern finishes fetching the index pattern and returns it instead of an "empty" index pattern. This proper object is passed down the components tree as indexPattern prop until it reaches the QueryBar component.
5. QueryBar's useEffect checks whether "indexPattern" prop is a data view. It is, so it sets it into state directly, without doing any async calls.
6. A moment later the await from step 3 resolves and returns an empty data view created from an "empty" indexPattern prop. This data view object is then set into state, replacing the good data view that was set in step 5.
7. QueryBar now has an empty data view in its state. It then passes it to the FilterItem component as a prop. FilterItem shows a warning.

To fix this we might consider:

• not passing indexPattern down to QueryBar before it's fetched
• not creating a data view from an "empty" index pattern object { fields: [], title: "" }

Similar SDH issue with some background info: https://github.com/elastic/sdh-kibana/issues/3254

[elasticmachine]

Pinging @elastic/security-detection-engine (Team:Detection Engine)

[vitaliidm]

@nikitaindik

Looks like this issue might have been fixed earlier.

Could not reproduce it on 8.16(main)

Screen.Recording.2024-09-30.at.14.15.34.mov

Can you look at it and verify whether this still an actual issue?

[nikitaindik]

@vitaliidm Interesting! Just checked on latest main and it does reproduce for me every time. I started with an empty ES, then added fake events using resolver_generator, then followed the same steps you did on a screen recording. Tried in Firefox and Chrome - reproduces in both.

[vitaliidm]

I was able to reproduce issue using AND filter.
After #194678, it appears to be fixed.

@nikitaindik , can you verify if you still can reproduce after the fix?

[nikitaindik]

Thanks @vitaliidm! I have checked on latest main and can confirm that it doesn't reproduce.

Do you think the fix should also be backported to previous Kibana versions?

[vitaliidm]

Backported to 8.15.3

[MadameSheema]

@pborgonovi 8.15.3 BC has been built already, please validate the fix and keep the ticket open until is validated in 8.16.0 BC, thanks!

[pborgonovi]

Hi @vitaliidm

I've checked 8.15.3 BC and issue persists as per the evidence below:

Screen.Recording.2024-10-11.at.11.20.53.AM.mov

As per what I could check the backport was succeeded. Would you please double check this one?

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[vitaliidm]

looks like this fix did not make to 8.15.3, despite PR being labeled as 8.15.3 by Kibana bot.
According to https://staging.elastic.co/8.15.3-82db83de/summary-8.15.3.html -
https://github.com/elastic/kibana/commits/3933429968aafb1ba31319fc38649d0f974044bf, 8.15.3 does not contain that fix.

[pborgonovi]

Thanks for the analysis @vitaliidm I'll make sure to validate 8.15.4 and 8.16 BC

cc @MadameSheema

[pborgonovi]

Issue is fixed in 8.16

Screen.Recording.2024-10-18.at.9.00.00.AM.mov

Waiting to validate in 8.15.4

[pborgonovi]

Issue is fixed in 8.15.4

Screen.Recording.2024-11-05.at.2.53.35.PM.mov

Closing this bug as fixed