[ML] Field Statistics not honoring negated filters

[eriroley]

Kibana version:
8.10.4

Elasticsearch version:
8.10.4

Server OS version:
Ubuntu 22.04 LTS

Browser version:
Chrome 118

Browser OS version:
Windows 11

Original install method (e.g. download page, yum, from source, etc.):
Repo - https://artifacts.elastic.co/packages/8.x/apt stable main

Describe the bug:
Field Statistics doesn't honor negative filters

Steps to reproduce:

1. Create query in discover using a negated filter
2. switch from documents to "Field Statistics"
3. number of documents exceeds number of hits - shows fields statistics that should have been excluded from search

Expected behavior:
number of documents equals number of hits - excluded documents not included in statistics

Screenshots (if relevant):
Note the inclusion of "administrator" in both the negative filter and the top values, and the number of documents exceeding the number of hits

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

So far I couldn't reproduce , neither in 8.10.4 nor in main.

I've a question, when reloading the page of your screenshot, so field statistics is displayed and the same filter is applied, is the result also false? How much values are you filtering out? Thx

[eriroley]

Sorry about the delay in responding - a reload of the page doesn't change things

I can replicate this with just one NOT, but the full query I am using is
NOT source.user.name: is one of shawn, walter, forti, administrator, fortiadmin, storage, super_admin, fortinet, jerry, Fortinet, superadmin, perry, user, jimmy, bruce, jaime, arthur, mark, christopher, anna, admin, angie, archive, ricoh, soporte, marketing, temp, warehouse, paul, margaret, edu, admin123, scanner, info, husson, kiosk, philip, craig, test123, ron, root, backups, shop, test, sysadmin, user1, backup, scan, james, user2, training, george, parent, sage, joe, sa, office, commercial, adm, fortik, charlie, victor, jeff, michael, jason, emma, conference, gloria, canon, dispatch, kathie, eddie, guest, thomas, FORTINET, alice, christine, juliana, scanning, security, class, rose, valentina, production, accounting, ecommerce, nicole, test1, andrew, karen, ashley, gabriela, engineer, tiffany, report, teresa, mila, katrina, lisa, vpn, frontdesk, janet, project, payroll, demo, caroline, leona, fortiuser, stephanie, miranda, alison, molly, christina, vicki, hr, sandra, jane, quality, cheryl, data, john, sharepoint, stephen, Admin, roger, accounts, chrysta, melody, sabrina, beatrice, tracey, loretta, lauren, lucy, angela, vera, beverly, debra, daisy, fax, deborah, isabel, 863, frank, stella, finance, luna, anca, david

Frank

[eriroley]

indeed - if I then explicitly exclude one of the specific names that is showing up in source.user.name in this case the top user) and then refresh the page, it still shows up in the top values (I did mask two valid usernames that I am negating in the screenshot)

[jughosta]

What top values are visible in a popover when source.user.name is pressed in the left sidebar?

[eriroley]

The popover on the left shows what is expected, and shows that is is calculated from the correct (matching) number of records
see screenshot attached

[qn895]

Thanks Frank @eriroley for documenting this issue. I suspect the multifields is what's causing your issue. To verify if I'm on the right track, do you mind doing the following?

In the popover for source.user.name, when you hover over the source.user.name.text, does it show a + or - ? If it shows a +, can you click on to add the field to the column? Once clicked, you should see source.user.name.text in the list of Selected fields.

Now, click to open that popover for source.user.name.text again, this time click on the - on itsupport. Does that correctly update the Field stats table?

What happen if you first include or exclude the terms directly from the Field stats table. Can you click on one of these buttons and share a screenshot of the filter bar? If it works, this could be a potential workaround (for now while we investigate the issue). This should also update the global filter for the rest of Discover.

[eriroley]

interestingly, we just did the update to 8.11.1 today, and this seems to have resolved it

[kertal]

@eriroley great to hear, so I guess we can close this? thx!

[kertal]

Closing for new, feel free to reopen if it's encountered again. Thx for reporting

[jughosta]

I was able to reproduce it on a cloud instance with logs sample data. It seems to be happening when no specific field is specified in the query input. A quick look at the component props shows that Discover passes current query and filters to Field Statistics embeddable.

cc @qn895

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[qn895]

Closing this issue via #176347