Mapping conflict in data view with hidden/system indices

[nflexfo]

Kibana version:

8.12.2

Elasticsearch version:

8.12.2

Server OS version:

Linux

Browser version:

Chrome

Original install method (e.g. download page, yum, from source, etc.):

Docker

Describe the bug:

Using the *-* wildcard in Index Pattern result in hidden/system indices being matched in mapping even if "Allow hidden and system indices" is turned off. This results in conflicts between our custom mapping (mostly keyword) and various fields, likes url which have object type instead, from the following indices:

.internal.alerts-observability.apm.alerts-default-000001, .internal.alerts-observability.logs.alerts-default-000001, .internal.alerts-observability.metrics.alerts-default-000001, .internal.alerts-observability.threshold.alerts-default-000001, .internal.alerts-observability.uptime.alerts-default-000001, .internal.alerts-security.alerts-default-000001

Also, hiddens/system indices are matched in ES|QL when using the following syntaxe FROM *-* which prevents us to performs full search on all our own indices .

Steps to reproduce:

1. Stack Management > (KIBANA) Data Views
2. Create data view
3. Index pattern: *-*
4. Show advanced settings
5. Disallow hidden and system indices

Expected behavior:

I do not expect any .internal.* indices to be match since the "Allow hidden and system indices" is turned off.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

This might be related to #168174, #168882 or #164652.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[jughosta]

Hi @nflexfo,
Thanks for reporting this use case!

As a workaround, you can exclude hidden indices by using this format *-*,-.* when defining an index pattern.

[kertal]

FYI @mattkime since this might be related to #168882

[mattkime]

I'm able to reproduce the problem locally on a plain dev instance with the default indices. It doesn't match the indices directly but they're paired with aliases which are matched with the index pattern.

My next step is to figure out who knows why the indices and aliases are configured this way.

Also, sometimes aliases are matched and fail to show up in the data source list during data view creation - #178587

[mattkime]

Closing this issue in favor of #178589 and #178587 - the first issue is the more more relevant to the original bug