[data views] Allow data views created on hidden and system indices - second attempt

[mattkime]

Summary

Previously, the 'Allow hidden and system indices' advanced option when creating a data view was only a UI convenience. It allowed you to see which hidden and system indices you were matching but they would be would be selected just the same once the data view was loaded. At some point something changed and now there are system and hidden indices that require expandWildcards: hidden to be passed to field caps in order to see anything. allowHidden: boolean is added to the DataView and DataViewSpec and passed through when field caps requests are made.

This is primarily a tool for troubleshooting. For instance, instead of hitting a full data stream across a number of data tiers you can select a specific index to compare its performance.

NOTE: This is a second attempt. What I learned - the whole expand_wildcards param is literal - you can directly query a hidden index without expandWildcards: hidden since its not using a wildcard. Tests now use a wildcard.

Closes: #164652

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[mbondyra]

LGTM 👌🏼

[davismcphee]

Thanks both for looking into and addressing the wildcard issue! I pulled and tested again and can confirm the issue no longer exists. It seems hidden and system index data views are now fully supported, LGTM 👍

[yngrdyn]

LGTM

[kertal]

@elasticmachine merge upstream

[kibana-ci]

💔 Build Failed

• Buildkite Build
• Commit: bc471b1
• Interpreting CI Failures

Failed CI Steps

• Serverless Security Cypress Tests #6
• Serverless Security Cypress Tests #6
• Security Solution Cypress Tests #16
• Security Solution Cypress Tests #16
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #4
• Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #4

Test Failures

• [job] [logs] Serverless Security Cypress Tests #6 / Add, edit and delete exception Add, Edit and delete Exception item should create exception item from Shared Exception List page and linked to a Rule should create exception item from Shared Exception List page and linked to a Rule
• [job] [logs] Security Solution Cypress Tests #16 / Add, edit and delete exception Add, Edit and delete Exception item should create exception item from Shared Exception List page and linked to a Rule should create exception item from Shared Exception List page and linked to a Rule
• [job] [logs] Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #4 / Add, edit and delete exception Add, Edit and delete Exception item should create exception item from Shared Exception List page and linked to a Rule should create exception item from Shared Exception List page and linked to a Rule
• [job] [logs] Security Solution Cypress Tests #16 / Add, edit and delete exception Add, Edit and delete Exception item should create exception item from Shared Exception List page and linked to a Rule should create exception item from Shared Exception List page and linked to a Rule
• [job] [logs] Rules, Alerts and Exceptions ResponseOps Cypress Tests on Security Solution #4 / Add, edit and delete exception Add, Edit and delete Exception item should create exception item from Shared Exception List page and linked to a Rule should create exception item from Shared Exception List page and linked to a Rule
• [job] [logs] Serverless Security Cypress Tests #6 / Add, edit and delete exception Add, Edit and delete Exception item should create exception item from Shared Exception List page and linked to a Rule should create exception item from Shared Exception List page and linked to a Rule

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
data	2542	2545	+3
dataViews	255	257	+2
total			+5

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
dataViewEditor	47.7KB	48.0KB	+254.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	413.3KB	413.3KB	+89.0B
dataViews	46.6KB	47.0KB	+429.0B
total			+518.0B

Unknown metric groups

API count

id	before	after	diff
data	3193	3197	+4
dataViews	918	922	+4
total			+8

History

• 💔 Build #175700 failed 32bdf63
• 💔 Build #175645 failed bfcf089
• 💔 Build #175550 failed 8b75fa4
• 💔 Build #175182 failed f6f115b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime

[yctercero]

The exceptions flake occurring seems to be completely unrelated to these changes. It's expecting one rule to show in the test instead of the two appearing for selection. Skipping and tracking in this PR here.