[DOCS][Rules][Data Views] - Data views available in rule creation flow

[yctercero]

Description

Users can specify data views wherever index pattern specification is available in order to take advantage of runtime fields, which are associated with a data view.

Issue: https://github.com/elastic/security-team/issues/2874
PR: elastic/kibana#130929

Acceptance Test Criteria

• When creating a rule, the user should be able to select a Data View to be used as the source of data
  • Note that this should be in addition to allowing them to select "source index patterns". The ability to select "source index patterns" is still needed because prebuilt rules rely on it.
  • For Indicator Match rules, a user should also be able to select a "Data View" as the indicator data source.
• After a "Data View" is selected, when entering a "custom query" the user should see runtime fields associated with the "Data View" in the field suggestions.
• After a "Data View" is selected, when entering an "EQL query" the user should be able to use runtime fields associated with the "Data View" in the EQL editor.
• After a "Data View" is selected, runtime fields associated with the "Data View" should be available as options for field overrides such as rule name override.
• When creating a Threshold Rule, the user should be able to select a runtime field as the group by field, if they configured a Data View as the data source.
• When creating an Indicator Match rules, runtime fields associated with the "Data View" should be available as options for the Indicator mapping field and Indicator index field configurations.
• When selecting a Data View, the user should have the option or a link to add a new runtime field to their selected Data View. Similar to the workflow in Discover.
• For the "alerts on alerts" scenario, the user should be guided towards using the .alerts Data View so they can take advantage of any runtime fields they may have added.

Notes

• Be sure to add any necessary screenshots, code text or console commands for clarity.
• Include any conditions or caveats that may affect customers.

[dhurley14]

related PR: elastic/kibana#130929

[dhurley14]

Some screenshots please let me know if you need more! cc @joepeeples

Data view selection

Example data view options

Rule details view

[nastasha-solomon]

@dhurley14 when you're ready, feel free to reach out to the @elastic/security-docs team for help crafting and reviewing the UI tour text for this feature.

[yctercero]

Error state when imported rule where data view does not exist - elastic/kibana#137841

[joepeeples]

Reassigning to @jmikell821 since she's documenting rule creation for 8.4 in #2258.

[yctercero]

Hey there! We'd discussed needing additional details for documenting. Hopefully the following helps:

API changes

• Rule creation
  • Can now specify a data_view_id
  • There is no schema check/validation disallowing a user from specifying both an index and a data_view_id, BUT our detection logic is that if a data_view_id is specified, that is what will be used to search
• Rule update/rule patch
  • Can update or patch data_view_id
• /api/detection_engine/rules/_bulk_action
  • There's no bulk actions specific to data views, but if a user selects to bulk edit indices of a bunch of rules, and some of those rules use data views the following behavior applies:
  • If overwrite_data_views is set to true - even rules using data views will be updated to include the index changes and the data_view_id will be removed
  • If overwrite_data_views is set to false - no updates or changes will be made to rules using data views
  • PR updating this param to snakecase
  • PR updating behavior

Affected flows

Rule Creation

• User can select to use index pattern, OR data view
• Data view selector currently displays the data view's title property, not name
• Only selection of a single data view, not multiple, is supported
• Runtime fields associated with data views are available
• Any runtime fields need to be created prior to rule creation. There's no options to create runtime fields or data views in rule creation.
• Options to overwrite fields will also display runtime fields if relevant (timestamp override, risk score...)
• 👀 If a user selects the Security default data view, we show a warning letting them know that using this data view could result in alerts on alerts. PR adding warning message.

Rule creation

Rule Edit

• User can select to use index pattern OR data view
• Data view selector currently displays the data view's title property, not name
• Only selection of a single data view, not multiple, is supported
• Runtime fields associated with data view are available
• 👀 If the selected data view no longer exists (like it was deleted), user is prompted with a warning, letting them know
  • To test create a rule with a data view, go to Stack Management --> Data Views --> delete the data view you selected, go back to the rule and click to edit

Rule Edit when error present

Rule Details

• User can see what data view they've selected - text displayed is the data view title, not name (name introduced 8.4 - we weren't aware till too late)
• 👀 If the selected data view no longer exists (like it was deleted), user is prompted with an error, letting them know that rule execution failed
  • To test create a rule with a data view, go to Stack Management --> Data Views --> delete the data view you selected, go back to the rule

Rule details when error present

Rule details

Rules Management

• User can select to bulk edit rule index patterns
  • If they select to apply changes to rules with data views, those rules will be updated as well
  • If they do not select to apply changes to rules with data views, those rules are not updated, but the rest are
• To test:
  • Create a rule with a data view and a rule with index patterns. Go to rule management and select both rules. Select 'Bulk actions' --> Add index patterns --> go through flow to add index pattern (don't select any boxes). The rule using index patterns should be updated, the rule using data views should not be.
  • Create a rule with a data view and a rule with index patterns. Go to rule management and select both rules. Select 'Bulk actions' --> Add index patterns --> go through flow to add index pattern --> check the box Apply changes to rules configured with data views. The rule using index patterns should be updated, the rule using data views should not be.

Bulk action flyout

Bulk action flyout success toast

Add/edit rule exception modal

• If rule list is assigned to is using runtime fields, runtime fields show as a field option to add an exception for