[Security Solution] Editing rules independently of source data (#180407…

…) (#191487)

## Summary

Addresses #180407
Addresses #178611

With these changes we allow user to create and update a rule even if
there are certain query bar validation error exist. Right now, we will
make any non-syntax validation errors in `EQL` and `ES|QL` rules types
to be non-blocking during the rule creation and rule updating workflows.

### Screenshot of the EQL rule creation workflow with existing
non-blocking validation errors:


https://github.com/user-attachments/assets/06b7f76c-e600-4a99-8ead-1445d429e9d3

### Screenshot of the EQL rule updating workflow with existing
non-blocking validation errors:


https://github.com/user-attachments/assets/9b35e113-b127-487b-bc23-afecf704db9d

## UPDATE

After discussing confirmation modal with @approksiu, we decided to
simplify it and show only title with generic description to avoid too be
too literal in the modal. User can see the full error description during
rule creation/editing workflows in the query bar where we show each
validation error as part of the query bar form item.

<img width="702" alt="Screenshot 2024-08-28 at 12 50 14"
src="https://github.com/user-attachments/assets/edfb791e-4e45-4fa5-8a46-c7e2772abdf9">

### Some test cases for local testing

<details>
  <summary><b>Create EQL rule with missing data source</b></summary>

#### Steps:
1. Open rules management page
2. Click create new rule button
3. Select EQL rule type
4. Set non-existing index in index patterns field
5. Add some valid EQL query (for example `any where true`)
6. Continue with other steps
7. Click create rule button

**Expected**: You will see the confirmation modal that warns user about
potentially failing rule executions. Clicking `Confirm` button will
create a rule.

</details>

<details>
  <summary><b>Create EQL rule with missing data field</b></summary>

#### Steps:
1. Open rules management page
2. Click create new rule button
3. Select EQL rule type
4. Set existing indices in index patterns field
5. Add some valid EQL query referring non-existing data field (for
example `any where agent.non_existing_field`)
6. Continue with other steps
7. Click create rule button

**Expected**: You will see the confirmation modal that warns user about
potentially failing rule executions. Clicking `Confirm` button will
create a rule.

</details>

<details>
<summary><b>Create EQL rule with syntax error in the query</b></summary>

#### Steps:
1. Open rules management page
2. Click create new rule button
3. Select EQL rule type
4. Set existing indices in index patterns field
5. Add some syntactically invalid EQL query (for example `hello world`)

**Expected**: The continue button does not allow user to proceed to the
About step due to existing syntax error.

</details>

<details>
  <summary><b>Create ES|QL rule with missing data source</b></summary>

#### Steps:
1. Open rules management page
2. Click create new rule button
3. Select ES|QL rule type
4. Add some valid ES|QL query with non-existing data source (for example
`from non-existing-index-* metadata _id, _version, _index | SORT
@timestamp`)
6. Continue with other steps
7. Click create rule button

**Expected**: You will see the confirmation modal that warns user about
potentially failing rule executions. Clicking `Confirm` button will
create a rule.

</details>

<details>
  <summary><b>Create ES|QL rule with missing data field</b></summary>

#### Steps:
1. Open rules management page
2. Click create new rule button
3. Select ES|QL rule type
4. Add some valid ES|QL query with non-existing data field (for example
`from logs-* metadata _id, _version, _index | SORT
agent.non_existing_field`)
6. Continue with other steps
7. Click create rule button

**Expected**: You will see the confirmation modal that warns user about
potentially failing rule executions. Clicking `Confirm` button will
create a rule.

</details>

<details>
<summary><b>Create ES|QL rule with syntax error in the
query</b></summary>

#### Steps:
1. Open rules management page
2. Click create new rule button
3. Select ES|QL rule type
4. Add some syntactically invalid ES|QL query (for example `hello
world`)

**Expected**: The continue button does not allow user to proceed to the
About step due to existing syntax error.

</details>

Same behaviour applies to the rule updating workflow. For example, you
can try to install one of the EQL or ES|QL rules that point to
non-existing data source or uses non-existing data field. User can still
update (add rule actions) to such installed pre-built rules.

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
  * elastic/security-docs#5758
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
* [Detection Engine -
Cypress](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6831)
(100 ESS & 100 Serverless)
* [Rule Management -
Cypress](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6830)
(100 ESS & 100 Serverless)

---------

Co-authored-by: Elastic Machine <[email protected]>
Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Vitalii Dmyterko <[email protected]>

x-pack/plugins/security_solution/public/common/hooks/eql/api.ts

@@ -36,17 +36,25 @@ interface Params {
   options: Omit<EqlOptionsSelected, 'query' | 'size'> | undefined;
 }
 
+export interface EqlResponseError {
+  code: EQL_ERROR_CODES;
+  messages?: string[];
+  error?: Error;
+}
+
+export interface ValidateEqlResponse {
+  valid: boolean;
+  error?: EqlResponseError;
+}
+
 export const validateEql = async ({
   data,
   dataViewTitle,
   query,
   signal,
   runtimeMappings,
   options,
-}: Params): Promise<{
-  valid: boolean;
-  error?: { code: EQL_ERROR_CODES; messages?: string[]; error?: Error };
-}> => {
+}: Params): Promise<ValidateEqlResponse> => {
   try {
     const { rawResponse: response } = await firstValueFrom(
       data.search.search<EqlSearchStrategyRequest, EqlSearchStrategyResponse>(

x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/eql_query_bar/validators.ts

@@ -12,6 +12,7 @@ import { isEqlRule } from '../../../../../common/detection_engine/utils';
 import { KibanaServices } from '../../../../common/lib/kibana';
 import type { DefineStepRule } from '../../../../detections/pages/detection_engine/rules/types';
 import { DataSourceType } from '../../../../detections/pages/detection_engine/rules/types';
+import type { EqlResponseError } from '../../../../common/hooks/eql/api';
 import { validateEql, EQL_ERROR_CODES } from '../../../../common/hooks/eql/api';
 import type { FieldValueQueryBar } from '../query_bar';
 import * as i18n from './translations';
@@ -47,6 +48,23 @@ export const debounceAsync = <Args extends unknown[], Result extends Promise<unk
   };
 };
 
+export const transformEqlResponseErrorToValidationError = (
+  responseError: EqlResponseError
+): ValidationError<EQL_ERROR_CODES> => {
+  if (responseError.error) {
+    return {
+      code: EQL_ERROR_CODES.FAILED_REQUEST,
+      message: i18n.EQL_VALIDATION_REQUEST_ERROR,
+      error: responseError.error,
+    };
+  }
+  return {
+    code: responseError.code,
+    message: '',
+    messages: responseError.messages,
+  };
+};
+
 export const eqlValidator = async (
   ...args: Parameters<ValidationFunc>
 ): Promise<ValidationError<EQL_ERROR_CODES> | void | undefined> => {
@@ -86,13 +104,8 @@ export const eqlValidator = async (
       options: eqlOptions,
     });
 
-    if (response?.valid === false) {
-      return {
-        code: response.error?.code,
-        message: '',
-        messages: response.error?.messages,
-        error: response.error?.error,
-      };
+    if (response?.valid === false && response.error) {
+      return transformEqlResponseErrorToValidationError(response.error);
     }
   } catch (error) {
     return {

x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/save_with_errors_confirmation/index.tsx

@@ -0,0 +1,41 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import React from 'react';
+
+import { EuiConfirmModal } from '@elastic/eui';
+
+import * as i18n from './translations';
+
+interface SaveWithErrorsModalProps {
+  errors: string[];
+  onCancel: () => void;
+  onConfirm: () => void;
+}
+
+const SaveWithErrorsModalComponent = ({
+  errors,
+  onCancel,
+  onConfirm,
+}: SaveWithErrorsModalProps) => {
+  return (
+    <EuiConfirmModal
+      data-test-subj="save-with-errors-confirmation-modal"
+      title={i18n.SAVE_WITH_ERRORS_MODAL_TITLE}
+      onCancel={onCancel}
+      onConfirm={onConfirm}
+      cancelButtonText={i18n.SAVE_WITH_ERRORS_CANCEL_BUTTON}
+      confirmButtonText={i18n.SAVE_WITH_ERRORS_CONFIRM_BUTTON}
+      defaultFocusedButton="confirm"
+    >
+      <>{i18n.SAVE_WITH_ERRORS_MODAL_MESSAGE(errors.length)}</>
+    </EuiConfirmModal>
+  );
+};
+
+export const SaveWithErrorsModal = React.memo(SaveWithErrorsModalComponent);
+SaveWithErrorsModal.displayName = 'SaveWithErrorsModal';

x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui/components/save_with_errors_confirmation/translations.ts

@@ -0,0 +1,36 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { i18n } from '@kbn/i18n';
+
+export const SAVE_WITH_ERRORS_MODAL_TITLE = i18n.translate(
+  'xpack.securitySolution.detectionEngine.createRule.saveWithErrorsModalTitle',
+  {
+    defaultMessage: 'This rule has validation errors',
+  }
+);
+
+export const SAVE_WITH_ERRORS_CANCEL_BUTTON = i18n.translate(
+  'xpack.securitySolution.detectionEngine.createRule.saveWithErrorsCancelButton',
+  {
+    defaultMessage: 'Cancel',
+  }
+);
+
+export const SAVE_WITH_ERRORS_CONFIRM_BUTTON = i18n.translate(
+  'xpack.securitySolution.detectionEngine.createRule.saveWithErrorsConfirmButton',
+  {
+    defaultMessage: 'Confirm',
+  }
+);
+
+export const SAVE_WITH_ERRORS_MODAL_MESSAGE = (errorsCount: number) =>
+  i18n.translate('xpack.securitySolution.detectionEngine.createRule.saveWithErrorsModalMessage', {
+    defaultMessage:
+      'This rule has {errorsCount} validation {errorsCount, plural, one {error} other {errors}} which can lead to failed rule executions, save anyway?',
+    values: { errorsCount },
+  });