Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[8.13] [Security Solution] Incorporates EQL options in EQL query vali…
…dation on both Rule Creation and Timeline (#178468) (#178994) # Backport This will backport the following commits from `main` to `8.13`: - [[Security Solution] Incorporates EQL options in EQL query validation on both Rule Creation and Timeline (#178468)](#178468) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ryland Herrick","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-03-19T17:52:35Z","message":"[Security Solution] Incorporates EQL options in EQL query validation on both Rule Creation and Timeline (#178468)\n\n## Summary\r\n\r\nThis PR updates the Detection Rule Creation and Timeline forms to\r\naccount for the new(er) EQL options fields (`timestamp_field`,\r\n`event_category_field`, and `tiebreaker_field`) when validating the EQL\r\nquery. While the rule query and timeline query (respectively) would\r\ncorrectly persist and use these options, they were unused during EQL\r\nvalidation, meaning that in certain situations it was impossible to\r\nproduce a \"valid\" (from the perspective of the form) EQL query if your\r\ndata necessitated one of those options (see\r\nhttps://github.com//issues/158326 for more details).\r\n\r\n### Changes\r\nThe above was accomplished by:\r\n\r\n1. Adding a hidden form field to the UI for `eqlOptions`\r\n* A lack of a field component meant that `formData` never held any value\r\nfor this field\r\n* This meant that the form-based validations (i.e. `eqlValidator`) would\r\nnot have access to `eqlOptions`\r\n2. Updating the `onOptionsChange` hook to additionally inform the form\r\nof changes to `eqlOptions`\r\n* This allows the form to have updated values for `eqlOptions` with\r\nminimal changes\r\n* Not ideal from a data-flow perspective, since the field changes are\r\nupdating two copies of `eqlOptions`. However, we're only using the \"form\r\ncopy\" for the validation call, and everything else works as it did\r\nbefore.\r\n\r\n \r\n### Additional changes\r\n__Update__: I've reverted the described changes to the default timeline\r\nvalues below, and moved that to a [separate\r\nissue](https://github.com/elastic/security-team/issues/8892).\r\n\r\nIn the course of performing the above tasks for timeline, I found the\r\nbehavior there to be slightly different: we were specifying default\r\nvalues for these fields, which was redundant in some cases\r\n(`timestamp_field`, `event_category_field`), and invalid in the case of\r\n`tiebreaker_field`. You can see here how the `tiebreaker_field` must be\r\nchanged to make the query valid:\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/657252/556bc998-c147-4825-8bde-5d8d03873e75\r\n\r\n\r\n~~I opted to remove these default values, as well as the defaulting\r\nbehavior from the search strategy call (see\r\n10a47fdaba5860a44638aada78c0cf8c37b2c580). While these values may be\r\npersisted in existing saved timelines, this makes the EQL workflow much\r\ncleaner for new users, and those users with persisted EQL options can\r\nnow modify and validate those options on the fly.~~\r\n\r\n#### Steps to Review\r\n\r\nFor @elastic/kibana-visualizations and/or @elastic/kibana-data-discovery\r\n: I've added a few tests to the EQL search strategy that document\r\nexisting behavior RE EQL option fields. Nothing about the search\r\nstrategy itself was changed. If you think the tests aren't useful, I'm\r\nhappy to delete them.\r\n\r\n1. Create an index without a `@timestamp` field, and insert some data:\r\n <details>\r\n <summary>Dev Tools</summary>\r\n\r\n PUT timestamp/\r\n {\r\n \"mappings\": {\r\n \"dynamic\": \"strict\",\r\n \"properties\": {\r\n \"timestamp\": {\r\n \"type\": \"date\",\r\n \"format\": \"epoch_second\"\r\n },\r\n \"locale\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"created_at\": {\r\n \"type\": \"date\",\r\n \"format\": \"epoch_second\"\r\n },\r\n \"event\": {\r\n \"properties\": {\r\n \"category\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n }\r\n\r\n PUT timestamp/_doc/4\r\n {\r\n \"timestamp\": 1710211630,\r\n \"locale\": \"es\",\r\n \"created_at\": 1710211630,\r\n \"event.category\": \"configuration\"\r\n }\r\n </details>\r\n2. Create a new EQL detection rule, and choose `timestamp` as the index\r\npattern\r\n3. Type `any where true` as the query, and observe a validation error\r\n(`Found 1 problem line -1:-1: Unknown column [@timestamp]`)\r\n4. Open EQL Settings popover, and select `timestamp` from the `Timestamp\r\nfield` dropdown\r\n5. Observe that the query has been revalidated, and found valid.\r\n\r\n\r\n### New Behavior:\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/657252/22a63363-d597-4dfd-8f9a-647f4084ec0e\r\n\r\n\r\n\r\n### Related Issues\r\n* Addresses https://github.com/elastic/kibana/issues/158326\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n### For maintainers\r\n\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"5be91c94b1c7e6b1aebd5fe75c86e2bc17c90ad0","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Timeline","Feature:Event Correlation (EQL) Rule","Feature:Rule Creation","backport:prev-minor","Team:Detection Engine","v8.14.0"],"title":"[Security Solution] Incorporates EQL options in EQL query validation on both Rule Creation and Timeline","number":178468,"url":"https://github.com/elastic/kibana/pull/178468","mergeCommit":{"message":"[Security Solution] Incorporates EQL options in EQL query validation on both Rule Creation and Timeline (#178468)\n\n## Summary\r\n\r\nThis PR updates the Detection Rule Creation and Timeline forms to\r\naccount for the new(er) EQL options fields (`timestamp_field`,\r\n`event_category_field`, and `tiebreaker_field`) when validating the EQL\r\nquery. While the rule query and timeline query (respectively) would\r\ncorrectly persist and use these options, they were unused during EQL\r\nvalidation, meaning that in certain situations it was impossible to\r\nproduce a \"valid\" (from the perspective of the form) EQL query if your\r\ndata necessitated one of those options (see\r\nhttps://github.com//issues/158326 for more details).\r\n\r\n### Changes\r\nThe above was accomplished by:\r\n\r\n1. Adding a hidden form field to the UI for `eqlOptions`\r\n* A lack of a field component meant that `formData` never held any value\r\nfor this field\r\n* This meant that the form-based validations (i.e. `eqlValidator`) would\r\nnot have access to `eqlOptions`\r\n2. Updating the `onOptionsChange` hook to additionally inform the form\r\nof changes to `eqlOptions`\r\n* This allows the form to have updated values for `eqlOptions` with\r\nminimal changes\r\n* Not ideal from a data-flow perspective, since the field changes are\r\nupdating two copies of `eqlOptions`. However, we're only using the \"form\r\ncopy\" for the validation call, and everything else works as it did\r\nbefore.\r\n\r\n \r\n### Additional changes\r\n__Update__: I've reverted the described changes to the default timeline\r\nvalues below, and moved that to a [separate\r\nissue](https://github.com/elastic/security-team/issues/8892).\r\n\r\nIn the course of performing the above tasks for timeline, I found the\r\nbehavior there to be slightly different: we were specifying default\r\nvalues for these fields, which was redundant in some cases\r\n(`timestamp_field`, `event_category_field`), and invalid in the case of\r\n`tiebreaker_field`. You can see here how the `tiebreaker_field` must be\r\nchanged to make the query valid:\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/657252/556bc998-c147-4825-8bde-5d8d03873e75\r\n\r\n\r\n~~I opted to remove these default values, as well as the defaulting\r\nbehavior from the search strategy call (see\r\n10a47fdaba5860a44638aada78c0cf8c37b2c580). While these values may be\r\npersisted in existing saved timelines, this makes the EQL workflow much\r\ncleaner for new users, and those users with persisted EQL options can\r\nnow modify and validate those options on the fly.~~\r\n\r\n#### Steps to Review\r\n\r\nFor @elastic/kibana-visualizations and/or @elastic/kibana-data-discovery\r\n: I've added a few tests to the EQL search strategy that document\r\nexisting behavior RE EQL option fields. Nothing about the search\r\nstrategy itself was changed. If you think the tests aren't useful, I'm\r\nhappy to delete them.\r\n\r\n1. Create an index without a `@timestamp` field, and insert some data:\r\n <details>\r\n <summary>Dev Tools</summary>\r\n\r\n PUT timestamp/\r\n {\r\n \"mappings\": {\r\n \"dynamic\": \"strict\",\r\n \"properties\": {\r\n \"timestamp\": {\r\n \"type\": \"date\",\r\n \"format\": \"epoch_second\"\r\n },\r\n \"locale\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"created_at\": {\r\n \"type\": \"date\",\r\n \"format\": \"epoch_second\"\r\n },\r\n \"event\": {\r\n \"properties\": {\r\n \"category\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n }\r\n\r\n PUT timestamp/_doc/4\r\n {\r\n \"timestamp\": 1710211630,\r\n \"locale\": \"es\",\r\n \"created_at\": 1710211630,\r\n \"event.category\": \"configuration\"\r\n }\r\n </details>\r\n2. Create a new EQL detection rule, and choose `timestamp` as the index\r\npattern\r\n3. Type `any where true` as the query, and observe a validation error\r\n(`Found 1 problem line -1:-1: Unknown column [@timestamp]`)\r\n4. Open EQL Settings popover, and select `timestamp` from the `Timestamp\r\nfield` dropdown\r\n5. Observe that the query has been revalidated, and found valid.\r\n\r\n\r\n### New Behavior:\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/657252/22a63363-d597-4dfd-8f9a-647f4084ec0e\r\n\r\n\r\n\r\n### Related Issues\r\n* Addresses https://github.com/elastic/kibana/issues/158326\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n### For maintainers\r\n\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"5be91c94b1c7e6b1aebd5fe75c86e2bc17c90ad0"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.14.0","branchLabelMappingKey":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/178468","number":178468,"mergeCommit":{"message":"[Security Solution] Incorporates EQL options in EQL query validation on both Rule Creation and Timeline (#178468)\n\n## Summary\r\n\r\nThis PR updates the Detection Rule Creation and Timeline forms to\r\naccount for the new(er) EQL options fields (`timestamp_field`,\r\n`event_category_field`, and `tiebreaker_field`) when validating the EQL\r\nquery. While the rule query and timeline query (respectively) would\r\ncorrectly persist and use these options, they were unused during EQL\r\nvalidation, meaning that in certain situations it was impossible to\r\nproduce a \"valid\" (from the perspective of the form) EQL query if your\r\ndata necessitated one of those options (see\r\nhttps://github.com//issues/158326 for more details).\r\n\r\n### Changes\r\nThe above was accomplished by:\r\n\r\n1. Adding a hidden form field to the UI for `eqlOptions`\r\n* A lack of a field component meant that `formData` never held any value\r\nfor this field\r\n* This meant that the form-based validations (i.e. `eqlValidator`) would\r\nnot have access to `eqlOptions`\r\n2. Updating the `onOptionsChange` hook to additionally inform the form\r\nof changes to `eqlOptions`\r\n* This allows the form to have updated values for `eqlOptions` with\r\nminimal changes\r\n* Not ideal from a data-flow perspective, since the field changes are\r\nupdating two copies of `eqlOptions`. However, we're only using the \"form\r\ncopy\" for the validation call, and everything else works as it did\r\nbefore.\r\n\r\n \r\n### Additional changes\r\n__Update__: I've reverted the described changes to the default timeline\r\nvalues below, and moved that to a [separate\r\nissue](https://github.com/elastic/security-team/issues/8892).\r\n\r\nIn the course of performing the above tasks for timeline, I found the\r\nbehavior there to be slightly different: we were specifying default\r\nvalues for these fields, which was redundant in some cases\r\n(`timestamp_field`, `event_category_field`), and invalid in the case of\r\n`tiebreaker_field`. You can see here how the `tiebreaker_field` must be\r\nchanged to make the query valid:\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/657252/556bc998-c147-4825-8bde-5d8d03873e75\r\n\r\n\r\n~~I opted to remove these default values, as well as the defaulting\r\nbehavior from the search strategy call (see\r\n10a47fdaba5860a44638aada78c0cf8c37b2c580). While these values may be\r\npersisted in existing saved timelines, this makes the EQL workflow much\r\ncleaner for new users, and those users with persisted EQL options can\r\nnow modify and validate those options on the fly.~~\r\n\r\n#### Steps to Review\r\n\r\nFor @elastic/kibana-visualizations and/or @elastic/kibana-data-discovery\r\n: I've added a few tests to the EQL search strategy that document\r\nexisting behavior RE EQL option fields. Nothing about the search\r\nstrategy itself was changed. If you think the tests aren't useful, I'm\r\nhappy to delete them.\r\n\r\n1. Create an index without a `@timestamp` field, and insert some data:\r\n <details>\r\n <summary>Dev Tools</summary>\r\n\r\n PUT timestamp/\r\n {\r\n \"mappings\": {\r\n \"dynamic\": \"strict\",\r\n \"properties\": {\r\n \"timestamp\": {\r\n \"type\": \"date\",\r\n \"format\": \"epoch_second\"\r\n },\r\n \"locale\": {\r\n \"type\": \"keyword\"\r\n },\r\n \"created_at\": {\r\n \"type\": \"date\",\r\n \"format\": \"epoch_second\"\r\n },\r\n \"event\": {\r\n \"properties\": {\r\n \"category\": {\r\n \"ignore_above\": 1024,\r\n \"type\": \"keyword\"\r\n }\r\n }\r\n }\r\n }\r\n }\r\n }\r\n\r\n PUT timestamp/_doc/4\r\n {\r\n \"timestamp\": 1710211630,\r\n \"locale\": \"es\",\r\n \"created_at\": 1710211630,\r\n \"event.category\": \"configuration\"\r\n }\r\n </details>\r\n2. Create a new EQL detection rule, and choose `timestamp` as the index\r\npattern\r\n3. Type `any where true` as the query, and observe a validation error\r\n(`Found 1 problem line -1:-1: Unknown column [@timestamp]`)\r\n4. Open EQL Settings popover, and select `timestamp` from the `Timestamp\r\nfield` dropdown\r\n5. Observe that the query has been revalidated, and found valid.\r\n\r\n\r\n### New Behavior:\r\n\r\n\r\nhttps://github.com/elastic/kibana/assets/657252/22a63363-d597-4dfd-8f9a-647f4084ec0e\r\n\r\n\r\n\r\n### Related Issues\r\n* Addresses https://github.com/elastic/kibana/issues/158326\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n### For maintainers\r\n\r\n- [x] This was checked for breaking API changes and was [labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"5be91c94b1c7e6b1aebd5fe75c86e2bc17c90ad0"}}]}] BACKPORT--> --------- Co-authored-by: Ryland Herrick <[email protected]>
- Loading branch information
1 parent
1faa537
commit 5814aa9