Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[8.x] [Cloud Security] Added filter support to graph API (elastic#199048
) (elastic#199702) # Backport This will backport the following commits from `main` to `8.x`: - [[Cloud Security] Added filter support to graph API (elastic#199048)](elastic#199048) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Kfir Peled","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-11T19:47:23Z","message":"[Cloud Security] Added filter support to graph API (elastic#199048)\n\n## Summary\r\n\r\nEnhances the graph API to support filtering by bool query.\r\n\r\nGraph API is an internal API that hasn't been released yet to ESS, and\r\nis not available yet on serverless (behind a feature-flag in\r\nkibana.config) due to the above I don't consider it a breaking change.\r\n\r\nPrevious API request body: \r\n\r\n```js\r\nquery: schema.object({\r\n actorIds: schema.arrayOf(schema.string()),\r\n eventIds: schema.arrayOf(schema.string()),\r\n // TODO: use zod for range validation instead of config schema\r\n start: schema.oneOf([schema.number(), schema.string()]),\r\n end: schema.oneOf([schema.number(), schema.string()]),\r\n```\r\n\r\nNew API request body:\r\n\r\n```js\r\n nodesLimit: schema.maybe(schema.number()), // Maximum number of nodes in the graph (currently the graph doesn't handle very well graph with over 100 nodes)\r\n showUnknownTarget: schema.maybe(schema.boolean()), // Whether or not to return events that miss target.entity.id\r\n query: schema.object({\r\n eventIds: schema.arrayOf(schema.string()), // Event ids that triggered the alert, would be marked in red\r\n // TODO: use zod for range validation instead of config schema\r\n start: schema.oneOf([schema.number(), schema.string()]),\r\n end: schema.oneOf([schema.number(), schema.string()]),\r\n esQuery: schema.maybe( // elasticsearch's dsl bool query\r\n schema.object({\r\n bool: schema.object({\r\n filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n }),\r\n })\r\n```\r\n\r\nNew field to the graph API response (pseudo):\r\n\r\n```js\r\nmessages?: ApiMessageCode[]\r\n\r\nenum ApiMessageCode {\r\n ReachedNodesLimit = 'REACHED_NODES_LIMIT',\r\n}\r\n```\r\n\r\n### How to test \r\n\r\nToggle feature flag in kibana.dev.yml\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental: ['graphVisualizationInFlyoutEnabled']\r\n```\r\n\r\nTo test through the UI you can use the mocked data\r\n\r\n```bash\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \\ \r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n --kibana-url http://elastic:changeme@localhost:5601\r\n\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \\\r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n --kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\n1. Go to the alerts page\r\n2. Change the query time range to show alerts from the 13th of October\r\n2024 (**IMPORTANT**)\r\n3. Open the alerts flyout\r\n5. Scroll to see the graph visualization : D\r\n\r\n\r\nTo test **only** the API you can use the mocked data\r\n\r\n```bash\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \\ \r\n--es-url http://elastic:changeme@localhost:9200 \\\r\n--kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\nAnd through dev tools:\r\n\r\n```\r\nPOST kbn:/internal/cloud_security_posture/graph?apiVersion=1\r\n{\r\n \"query\": {\r\n \"eventIds\": [],\r\n \"start\": \"now-1y/y\",\r\n \"end\": \"now/d\",\r\n \"esQuery\": {\r\n \"bool\": {\r\n \"filter\": [\r\n {\r\n \"match_phrase\": {\r\n \"actor.entity.id\": \"[email protected]\"\r\n }\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\n### Related PRs\r\n\r\n- https://github.com/elastic/kibana/pull/196034\r\n- https://github.com/elastic/kibana/pull/195307\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"160e626ab58bda7cfe442dbb276744f878eaaf90","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor"],"title":"[Cloud Security] Added filter support to graph API","number":199048,"url":"https://github.com/elastic/kibana/pull/199048","mergeCommit":{"message":"[Cloud Security] Added filter support to graph API (elastic#199048)\n\n## Summary\r\n\r\nEnhances the graph API to support filtering by bool query.\r\n\r\nGraph API is an internal API that hasn't been released yet to ESS, and\r\nis not available yet on serverless (behind a feature-flag in\r\nkibana.config) due to the above I don't consider it a breaking change.\r\n\r\nPrevious API request body: \r\n\r\n```js\r\nquery: schema.object({\r\n actorIds: schema.arrayOf(schema.string()),\r\n eventIds: schema.arrayOf(schema.string()),\r\n // TODO: use zod for range validation instead of config schema\r\n start: schema.oneOf([schema.number(), schema.string()]),\r\n end: schema.oneOf([schema.number(), schema.string()]),\r\n```\r\n\r\nNew API request body:\r\n\r\n```js\r\n nodesLimit: schema.maybe(schema.number()), // Maximum number of nodes in the graph (currently the graph doesn't handle very well graph with over 100 nodes)\r\n showUnknownTarget: schema.maybe(schema.boolean()), // Whether or not to return events that miss target.entity.id\r\n query: schema.object({\r\n eventIds: schema.arrayOf(schema.string()), // Event ids that triggered the alert, would be marked in red\r\n // TODO: use zod for range validation instead of config schema\r\n start: schema.oneOf([schema.number(), schema.string()]),\r\n end: schema.oneOf([schema.number(), schema.string()]),\r\n esQuery: schema.maybe( // elasticsearch's dsl bool query\r\n schema.object({\r\n bool: schema.object({\r\n filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n }),\r\n })\r\n```\r\n\r\nNew field to the graph API response (pseudo):\r\n\r\n```js\r\nmessages?: ApiMessageCode[]\r\n\r\nenum ApiMessageCode {\r\n ReachedNodesLimit = 'REACHED_NODES_LIMIT',\r\n}\r\n```\r\n\r\n### How to test \r\n\r\nToggle feature flag in kibana.dev.yml\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental: ['graphVisualizationInFlyoutEnabled']\r\n```\r\n\r\nTo test through the UI you can use the mocked data\r\n\r\n```bash\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \\ \r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n --kibana-url http://elastic:changeme@localhost:5601\r\n\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \\\r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n --kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\n1. Go to the alerts page\r\n2. Change the query time range to show alerts from the 13th of October\r\n2024 (**IMPORTANT**)\r\n3. Open the alerts flyout\r\n5. Scroll to see the graph visualization : D\r\n\r\n\r\nTo test **only** the API you can use the mocked data\r\n\r\n```bash\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \\ \r\n--es-url http://elastic:changeme@localhost:9200 \\\r\n--kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\nAnd through dev tools:\r\n\r\n```\r\nPOST kbn:/internal/cloud_security_posture/graph?apiVersion=1\r\n{\r\n \"query\": {\r\n \"eventIds\": [],\r\n \"start\": \"now-1y/y\",\r\n \"end\": \"now/d\",\r\n \"esQuery\": {\r\n \"bool\": {\r\n \"filter\": [\r\n {\r\n \"match_phrase\": {\r\n \"actor.entity.id\": \"[email protected]\"\r\n }\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\n### Related PRs\r\n\r\n- https://github.com/elastic/kibana/pull/196034\r\n- https://github.com/elastic/kibana/pull/195307\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"160e626ab58bda7cfe442dbb276744f878eaaf90"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199048","number":199048,"mergeCommit":{"message":"[Cloud Security] Added filter support to graph API (elastic#199048)\n\n## Summary\r\n\r\nEnhances the graph API to support filtering by bool query.\r\n\r\nGraph API is an internal API that hasn't been released yet to ESS, and\r\nis not available yet on serverless (behind a feature-flag in\r\nkibana.config) due to the above I don't consider it a breaking change.\r\n\r\nPrevious API request body: \r\n\r\n```js\r\nquery: schema.object({\r\n actorIds: schema.arrayOf(schema.string()),\r\n eventIds: schema.arrayOf(schema.string()),\r\n // TODO: use zod for range validation instead of config schema\r\n start: schema.oneOf([schema.number(), schema.string()]),\r\n end: schema.oneOf([schema.number(), schema.string()]),\r\n```\r\n\r\nNew API request body:\r\n\r\n```js\r\n nodesLimit: schema.maybe(schema.number()), // Maximum number of nodes in the graph (currently the graph doesn't handle very well graph with over 100 nodes)\r\n showUnknownTarget: schema.maybe(schema.boolean()), // Whether or not to return events that miss target.entity.id\r\n query: schema.object({\r\n eventIds: schema.arrayOf(schema.string()), // Event ids that triggered the alert, would be marked in red\r\n // TODO: use zod for range validation instead of config schema\r\n start: schema.oneOf([schema.number(), schema.string()]),\r\n end: schema.oneOf([schema.number(), schema.string()]),\r\n esQuery: schema.maybe( // elasticsearch's dsl bool query\r\n schema.object({\r\n bool: schema.object({\r\n filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),\r\n }),\r\n })\r\n```\r\n\r\nNew field to the graph API response (pseudo):\r\n\r\n```js\r\nmessages?: ApiMessageCode[]\r\n\r\nenum ApiMessageCode {\r\n ReachedNodesLimit = 'REACHED_NODES_LIMIT',\r\n}\r\n```\r\n\r\n### How to test \r\n\r\nToggle feature flag in kibana.dev.yml\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental: ['graphVisualizationInFlyoutEnabled']\r\n```\r\n\r\nTo test through the UI you can use the mocked data\r\n\r\n```bash\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit \\ \r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n --kibana-url http://elastic:changeme@localhost:5601\r\n\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_functional/es_archives/security_alerts \\\r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n --kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\n1. Go to the alerts page\r\n2. Change the query time range to show alerts from the 13th of October\r\n2024 (**IMPORTANT**)\r\n3. Open the alerts flyout\r\n5. Scroll to see the graph visualization : D\r\n\r\n\r\nTo test **only** the API you can use the mocked data\r\n\r\n```bash\r\nnode scripts/es_archiver load x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \\ \r\n--es-url http://elastic:changeme@localhost:9200 \\\r\n--kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\nAnd through dev tools:\r\n\r\n```\r\nPOST kbn:/internal/cloud_security_posture/graph?apiVersion=1\r\n{\r\n \"query\": {\r\n \"eventIds\": [],\r\n \"start\": \"now-1y/y\",\r\n \"end\": \"now/d\",\r\n \"esQuery\": {\r\n \"bool\": {\r\n \"filter\": [\r\n {\r\n \"match_phrase\": {\r\n \"actor.entity.id\": \"[email protected]\"\r\n }\r\n }\r\n ]\r\n }\r\n }\r\n }\r\n}\r\n```\r\n\r\n### Related PRs\r\n\r\n- https://github.com/elastic/kibana/pull/196034\r\n- https://github.com/elastic/kibana/pull/195307\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"160e626ab58bda7cfe442dbb276744f878eaaf90"}}]}] BACKPORT--> Co-authored-by: Kfir Peled <[email protected]>
- Loading branch information