Skip to content

Commit

Permalink
[8.x] [Cloud Security] Added filter support to graph API (elastic#199048
Browse files Browse the repository at this point in the history
) (elastic#199702)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Cloud Security] Added filter support to graph API
(elastic#199048)](elastic#199048)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Kfir
Peled","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-11T19:47:23Z","message":"[Cloud
Security] Added filter support to graph API (elastic#199048)\n\n##
Summary\r\n\r\nEnhances the graph API to support filtering by bool
query.\r\n\r\nGraph API is an internal API that hasn't been released yet
to ESS, and\r\nis not available yet on serverless (behind a feature-flag
in\r\nkibana.config) due to the above I don't consider it a breaking
change.\r\n\r\nPrevious API request body: \r\n\r\n```js\r\nquery:
schema.object({\r\n actorIds: schema.arrayOf(schema.string()),\r\n
eventIds: schema.arrayOf(schema.string()),\r\n // TODO: use zod for
range validation instead of config schema\r\n start:
schema.oneOf([schema.number(), schema.string()]),\r\n end:
schema.oneOf([schema.number(), schema.string()]),\r\n```\r\n\r\nNew API
request body:\r\n\r\n```js\r\n nodesLimit:
schema.maybe(schema.number()), // Maximum number of nodes in the graph
(currently the graph doesn't handle very well graph with over 100
nodes)\r\n showUnknownTarget: schema.maybe(schema.boolean()), // Whether
or not to return events that miss target.entity.id\r\n query:
schema.object({\r\n eventIds: schema.arrayOf(schema.string()), // Event
ids that triggered the alert, would be marked in red\r\n // TODO: use
zod for range validation instead of config schema\r\n start:
schema.oneOf([schema.number(), schema.string()]),\r\n end:
schema.oneOf([schema.number(), schema.string()]),\r\n esQuery:
schema.maybe( // elasticsearch's dsl bool query\r\n schema.object({\r\n
bool: schema.object({\r\n filter:
schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow'
}))),\r\n must: schema.maybe(schema.arrayOf(schema.object({}, {
unknowns: 'allow' }))),\r\n should:
schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow'
}))),\r\n must_not: schema.maybe(schema.arrayOf(schema.object({}, {
unknowns: 'allow' }))),\r\n }),\r\n })\r\n```\r\n\r\nNew field to the
graph API response (pseudo):\r\n\r\n```js\r\nmessages?:
ApiMessageCode[]\r\n\r\nenum ApiMessageCode {\r\n ReachedNodesLimit =
'REACHED_NODES_LIMIT',\r\n}\r\n```\r\n\r\n### How to test \r\n\r\nToggle
feature flag in
kibana.dev.yml\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental:
['graphVisualizationInFlyoutEnabled']\r\n```\r\n\r\nTo test through the
UI you can use the mocked data\r\n\r\n```bash\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit
\\ \r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n
--kibana-url http://elastic:changeme@localhost:5601\r\n\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_functional/es_archives/security_alerts
\\\r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n
--kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\n1. Go
to the alerts page\r\n2. Change the query time range to show alerts from
the 13th of October\r\n2024 (**IMPORTANT**)\r\n3. Open the alerts
flyout\r\n5. Scroll to see the graph visualization : D\r\n\r\n\r\nTo
test **only** the API you can use the mocked data\r\n\r\n```bash\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \\
\r\n--es-url http://elastic:changeme@localhost:9200 \\\r\n--kibana-url
http://elastic:changeme@localhost:5601\r\n```\r\n\r\nAnd through dev
tools:\r\n\r\n```\r\nPOST
kbn:/internal/cloud_security_posture/graph?apiVersion=1\r\n{\r\n
\"query\": {\r\n \"eventIds\": [],\r\n \"start\": \"now-1y/y\",\r\n
\"end\": \"now/d\",\r\n \"esQuery\": {\r\n \"bool\": {\r\n \"filter\":
[\r\n {\r\n \"match_phrase\": {\r\n \"actor.entity.id\":
\"[email protected]\"\r\n }\r\n }\r\n ]\r\n }\r\n }\r\n
}\r\n}\r\n```\r\n\r\n### Related PRs\r\n\r\n-
https://github.com/elastic/kibana/pull/196034\r\n-
https://github.com/elastic/kibana/pull/195307\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"160e626ab58bda7cfe442dbb276744f878eaaf90","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor"],"title":"[Cloud
Security] Added filter support to graph
API","number":199048,"url":"https://github.com/elastic/kibana/pull/199048","mergeCommit":{"message":"[Cloud
Security] Added filter support to graph API (elastic#199048)\n\n##
Summary\r\n\r\nEnhances the graph API to support filtering by bool
query.\r\n\r\nGraph API is an internal API that hasn't been released yet
to ESS, and\r\nis not available yet on serverless (behind a feature-flag
in\r\nkibana.config) due to the above I don't consider it a breaking
change.\r\n\r\nPrevious API request body: \r\n\r\n```js\r\nquery:
schema.object({\r\n actorIds: schema.arrayOf(schema.string()),\r\n
eventIds: schema.arrayOf(schema.string()),\r\n // TODO: use zod for
range validation instead of config schema\r\n start:
schema.oneOf([schema.number(), schema.string()]),\r\n end:
schema.oneOf([schema.number(), schema.string()]),\r\n```\r\n\r\nNew API
request body:\r\n\r\n```js\r\n nodesLimit:
schema.maybe(schema.number()), // Maximum number of nodes in the graph
(currently the graph doesn't handle very well graph with over 100
nodes)\r\n showUnknownTarget: schema.maybe(schema.boolean()), // Whether
or not to return events that miss target.entity.id\r\n query:
schema.object({\r\n eventIds: schema.arrayOf(schema.string()), // Event
ids that triggered the alert, would be marked in red\r\n // TODO: use
zod for range validation instead of config schema\r\n start:
schema.oneOf([schema.number(), schema.string()]),\r\n end:
schema.oneOf([schema.number(), schema.string()]),\r\n esQuery:
schema.maybe( // elasticsearch's dsl bool query\r\n schema.object({\r\n
bool: schema.object({\r\n filter:
schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow'
}))),\r\n must: schema.maybe(schema.arrayOf(schema.object({}, {
unknowns: 'allow' }))),\r\n should:
schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow'
}))),\r\n must_not: schema.maybe(schema.arrayOf(schema.object({}, {
unknowns: 'allow' }))),\r\n }),\r\n })\r\n```\r\n\r\nNew field to the
graph API response (pseudo):\r\n\r\n```js\r\nmessages?:
ApiMessageCode[]\r\n\r\nenum ApiMessageCode {\r\n ReachedNodesLimit =
'REACHED_NODES_LIMIT',\r\n}\r\n```\r\n\r\n### How to test \r\n\r\nToggle
feature flag in
kibana.dev.yml\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental:
['graphVisualizationInFlyoutEnabled']\r\n```\r\n\r\nTo test through the
UI you can use the mocked data\r\n\r\n```bash\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit
\\ \r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n
--kibana-url http://elastic:changeme@localhost:5601\r\n\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_functional/es_archives/security_alerts
\\\r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n
--kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\n1. Go
to the alerts page\r\n2. Change the query time range to show alerts from
the 13th of October\r\n2024 (**IMPORTANT**)\r\n3. Open the alerts
flyout\r\n5. Scroll to see the graph visualization : D\r\n\r\n\r\nTo
test **only** the API you can use the mocked data\r\n\r\n```bash\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \\
\r\n--es-url http://elastic:changeme@localhost:9200 \\\r\n--kibana-url
http://elastic:changeme@localhost:5601\r\n```\r\n\r\nAnd through dev
tools:\r\n\r\n```\r\nPOST
kbn:/internal/cloud_security_posture/graph?apiVersion=1\r\n{\r\n
\"query\": {\r\n \"eventIds\": [],\r\n \"start\": \"now-1y/y\",\r\n
\"end\": \"now/d\",\r\n \"esQuery\": {\r\n \"bool\": {\r\n \"filter\":
[\r\n {\r\n \"match_phrase\": {\r\n \"actor.entity.id\":
\"[email protected]\"\r\n }\r\n }\r\n ]\r\n }\r\n }\r\n
}\r\n}\r\n```\r\n\r\n### Related PRs\r\n\r\n-
https://github.com/elastic/kibana/pull/196034\r\n-
https://github.com/elastic/kibana/pull/195307\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"160e626ab58bda7cfe442dbb276744f878eaaf90"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199048","number":199048,"mergeCommit":{"message":"[Cloud
Security] Added filter support to graph API (elastic#199048)\n\n##
Summary\r\n\r\nEnhances the graph API to support filtering by bool
query.\r\n\r\nGraph API is an internal API that hasn't been released yet
to ESS, and\r\nis not available yet on serverless (behind a feature-flag
in\r\nkibana.config) due to the above I don't consider it a breaking
change.\r\n\r\nPrevious API request body: \r\n\r\n```js\r\nquery:
schema.object({\r\n actorIds: schema.arrayOf(schema.string()),\r\n
eventIds: schema.arrayOf(schema.string()),\r\n // TODO: use zod for
range validation instead of config schema\r\n start:
schema.oneOf([schema.number(), schema.string()]),\r\n end:
schema.oneOf([schema.number(), schema.string()]),\r\n```\r\n\r\nNew API
request body:\r\n\r\n```js\r\n nodesLimit:
schema.maybe(schema.number()), // Maximum number of nodes in the graph
(currently the graph doesn't handle very well graph with over 100
nodes)\r\n showUnknownTarget: schema.maybe(schema.boolean()), // Whether
or not to return events that miss target.entity.id\r\n query:
schema.object({\r\n eventIds: schema.arrayOf(schema.string()), // Event
ids that triggered the alert, would be marked in red\r\n // TODO: use
zod for range validation instead of config schema\r\n start:
schema.oneOf([schema.number(), schema.string()]),\r\n end:
schema.oneOf([schema.number(), schema.string()]),\r\n esQuery:
schema.maybe( // elasticsearch's dsl bool query\r\n schema.object({\r\n
bool: schema.object({\r\n filter:
schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow'
}))),\r\n must: schema.maybe(schema.arrayOf(schema.object({}, {
unknowns: 'allow' }))),\r\n should:
schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow'
}))),\r\n must_not: schema.maybe(schema.arrayOf(schema.object({}, {
unknowns: 'allow' }))),\r\n }),\r\n })\r\n```\r\n\r\nNew field to the
graph API response (pseudo):\r\n\r\n```js\r\nmessages?:
ApiMessageCode[]\r\n\r\nenum ApiMessageCode {\r\n ReachedNodesLimit =
'REACHED_NODES_LIMIT',\r\n}\r\n```\r\n\r\n### How to test \r\n\r\nToggle
feature flag in
kibana.dev.yml\r\n\r\n```yaml\r\nxpack.securitySolution.enableExperimental:
['graphVisualizationInFlyoutEnabled']\r\n```\r\n\r\nTo test through the
UI you can use the mocked data\r\n\r\n```bash\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_functional/es_archives/logs_gcp_audit
\\ \r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n
--kibana-url http://elastic:changeme@localhost:5601\r\n\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_functional/es_archives/security_alerts
\\\r\n --es-url http://elastic:changeme@localhost:9200 \\\r\n
--kibana-url http://elastic:changeme@localhost:5601\r\n```\r\n\r\n1. Go
to the alerts page\r\n2. Change the query time range to show alerts from
the 13th of October\r\n2024 (**IMPORTANT**)\r\n3. Open the alerts
flyout\r\n5. Scroll to see the graph visualization : D\r\n\r\n\r\nTo
test **only** the API you can use the mocked data\r\n\r\n```bash\r\nnode
scripts/es_archiver load
x-pack/test/cloud_security_posture_api/es_archives/logs_gcp_audit \\
\r\n--es-url http://elastic:changeme@localhost:9200 \\\r\n--kibana-url
http://elastic:changeme@localhost:5601\r\n```\r\n\r\nAnd through dev
tools:\r\n\r\n```\r\nPOST
kbn:/internal/cloud_security_posture/graph?apiVersion=1\r\n{\r\n
\"query\": {\r\n \"eventIds\": [],\r\n \"start\": \"now-1y/y\",\r\n
\"end\": \"now/d\",\r\n \"esQuery\": {\r\n \"bool\": {\r\n \"filter\":
[\r\n {\r\n \"match_phrase\": {\r\n \"actor.entity.id\":
\"[email protected]\"\r\n }\r\n }\r\n ]\r\n }\r\n }\r\n
}\r\n}\r\n```\r\n\r\n### Related PRs\r\n\r\n-
https://github.com/elastic/kibana/pull/196034\r\n-
https://github.com/elastic/kibana/pull/195307\r\n\r\n###
Checklist\r\n\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine
<[email protected]>","sha":"160e626ab58bda7cfe442dbb276744f878eaaf90"}}]}]
BACKPORT-->

Co-authored-by: Kfir Peled <[email protected]>
  • Loading branch information
kibanamachine and kfirpeled authored Nov 13, 2024
1 parent 1456012 commit 7a39e33
Show file tree
Hide file tree
Showing 13 changed files with 693 additions and 176 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,26 @@
*/

import { schema } from '@kbn/config-schema';
import { ApiMessageCode } from '../../types/graph/v1';

export const graphRequestSchema = schema.object({
nodesLimit: schema.maybe(schema.number()),
showUnknownTarget: schema.maybe(schema.boolean()),
query: schema.object({
actorIds: schema.arrayOf(schema.string()),
eventIds: schema.arrayOf(schema.string()),
// TODO: use zod for range validation instead of config schema
start: schema.oneOf([schema.number(), schema.string()]),
end: schema.oneOf([schema.number(), schema.string()]),
esQuery: schema.maybe(
schema.object({
bool: schema.object({
filter: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
must: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
should: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
must_not: schema.maybe(schema.arrayOf(schema.object({}, { unknowns: 'allow' }))),
}),
})
),
}),
});

Expand All @@ -23,6 +35,9 @@ export const graphResponseSchema = () =>
schema.oneOf([entityNodeDataSchema, groupNodeDataSchema, labelNodeDataSchema])
),
edges: schema.arrayOf(edgeDataSchema),
messages: schema.maybe(
schema.arrayOf(schema.oneOf([schema.literal(ApiMessageCode.ReachedNodesLimit)]))
),
});

export const colorSchema = schema.oneOf([
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,5 +20,6 @@
"@kbn/i18n",
"@kbn/analytics",
"@kbn/usage-collection-plugin",
"@kbn/es-query",
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
*/

import type { TypeOf } from '@kbn/config-schema';
import type { BoolQuery } from '@kbn/es-query';
import {
colorSchema,
edgeDataSchema,
Expand All @@ -17,13 +18,21 @@ import {
nodeShapeSchema,
} from '../../schema/graph/v1';

export type GraphRequest = TypeOf<typeof graphRequestSchema>;
export type GraphResponse = TypeOf<typeof graphResponseSchema>;
export type GraphRequest = Omit<TypeOf<typeof graphRequestSchema>, 'query.esQuery'> & {
query: { esQuery?: { bool: Partial<BoolQuery> } };
};
export type GraphResponse = Omit<TypeOf<typeof graphResponseSchema>, 'messages'> & {
messages?: ApiMessageCode[];
};

export type Color = typeof colorSchema.type;

export type NodeShape = TypeOf<typeof nodeShapeSchema>;

export enum ApiMessageCode {
ReachedNodesLimit = 'REACHED_NODES_LIMIT',
}

export type EntityNodeDataModel = TypeOf<typeof entityNodeDataSchema>;

export type GroupNodeDataModel = TypeOf<typeof groupNodeDataSchema>;
Expand Down
20 changes: 12 additions & 8 deletions x-pack/plugins/cloud_security_posture/server/routes/graph/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import {
graphResponseSchema,
} from '@kbn/cloud-security-posture-common/schema/graph/latest';
import { transformError } from '@kbn/securitysolution-es-utils';
import type { GraphRequest } from '@kbn/cloud-security-posture-common/types/graph/v1';
import { GRAPH_ROUTE_PATH } from '../../../common/constants';
import { CspRouter } from '../../types';
import { getGraph as getGraphV1 } from './v1';
Expand Down Expand Up @@ -39,26 +40,29 @@ export const defineGraphRoute = (router: CspRouter) =>
},
},
async (context, request, response) => {
const { actorIds, eventIds, start, end } = request.body.query;
const { nodesLimit, showUnknownTarget = false } = request.body;
const { eventIds, start, end, esQuery } = request.body.query as GraphRequest['query'];
const cspContext = await context.csp;
const spaceId = (await cspContext.spaces?.spacesService?.getActiveSpace(request))?.id;

try {
const { nodes, edges } = await getGraphV1(
{
const resp = await getGraphV1({
services: {
logger: cspContext.logger,
esClient: cspContext.esClient,
},
{
actorIds,
query: {
eventIds,
spaceId,
start,
end,
}
);
esQuery,
},
showUnknownTarget,
nodesLimit,
});

return response.ok({ body: { nodes, edges } });
return response.ok({ body: resp });
} catch (err) {
const error = transformError(err);
cspContext.logger.error(`Failed to fetch graph ${err}`);
Expand Down
23 changes: 0 additions & 23 deletions x-pack/plugins/cloud_security_posture/server/routes/graph/types.ts

This file was deleted.

Loading

0 comments on commit 7a39e33

Please sign in to comment.