Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Restructuring folders of Detection Engine + refactoring Rule Management #142950

Merged
merged 29 commits into from
Oct 21, 2022
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
2eef5de
Preliminary refactoring: folder restructuring and creating API hooks
banderror Oct 13, 2022
20a7de5
Further rules table refactoring
xcrzx Oct 13, 2022
f03f3d1
Fix CI: linting errors, type errors, tests
banderror Oct 13, 2022
5babefe
Restructuring common folder, take 1
banderror Oct 14, 2022
4657f76
Cleanup
xcrzx Oct 14, 2022
872394b
Update query hooks
xcrzx Oct 14, 2022
323db29
Fix name collision
xcrzx Oct 14, 2022
9eb427b
Restructuring common folder, take 2
banderror Oct 14, 2022
87b7299
Restructuring common folder, take 3
banderror Oct 15, 2022
fe818a1
Fix tests
xcrzx Oct 16, 2022
1a0f5ba
Fix tests
xcrzx Oct 16, 2022
31c5a82
Restructuring common folder, take 4
banderror Oct 16, 2022
1e79428
Restructuring common folder, take 5
banderror Oct 16, 2022
4a08b8b
Restructuring common folder, take 6
banderror Oct 17, 2022
5abd594
Restructuring common folder, take 7
banderror Oct 17, 2022
ae573dc
Fix errors
banderror Oct 17, 2022
49d6f94
Update CODEOWNERS
banderror Oct 17, 2022
95c0926
Fix tests
banderror Oct 17, 2022
1bb6651
Fix tests
xcrzx Oct 17, 2022
38c27c5
Test fixes
xcrzx Oct 17, 2022
c9476c6
Fix tests
xcrzx Oct 18, 2022
a55d91d
Fix errors after rebase
banderror Oct 18, 2022
cf04cf4
Fix some TODOs
xcrzx Oct 18, 2022
25cc4f8
Update and unskip more tests
xcrzx Oct 18, 2022
66dc2ad
Import bulk action types NOT via index.ts
banderror Oct 18, 2022
0b78136
Merge main into onweek-refactoring-rule-management
banderror Oct 20, 2022
8f55566
Fix post-merge errors
banderror Oct 20, 2022
d6363ee
Address comments
banderror Oct 20, 2022
df5dfd7
Merge branch 'main' into onweek-refactoring-rule-management
banderror Oct 21, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
37 changes: 21 additions & 16 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -477,26 +477,31 @@ x-pack/examples/files_example @elastic/kibana-app-services
/x-pack/plugins/security_solution/common/detection_engine/schemas/alerts @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/common/field_maps @elastic/security-detections-response-alerts

/x-pack/plugins/security_solution/public/detection_engine/rule_creation_ui @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/public/detections/pages/alerts @elastic/security-detections-response-alerts

/x-pack/plugins/security_solution/server/lib/detection_engine/migrations @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/notifications @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/schemas @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/rule_preview @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/rule_types @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/signals @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/index @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/signals @elastic/security-detections-response-alerts

## Security Solution sub teams - Detections and Response Rules
/x-pack/plugins/security_solution/common/detection_engine/fleet_integrations @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/prebuilt_rules @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/rule_management @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/rule_monitoring @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/schemas/common @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/schemas/request @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/schemas/response @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/common/detection_engine/rule_schema @elastic/security-detections-response-rules
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to add @elastic/security-detections-response-alerts here and on /x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema since our area does a fair amount of work adding fields to the schema, migrating rules, etc. The rule schemas are one of the few places I think shared ownership makes sense so both teams get pinged.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marshallmain I thought you would like to stop owning it, but apparently, I misunderstood you. Shared ownership makes total sense to me here. Will add.


/x-pack/plugins/security_solution/public/common/components/health_truncate_text @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/common/components/links_to_docs @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/common/components/ml_popover @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/common/components/popover_items @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detection_engine/fleet_integrations @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detection_engine/rule_details_ui @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detection_engine/rule_management @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detection_engine/rule_management_ui @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detection_engine/rule_monitoring @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detections/components/callouts @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/detections/components/modals/ml_job_upgrade_modal @elastic/security-detections-response-rules
Expand All @@ -507,17 +512,12 @@ x-pack/examples/files_example @elastic/kibana-app-services
/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/public/rules @elastic/security-detections-response-rules

/x-pack/plugins/security_solution/server/lib/detection_engine/routes/fleet @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/create_rule_exceptions_route* @elastic/security-solution-platform
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rule_exceptions_route* @elastic/security-solution-platform
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/import_rules_route* @elastic/security-solution-platform
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/preview_rules_route* @elastic/security-detections-response-alerts
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/utils @elastic/security-solution-platform
/x-pack/plugins/security_solution/server/lib/detection_engine/routes/tags @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/fleet_integrations @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/rule_management @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/rule_monitoring @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/rules @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/tags @elastic/security-detections-response-rules
/x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema @elastic/security-detections-response-rules

/x-pack/plugins/security_solution/server/utils @elastic/security-detections-response-rules

## Security Solution sub teams - Security Platform
Expand All @@ -527,12 +527,17 @@ x-pack/examples/files_example @elastic/kibana-app-services
/x-pack/plugins/security_solution/cypress/e2e/exceptions @elastic/security-solution-platform
/x-pack/plugins/security_solution/cypress/e2e/value_lists @elastic/security-solution-platform

/x-pack/plugins/security_solution/common/detection_engine/rule_exceptions @elastic/security-solution-platform

/x-pack/plugins/security_solution/public/detection_engine/rule_exceptions @elastic/security-solution-platform
/x-pack/plugins/security_solution/public/detection_engine/rule_exceptions_ui @elastic/security-solution-platform
/x-pack/plugins/security_solution/public/common/components/exceptions @elastic/security-solution-platform
/x-pack/plugins/security_solution/public/exceptions @elastic/security-solution-platform
/x-pack/plugins/security_solution/public/detections/containers/detection_engine/lists @elastic/security-solution-platform
/x-pack/plugins/security_solution/public/common/components/sourcerer @elastic/security-solution-platform

/x-pack/plugins/security_solution/server/lib/detection_engine/rule_actions_legacy @elastic/security-solution-platform
/x-pack/plugins/security_solution/server/lib/detection_engine/rule_exceptions @elastic/security-solution-platform
/x-pack/plugins/security_solution/server/lib/sourcerer @elastic/security-solution-platform

## Security Threat Intelligence - Under Security Platform
Expand Down Expand Up @@ -595,7 +600,7 @@ x-pack/test/threat_intelligence_cypress @elastic/protections-experience


# Security Intelligence And Analytics
/x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_rules @elastic/security-intelligence-analytics
/x-pack/plugins/security_solution/server/lib/detection_engine/prebuilt_rules/content/prepackaged_rules @elastic/security-intelligence-analytics


# Security Asset Management
Expand Down
2 changes: 1 addition & 1 deletion packages/kbn-optimizer/limits.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ pageLoadAssetSize:
screenshotting: 22870
searchprofiler: 67080
security: 65433
securitySolution: 273763
securitySolution: 339077
Copy link
Contributor Author

@banderror banderror Oct 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be addressed shortly in a follow-up PR. #143532

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can resolving #95870 please be prioritized? It's affecting the end-user performance of Kibana as a whole and just appears to be getting worse, not better.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That will come with this PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think after updating this PR with the main, which contains the recent bundle changes, we can set securitySolution: 66738 back.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YulNaumenko @tylersmalley I managed to keep 66738 in this PR. Will see if I can reduce it a bit more by doing #143532

sessionView: 77750
share: 71239
snapshotRestore: 79032
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,14 +20,14 @@ export * from './src/default_severity_mapping_array';
export * from './src/default_threat_array';
export * from './src/default_to_string';
export * from './src/default_uuid';
export * from './src/from';
export * from './src/language';
export * from './src/machine_learning_job_id';
export * from './src/max_signals';
export * from './src/normalized_ml_job_id';
export * from './src/references_default_array';
export * from './src/risk_score';
export * from './src/risk_score_mapping';
export * from './src/rule_schedule';
export * from './src/saved_object_attributes';
export * from './src/severity';
export * from './src/severity_mapping';
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,42 +6,47 @@
* Side Public License, v 1.
*/

/* eslint-disable @typescript-eslint/naming-convention */

import * as t from 'io-ts';
import { saved_object_attributes } from '../saved_object_attributes';

export type RuleActionGroup = t.TypeOf<typeof RuleActionGroup>;
export const RuleActionGroup = t.string;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reasoning behind using PascalCase for the io-ts runtime values now? In the past we've used camelCase for objects and PascalCase for types, which generally makes it simple to distinguish between types and concrete instances of that type.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marshallmain This is a TypeScript feature called aliases. In this case, it's handy because it allows to write only one JSDoc comment which will be applied to both the TS type and the io-ts schema. So when you hover over them in the IDE you will see the same comment. Without that, we'd need to either duplicate the comment or leave either the TS type or the io-ts schema uncommented. Personally, I also like the fact that it's 1 import instead of 2 when you need both (e.g. in routes).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One of the examples of how JSDoc comments for rule attributes could look like: x-pack/plugins/security_solution/common/detection_engine/rule_schema/model/common_attributes/related_integrations.ts


export type RuleActionId = t.TypeOf<typeof RuleActionId>;
export const RuleActionId = t.string;

export type RuleActionTypeId = t.TypeOf<typeof RuleActionTypeId>;
export const RuleActionTypeId = t.string;

/**
* Params is an "object", since it is a type of RuleActionParams which is action templates.
* @see x-pack/plugins/alerting/common/rule.ts
*/
export const action_group = t.string;
export const action_id = t.string;
export const action_action_type_id = t.string;
export const action_params = saved_object_attributes;
export type RuleActionParams = t.TypeOf<typeof RuleActionParams>;
export const RuleActionParams = saved_object_attributes;

export const action = t.exact(
export type RuleAction = t.TypeOf<typeof RuleAction>;
export const RuleAction = t.exact(
t.type({
group: action_group,
id: action_id,
action_type_id: action_action_type_id,
params: action_params,
group: RuleActionGroup,
id: RuleActionId,
action_type_id: RuleActionTypeId,
params: RuleActionParams,
})
);

export type Action = t.TypeOf<typeof action>;
export type RuleActionArray = t.TypeOf<typeof RuleActionArray>;
export const RuleActionArray = t.array(RuleAction);

export const actions = t.array(action);
export type Actions = t.TypeOf<typeof actions>;

export const actionsCamel = t.array(
t.exact(
t.type({
group: action_group,
id: action_id,
actionTypeId: action_action_type_id,
params: action_params,
})
)
export type RuleActionCamel = t.TypeOf<typeof RuleActionCamel>;
export const RuleActionCamel = t.exact(
t.type({
group: RuleActionGroup,
id: RuleActionId,
actionTypeId: RuleActionTypeId,
params: RuleActionParams,
})
);
export type ActionsCamel = t.TypeOf<typeof actions>;

export type RuleActionArrayCamel = t.TypeOf<typeof RuleActionArrayCamel>;
export const RuleActionArrayCamel = t.array(RuleActionCamel);
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,16 @@

import * as t from 'io-ts';
import { Either } from 'fp-ts/lib/Either';
import { actions, Actions } from '../actions';
import { RuleActionArray } from '../actions';

export const DefaultActionsArray = new t.Type<Actions, Actions | undefined, unknown>(
export const DefaultActionsArray = new t.Type<
RuleActionArray,
RuleActionArray | undefined,
unknown
>(
'DefaultActionsArray',
actions.is,
(input, context): Either<t.Errors, Actions> =>
input == null ? t.success([]) : actions.validate(input, context),
RuleActionArray.is,
(input, context): Either<t.Errors, RuleActionArray> =>
input == null ? t.success([]) : RuleActionArray.validate(input, context),
t.identity
);
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

import * as t from 'io-ts';
import { Either } from 'fp-ts/lib/Either';
import { from } from '../from';
import { From } from '../from';

/**
* Types the DefaultFromString as:
Expand All @@ -21,7 +21,7 @@ export const DefaultFromString = new t.Type<string, string | undefined, unknown>
if (input == null) {
return t.success('now-6m');
}
return from.validate(input, context);
return From.validate(input, context);
},
t.identity
);
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@

import * as t from 'io-ts';
import { Either } from 'fp-ts/lib/Either';
import { RiskScoreMapping, risk_score_mapping } from '../risk_score_mapping';
import { RiskScoreMapping } from '../risk_score_mapping';

/**
* Types the DefaultStringArray as:
* - If null or undefined, then a default risk_score_mapping array will be set
* - If null or undefined, then a default RiskScoreMapping array will be set
*/
export const DefaultRiskScoreMappingArray = new t.Type<
RiskScoreMapping,
RiskScoreMapping | undefined,
unknown
>(
'DefaultRiskScoreMappingArray',
risk_score_mapping.is,
RiskScoreMapping.is,
(input, context): Either<t.Errors, RiskScoreMapping> =>
input == null ? t.success([]) : risk_score_mapping.validate(input, context),
input == null ? t.success([]) : RiskScoreMapping.validate(input, context),
t.identity
);
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@

import * as t from 'io-ts';
import { Either } from 'fp-ts/lib/Either';
import { SeverityMapping, severity_mapping } from '../severity_mapping';
import { SeverityMapping } from '../severity_mapping';

/**
* Types the DefaultStringArray as:
* - If null or undefined, then a default severity_mapping array will be set
* - If null or undefined, then a default SeverityMapping array will be set
*/
export const DefaultSeverityMappingArray = new t.Type<
SeverityMapping,
SeverityMapping | undefined,
unknown
>(
'DefaultSeverityMappingArray',
severity_mapping.is,
SeverityMapping.is,
(input, context): Either<t.Errors, SeverityMapping> =>
input == null ? t.success([]) : severity_mapping.validate(input, context),
input == null ? t.success([]) : SeverityMapping.validate(input, context),
t.identity
);
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ import { parseScheduleDates } from '@kbn/securitysolution-io-ts-utils';

const stringValidator = (input: unknown): input is string => typeof input === 'string';

export const from = new t.Type<string, string, unknown>(
export type From = t.TypeOf<typeof From>;
export const From = new t.Type<string, string, unknown>(
'From',
t.string.is,
(input, context): Either<t.Errors, string> => {
Expand All @@ -23,7 +24,3 @@ export const from = new t.Type<string, string, unknown>(
},
t.identity
);
export type From = t.TypeOf<typeof from>;

export const fromOrUndefined = t.union([from, t.undefined]);
export type FromOrUndefined = t.TypeOf<typeof fromOrUndefined>;
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,6 @@
* Side Public License, v 1.
*/

/* eslint-disable @typescript-eslint/naming-convention */

import * as t from 'io-ts';
import { Either } from 'fp-ts/lib/Either';

Expand All @@ -16,6 +14,7 @@ import { Either } from 'fp-ts/lib/Either';
* - Natural Number (positive integer and not a float),
* - Between the values [0 and 100] inclusive.
*/
export type RiskScore = t.TypeOf<typeof RiskScore>;
export const RiskScore = new t.Type<number, number, unknown>(
'RiskScore',
t.number.is,
Expand All @@ -26,11 +25,3 @@ export const RiskScore = new t.Type<number, number, unknown>(
},
t.identity
);

export type RiskScoreC = typeof RiskScore;

export const risk_score = RiskScore;
export type RiskScore = t.TypeOf<typeof risk_score>;

export const riskScoreOrUndefined = t.union([risk_score, t.undefined]);
export type RiskScoreOrUndefined = t.TypeOf<typeof riskScoreOrUndefined>;
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,19 @@
* Side Public License, v 1.
*/

/* eslint-disable @typescript-eslint/naming-convention */

import * as t from 'io-ts';
import { operator } from '@kbn/securitysolution-io-ts-types';
import { riskScoreOrUndefined } from '../risk_score';
import { RiskScore } from '../risk_score';

export const risk_score_mapping_field = t.string;
export const risk_score_mapping_value = t.string;
export const risk_score_mapping_item = t.exact(
export type RiskScoreMappingItem = t.TypeOf<typeof RiskScoreMappingItem>;
export const RiskScoreMappingItem = t.exact(
t.type({
field: risk_score_mapping_field,
value: risk_score_mapping_value,
field: t.string,
value: t.string,
operator,
risk_score: riskScoreOrUndefined,
risk_score: t.union([RiskScore, t.undefined]),
})
);

export const risk_score_mapping = t.array(risk_score_mapping_item);
export type RiskScoreMapping = t.TypeOf<typeof risk_score_mapping>;

export const riskScoreMappingOrUndefined = t.union([risk_score_mapping, t.undefined]);
export type RiskScoreMappingOrUndefined = t.TypeOf<typeof riskScoreMappingOrUndefined>;
export type RiskScoreMapping = t.TypeOf<typeof RiskScoreMapping>;
export const RiskScoreMapping = t.array(RiskScoreMappingItem);
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import * as t from 'io-ts';
import { From } from '../from';

export type RuleInterval = t.TypeOf<typeof RuleInterval>;
export const RuleInterval = t.string; // we need a more specific schema

export type RuleIntervalFrom = t.TypeOf<typeof RuleIntervalFrom>;
export const RuleIntervalFrom = From;

/**
* TODO: Create a regular expression type or custom date math part type here
*/
export type RuleIntervalTo = t.TypeOf<typeof RuleIntervalTo>;
export const RuleIntervalTo = t.string; // we need a more specific schema
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,5 @@

import * as t from 'io-ts';

export const severity = t.keyof({ low: null, medium: null, high: null, critical: null });
export type Severity = t.TypeOf<typeof severity>;

export const severityOrUndefined = t.union([severity, t.undefined]);
export type SeverityOrUndefined = t.TypeOf<typeof severityOrUndefined>;
export type Severity = t.TypeOf<typeof Severity>;
export const Severity = t.keyof({ low: null, medium: null, high: null, critical: null });
Loading