Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RAC] Alerts as Data Reference Rules #95735

Closed
12 of 17 tasks
spong opened this issue Mar 30, 2021 · 4 comments
Closed
12 of 17 tasks

[RAC] Alerts as Data Reference Rules #95735

spong opened this issue Mar 30, 2021 · 4 comments
Labels
Team:Detections and Resp Security Detection Response Team Team:Observability Team label for Observability Team (for things that are handled across all of observability) Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: rac label obsolete

Comments

@spong
Copy link
Member

spong commented Mar 30, 2021 •
edited by madirey
Loading

This issue is for building out the reference rule implementations necessary for finalizing the Alerts as Data Schema Definition (#93728), and in prep for the migrating the Security Solution rules to use the rule-registry for registering rules and writing alerts.

Reference rules needed to exercise the following:

  • Rule leveraging alert hierarchy (alerts-on-alerts) @madirey
  • Rule using Runtime Fields @spong
  • Building block rules (e.g. EQL Sequences)
  • Rule with Exceptions (Endpoint/Rule)
  • Rule implementing field overrides
  • Workflow fields (status/assignment)
  • Existing Alerting Platform Rules (Maps, ES DSL Rule, etc)
  • Existing Security Rules
    • Query Rule
    • EQL Rule
    • Threshold Rule
    • ML Rule
    • Indicator Match Rule
  • ...

Tangential Rules for testing non-schema related functionality:

  • Searchable alertParams (at the rule level)
  • Common logging via Event Log (most likely porting over @banderror's work with event_log monitoring?)
  • Rule exercising legacy max_signals concept so we can determine how to abstract things for features like max_alerts aggregations/groups
@spong spong added Team:Observability Team label for Observability Team (for things that are handled across all of observability) Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: rac label obsolete labels Mar 30, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

This was referenced Mar 30, 2021
Closed
Merged
@spong spong mentioned this issue Apr 8, 2021
Open
spong added a commit that referenced this issue May 28, 2021
@spong
[RAC][Security Solution] Register Security Detection Rules with Rule …
4c48993 
…Registry (#96015)

## Summary

This PR starts the migration of the Security Solution rules to use the rule-registry introduced in #95903. This is a pathfinding effort in porting over the existing Security Solution rules, and may include some temporary reference rules for testing out different paradigms as we move the rules over. See #95735 for details


Enable via the following feature flags in your `kibana.dev.yml`:

```
# Security Solution Rules on Rule Registry
xpack.ruleRegistry.index: '.kibana-[USERNAME]-alerts' # Only necessary to scope from other devs testing, if not specified defaults to `.alerts-security-solution`
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']
```

> Note: if setting a custom `xpack.ruleRegistry.index`, for the time being you must also update the [DEFAULT_ALERTS_INDEX](https://github.com/elastic/kibana/blob/9e213fb7a5a0337591a50a0567924ebe950b9791/x-pack/plugins/security_solution/common/constants.ts#L28) in order for the UI to display alerts within the alerts table.

---

Three reference rule types have been added (`query`, `eql`, `threshold`), along with scripts for creating them located in:

```
x-pack/plugins/security_solution/server/lib/detection_engine/reference_rules/scripts/
```

Main Detection page TGrid queries have been short-circuited to query `.alerts-security-solution*` for displaying alerts from the new alerts as data indices.

To test, checkout, enable the above feature flag(s), and run one of the scripts from the above directory, e.g.  `./create_reference_rule_query.sh` (ensure your ENV vars as set! :)


Alerts as data within the main Detection Page 🎉 
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/119911768-39cfba00-bf17-11eb-8996-63c0b813fdcc.png" />
</p>




cc @madirey @dgieselaar @pmuellr @yctercero @dhurley14 @marshallmain
spong added a commit to spong/kibana that referenced this issue May 28, 2021
@spong
[RAC][Security Solution] Register Security Detection Rules with Rule …
16cd13d 
…Registry (elastic#96015)

## Summary

This PR starts the migration of the Security Solution rules to use the rule-registry introduced in elastic#95903. This is a pathfinding effort in porting over the existing Security Solution rules, and may include some temporary reference rules for testing out different paradigms as we move the rules over. See elastic#95735 for details

Enable via the following feature flags in your `kibana.dev.yml`:

```
# Security Solution Rules on Rule Registry
xpack.ruleRegistry.index: '.kibana-[USERNAME]-alerts' # Only necessary to scope from other devs testing, if not specified defaults to `.alerts-security-solution`
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']
```

> Note: if setting a custom `xpack.ruleRegistry.index`, for the time being you must also update the [DEFAULT_ALERTS_INDEX](https://github.com/elastic/kibana/blob/9e213fb7a5a0337591a50a0567924ebe950b9791/x-pack/plugins/security_solution/common/constants.ts#L28) in order for the UI to display alerts within the alerts table.

---

Three reference rule types have been added (`query`, `eql`, `threshold`), along with scripts for creating them located in:

```
x-pack/plugins/security_solution/server/lib/detection_engine/reference_rules/scripts/
```

Main Detection page TGrid queries have been short-circuited to query `.alerts-security-solution*` for displaying alerts from the new alerts as data indices.

To test, checkout, enable the above feature flag(s), and run one of the scripts from the above directory, e.g.  `./create_reference_rule_query.sh` (ensure your ENV vars as set! :)

Alerts as data within the main Detection Page 🎉
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/119911768-39cfba00-bf17-11eb-8996-63c0b813fdcc.png" />
</p>

cc @madirey @dgieselaar @pmuellr @yctercero @dhurley14 @marshallmain

# Conflicts:
#	x-pack/plugins/security_solution/server/plugin.ts
spong added a commit that referenced this issue May 28, 2021
@spong
[RAC][Security Solution] Register Security Detection Rules with Rule …
8ca9d3f 
…Registry (#96015) (#100940)

## Summary

This PR starts the migration of the Security Solution rules to use the rule-registry introduced in #95903. This is a pathfinding effort in porting over the existing Security Solution rules, and may include some temporary reference rules for testing out different paradigms as we move the rules over. See #95735 for details

Enable via the following feature flags in your `kibana.dev.yml`:

```
# Security Solution Rules on Rule Registry
xpack.ruleRegistry.index: '.kibana-[USERNAME]-alerts' # Only necessary to scope from other devs testing, if not specified defaults to `.alerts-security-solution`
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']
```

> Note: if setting a custom `xpack.ruleRegistry.index`, for the time being you must also update the [DEFAULT_ALERTS_INDEX](https://github.com/elastic/kibana/blob/9e213fb7a5a0337591a50a0567924ebe950b9791/x-pack/plugins/security_solution/common/constants.ts#L28) in order for the UI to display alerts within the alerts table.

---

Three reference rule types have been added (`query`, `eql`, `threshold`), along with scripts for creating them located in:

```
x-pack/plugins/security_solution/server/lib/detection_engine/reference_rules/scripts/
```

Main Detection page TGrid queries have been short-circuited to query `.alerts-security-solution*` for displaying alerts from the new alerts as data indices.

To test, checkout, enable the above feature flag(s), and run one of the scripts from the above directory, e.g.  `./create_reference_rule_query.sh` (ensure your ENV vars as set! :)

Alerts as data within the main Detection Page 🎉
<p align="center">
  <img width="500" src="https://user-images.githubusercontent.com/2946766/119911768-39cfba00-bf17-11eb-8996-63c0b813fdcc.png" />
</p>

cc @madirey @dgieselaar @pmuellr @yctercero @dhurley14 @marshallmain

# Conflicts:
#	x-pack/plugins/security_solution/server/plugin.ts
@spong
Copy link
Member Author

spong commented Aug 9, 2021

Closing as remaining migration of Security rules is being tracked as part of referenced issues above.

@spong spong closed this as completed Aug 9, 2021
@kobelb kobelb added the needs-team Issues missing a team label label Jan 31, 2022
@botelastic botelastic bot removed the needs-team Issues missing a team label label Jan 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Team:Detections and Resp Security Detection Response Team Team:Observability Team label for Observability Team (for things that are handled across all of observability) Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Theme: rac label obsolete
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@madirey @kobelb @spong @elasticmachine @marshallmain @dplumlee