Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Entity Analytics] New API endpoint to cleanup the risk engine installation and data #191843

Conversation

abhishekbhatia1710
Copy link
Contributor

@abhishekbhatia1710 abhishekbhatia1710 commented Aug 30, 2024

Summary

  1. Create a new public API endpoint : DELETE /api/risk_score/engine/dangerously_delete_data

Test cases Result

  PASS  x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)
  risk engine cleanup route
    invokes the risk engine cleanup route
      ✓ should call the router with the correct route and handler (71 ms)
      ✓ returns a 200 when cleanup is successful (64 ms)
      ✓ returns a 500 when cleanup is unsuccessful (57 ms)
      ✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)
    when task manager is unavailable
      ✓ returns a 400 when task manager is unavailable (55 ms)
    when user does not have the required privileges
      ✓ returns a 403 when user does not have the required privileges (88 ms)

API Responses

When multiple errors encountered

{
    "risk_engine_cleanup": false,
    "errors": [
        {
            "seq": 1,
            "error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found"
        },
        {
            "seq": 2,
            "error": "index_not_found_exception\n\tRoot causes:\n\t\tindex_not_found_exception: no such index [risk-score.risk-score-default]"
        },
        {
            "seq": 3,
            "error": "index_template_missing_exception\n\tRoot causes:\n\t\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing"
        },
        {
            "seq": 4,
            "error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: .risk-score-mappings"
        }
    ],
    "status_code": 500
}

Success

{
    "risk_engine_cleanup": true
}

Checklist

Delete any items that are not applicable to this PR.

OpenAPI spec

image

For maintainers

@abhishekbhatia1710 abhishekbhatia1710 added release_note:enhancement Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Entity Analytics Security Solution Entity Analytics features Team:Entity Analytics Security Entity Analytics Team labels Aug 30, 2024
@abhishekbhatia1710 abhishekbhatia1710 self-assigned this Aug 30, 2024
@abhishekbhatia1710 abhishekbhatia1710 changed the title [SecuritySolution] New API endpoint to cleanup the risk engine installation and data [SecuritySolution] [Entity Analytics] New API endpoint to cleanup the risk engine installation and data Aug 30, 2024
@abhishekbhatia1710 abhishekbhatia1710 force-pushed the ea-66-api-delete-risk-engine-installation-data branch from 2abeef6 to a8b4d04 Compare September 5, 2024 12:49
@abhishekbhatia1710 abhishekbhatia1710 marked this pull request as ready for review September 5, 2024 13:44
@abhishekbhatia1710 abhishekbhatia1710 requested a review from a team as a code owner September 5, 2024 13:44
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

@abhishekbhatia1710 abhishekbhatia1710 changed the title [SecuritySolution] [Entity Analytics] New API endpoint to cleanup the risk engine installation and data [Entity Analytics] New API endpoint to cleanup the risk engine installation and data Sep 6, 2024
@abhishekbhatia1710
Copy link
Contributor Author

@elasticmachine merge upstream

Copy link
Member

@machadoum machadoum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! 👏 👏 👏

@hop-dev hop-dev added the backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) label Sep 23, 2024
@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7004

[❌] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 0/1 tests passed.
[❌] x-pack/test/cases_api_integration/security_and_spaces/config_trial.ts: 0/1 tests passed.

see run history

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🟠 Some tests failed. - kibana-flaky-test-suite-runner#7006

[❌] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 0/1 tests passed.
[❌] x-pack/test/cases_api_integration/security_and_spaces/config_trial.ts: 0/1 tests passed.

see run history

@abhishekbhatia1710
Copy link
Contributor Author

The integration test for the tearDown failed because the other PR for allowing risk score in multiple spaces got merged today and the key is_max_amount_of_risk_engines_reached is removed now.

I have changed the integration test to not to check the key is_max_amount_of_risk_engines_reached

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 20.4MB 20.4MB +330.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 87.1KB 87.3KB +116.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @abhishekbhatia1710

@kibanamachine
Copy link
Contributor

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7011

[✅] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 5/5 tests passed.
[✅] x-pack/test/cases_api_integration/security_and_spaces/config_trial.ts: 5/5 tests passed.

see run history

@abhishekbhatia1710 abhishekbhatia1710 merged commit 22f451b into elastic:main Sep 25, 2024
41 checks passed
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.x Backport failed because of merge conflicts

You might need to backport the following PRs to 8.x:
- [Security Solution] Omit Solution from tags and titles in Security Solution's OpenAPI bundles (#193669)
- [Threat Hunting Investigations] Use OpenAPI types in more timeline routes (#189977)

Manual backport

To create the backport manually run:

node scripts/backport --pr 191843

Questions ?

Please refer to the Backport tool documentation

@abhishekbhatia1710
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

abhishekbhatia1710 added a commit to abhishekbhatia1710/kibana that referenced this pull request Sep 25, 2024
…lation and data (elastic#191843)

## Summary

1. Create a new public API endpoint : `DELETE
/api/risk_score/engine/dangerously_delete_data`

## Test cases Result

```
  PASS  x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts (7.017 s)
  risk engine cleanup route
    invokes the risk engine cleanup route
      ✓ should call the router with the correct route and handler (71 ms)
      ✓ returns a 200 when cleanup is successful (64 ms)
      ✓ returns a 500 when cleanup is unsuccessful (57 ms)
      ✓ returns a 500 when cleanup is unsuccessful with multiple errors (53 ms)
    when task manager is unavailable
      ✓ returns a 400 when task manager is unavailable (55 ms)
    when user does not have the required privileges
      ✓ returns a 403 when user does not have the required privileges (88 ms)
```

### API Responses

## When multiple errors encountered
```
{
    "risk_engine_cleanup": false,
    "errors": [
        {
            "seq": 1,
            "error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: Transform with id [risk_score_latest_transform_default] could not be found"
        },
        {
            "seq": 2,
            "error": "index_not_found_exception\n\tRoot causes:\n\t\tindex_not_found_exception: no such index [risk-score.risk-score-default]"
        },
        {
            "seq": 3,
            "error": "index_template_missing_exception\n\tRoot causes:\n\t\tindex_template_missing_exception: index_template [.risk-score.risk-score-default-index-template] missing"
        },
        {
            "seq": 4,
            "error": "resource_not_found_exception\n\tRoot causes:\n\t\tresource_not_found_exception: .risk-score-mappings"
        }
    ],
    "status_code": 500
}
```

## Success

```
{
    "risk_engine_cleanup": true
}
```

### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

### OpenAPI spec

![image](https://github.com/user-attachments/assets/56d69602-061d-4a01-9d2b-01a8398ffc76)

### For maintainers

- [ ] This was checked for breaking API changes and was [labeled
appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

(cherry picked from commit 22f451b)

# Conflicts:
#	oas_docs/output/kibana.serverless.staging.yaml
#	oas_docs/output/kibana.staging.yaml
abhishekbhatia1710 added a commit that referenced this pull request Sep 26, 2024
…installation and data (#191843) (#193989)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Entity Analytics] New API endpoint to cleanup the risk engine
installation and data
(#191843)](#191843)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Abhishek
Bhatia","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-25T11:35:24Z","message":"[Entity
Analytics] New API endpoint to cleanup the risk engine installation and
data (#191843)\n\n## Summary\r\n\r\n1. Create a new public API endpoint
:
`DELETE\r\n/api/risk_score/engine/dangerously_delete_data`\r\n\r\n\r\n##
Test cases Result\r\n\r\n```\r\n PASS
x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts
(7.017 s)\r\n risk engine cleanup route\r\n invokes the risk engine
cleanup route\r\n ✓ should call the router with the correct route and
handler (71 ms)\r\n ✓ returns a 200 when cleanup is successful (64
ms)\r\n ✓ returns a 500 when cleanup is unsuccessful (57 ms)\r\n ✓
returns a 500 when cleanup is unsuccessful with multiple errors (53
ms)\r\n when task manager is unavailable\r\n ✓ returns a 400 when task
manager is unavailable (55 ms)\r\n when user does not have the required
privileges\r\n ✓ returns a 403 when user does not have the required
privileges (88 ms)\r\n```\r\n\r\n\r\n### API Responses\r\n\r\n## When
multiple errors encountered\r\n```\r\n{\r\n \"risk_engine_cleanup\":
false,\r\n \"errors\": [\r\n {\r\n \"seq\": 1,\r\n \"error\":
\"resource_not_found_exception\\n\\tRoot
causes:\\n\\t\\tresource_not_found_exception: Transform with id
[risk_score_latest_transform_default] could not be found\"\r\n },\r\n
{\r\n \"seq\": 2,\r\n \"error\": \"index_not_found_exception\\n\\tRoot
causes:\\n\\t\\tindex_not_found_exception: no such index
[risk-score.risk-score-default]\"\r\n },\r\n {\r\n \"seq\": 3,\r\n
\"error\": \"index_template_missing_exception\\n\\tRoot
causes:\\n\\t\\tindex_template_missing_exception: index_template
[.risk-score.risk-score-default-index-template] missing\"\r\n },\r\n
{\r\n \"seq\": 4,\r\n \"error\":
\"resource_not_found_exception\\n\\tRoot
causes:\\n\\t\\tresource_not_found_exception: .risk-score-mappings\"\r\n
}\r\n ],\r\n \"status_code\": 500\r\n}\r\n```\r\n\r\n##
Success\r\n\r\n```\r\n{\r\n \"risk_engine_cleanup\":
true\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n\r\n###
OpenAPI
spec\r\n\r\n\r\n![image](https://github.com/user-attachments/assets/56d69602-061d-4a01-9d2b-01a8398ffc76)\r\n\r\n\r\n###
For maintainers\r\n\r\n- [ ] This was checked for breaking API changes
and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"22f451b30df3f80974b73314c9db4c8f8a509789","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","v9.0.0","Team:
SecuritySolution","backport:prev-minor","Feature:Entity
Analytics","Team:Entity
Analytics"],"number":191843,"url":"https://github.com/elastic/kibana/pull/191843","mergeCommit":{"message":"[Entity
Analytics] New API endpoint to cleanup the risk engine installation and
data (#191843)\n\n## Summary\r\n\r\n1. Create a new public API endpoint
:
`DELETE\r\n/api/risk_score/engine/dangerously_delete_data`\r\n\r\n\r\n##
Test cases Result\r\n\r\n```\r\n PASS
x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts
(7.017 s)\r\n risk engine cleanup route\r\n invokes the risk engine
cleanup route\r\n ✓ should call the router with the correct route and
handler (71 ms)\r\n ✓ returns a 200 when cleanup is successful (64
ms)\r\n ✓ returns a 500 when cleanup is unsuccessful (57 ms)\r\n ✓
returns a 500 when cleanup is unsuccessful with multiple errors (53
ms)\r\n when task manager is unavailable\r\n ✓ returns a 400 when task
manager is unavailable (55 ms)\r\n when user does not have the required
privileges\r\n ✓ returns a 403 when user does not have the required
privileges (88 ms)\r\n```\r\n\r\n\r\n### API Responses\r\n\r\n## When
multiple errors encountered\r\n```\r\n{\r\n \"risk_engine_cleanup\":
false,\r\n \"errors\": [\r\n {\r\n \"seq\": 1,\r\n \"error\":
\"resource_not_found_exception\\n\\tRoot
causes:\\n\\t\\tresource_not_found_exception: Transform with id
[risk_score_latest_transform_default] could not be found\"\r\n },\r\n
{\r\n \"seq\": 2,\r\n \"error\": \"index_not_found_exception\\n\\tRoot
causes:\\n\\t\\tindex_not_found_exception: no such index
[risk-score.risk-score-default]\"\r\n },\r\n {\r\n \"seq\": 3,\r\n
\"error\": \"index_template_missing_exception\\n\\tRoot
causes:\\n\\t\\tindex_template_missing_exception: index_template
[.risk-score.risk-score-default-index-template] missing\"\r\n },\r\n
{\r\n \"seq\": 4,\r\n \"error\":
\"resource_not_found_exception\\n\\tRoot
causes:\\n\\t\\tresource_not_found_exception: .risk-score-mappings\"\r\n
}\r\n ],\r\n \"status_code\": 500\r\n}\r\n```\r\n\r\n##
Success\r\n\r\n```\r\n{\r\n \"risk_engine_cleanup\":
true\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n\r\n###
OpenAPI
spec\r\n\r\n\r\n![image](https://github.com/user-attachments/assets/56d69602-061d-4a01-9d2b-01a8398ffc76)\r\n\r\n\r\n###
For maintainers\r\n\r\n- [ ] This was checked for breaking API changes
and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"22f451b30df3f80974b73314c9db4c8f8a509789"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191843","number":191843,"mergeCommit":{"message":"[Entity
Analytics] New API endpoint to cleanup the risk engine installation and
data (#191843)\n\n## Summary\r\n\r\n1. Create a new public API endpoint
:
`DELETE\r\n/api/risk_score/engine/dangerously_delete_data`\r\n\r\n\r\n##
Test cases Result\r\n\r\n```\r\n PASS
x-pack/plugins/security_solution/server/lib/entity_analytics/risk_engine/routes/delete.test.ts
(7.017 s)\r\n risk engine cleanup route\r\n invokes the risk engine
cleanup route\r\n ✓ should call the router with the correct route and
handler (71 ms)\r\n ✓ returns a 200 when cleanup is successful (64
ms)\r\n ✓ returns a 500 when cleanup is unsuccessful (57 ms)\r\n ✓
returns a 500 when cleanup is unsuccessful with multiple errors (53
ms)\r\n when task manager is unavailable\r\n ✓ returns a 400 when task
manager is unavailable (55 ms)\r\n when user does not have the required
privileges\r\n ✓ returns a 403 when user does not have the required
privileges (88 ms)\r\n```\r\n\r\n\r\n### API Responses\r\n\r\n## When
multiple errors encountered\r\n```\r\n{\r\n \"risk_engine_cleanup\":
false,\r\n \"errors\": [\r\n {\r\n \"seq\": 1,\r\n \"error\":
\"resource_not_found_exception\\n\\tRoot
causes:\\n\\t\\tresource_not_found_exception: Transform with id
[risk_score_latest_transform_default] could not be found\"\r\n },\r\n
{\r\n \"seq\": 2,\r\n \"error\": \"index_not_found_exception\\n\\tRoot
causes:\\n\\t\\tindex_not_found_exception: no such index
[risk-score.risk-score-default]\"\r\n },\r\n {\r\n \"seq\": 3,\r\n
\"error\": \"index_template_missing_exception\\n\\tRoot
causes:\\n\\t\\tindex_template_missing_exception: index_template
[.risk-score.risk-score-default-index-template] missing\"\r\n },\r\n
{\r\n \"seq\": 4,\r\n \"error\":
\"resource_not_found_exception\\n\\tRoot
causes:\\n\\t\\tresource_not_found_exception: .risk-score-mappings\"\r\n
}\r\n ],\r\n \"status_code\": 500\r\n}\r\n```\r\n\r\n##
Success\r\n\r\n```\r\n{\r\n \"risk_engine_cleanup\":
true\r\n}\r\n```\r\n\r\n### Checklist\r\n\r\nDelete any items that are
not applicable to this PR.\r\n\r\n- [x] Any text added follows [EUI's
writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing),
uses\r\nsentence case text and includes
[i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n-
[x]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas
added for features that require explanation or tutorials\r\n- [x] [Unit
or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common scenarios\r\n\r\n\r\n###
OpenAPI
spec\r\n\r\n\r\n![image](https://github.com/user-attachments/assets/56d69602-061d-4a01-9d2b-01a8398ffc76)\r\n\r\n\r\n###
For maintainers\r\n\r\n- [ ] This was checked for breaking API changes
and was
[labeled\r\nappropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)","sha":"22f451b30df3f80974b73314c9db4c8f8a509789"}}]}]
BACKPORT-->

---------

Co-authored-by: kibanamachine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Feature:Entity Analytics Security Solution Entity Analytics features release_note:enhancement Team:Entity Analytics Security Entity Analytics Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.16.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants