Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Response Ops][Alerting] Deleting ad hoc run task if the last schedule entry ends in a timeout #187496

Merged
merged 4 commits into from
Jul 9, 2024

Conversation

ymao1
Copy link
Contributor

@ymao1 ymao1 commented Jul 3, 2024

Summary

Fixing bug where ad hoc run task was not getting deleted if running the last schedule entry and the run times out.

To Verify

  1. Create a detection rule
  2. Add a delay in the ad hoc task runner
--- a/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts
+++ b/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts
@@ -530,6 +530,7 @@ export class AdHocTaskRunner {
     } catch (err) {
       runMetrics = asErr(err);
     }
+    await new Promise((resolve) => setTimeout(resolve, 3100000));
     await this.processAdHocRunResults(runMetrics);
  1. Schedule a backfill for the rule with only one schedule entry
POST https://localhost:5601/internal/alerting/rules/backfill/_schedule
[
  {
    "rule_id": <ruleId>,
    "start": "2024-07-03T13:05:00.000Z"
  }
]
  1. Wait for the run to get cancelled then verify that the task was deleted.

@ymao1 ymao1 changed the title Deleting ad hoc run task if the last schedule entry ends in a timeout [Response Ops][Alerting] Deleting ad hoc run task if the last schedule entry ends in a timeout Jul 3, 2024
@ymao1 ymao1 self-assigned this Jul 3, 2024
@ymao1 ymao1 added Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.15.0 labels Jul 3, 2024
@ymao1
Copy link
Contributor Author

ymao1 commented Jul 3, 2024

@elasticmachine merge upstream

@ymao1 ymao1 marked this pull request as ready for review July 3, 2024 20:31
@ymao1 ymao1 requested a review from a team as a code owner July 3, 2024 20:31
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@ymao1
Copy link
Contributor Author

ymao1 commented Jul 8, 2024

@elasticmachine merge upstream

@ymao1 ymao1 added the v8.16.0 label Jul 8, 2024
@ymao1 ymao1 requested review from pmuellr and js-jankisalvi July 8, 2024 11:53
Copy link
Contributor

@js-jankisalvi js-jankisalvi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified locally, works as expected 🎉

Thank you for helping me with testing!! ❤️

Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, left a comment about annotating a type for improved searchability in VS Code

@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Jest Tests #12 / UnifiedFieldList should render correctly

Metrics [docs]

✅ unchanged

History

cc @ymao1

@ymao1 ymao1 merged commit c2be810 into elastic:main Jul 9, 2024
38 of 39 checks passed
@ymao1 ymao1 deleted the backfill-timeout-delete-task branch July 9, 2024 00:08
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jul 9, 2024
…e entry ends in a timeout (elastic#187496)

## Summary

Fixing bug where ad hoc run task was not getting deleted if running the
last schedule entry and the run times out.

## To Verify
1. Create a detection rule
2. Add a delay in the ad hoc task runner
```
--- a/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts
+++ b/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts
@@ -530,6 +530,7 @@ export class AdHocTaskRunner {
     } catch (err) {
       runMetrics = asErr(err);
     }
+    await new Promise((resolve) => setTimeout(resolve, 3100000));
     await this.processAdHocRunResults(runMetrics);
```
3. Schedule a backfill for the rule with only one schedule entry

```
POST https://localhost:5601/internal/alerting/rules/backfill/_schedule
[
  {
    "rule_id": <ruleId>,
    "start": "2024-07-03T13:05:00.000Z"
  }
]
```

4. Wait for the run to get cancelled then verify that the task was
deleted.

---------

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit c2be810)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.15

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Jul 9, 2024
…schedule entry ends in a timeout (#187496) (#187810)

# Backport

This will backport the following commits from `main` to `8.15`:
- [[Response Ops][Alerting] Deleting ad hoc run task if the last
schedule entry ends in a timeout
(#187496)](#187496)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ying
Mao","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-07-09T00:07:57Z","message":"[Response
Ops][Alerting] Deleting ad hoc run task if the last schedule entry ends
in a timeout (#187496)\n\n## Summary\r\n\r\nFixing bug where ad hoc run
task was not getting deleted if running the\r\nlast schedule entry and
the run times out.\r\n\r\n## To Verify\r\n1. Create a detection
rule\r\n2. Add a delay in the ad hoc task runner\r\n```\r\n---
a/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts\r\n+++
b/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts\r\n@@
-530,6 +530,7 @@ export class AdHocTaskRunner {\r\n } catch (err) {\r\n
runMetrics = asErr(err);\r\n }\r\n+ await new Promise((resolve) =>
setTimeout(resolve, 3100000));\r\n await
this.processAdHocRunResults(runMetrics);\r\n```\r\n3. Schedule a
backfill for the rule with only one schedule entry\r\n\r\n```\r\nPOST
https://localhost:5601/internal/alerting/rules/backfill/_schedule\r\n[\r\n
{\r\n \"rule_id\": <ruleId>,\r\n \"start\":
\"2024-07-03T13:05:00.000Z\"\r\n }\r\n]\r\n```\r\n\r\n4. Wait for the
run to get cancelled then verify that the task
was\r\ndeleted.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"c2be810de37479d43661e722e7d7454dacc35c79","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Alerting","release_note:skip","Team:ResponseOps","v8.15.0","v8.16.0"],"title":"[Response
Ops][Alerting] Deleting ad hoc run task if the last schedule entry ends
in a
timeout","number":187496,"url":"https://github.com/elastic/kibana/pull/187496","mergeCommit":{"message":"[Response
Ops][Alerting] Deleting ad hoc run task if the last schedule entry ends
in a timeout (#187496)\n\n## Summary\r\n\r\nFixing bug where ad hoc run
task was not getting deleted if running the\r\nlast schedule entry and
the run times out.\r\n\r\n## To Verify\r\n1. Create a detection
rule\r\n2. Add a delay in the ad hoc task runner\r\n```\r\n---
a/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts\r\n+++
b/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts\r\n@@
-530,6 +530,7 @@ export class AdHocTaskRunner {\r\n } catch (err) {\r\n
runMetrics = asErr(err);\r\n }\r\n+ await new Promise((resolve) =>
setTimeout(resolve, 3100000));\r\n await
this.processAdHocRunResults(runMetrics);\r\n```\r\n3. Schedule a
backfill for the rule with only one schedule entry\r\n\r\n```\r\nPOST
https://localhost:5601/internal/alerting/rules/backfill/_schedule\r\n[\r\n
{\r\n \"rule_id\": <ruleId>,\r\n \"start\":
\"2024-07-03T13:05:00.000Z\"\r\n }\r\n]\r\n```\r\n\r\n4. Wait for the
run to get cancelled then verify that the task
was\r\ndeleted.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"c2be810de37479d43661e722e7d7454dacc35c79"}},"sourceBranch":"main","suggestedTargetBranches":["8.15"],"targetPullRequestStates":[{"branch":"8.15","label":"v8.15.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/187496","number":187496,"mergeCommit":{"message":"[Response
Ops][Alerting] Deleting ad hoc run task if the last schedule entry ends
in a timeout (#187496)\n\n## Summary\r\n\r\nFixing bug where ad hoc run
task was not getting deleted if running the\r\nlast schedule entry and
the run times out.\r\n\r\n## To Verify\r\n1. Create a detection
rule\r\n2. Add a delay in the ad hoc task runner\r\n```\r\n---
a/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts\r\n+++
b/x-pack/plugins/alerting/server/task_runner/ad_hoc_task_runner.ts\r\n@@
-530,6 +530,7 @@ export class AdHocTaskRunner {\r\n } catch (err) {\r\n
runMetrics = asErr(err);\r\n }\r\n+ await new Promise((resolve) =>
setTimeout(resolve, 3100000));\r\n await
this.processAdHocRunResults(runMetrics);\r\n```\r\n3. Schedule a
backfill for the rule with only one schedule entry\r\n\r\n```\r\nPOST
https://localhost:5601/internal/alerting/rules/backfill/_schedule\r\n[\r\n
{\r\n \"rule_id\": <ruleId>,\r\n \"start\":
\"2024-07-03T13:05:00.000Z\"\r\n }\r\n]\r\n```\r\n\r\n4. Wait for the
run to get cancelled then verify that the task
was\r\ndeleted.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"c2be810de37479d43661e722e7d7454dacc35c79"}}]}]
BACKPORT-->

Co-authored-by: Ying Mao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Alerting release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.15.0 v8.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants