Stop Streaming Jobs When datasource is disabled/deleted.

[vmmusings]

Description

• This PR is to address running streaming jobs when a datasource is deleted/disabled. The idea is to run a cron for every 15 minutes(debatable) and check for hanging streaming jobs.
• When a datasource is deleted, we drop all the auto refresh indices.
• When a datasource is disabled, we alter the auto refresh indices to manual ones.

Documentation: https://github.com/vamsi-amazon/sql/blob/ss-jobs/docs/user/admin/settings.rst#pluginsqueryexecutionenginesparkstreamingjobshousekeeperinterval

Testing

PUT {{baseUrl}}/_cluster/settings
content-type: application/json

{
  "transient" :{
    "plugins.query.executionengine.spark.streamingjobs.housekeeper.interval":"2m"
  }
}

{
  "acknowledged": true,
  "persistent": {},
  "transient": {
    "plugins": {
      "query": {
        "executionengine": {
          "spark": {
            "streamingjobs": {
              "housekeeper": {
                "interval": "2m"
              }
            }
          }
        }
      }
    }
  }
}

Stats API with new metric.

GET {{baseUrl}}/_plugins/_ppl/stats
content-type: application/json

{
 "streaming_job_housekeeper_task_failure_count": 0
}

Verified Interval Change effect on cron task.

2024-03-19T14:16:46,586][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:17:46,599][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:18:46,605][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:19:46,622][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:20:46,625][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.




[2024-03-19T14:23:23,060][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:25:23,069][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:27:23,087][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:29:23,094][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:31:23,102][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:33:23,112][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:35:23,125][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:37:23,162][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.
[2024-03-19T14:39:23,209][ERROR][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [b0de28f0d8d1] Starting the cleaner task for disabled and deleted data sources.

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 78.94737% with 20 lines in your changes are missing coverage. Please review.

Project coverage is 95.40%. Comparing base (e17962f) to head (31a0efa).

Files	Patch %	Lines
...sql/spark/cluster/ClusterManagerEventListener.java	0.00%	20 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2559      +/-   ##
============================================
- Coverage     95.50%   95.40%   -0.11%     
- Complexity     5117     5129      +12     
============================================
  Files           489      490       +1     
  Lines         14318    14412      +94     
  Branches        963      965       +2     
============================================
+ Hits          13675    13750      +75     
- Misses          618      637      +19     
  Partials         25       25              

Flag	Coverage Δ
sql-engine	95.40% <78.94%> (-0.11%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.