Implement Abandoned Job Detection and Recovery

[jgambarios]

Parent Issue

#29474

Task

We need to enhance our job queue system to handle abandoned jobs. These are jobs that may have been interrupted due to server crashes, network failures, or other unexpected issues, leaving them in an inconsistent state.

Objective:

Implement mechanisms to detect abandoned jobs and provide recovery strategies to ensure system reliability and data consistency.

Proposed Strategies:

1. Job Heartbeats:

   • Implement periodic heartbeat updates for running jobs
   • Create a background process to identify jobs with stale heartbeats

2. Timeout Mechanisms:

   • Add a max_execution_time field to job configurations
   • Implement a background process to check for jobs exceeding their maximum execution time

3. Recovery Procedures:

   • Develop a recovery process
   • Identify jobs in inconsistent states and apply appropriate recovery actions

Additional Considerations:

• Ensure that abandoned job recovery doesn't conflict with distributed locking mechanisms
• Consider the impact on job queue performance and optimize where necessary
• Evaluate and document any changes to the system's fault tolerance and high availability characteristics

Proposed Objective

Core Features

Proposed Priority

Priority 2 - Important

Acceptance Criteria

1. System can detect jobs that have been abandoned due to server crashes or other issues
2. Abandoned jobs are automatically handled according to configured recovery strategies
3. All new functionality is covered by appropriate tests
4. System performance is not significantly impacted by new abandoned job handling processes

[nollymar]

A job will be considered abandoned after 30 min (configurable)

[github-actions]

PRs:

• Feat (Core): Implement abandoned job detection and recovery #30710
• #30367 Refactor job system and enhance SSE monitoring. #30816

[jgambarios]

Changes applied as part of the feedback:

1. Streamlined job state management by introducing more precise states such as FAILED_PERMANENTLY and ABANDONED_PERMANENTLY.
2. The JobQueueResource now allows to list: active, completed, successful, canceled, failed and abandoned jobs.
3. Improved SSE handling.

[fabrizzio-dotCMS]

It looks great now!
The job monitor moves smoothly, with no connection leaks and no stuck jobs.

Some potential improvements

• But people need to know what they're doing when configuring this functionality

JOB_ABANDONMENT_THRESHOLD_MINUTES should always be greater than
JOB_ABANDONMENT_DETECTION_INTERVAL_MINUTES, allowing the system to verify consistently
As an improvement, we could validate or apply defaults if a minimum window is violated.

• When an abandoned job lacks a retry policy, it will get automatically marked as permanently abandoned, which makes complete sense. Perpahs this policy should be shown when we list the available queues.
• Perhaps we could fusion these two annotations into a single one:

@Queue("importContentlets")
@NoRetryPolicy

• And finally, we can not forget that we need to initiate the thread pool automatically. Currently, it gets started until the endpoint is consumed, otherwise, abandoned jobs will never be detected until someone kicks off the thread-pool. I know we're taking care of that in another ticket.