Implementation of the REST Endpoint and Job Queue Abstraction

[fabrizzio-dotCMS]

Parent Issue

#29575

User Story

As a developer, I need to implement a REST endpoint and a job queue abstraction to manage the creation, execution, and monitoring of system tasks. This endpoint will serve as the main entry point to specify which sub-task to execute and the parameters required for the sub-task. The sub-task will be encapsulated in a Job class and executed using a job queue implementation. The job queue implementation should be an abstraction easily replaced without affecting the business logic.

The underlying logic behind the queue needs to be seamless and easily replaceable by a different Queue Service Provider.
The default that will serve as a starting point will be done using Postgres.

Acceptance Criteria

1. Separation of Concerns:
   1.1. Job Queue Abstraction: A clear abstraction for the job queue that can be easily swapped without affecting the core business logic.
   1.2 Default Implementation: A default job queue implementation using PostgreSQL, with well-defined methods for enqueuing, status checking, cancellation, and retrying jobs.

2. Job Creation and Enqueueing:
   2.1 Endpoint: A REST endpoint /jobs to create and enqueue jobs, accepting parameters and an optional queue name.
   2.2 Response: Returns a unique job identifier upon job creation.

3. Job Status Inquiry:
   3.1 Endpoint: A REST endpoint /jobs/{id}/status to query the status of a job.
   3.2 Details: The status response includes the job state, progress percentage, and the node executing the job.

4. Job Execution, Cancellation and Retry:
   4.1 Job Execution: A REST endpoint /jobs/{id}/exec to dispatch a running job.
   4.2. Cancellation Endpoint: A REST endpoint /jobs/{id}/cancel to cancel a running job.

5. List Enqueued Jobs and Status:
   5.1 Endpoint: A REST endpoint /jobs to list all enqueued jobs and their statuses.

6. Job Progress Metrics:
   6.1 Tracking: Implement logic to compute and report the completion percentage of jobs.
   6.2 Data Sources: Handle records from various sources (files, databases, lists) and track progress accurately.

7. Real-Time Job Monitoring:
   7.1 Endpoint: A REST endpoint /jobs/{id}/monitor to monitor the status of a job in real-time using Server-Sent Events (SSE).
   7.2 Updates: Stream updates on job status changes, progress, and completion to the client in real time.

Implementation Details:

8. Default Implementation: PostgreSQL tables should be defined to store job details, parameters, and execution results in JSONB fields.
   8.1 Error Handling: Implement fault tolerance mechanisms to handle job execution failures and support retries.
   8.2 Documentation: Provide comprehensive documentation for the API endpoints, job queue abstraction, and the default PostgreSQL implementation.

9. Transaction Management: The Job Processor must keep the job execution and completion transactional. While the task being executed should have a transaction of its own. Therefore it can commit every n rows if desired. This should not interfere with the parent job transaction which only controls the state of the job execution.

All these endpoints can also take an optional queue name

Proposed Objective

Core Features

Proposed Priority

Priority 3 - Average

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

https://dotcms.slack.com/archives/CQNF9PCFQ/p1722959987326289

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

Tasks

Preview Give feedback

1. Create Model and Classes Abstraction for Job Handling #29478
   QA : Not Needed Team : Scout Type : Task +3
2. Integrate Context and Dependency Injection (CDI) into Job Queue System #29898
   QA : Not Needed Team : Scout Type : Task +3
3. Create a default JobQueue Implementation using Postgres #29479
   QA : Not Needed Release : 24.10.02 Team : Scout Type : Task +4
4. Implement the REST Endpoint to manage the job queue #29480
   Merged QA : Approved QA : Passed Internal Release : 24.10.25 Team : Scout Type : Task +6
5. Ensure Proper Transaction Management in Job Handling #29485
   Team : Scout Triage Type : Task +3
6. addJob method should perform security validation when registering a JobClass #29544
   Team : Scout Triage Type : Task +3
7. Refactor: Migrate JobProcessor Discovery from Jandex to CDI #30545
   Merged QA : Needs Internal Team : Scout Triage Type : Task +5
   
8. New REST Endpoints for Content Import Job Management #30550
   4 of 4
   Epic Team : Scout +2
9. Create ExportContentletProcessor reusing existing export logic #30557
   0 of 6
   Epic Team : Scout +2
10. New REST Endpoints for Content Export Job Management #30559
    Epic Team : Scout +2