Implement the REST Endpoint to manage the job queue

[fabrizzio-dotCMS]

Parent Issue

#29474

Task

As a developer, I need to create a REST endpoint to manage the creation, execution, monitoring, and querying of system tasks. This endpoint will serve as the main interface for users to interact with the job system, allowing them to enqueue new jobs, check job status, cancel running jobs, retry failed jobs, and monitor job progress in real-time using Server-Sent Events (SSE).real time

Proposed Objective

Core Features

Proposed Priority

Priority 3 - Average

Acceptance Criteria

1. Job Creation Endpoint: Implement a REST endpoint to create and enqueue new jobs.
   1.1. The endpoint should accept job parameters and optionally a queue name.
   1.2. The response should include a unique job identifier.
2. Job Status Inquiry Endpoint: Implement a REST endpoint to query the status of a specific job.
   2.1. The status should include the job state, progress percentage, and the node executing the job.
3. Job Cancellation Endpoint: Implement a REST endpoint to cancel a running job.
   3.1. The endpoint should update the job status to "canceled."
4. Job Retry Endpoint: Implement a REST endpoint to retry a failed job.
   4.1. The endpoint should reset the job status to "pending" and re-enqueue it.
5. List Jobs Endpoint: Implement a REST endpoint to list all enqueued jobs and their statuses.
6. Real-Time Job Monitoring Endpoint: Implement a REST endpoint to monitor job status in real time using Server-Sent Events (SSE).
   6.1. The endpoint should stream updates on job status changes, progress, and completion to the client.
7. Error Handling: Ensure all endpoints handle errors gracefully and return appropriate HTTP status codes and messages.
8. Documentation: Document the REST API endpoints, including request and response formats and examples.
9. Write the adequate postman test if applicable

Let the following code serve as a high-level example:

// JobResource.java (JAX-RS)
@Path("/jobs")
public class JobResource {
    @Inject
    private JobQueue jobQueue;

    @POST
    @Path("/{queueName}")
    public Response addJob(@PathParam("queueName") String queueName, Map<String, Object> parameters) {
        String jobId = jobQueue.addJob(queueName, parameters);
        return Response.ok(jobId).build();
    }

    @GET
    @Path("/{jobId}")
    public Response getJob(@PathParam("jobId") String jobId) {
        Job job = jobQueue.getJob(jobId);
        return Response.ok(job).build();
    }

    @GET
    @Path("/{queueName}/active")
    public Response getActiveJobs(@PathParam("queueName") String queueName,
                                  @QueryParam("page") @DefaultValue("1") int page,
                                  @QueryParam("pageSize") @DefaultValue("10") int pageSize) {
        List<Job> jobs = jobQueue.getActiveJobs(queueName, page, pageSize);
        return Response.ok(jobs).build();
    }

    @GET
    @Path("/{queueName}/completed")
    public Response getCompletedJobs(@PathParam("queueName") String queueName,
                                     @QueryParam("startDate") String startDate,
                                     @QueryParam("endDate") String endDate,
                                     @QueryParam("page") @DefaultValue("1") int page,
                                     @QueryParam("pageSize") @DefaultValue("10") int pageSize) {
        LocalDateTime start = LocalDateTime.parse(startDate);
        LocalDateTime end = LocalDateTime.parse(endDate);
        List<Job> jobs = jobQueue.getCompletedJobs(queueName, start, end, page, pageSize);
        return Response.ok(jobs).build();
    }

    @GET
    @Path("/{jobId}/watch")
    @Produces(MediaType.SERVER_SENT_EVENTS)
    public void watchJob(@PathParam("jobId") String jobId, @Context SseEventSink eventSink) {
        CompletableFuture<Job> future = jobQueue.watchJob(jobId);
        future.thenAccept(job -> {
            eventSink.send(sse.newEvent(Json.encode(job)));
            eventSink.close();
        });
    }
}

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response

[wezell]

Just a note, and I am sure you are on it, but we will need to be careful with the addJob and passing in clazz names to run/instantiate. We will probably need an allow list of job classes that can be called (and allow customers to add to it via osgi).

[fabrizzio-dotCMS]

Just a note, and I am sure you are on it, but we will need to be careful with the addJob and passing in clazz names to run/instantiate. We will probably need an allow list of job classes that can be called (and allow customers to add to it via osgi).

#29544

[fabrizzio-dotCMS]

@bryanboza, The following endpoint {{baseUrl}}/api/v1/jobs/{{jobId}}/status has been updated to include all additional details as shown on the job-list endpoints.
Postman was also updated to work with JWT

[bryanboza]

• Status

Tested every one of the endpoints and here some feedback

• For the /status endpoint, we need to return more information because this is the only way to retrieve details about a specific job by ID. Therefore, we should return all the complete information about the job, just as we do for failed jobs. In the status response, all job-related information must be included.

Job Status:

Failed job:

• Multipart is always required when creating a new job. If you only send an 'application/json' instead of "multipart/form-data" in the request without including the multipart, it will fail. We need to allow job creation without the need to send files—just by sending a regular JSON with the necessary information.

• Finally, creating a new endpoint to get jobs by status could be useful, as we currently don't have a way to filter all the jobs with a specific status.

So to summarize this fix:

• Point 1, was fixed and now the status endpoint is returning the complete information. ✅
  

• Point 2, according to @fabrizzio-dotCMS is a jersey limitation. We need to create a new endpoint to handle this case in the future. New card for it Allow job creation without requiring multipart/form-data #30429

• Point 3, need to be reported as an enhancement. New card for it Create a new endpoint to get jobs by status #30430