[Proposal] Implementing an Automatic Re-Queuing and Status Transition Mechanism

[flyingfang]

What is the problem you're trying to solve

We can implement a mechanism that supports automatically re-queuing tasks that have entered the "Inqueue" status but remain unscheduled for an extended period of time. These tasks would then be transitioned to the “Pending” status.
This will help optimize the queue’s efficiency and ensure that tasks are processed in a timely manner.

When tasks in the scheduling queue remain unscheduled for an extended period due to factors such as affinity or resource fragmentation, they are occupying quota while blocking subsequent jobs from entering the queue.
The system can automatically release these resources to improve the allocation rate of the queue. This ensures that the system can more efficiently utilize its resources and process jobs in a timely manner.

Describe the solution you'd like

We can add a requeue action，to incorporate a re-enqueue process.
The main process in the entire action is to traverse all jobs in Pending and Inqueue status, as well as jobs with pending tasks. By a registerable function ssn.JobRequeueable, we determine whether the job needs to be re-queued.
Next, we will traverse the tasks that need to be re-queued, and call another registerable function ssn.JobRequeue, to implement the re-queueing of the tasks.

// Session information for the current session
type Session struct {
	UID types.UID
         // ....
	requeueableFns               map[string]api.VoteFn
	jobRequeueFns                map[string]api.JobRequeueFn
         // ....
}

type JobRequeueFn func(*JobInfo) error
type VoteFn func(interface{}) int

Additional context

No response

[Monokaix]

Can you also paste the related issues？

[Monokaix]

It's a good idea, but what I concern is that add a new action is a little heavy and it's a little cut off from enqueue action.
How about just use jobReady or other callback func to rollback the pg status?

[flyingfang]

It's a good idea, but what I concern is that add a new action is a little heavy and it's a little cut off from enqueue action. How about just use jobReady or other callback func to rollback the pg status?

I think "requeue" and "enqueue" are two decoupled logical operations, so I didn’t consider placing them within the same action. From the current perspective, besides increasing the overhead of invoking queueOrder and jobOrder, there do not seem to be any additional costs involved. In light of this, would keeping these as separate actions provide higher flexibility?

[Monokaix]

It doesn’t have to be placed in the enqueue action. The func UpdateJobStatus will update pg when a scheduling ends, can we just check the pg in inqueue status to rollback to pending if it's unschedulable instead of in a seperate action, becasue the ssn can hold the unschedulable message and we can use them to check whether change the pg status.

[flyingfang]

It doesn’t have to be placed in the enqueue action. The func UpdateJobStatus will update pg when a scheduling ends, can we just check the pg in inqueue status to rollback to pending if it's unschedulable instead of in a seperate action, becasue the ssn can hold the unschedulable message and we can use them to check whether change the pg status.

Keeping jobs in Inqueue status still makes sense in general scenarios. Because the jobs have already acquired their quotas and will be scheduled util the required resources are available.
What we are proposing is merely to open up a mechanism for the logic of automatically re-enqueuing jobs following specific strategies, while the default behavior remains to retain the jobs in the Inqueue status.

[Monokaix]

It doesn’t have to be placed in the enqueue action. The func UpdateJobStatus will update pg when a scheduling ends, can we just check the pg in inqueue status to rollback to pending if it's unschedulable instead of in a seperate action, becasue the ssn can hold the unschedulable message and we can use them to check whether change the pg status.

Keeping jobs in Inqueue status still makes sense in general scenarios. Because the jobs have already acquired their quotas and will be scheduled util the required resources are available. What we are proposing is merely to open up a mechanism for the logic of automatically re-enqueuing jobs following specific strategies, while the default behavior remains to retain the jobs in the Inqueue status.

So what's the difference between re-enqueuing jobs following specific strategies logs and re-enqueue when job schedulable?

[Monokaix]

How about discussing in weekly meeting?