Fix panic issue with proportional scheduling

[Cdayz]

No description provided.

[Cdayz]

/assign @Thor-wl

[lowang-bh]

/hold

[Cdayz]

@lowang-bh Hi can you describe why you set hold label? Is there any problems with this PR, can i change it to make better?

[lowang-bh]

Hi, @Cdayz can you see this comment? In my opinion, a more elegent fix is to replace comapre function with comparing cpu and memory independently.

[lowang-bh]

I think a good way is don't use the compare function. Just compare CPU and Memory differently at this position:

volcano/pkg/scheduler/plugins/predicates/proportional.go

Lines 40 to 42 in 5302995

	r := node.Idle.Clone()
	r = r.Sub(task.Resreq)
	if r.MilliCPU < cpuReserved || r.Memory < memoryReserved {

[Cdayz]

Oh, probably it's not a right decision, this PR fixes bug when node have ANY of resources lower than requested by task, and it leads to panic on this line.

volcano/pkg/scheduler/plugins/predicates/proportional.go

Line 41 in 5302995

r = r.Sub(task.Resreq)

[Cdayz]

Also I think that maybe this pr made in a wrong place, because for example in allocate action of scheduler, checking that node resources sufficient for run task made before run other checks.

volcano/pkg/scheduler/actions/allocate/allocate.go

Lines 100 to 118 in 5302995

	predicateFn := func(task *api.TaskInfo, node *api.NodeInfo) ([]*api.Status, error) {
	// Check for Resource Predicate
	if ok, reason := task.InitResreq.LessEqualWithReason(node.FutureIdle(), api.Zero); !ok {
	return nil, api.NewFitError(task, node, reason)
	}
	var statusSets util.StatusSets
	statusSets, err := ssn.PredicateFn(task, node)
	if err != nil {
	return nil, fmt.Errorf("predicates failed in allocate for task <%s/%s> on node <%s>: %v",
	task.Namespace, task.Name, node.Name, err)
	}
	if statusSets.ContainsUnschedulable() || statusSets.ContainsUnschedulableAndUnresolvable() ||
	statusSets.ContainsErrorSkipOrWait() {
	return nil, fmt.Errorf("predicates failed in allocate for task <%s/%s> on node <%s>, status is not success",
	task.Namespace, task.Name, node.Name)
	}
	return nil, nil
	}

[Cdayz]

Maybe we can add this check to all actions in predicateFn ?

[Cdayz]

@lowang-bh check my comments please

[lowang-bh]

Maybe we can add this check to all actions in predicateFn ?

It doesn't all need resorce compare in predicateFn, eg, preempt and backfill action.

[lowang-bh]

how about use:

if r.MilliCPU - task.Resreq.MilliCPU  < cpuReserved || r.Memory - task.Resreq.Memory < memoryReserved 

to replace

 r := node.Idle.Clone() 
 r = r.Sub(task.Resreq) 
 if r.MilliCPU < cpuReserved || r.Memory < memoryReserved { 

[Cdayz]

@lowang-bh hey, i changed as you suggested

[lowang-bh]

/hold cancel

[Thor-wl]

@Cdayz Hi, I'm sorry for the late review. Can you demonstrate how this introduce panic? I've not gotten the point here.

[lowang-bh]

if node.idle has some dimension less than requst, there will be a assert.

[Thor-wl]

if node.idle has some dimension less than requst, there will be a assert.

OK. That's reasonable. I'm OK about the fix.

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Thor-wl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/scheduler/plugins/OWNERS [Thor-wl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment