Do not create the launcher job if the job starts suspended #670
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| // If the job is suspended, the list of worker pods will be incorrect. We also do | ||
| // not want to start the launcher job if the MPIJob starts suspended. | ||
| if launcher == nil && !isMPIJobSuspended(mpiJob) { |
There was a problem hiding this comment.
This opens the question on what should be done when unsuspending the launcher job in case kueue has decided to change the NodeSelector? Should we instead recreate the job since NodeSelector is immutable?
There was a problem hiding this comment.
Yeah, I think this question does not have a straigtforward answer, I can see at least two approaches possible:
- as in JobSet- update the NodeSelector field in pod template when resuming the Job
- Recreate the launcher/worker Jobs , this can be probably achieved easily by deleting the jobs on suspending the MPIJob
I'm ok with any of those that is simpler to implement. Any optinion @tenzen-y ?
There was a problem hiding this comment.
In any case I think it is safe to decouple the fixes.
|
@GonzaloSaez please fix the DCO |
Signed-off-by: GonzaloSaez <[email protected]>
Signed-off-by: GonzaloSaez <[email protected]>
GonzaloSaez
force-pushed
the
g/fix_job_launch_wait_for_pods_ready
branch
from
802bd49
to
804cfd8
Compare
|
For context, linking it back to the related Kueue issue: kubernetes-sigs/kueue#3400 and the slack discussion https://kubernetes.slack.com/archives/C032ZE66A2X/p1730369507818399 |
When the MPIJob starts suspended, we were creating the launcher job no matter the initial suspended state. This causes issues with kueue, since it will suspend the MPIJob but it will create a job with the wrong NodeSelector coming from the kueue flavour. I think avoiding creating the launcher in this scenario is the right thing to do but I'm not sure if others have different thoughts.