Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dead lock protection #13

Open
idelvall opened this issue Nov 15, 2016 · 0 comments
Open

Dead lock protection #13

idelvall opened this issue Nov 15, 2016 · 0 comments

Comments

@idelvall
Copy link
Member

idelvall commented Nov 15, 2016

Several deadlock scenarios can occur:

  1. Submitted job with excessive RSS promise (and schedulerCfg.maxJobRSSBytes=-1)
  2. Submitted child job (job launched by other job that waits for it) and the sum of the promises from the root ancestor to it exceeding schedulerCfg.maxTotalRSSBytes
  3. All running jobs waiting for queued jobs to end

The two first are irresolvable scenarios, so submission should fail.

The last one can be addressed in different ways.

  • Making the scheduler preemptive, so running jobs could be stopped for later execution. A way of freezing a process is needed for this. This would allow to freeze a process to make room to other process descendants (not its own descendants)
  • Canceling running jobs, maybe introducing the concept of job idempotence for later relaunching
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant