Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

two jobs of different queue contend for more resource than weight will get into lock status #3725

Open
cocodee opened this issue Sep 14, 2024 · 1 comment
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@cocodee
Copy link

cocodee commented Sep 14, 2024

Description

there are two queues: A ,B and the cluster have 10GPUs, and both A and B have a deserved with 5GPUs and Capability with 10GPUs.
Now, user submit jobs that use 6GPUs to queue A. After A begins running. user then submit not preemptable tasks to queue A and use up 6GPUs,then sub mit another task to queue B and use 6GPUs.
Then ,there willl be one job running and two job in queue. Then delete the running job. then another two jobs will stay in pending status for ever

Steps to reproduce the issue

Scenario Description
Queue Configuration:

Queue A and B each have a quota (deserved) of 5 GPUs, but each queue has a capacity (capability) of 10 GPUs.

Job Submission:

A user submits a job that uses 6 GPUs to Queue A.

Queue A starts running this job.

The user submits two non-preemptable tasks to Queue A, each using 6 GPUs.

The user submits another job that uses 6 GPUs to Queue B.

Job Status:

One job is running, and two jobs are waiting in the queue.

The running job is deleted.

The two jobs remain in a pending state.

Describe the results you received and expected

The two pending jobs could be scheduled.

What version of Volcano are you using?

v1.9.0

Any other relevant information

No response

@cocodee cocodee added the kind/bug Categorizes issue or PR as related to a bug. label Sep 14, 2024
@lowang-bh
Copy link
Member

please paste your scheduler configmap, and your job/queue's yaml.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

2 participants