Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Rely on blocked_executions.expires_at to select candidates to be rele…
…ased Sempahores' expiry times are bumped when new jobs are successfully queued for execution or when jobs finish and release the semaphores. This means that if a blocked execution's expiry time is in the past, the semaphore's expiry time is most likely in the past too. Here's why: we know that if we still have a blocked job execution after its expiry time has passed, it's because: 1. A job holding the semaphore hasn't finished yet, and in that case, the semaphore's expiry time would have expired as well and would be cleared up right before checking blocked jobs. 2. The job holding the semaphore finished and released the semaphore but failed to unblock the next job. In that case, when we inspect the blocked job's concurrency key, we'll see the semaphore released. a. It's possible a new job is enqueued in the meantime and claims the semaphore, so we wouldn't be able to unblock that blocked job. However, if this happens, it's also more likely that this new job will succeed at unblocking it when it is finished. The more jobs that are enqueued and run, bumping the semaphore's expiry time, the more likely we are to unblock the blocked jobs via the normal method. 3. The job holding the semaphore finished but failed to release the semaphore: this case is the same as 1, the semaphore will be cleared before unblocking the execution. We take advantage of this to select X (scheduler's batch size) distinct concurrency keys from expired blocked executions, and for that we can use the index on (expires_at, concurrency_key), that filters the elements, even if we have to scan all of them to find the distcint concurrency keys using a temporary table that would get at most X items long. Then, we'll check whether these (up to) X concurrency keys are releasable and try to release them. A potential problem would be if we happen to select X concurrency keys that are expired but turn out not to be releasable. I think this should be very unlikely because for this to happen, we'd have to failed to unblock X jobs via the regular method and that other jobs using the same concurrency keys were enqueued, claiming the semaphore (case 2.a. described above), before we had the chance to unblock them, for all of them. We'd need two exceptional things to happen at once: a large backlog of concurrent jobs (using different keys) and a large amount of failed unblocked jobs. Thanks to @djmb for thinking through this with me and all the help!
- Loading branch information