fix: cache cancellations to prevent race conditions in pod creation #230
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
There exists a condition where a cancellation can be actioned before the task or build has been started, so that when the task or build are started they have no idea that it was cancelled. This is because there are two queues used, one for builds and tasks, the other for misc tasks. Cancellations enter via the misc queue, and because they are handled in their own queue, it is possible that the cancellation message could be processed before the build or task message is handled completely.
This implements an expiring cache that will cache cancellations, so that if the condition is met and the pod gets started, the pod controller will check the cache periodically and if the cancellation exists in cache will cancel the task or build in the way it would normally be cancelled.