running tasks get spuriously killed

[jcristau]

From time to time we hit an issue where a scriptworker task gets killed: when it periodically scans each pool, k8s-autoscale counts pending tasks and running workers, and if it thinks there are too many running workers it tells k8s to stop them. scriptworker gets SIGUSR1, which tells it to stop after the current task. However if it's not done after terminationGracePeriodSeconds (currently 20 minutes, except for treescript where it's 1 hour), it gets SIGTERM and terminates the running task, which then has to be rerun for no good reason.

[gabrielBusta]

Maybe the workers could shutdown gracefully and tell auto-scale when they are done with the current task?