Env vars not expanded for scheduler service

[mcguipat]

In dask_kubernetes/core.py, the templates for the worker and the scheduler have environment variables expanded when loading configured templates; however, the scheduler-service-template does not. At present, I currently specify ownerReferences on the scheduler and workers with the originating pod (via environment variables set using the k8s downward api). In the event the originating pod goes down. Any dangling resources can be cleaned up automatically; however, the scheduler cannot in this case.

[jacobtomlinson]

Thanks for raising this @mcguipat.

Do you feel you could raise a PR to fix this?

[mcguipat]

I think there is probably a simple fix just wrapping the template in the expand function; however, when looking at the generate structure of the how the service template is handled. It seems quite different from the scheduler/worker templates. Is there any desire to align this or simply make a very targeted fix?

[jacobtomlinson]

@mcguipat I have no preference. It is down to you and how much time you are willing to spend on it.

[mcguipat]

The scheduler service generally is something that feels quite jammed in. As far as I can tell, there is not actually another way to manually configure a template outside of through dask.config, unlike the worker and scheduler. I think its best to probably do something proper here as it should give sufficient flexiblity for more involved service configurations. I will try to get to this over the weekend.

[jacobtomlinson]

The classic KubeCluster was removed in #890. All users will need to migrate to the Dask Operator. Closing.