Add tolerations to allow scheduling on tainted nodes

[nateguchi]

We want to have dedicated nodes in our K8s cluster that will only schedule the pods we want on them.

To achieve this we need to be able to set tolerations on the pods generated by Deis.

Is there any way this functionality could be implemented?

[kingdonb]

Sounds like a useful feature; I don't see any reason why it couldn't be implemented.

I know a little bit about node tolerations but my knowledge is a little dated (from when they were called "taints")

Would you be willing to take a stab at writing a design document for this? https://deis.com/docs/workflow/contributing/design-documents/

This is the first step towards any major change to behavior of Workflow components.

(I don't know if there are parts of the idea that you're not sure about, it would be fine to leave "WIP" notes in sections that you're not comfortable writing in detail, like for example maybe Code Changes. These sections may require some research. Starting a design document is still worthwhile even if you're not completely sure about the details.)