kwok controller deployment shouldn't schedule onto a kwok node

[njtran]

What would you like to be added?

Currently I'm adding in a node affinity so that the kwok controller deployment doesn't schedule onto a kwok node here in Karpenter: https://github.com/kubernetes-sigs/karpenter/blob/main/hack/install-kwok.sh#L34-L40

      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kwok.x-k8s.io/node
                operator: DoesNotExist

I see the current deployment spec doesn't have this, https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/kwok/deployment.yaml

Which means that I have to orchestrate patching this in my kustomize, when I think it's pretty normal to not want to schedule the pod that needs to run to a fake node.

Happy to help add this in if it would be supported.

Why is this needed?

Kwok controller pods shouldn't schedule onto a kwok node, since the controller wouldn't be able to run.

[wzshiming]

That makes sense, soon I will support helm chart #1138. I noticed that karpenter also uses chart, this one in the chart which may be more appropriate. WDYT?

[njtran]

If you're porting over everything to a helm chart, that makes sense to me. I'm also thinking it probably could be added to the existing deployment spec in the meantime, right? Unless you think the chart will be coming out with the next release.

[wzshiming]

I've released helm chart 0.0.1, looks pretty well.

https://github.com/kubernetes-sigs/kwok/blob/main/charts/kwok/
https://artifacthub.io/packages/helm/kwok/kwok

I plan to release kwok 0.6 and kwok chart 0.1 this week.

[njtran]

Got it. I'm using Kwok in Kind clusters in Karpenter presubmit tests. I currently have 10 replicated copies of the kwok install, all pointing to different partitions to get around the scalability limits.

I'm currently upgrading our kwok from v0.3.0 to v0.5.2.

Is it possible to add configuration in the helm chart for https://github.com/kubernetes-sigs/kwok/blob/main/kustomize/kwok/deployment.yaml#L16

    value: --manage-nodes-with-label-selector=kwok-partition=a

or make it so that there's some way in the helm chart to replicate the deployment?

[wzshiming]

The roadmap had added an operator for that.

• Add operator to manage controller in cluster #1122
• Add operator to manage any resource created and deleted #1126

[wzshiming]

value: --manage-nodes-with-label-selector=kwok-partition=a

Sure, it just needs to add a new parameter in chart.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten