Karpenter CRD upgrade support via karpenter-crd chart

[armujahid]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Issue

The Karpenter helm chart does not manage the lifecycle of CRDs; it will only install the CRD during the first installation of the helm chart. Subsequent chart upgrades will not add or remove CRDs, even if the CRDs have changed.

Describe the solution you would like

Karpenter now provides a separate karpenter-crd https://github.com/aws/karpenter/tree/main/charts/karpenter-crd helm chart to manage the lifecycle of CRDs https://karpenter.sh/docs/upgrade-guide/#custom-resource-definition-crd-upgrades. I think that we should also include this in the Karpenter addon.

Alternatives

Since upgrading the CRDs is a risky process, users can manually check release notes and upgrade CRDs if required. This is what we currently do.

kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.28.0/pkg/apis/crds/karpenter.sh_provisioners.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.28.0/pkg/apis/crds/karpenter.sh_machines.yaml
kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.28.0/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml

Additional context

[askulkarni2]

@armujahid thanks for the issue! We should add this.

[mikesplain]

Frankly, couldn't this go for any resource in this repo or users of https://github.com/aws-ia/terraform-aws-eks-blueprints-addon that use a CRD? We have a separate process that installs the karpenter crd chart for us and it works well but we don't have a good solution for the CRDs for things like AWS Load Balancer Controller.

@askulkarni2 Where's the right spot for an issue more generalized around "CRD Management" for AWS supplied Addons?

Seems like reasonable solutions would be something along the line of:

• Skipping the CRD install within helm charts since it doesn't keep them up to date
• Managing the install of either "a helm chart of CRDS" like karpenter or automatically pulling the correct version of CRDs from the dependencies repository or helm chart.
• The Addons repository can supply those values for known AWS dependencies but users of the https://github.com/aws-ia/terraform-aws-eks-blueprints-addon will have to determine them for themselves (which sounds reasonable to me).

[askulkarni2]

Besides the github issues of the individual add-on repos, https://github.com/aws/containers-roadmap/issues would be the right place.