[EKS] [request]: Add Node Selectors for Amazon CloudWatch Observability EKS add-on

[fluckx-cs]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I want the amazon cloudwatch observability EKS addon to support node-selectors.
Please allow us to specify node-selectors for any controller pods and for the daemonset.
Both controller pods and daemonset nodeselector can be different and should be configurable separately!

Which service(s) is this request for?
EKS Addon -- Amazon Cloudwatch Observability

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I'm trying to install the EKS addon and have the fluent-bit daemonset and other controller pods only choose specific instances.
The reason is mainly related to cost. We want to run the observability on a subset of instances and not all of them.

Today this isn't possible through installation and configuration of the EKS addon. We have to patch it in post-install.
The problem is that the nodes where any "infra" is hosted is not shared with pods where the applications run ( where possible ).

Before the toleration change ( Issue #2195 ) we had to patch both the toleration and the node-selector.
As the configuration now supports tolerations, we only need to support the node-selectors.

Are you currently working around this issue?
Yes.
First we install the eks add-on. Afterwards we patch the deployment/daemonset with the node selector.

Additional context

Attachments

[rickypeng99]

Would also be great if we can configure toleration and node selectors for each individual components.

[loganmzz]

@rickypeng99 I opened #2515