AAW Infra: scale down gerenal nodepool

[Souheil-Yazji]

Is your feature request related to a problem? Please link issue ticket

Let's look at scaling down the General Nodepool to saturate the nodes better.
Currently, with some observation using grafana/lens, I can see that we are looking at maximum utilization of ~30% for CPU/mem in terms of node resource usage.

Another issue is that most of the daemonsets that don't necessarily need to be on the general nodes do have a pod on them. This is costly at no added value.

Describe the solution you'd like

• Investigate the resource saturation on the average general nodepool node.
• Investigate the daemonsets which deploy to general nodes
• Identify possible clean up for those daemonsets, we can use taints/toleration to prevent them from scheduling pods to the general nodes
• Apply step 3
• Collect metrics on new resource utilization
• Adjust the size of the nodes by changing the VMSS VM type
• Save lots of money.

Describe alternatives you've considered

NA

Additional context