CAPA Kubernetes API allow lists

[alex-dabija]

Story

-As a cluster admin, I want to have the Kubernetes API protected by allow lists in order to improve the security of clusters.

Background

The Kubernetes API is not protected for public CAPA clusters. This, of course, is really bad for security and needs to be changed.

Requirements

• Access is restricted by default;
• Firewall rules to allow access only from specific IPs or CIDRs need to be configured for the Kubernetes API and bastions hosts of management clusters and workload clusters;
• Firewall rules must contain the specific port to which access is granted;
• The default list should include Giant Swarm's VPN endpoints;
• The customer needs to be able to configure the list of IPs or CIDRs (values.yml);

Questions

• Should firewall rules be implemented for private clusters?

Links

• capg-firewall-rule-operator;

[bdehri]

Are we going to create a new operator for that?

[fiunchinho]

Regarding the creation of a new operator, I don't think that's going to work. CAPA is reconciling the security groups and the ingress rules. I tested changing the ingress rules on one of the security groups and it got reverted by CAPA. So creating a new controller wouldn't help because it would be fighting CAPA all the time, re updating the ingress rule forever.

I submitted a PR upstream that would provide us with this functionality. The problem with that is that it will probably take a while until it's merged. Maybe we could use our fork if we really need this and we can't wait.

[alex-dabija]

I submitted a PR upstream that would provide us with this functionality. The problem with that is that it will probably take a while until it's merged. Maybe we could use our fork if we really need this and we can't wait.

I think we should be integrating in our fork any patch / PR we submit upstream because we would be testing the change before we get feedback from them.

[fiunchinho]

I submitted a PR upstream that would provide us with this functionality. The problem with that is that it will probably take a while until it's merged. Maybe we could use our fork if we really need this and we can't wait.

I think we should be integrating in our fork any patch / PR we submit upstream because we would be testing the change before we get feedback from them.

Yeah, agree. I'll do that. Regarding solving this issue, I think we can still wait and we can consider deploying our fork to all MCs later on if upstream doesn't move fast enough?

[fiunchinho]

This has been merged but still not released.

[fiunchinho]

Waiting on this PR now kubernetes-sigs/cluster-api-provider-aws#4406

[fiunchinho]

This PR exposes a new value in our cluster-aws chart so that users can specify which cidrs to allow in the k8s api. Once that's merged and released, both GiantSwarm and customers can start using it. To secure our own MCs, we need to pass the MC own Nat Gateway IPs using the new exposed value loadBalancerIngressAllowCidrBlocks in the cluster-app values in the gitops repository. Our VPN IPs are automatically added whenever another cidr is sets, so there is no need to set it: only the MC Nat Gateway IPs are necessary.

[fiunchinho]

To make it easier for customers to template new clusters without asking us the MC Nat Gateway IPs, we talked about adding a new defaulting to kubectl-gs that would read the MC Nat Gateway IPs when templating a cluster, and adding those IP's to the ones passed by the user.

The problem is that the field containing the Nat Gateway IPs was added in the latest CAPA release v2.2.4, which means we need to bump version of dependencies on kubectl-gs, and I tried and it breaks everything. So that part is not done yet.

[AndiDog]

I asked in Slack whether CAPA maintainers would consider upgrading controller-runtime, but I don't think this will come very soon. So let's leave the feature as-is for now. I'll write a documentation page and then we can improve things later if it turns out problematic or solvable.