CAPI providers work behind an HTTP(S) proxy

[glitchcrab]

User Story

• As a giant swarm customer on CAP(VCD|V|O..) I want my environment to use a central HTTP(S) Proxy so that I have a central point to block traffic if I need to.

Tasks

in Progress

• create follow up items from document current proxy implementation rfc#50

open

done

• Find a solution for dependencies on the mutating webhooks app
• inject proxy variables to containerd (for capvcd this is already done via the vcdcluster.spec)
• Support creating Management Clusters using a proxy in mc-bootstrap #1423 (PR is open and ready for review)
• Support creating Clusters using an http proxy in cluster-cloud-director #1421
• extend cluster-apps-operator to handle a installation wide proxy secret (see also this slack thread)
• Create an app deploying Kyverno Mutating webhooks for automatic proxy configuration #1419
• Configure a VCD environment with a proxy for testing #1425

Background:

The new environment will have to be behind a proxy - this is a hard requirement from their security team. It is currently a transparent proxy but will be changed to authenticated in the near future.

Team rocket already did this for kvm/legacy . Here is documented how https://github.com/giantswarm/giantswarm/issues/17226#issuecomment-1226904904.

Related stories

• Proxy test environment #1427

[JosephSalisbury]

We need to work out what we need to do / what the current status is

[JosephSalisbury]

blocked by https://github.com/giantswarm/giantswarm/issues/17226, which documents status quo

[kopiczko]

There are two different approaches:

1. Using kyverno to inject proxy env vars.
2. Converting all our apps to support proxy settings (in values.yaml)

Solution 2. has the advantage of not using kyverno and ability to do some very app specific in case it's needed but requires way more work.

[cornelius-keller]

@alex-dabija @puja108 from discussion with @giantswarm/team-rocket the question what should be the user-experience for the customer came up.
As team-rocket we would prefer to configure the web-hooks to only touch Giantswarm Workloads and Apps, and don't touch customer workloads.
Do we have any expections on the customer side? If we leave the customer workloads alone the customer would need to either configure all there apps to be proxy aware, or have its own mutating web-hook.

[cornelius-keller]

@kopiczko @bavarianbidi how would we distinct customer workloads from managed apps installed in customer name spaces?

[alex-dabija]

Personally, I think it's fine to focus only on our apps for now. I don't yet know if our customer would be fine though.

[kopiczko]

@kopiczko @bavarianbidi how would we distinct customer workloads from managed apps installed in customer name spaces?

Do we keep any workloads outside giantswarm, kube-system and flux* namespaces?

[kopiczko]

BTW there is also a chicken-egg problem because flux and app platform are installed well before kyverno so they will need to support proxy mode without webhooks. Same applies to kyverno itself possibly (if it needs internet for anything).

[bavarianbidi]

As kyverno is currently deployed in namespace giantswarm, it's not possible to create a mutating config which affects pods in namespace giantswarm.

The future goal in general would be moving kyverno out of namespace/giantswarm, but this isn't implemented yet:

• https://github.com/giantswarm/giantswarm/issues/23971
• slack thread #1
• slack thread #2

[bavarianbidi]

Current status:

With the below make create-mc target in mc-bootstrap (based on mc-bootstrap PR #264) it's possible to create a workload-cluster behind a corporate proxy.

INSTALLATION=gario CUSTOMER=GS PROVIDER=cloud-director NEW_TEST_INSTALLATION=1 EDITOR=vim MC_PROXY_ENABLED=true MC_HTTP_PROXY="http://giantswarm:[email protected]:3128" MC_HTTPS_PROXY="http://giantswarm:[email protected]:3128" MC_NO_PROXY="vmware.ikoula.com" make create-mc

There are currently two "limitations":

• CAPVCD: due an issue in capvcd i have to edit the status of vcdcluster during clusterctl move phase and add existing RDE_ID (from initial cluster creation) there (to keep track of that, issue #24517 got created)
• kyverno-policies-connectivity: once the cluster got created, installations/<INSTALLATION_NAME>/apps/kyverno-policies-connectivity/configmap-values.yaml.patch in config must be updated to set the proper noProxy list for this cluster. This can be improved in two different ways:
  • kyverno-policies-connectivity could take care of cluster-values --> open topic in the rfc discussion
  • add an additional step in mc-bootstrap where we update the already created config in config-repo and write the cluster-specific noProxy values there.

  NOTE: as we apply the kyverno-policies-connectivity in a very early stage with the right noProxy configuration it's fine during the initial cluster creation. Only before the app-platform takes over the configuration of this app, we have to updated the configuration in config repo

A more detailed view/description of the current implementation can be found in this RFC where still some open questions should be discussed.

[kopiczko]

add an additional step in mc-bootstrap where we update the already created config in config-repo and write the cluster-specific noProxy values there.

Do we need to configure the config with the extra values? Can't kyverno read those extra values directly from cluster object in the cluster?

[bavarianbidi]

Do we need to configure the config with the extra values? Can't kyverno read those extra values directly from cluster object in the cluster?

Yes, as kyverno-policy-connectivity will lead to a kyverno cluster policy which will inject the http_proxy, https_proxy and no_proxy variable to all pods (outside namespace/kube-system) with the proxy configuration all pods need in this MC.

As the kyverno-policy-connectivity App will get deployed as Kind: Konfigure we have to make use of the cluster-secret there.
(Please see this comment in the proxy-RFC about the issue)

[kopiczko]

It looks like kyverno policies are able to dynamically read values from arbitrary kubernetes objects https://kyverno.io/docs/writing-policies/external-data-sources/#variables-from-kubernetes-api-server-calls.

We are/were even using this ourselves https://github.com/giantswarm/kyverno-policies/blob/f7e3ed0c0e9e917ed5b3495e74b7da604831dad2/policies/aws/AWSCAPI.yaml#L49-L57

It is also possible to use the context value in the mutate: part of the policy

[bavarianbidi]

Good catch! But this implementation is limited to management clusters (as Workload clusters don't have the cluster specific CR in place).

Theoretically it should be possible to extend the existing kyverno-policy-connectivity chart to use the values from cluster.proxy.noProxy, proxy.noProxy or dynamic discovered values via api server calls.

[erkanerol]

While creating guppy by enabling proxy and blocking all internet access, I hit the issues here #1424 (comment) and 3 more issues.

• chart-operator is crashing because of the init container. Related thread
  https://gigantic.slack.com/archives/C01F7T2MNRL/p1669707288159799

UPDATE: Fixed with giantswarm/cluster-apps-operator#316

• cert-manager doesn't work since noProxy contains baseDomain. Related thread https://gigantic.slack.com/archives/C01F7T2MNRL/p1669720827283559

UPDATE: Fixed with giantswarm/cluster-cloud-director#57

• silence-operator-sync jobs don't work since env variables in the init container are missing.

UPDATE: Resolved when the version of kyverno-policies are updated.

[erkanerol]

guppy is up&running behind a proxy without direct internet access. The only issue is the dynamic kyverno policy as discussed above. I am going to try to implement it.

[erkanerol]

Dynamic kyverno policy will be supported with this PR
giantswarm/kyverno-policies-connectivity#38

[erkanerol]

CAPA and CAPVCD work fine with proxy. Dynamic kyverno policy is implemented. We decided to handle CAPO with a separate issue. #1783