EKS SSO with dex

[T-Kukawka]

User Story

Currently when creating EKS CAPA cluster, customer is required to use AWS authentication in order to connect with the clusters. What would be perfect, is that the GS dex application is easily deployable on the EKS clusters, such that customers can use the OIDC provider they would usually use for any other CAPA or Vintage clusters.

Acceptance Criteria

• Dex application works on EKS clusters
• Documentation for customers is provided for the dex configuration on EKS clusters

Dependencies (optional)

• EKS instructions - https://intranet.giantswarm.io/docs/dev-and-releng/capi-eks-internal-doc/

Implementation details

• Implementation info to be filled out by engineer

[tuladhar]

In standup, Spyros mentioned this post: https://aws.amazon.com/blogs/containers/using-dex-dex-k8s-authenticator-to-authenticate-to-amazon-eks/

[T-Kukawka]

@gawertm @anvddriesch this might be nice to show with demo next week

[anvddriesch]

From my perspective this should just be the same as any other cluster since the eks chart supports setting oidc flags in the same way as the other providers https://github.com/giantswarm/cluster-eks/tree/main/helm/cluster-eks#control-plane
However, I'll confirm by installing the auth-bundle on an eks cluster to double check. 😄

[gawertm]

not sure we need that for customer already. as we don't even know if they run on EKS?

[T-Kukawka]

@gawertm yes they want EKS clusters - that was clear from the talk we had at least

[anvddriesch]

So I created an eks cluster with oidc flags set up as described in our docs. Then I installed the auth-bundle without any extra configuration.
There were no issues installing the bundle and the actual oidc flow is successful.

However, when validating the token, kubectl gs fails with the following message.

Error: Token verification process failed: Get "https://api.ant1.gaws.gigantic.io:443/version?timeout=32s": tls: failed to verify certificate: x509: certificate is valid for 3f7cff008919835d5005fffa7a17f532.gr7.eu-central-1.eks.amazonaws.com, ip-172-16-160-86.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not api.ant1.gaws.gigantic.io

So it looks like the issue here is the api address we are trying to use for the cluster. I'll see if I can fix this.

[anvddriesch]

Issues so far:

• oidc flags are not applied when using the template command. I had to edit the controlplane values myself in order to propagate them
• after this not being an issue initially, in subsequent tests I was unable to make cert-manager issue a valid certificate for the ingresses in the auth bundle. The problem is that the http01 challenge is timing out.
  I confirmed that a public lb is created and used for the ingresses, so it should work. Furthermore, this problem also occured when the auth bundle was deployed in giantswarm namespace which has a specific netpol to allow traffic for http01 solvers.

[ssyno]

auth-bundle deployed successfully on spy0 EKS on snail-MC
CAPEKS template:

---
apiVersion: v1
data:
  values: |
    global:
      metadata:
        name: spy0
        organization: bigmac
      controlPlane:
        oidcIdentityProviderConfig:
          clientId: dex-k8s-authenticator
          groupsClaim: groups
          usernameClaim: email
          issuerUrl: https://dex.spy0.gaws.gigantic.io
          identityProviderConfigName: dex-k8s-authenticator
      podSecurityStandards:
        enforced: true
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    giantswarm.io/cluster: spy0
  name: spy0-userconfig
  namespace: org-bigmac
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  labels:
    app-operator.giantswarm.io/version: 0.0.0
  name: spy0
  namespace: org-bigmac
spec:
  catalog: cluster
  config:
    configMap:
      name: ""
      namespace: ""
    secret:
      name: ""
      namespace: ""
  kubeConfig:
    context:
      name: ""
    inCluster: true
    secret:
      name: ""
      namespace: ""
  name: cluster-eks
  namespace: org-bigmac
  userConfig:
    configMap:
      name: spy0-userconfig
      namespace: org-bigmac
  version: 0.16.0
---
apiVersion: v1
data:
  values: |
    clusterName: spy0
    organization: bigmac
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    giantswarm.io/cluster: spy0
  name: spy0-default-apps-userconfig
  namespace: org-bigmac
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  labels:
    app-operator.giantswarm.io/version: 0.0.0
    giantswarm.io/cluster: spy0
    giantswarm.io/managed-by: cluster
  name: spy0-default-apps
  namespace: org-bigmac
spec:
  catalog: cluster
  config:
    configMap:
      name: spy0-cluster-values
      namespace: org-bigmac
    secret:
      name: ""
      namespace: ""
  kubeConfig:
    context:
      name: ""
    inCluster: true
    secret:
      name: ""
      namespace: ""
  name: default-apps-eks
  namespace: org-bigmac
  userConfig:
    configMap:
      name: spy0-default-apps-userconfig
      namespace: org-bigmac
  version: 0.7.0

auth-bundle values

---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  finalizers:
  - operatorkit.giantswarm.io/app-operator-app
  generation: 1
  labels:
    app-operator.giantswarm.io/version: 0.0.0
    app.kubernetes.io/name: auth-bundle
    giantswarm.io/cluster: spy0
    policy.giantswarm.io/psp-status: disabled
  name: spy0-auth-bundle
  namespace: org-bigmac
spec:
  catalog: giantswarm
  config:
    configMap:
      name: spy0-cluster-values
      namespace: org-bigmac
  extraConfigs:
  - kind: configMap
    name: psp-removal-patch
    namespace: org-bigmac
    priority: 150
  kubeConfig:
    context:
      name: spy0
    inCluster: true
    secret:
      name: spy0-kubeconfig
      namespace: org-bigmac
  name: auth-bundle
  namespace: org-bigmac
  userConfig:
    configMap:
      name: spy0-auth-bundle-user-values
      namespace: org-bigmac
  version: 0.1.3
---
apiVersion: v1
data:
  values: |
    apps:
      athena:
        userConfig:
          configMap:
            values: |
              managementCluster:
                name: snail
      dex-app:
        userConfig:
          configMap:
            values: |
              isWorkloadCluster: true
      ingress-nginx:
        enabled: true
kind: ConfigMap
metadata:
  labels:
    app-operator.giantswarm.io/watching: "true"
  name: spy0-auth-bundle-user-values
  namespace: org-bigmac

[ssyno]

Error: Token verification process failed: Get "[https://api.spy0.gaws.gigantic.io:443/version?timeout=32s](https://api.spy0.gaws.gigantic.io/version?timeout=32s)": tls: failed to verify certificate: x509: certificate is valid for 6eae2f2e28f9d92eff54dfa57c37081d.gr7.eu-central-1.eks.amazonaws.com, ip-172-16-121-133.eu-central-1.compute.internal, kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, not api.spy0.gaws.gigantic.io

The issue still persist as there doesn't seem to be an option yet to define extraSANs or global.controlPlane.apiExtraCertSANs like on cluster-aws on clusterEKS yet. As of this certificates are not valid for the API-URL:api.spy0.gaws.gigantic.io

[anvddriesch]

I tried using the eks api-endpoint as address in athena and that resulted in a valid token.
So in addition to the oidc settings seen above in the comment by @ssyno, this was needed:

apiVersion: v1
data:
  values: |
    apps:
      athena:
        userConfig:
          configMap:
            values: |
              managementCluster:
                name: snail
              kubernetes:
                api:
                  address: https://6EAE2F2E28F9D92EFF54DFA57C37081D.gr7.eu-central-1.eks.amazonaws.com
[...]

[tuladhar]

oidc flags are not applied when using the template command. I had to edit the controlplane values myself in order to propagate them

Phoenix confirmed that it's not implemented on CAPA/EKS. See more

[tuladhar]

Here's the PR for updating our OIDC Dex documentation with changes related to EKS.

[tuladhar]

✅ Dex application works on EKS clusters
✅ Documentation for customers is provided for the dex configuration on EKS clusters. Doc link

@T-Kukawka @alex-dabija

[T-Kukawka]

closing this, thanks a lot!