tEKS

tEKS is a set of Terraform / Terragrunt modules designed to get you everything you need to run a production EKS cluster on AWS. It ships with sensible defaults, and add a lot of common addons with their configurations that work out of the box.

Roadmap

When this projet started, it did not rely on the official Terraform EKS module which is now quite stable and allows advanced configurations. The goal is now to migrate parts of this project to the upstream one and offloading part of the work to official modules and integrating them with Terragrunt:

• AWS VPC
• EKS
• [eks-addons-upstream] will be a rework of the old eks-addons module to make it compatible with upstream.

The following modules will be kept for compatibility:

• eks
• eks-addons

Branches

• master: Backward incompatible, development will continue with Terraform 0.12.X and Terragrunt 0.19.X. Releases bumped to v2.X.X
• release-1.X: Compatible with Terraform < 0.12 and Terragrunt < 0.19. Be sure to target the same modules version.

Upgrading from older version

v1.X is compatible with Terraform < 0.12 and Terragrunt < 0.19. The upgrade path to v2.x is simple:

• update tooling locally
• migrate from terraform.tfvars files to terragrunt.hcl as shown in live folder

Main features

• Node pools with customizable labels / taints
• Fully customizable kubelet args
• Supports new or existing VPC
• Calico for network policies
• Common addons with associated IAM permissions if needed:
  • cluster-autoscaler: scale worker nodes based on workload
  • external-dns: sync ingress and service records in route53
  • cert-manager: automatically generate TLS certificates, supports ACME v2
  • kiam: prevents pods to access EC2 metadata and enables pods to assume specific AWS IAM roles
  • nginx-ingress: processes Ingress object and acts as a HTTP/HTTPS proxy (compatible with cert-manager)
  • metrics-server: enable metrics API and horizontal pod scaling (HPA)
  • prometheus-operator: Monitoring / Alerting / Dashboards
  • virtual-kubelet: enables using ECS Fargate as a provider to run workload without EC2 instances
  • fluentd-cloudwatch: forwards logs to AWS Cloudwatch
  • node-problem-detector: Forwards node problems to Kubernetes events
  • flux: Continous Delivery with Gitops workflow.
  • sealed-secrets: Technology agnostic, store secrets on git.

Requirements

• Terraform
• Terragrunt
• kubectl
• helm
• aws-iam-authenticator

Documentation

User guides, feature documentation and examples are available here

About Kiam

Kiam prevents pods from accessing EC2 instances IAM role and therefore using the instances role to perform actions on AWS. It also allows pods to assume specific IAM roles if needed. To do so kiam-agent acts as an iptables proxy on nodes. It intercepts requests made to EC2 metadata and redirect them to a kiam-server that fetches IAM credentials and pass them to pods.

For security reasons, because Kiam needs to assume an IAM role that can assume other roles, it is best to run it on a dedicated node with specific IAM permission where no other workload are running and where there is no kiam-agent (because kiam-server need access to EC2 metadata). This is taken care of by default but it is customizable.

Addons that require specific IAM permissions

Some addons interface with AWS API, for example:

• cluster-autoscaler
• external-dns
• cert-manager
• virtual-kubelet: only with Kiam enable

Without KIAM

If you are not using Kiam, addons must access EC2 instances metdata to get credentials and access Kubernetes API, it is best for security reason to use a dedicated node for addons in that case to avoid other pods to access IAM roles and messed up route53, or scale down your cluster for example.

With Kiam

If you are using Kiam, these addons can run either with instances IAM roles on the same dedicated nodes where kiam-server is running amd bypass kiam-agent or they can run anywhere and assume a specific role through Kiam.

The following matrix tries to explain the possible combinations:

					virtual-kubelet (assume role with Kiam)
cluster-autoscaler		cluster-autoscaler			cert-manager (assume role with Kiam)	cert-manager (assume role with Kiam)
external-dns		external-dns			cluster-autoscaler (assume role with Kiam)	cluster-autoscaler (assume role with Kiam)
cert-manager		cert-manager	virtual-kubelet (assume role with Kiam)		external-dns (assume role with Kiam)	external-dns (assume role with Kiam)
kiam-server		kiam-server	kiam-agent	kiam-server	kiam-agent	kiam-agent (in HostNetwork, bypass kiam-agent)
Dedicated node(s) with IAM roles attached	Worker node(s)	Dedicated node(s) with IAM roles attached (bypass kiam-agent)	Worker node(s)	Dedicated node(s) with only IAM role for Kiam (bypass kiam-agent)	Worker node(s)	Worker node(s) with only IAM role for Kiam (bypass kiam-agent) !!! Security concerns !!!
Without Kiam		With Kiam		With Kiam		With Kiam

License