This tutorial will walk you through how to install a Kubernetes cluster on AWS using kops.
kops, short for Kubernetes Operations, is a set of tools for installing, operating, and deleting Kubernetes clusters. kops can also perform rolling upgrades from older versions of Kubernetes to newer ones. kops also manages the cluster add-ons. After the cluster is created, the Kubernetes CLI can be used to manage resources in the cluster. If you have not gone through the Prerequisites section, please do so now before continuing.
kops is available on Mac OSX, Linux, or the Windows 10 Linux Subsystem.
There is no need to download a Kubernetes binary distribution for creating a cluster using kops. However, you do need to download the kops CLI. It takes care of downloading the right Kubernetes binary in the cloud, and then provisions and manages the cluster.
Install kops using brew.
$ brew update && brew install kops
If kops is already installed, then it can be upgraded to the latest version using the following command:
$ brew upgrade kops
curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64 chmod +x kops-linux-amd64 sudo mv kops-linux-amd64 /usr/local/bin/kops
kops is available on Mac OSX, Linux, Container image, or using with the Windows 10 Linux Subsystem. Complete installation instructions are available at https://github.com/kubernetes/kops#installing.
Latest and early versions of kops can be downloaded from https://github.com/kubernetes/kops/releases.
You will also need the Kubernetes CLI (kubectl) to manage the resources on the Kubernetes cluster. If you don’t have the Kubernetes CLI installed, please follow the instructions here to do so.
kops uses a public SSH key while creating a cluster. The location of this file defaults to ~/.ssh/id_rsa.pub. Please generate an SSH key using ssh-keygen command.
$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (~/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in ~/.ssh/id_rsa Your public key has been saved in ~/.ssh/id_rsa.pub. The key fingerprint is: SHA256:bQXH2zAAcFVtpEwcOL5D1QZnXp3zkiC123456789 The key's randomart image is: +---[RSA 2048]----+ | ..oo=B**+ +| | . o+=B*=+.| | +..+*O+E=| | . =o=.o=oo| | S.*.. o | | oo | | . | | | | | +----[SHA256]-----+
Alternatively, you can use the --ssh-public-key option to specify a custom location.
More details about kops security can be found in the documentation for kops.
Make sure the latest version of the AWS CLI is installed.
$ pip install --upgrade awscli
The AWS user profile used in this workshop must have these IAM policies attached.
AmazonEC2FullAccess AmazonRoute53FullAccess AmazonS3FullAccess IAMFullAccess AmazonVPCFullAccess
If you are using the AWS user and credentials created as part of the Prerequisites, then you should already have these permissions.
Please review these links for additional info on IAM permissions: https://github.com/kubernetes/kops/blob/master/docs/aws.md#setup-iam-user. https://github.com/kubernetes/kops/blob/master/docs/iam_roles.md
kops needs a “state store” to store configuration information of the cluster. For example, how many nodes in the cluster, the instance type of each node, and the Kubernetes version. The state is stored during the initial cluster creation. Any subsequent changes to the cluster are also persisted to this store. Amazon S3 is the recommended storage mechanism when running a cluster in AWS Cloud. Create an S3 bucket and pass that to the kops CLI during cluster creation. A state store can work with multiple kops clusters.
|
Note
|
The bucket name must be unique otherwise you will encounter an error on deployment. We will use an example bucket name of example-state-store- and add a randomly generated string to the end.
|
# create variables for bucket, state store, and cluster names
$ export S3_BUCKET=example-state-store-$(cat /dev/random | LC_ALL=C tr -dc "[:alpha:]" | tr '[:upper:]' '[:lower:]' | head -c 32)
$ export KOPS_STATE_STORE=s3://${S3_BUCKET}
# use AWS CLI to create the bucket $ aws s3 mb $KOPS_STATE_STORE
# enable versioning $ aws s3api put-bucket-versioning \ --bucket $S3_BUCKET \ --versioning-configuration \ Status=Enabled
The kops CLI can be used to create a highly available cluster, with multiple master nodes spread across multiple Availability Zones. Workers can be spread across multiple zones as well. Some of the tasks that happen behind the scene during cluster creation are:
-
Provisioning EC2 instances
-
Creating and configuring AWS resources such as a VPC, Auto Scaling Groups, IAM users, and security groups
-
Installing Kubernetes
-
If required configuring Route53 DNS
When setting up a cluster you have two options on how the nodes in the cluster communicate:
-
Using the gossip protocol - kops has support for a gossip-based cluster. This does not require a domain, subdomain, or Route53 hosted zone to be registered. A gossip-based cluster is therefore easier and quicker to setup, and is the preferred method for creating a cluster for use with this workshop.
-
Using DNS - Creating a Kubernetes cluster that uses DNS for node discovery requires your own domain (or subdomain) and setting up Route 53 hosted zones. This allows the various Kubernetes components to use DNS resolutions find and communicate with each other, and for kubectl to be able to talk directly with the master node(s).
Instructions for creating a gossip-based cluster are provided below, however, the examples in the workshop should work with either option. Instructions for creating a DNS-based cluster are provided as an appendix at the bottom of this page.
kops supports creating a gossip-based cluster, which uses Weave Mesh behind the scenes. This makes the process of creating a Kubernetes cluster using kops DNS-free, and therefore much simpler. This also means a top-level domain or a subdomain is no longer required to create the cluster. To create a cluster using the gossip protocol, indicate this to by using a cluster name with a suffix of .k8s.local. In the following steps, we will use example.cluster.k8s.local as a sample gossip cluster name. You may choose a different name as long as it ends with .k8s.local.
Information on setting up a DNS-based cluster can be found at the bottom of this page in the Appendix. However, setting up a gossip-based cluster allows you to get started quickly.
We show two examples of creating gossip-based clusters below. You can choose whether to create a single-master or multi-master cluster. Workshop exercises will work on both types of cluster.
By default, create cluster command creates a single master node and two worker nodes in the specified zones.
Create a Kubernetes cluster using the following command. This will create a cluster with a single master, multi-node and multi-az configuration:
$ kops create cluster \ --name example.cluster.k8s.local \ --zones $AWS_AVAILABILITY_ZONES \ --yes
You can find the command for creating the AWS_AVAILABILITY_ZONES environment variable at ../prereqs.adoc#aws-availability-zones.
The create cluster command only creates and stores the cluster config in the S3 bucket. Adding the --yes flag ensures that the cluster is immediately created as well.
Alternatively, you may not specify the --yes flag as part of the kops create cluster command. Then you can use kops edit cluster example.cluster.k8s.local command to view the current cluster state and make changes. The cluster creation, in that case, is started with the following command:
$ kops update cluster example.cluster.k8s.local --yes
Once the kops create cluster command is issued, it provisions the EC2 instances, sets up Auto Scaling Groups, IAM users, Security Groups, installs Kubernetes on each node, then configures the master and worker nodes. This process can take some time based upon the number of master and worker nodes.
Wait for 5-8 minutes and then the cluster can be validated as shown:
$ kops validate cluster
Using cluster from kubectl context: example.cluster.k8s.local
Validating cluster example.cluster.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-central-1a Master m3.medium 1 1 eu-central-1a
nodes Node t2.medium 2 2 eu-central-1a,eu-central-1b
NODE STATUS
NAME ROLE READY
ip-172-20-57-94.ec2.internal master True
ip-172-20-63-55.ec2.internal node True
ip-172-20-75-78.ec2.internal node True
Your cluster example.cluster.k8s.local is readyCreate a cluster with multi-master, multi-node and multi-az configuration. We can create and build the cluster in one step by passing the --yes flag.
$ kops create cluster \ --name example.cluster.k8s.local \ --master-count 3 \ --node-count 5 \ --zones $AWS_AVAILABILITY_ZONES \ --yes
A multi-master cluster can be created by using the --master-count option and specifying the number of master nodes. An odd value is recommended. By default, the master nodes are spread across the AZs specified using the --zones option. Alternatively, --master-zones option can be used to explicitly specify the zones for the master nodes.
The --zones option is also used to distribute the worker nodes. The number of workers is specified using the --node-count option.
As mentioned above, wait for 5-8 minutes for the cluster to be created. Validate the cluster:
$ kops validate cluster
Using cluster from kubectl context: example.cluster.k8s.local
Validating cluster example.cluster.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-central-1a Master m3.medium 1 1 eu-central-1a
master-eu-central-1b Master m3.medium 1 1 eu-central-1b
master-eu-central-1c Master c4.large 1 1 eu-central-1c
nodes Node t2.medium 5 5 eu-central-1a,eu-central-1b,eu-central-1c
NODE STATUS
NAME ROLE READY
ip-172-20-101-97.ec2.internal node True
ip-172-20-119-53.ec2.internal node True
ip-172-20-124-138.ec2.internal master True
ip-172-20-35-15.ec2.internal master True
ip-172-20-63-104.ec2.internal node True
ip-172-20-69-241.ec2.internal node True
ip-172-20-84-65.ec2.internal node True
ip-172-20-93-167.ec2.internal master True
Your cluster example.cluster.k8s.local is readyNote that all masters are spread across different AZs.
Your output may differ slightly from the one shown here based up on the type of cluster you created.
kops can create a private Kubernetes cluster, where the master and worker nodes are launched in private subnets in a VPC. This is possible with both Gossip and DNS-based clusters. This reduces the attack surface on your instances by protecting them behind security groups inside private subnets. The services hosted in the cluster can still be exposed via internet-facing ELBs if required. It’s necessary to run a CNI network provider in the Kubernetes cluster when using a private topology. We have used Calico below, though other options such as kopeio-vxlan, weave, romano and others are available.
To print full list of CNI providers:
kops create cluster --help
Create a gossip-based private cluster with master and worker nodes in private subnets:
$ kops create cluster \ --networking calico \ --topology private \ --name example.cluster.k8s.local \ --zones $AWS_AVAILABILITY_ZONES \ --yes
Once the kops create cluster command is issued, it provisions the EC2 instances, sets up AutoScaling Groups, IAM users, Security Groups, installs Kubernetes on each node, then configures the master and worker nodes. This process can take some time based upon the number of master and worker nodes.
Wait for 5-8 minutes and then the cluster can be validated as shown:
$ kops validate cluster
Using cluster from kubectl context: example.cluster.k8s.local
Validating cluster example.cluster.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-central-1a Master m3.medium 1 1 eu-central-1a
nodes Node t2.medium 2 2 eu-central-1a,eu-central-1b,eu-central-1c
NODE STATUS
NAME ROLE READY
ip-172-20-124-144.eu-central-1.compute.internal node True
ip-172-20-58-179.eu-central-1.compute.internal master True
ip-172-20-93-220.eu-central-1.compute.internal node True
Your cluster example.cluster.k8s.local is readyIt is also possible to create a DNS-based cluster where the master and worker nodes are in private subnets. For more information about creating DNS-based clusterssee Appendix: Create a DNS-based Kubernetes cluster below.
If --dns private is also specified, a Route53 private hosted zone is created for routing the traffic for the domain within one or more VPCs. The Kubernetes API can therefore only be accessed from within the VPC. This is a current issue with kops (see kubernetes/kops#2032). A possible workaround is to mirror the private Route53 hosted zone with a public hosted zone that exposes only the API server ELB endpoint. This workaround is discussed here.
Although most of the exercises in this workshop should work on a cluster with a private VPC, some commands won’t, specifically those that use a proxy to access internally hosted services.
You may create multiple Kubernetes clusters. The configuration for each cluster is stored in a configuration file, referred to as “kubeconfig file”. By default, kubectl looks for a file named config in the directory ~/.kube. The kubectl CLI uses kubeconfig file to find the information it needs to choose a cluster and communicate with the API server of a cluster.
This allows you to deploy your applications to different environments by just changing the context. For example, here is a typical flow for application development:
-
Build your application using minikube (See Set up Local Development Environment for more information)
-
Change the context to a test cluster created on AWS
-
Use the same command to deploy to test environment
-
Once satisfied, change the context again to a production cluster on AWS
-
Once again, use the same command to deploy to production environment
Get a summary of available contexts:
$ kubectl config get-contexts
kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
* example.cluster.k8s.local example.cluster.k8s.local example.cluster.k8s.local
minikube minikube minikube
The output shows dfferent contexts, one per cluster, that are available to kubectl. NAME column shows the context name. * indicates the current context.
View the current context:
$ kubectl config current-context example.cluster.k8s.local
If multiple clusters exist, then you can change the context:
$ kubectl config use-context <config-name>
Note: This section is for Kubebernetes 1.7.x, in 1.8.x the api is batch/v1beta1.
Kubernetes resources are created with a specific API version. The exact value is defined by the apiVersion attribute in the resource configuration file. Some of the values are v1, extensions/v1beta1 or batch/v1. By default, resources with apiVersion values X, Y, Z are enabled. If a resource has a version with the word alpha in it, then that version needs to be explicitly enabled in the cluster. For example, if you are running a Kubernetes cluster of version 1.7.x, then Cron Job resource cannot be created unless batch/v2alpha1 is explicitly enabled.
This section shows how to turn on an API version for your cluster. It will use batch/v2alpha1 as an example.
Specific API versions can be turned on or off by passing --runtime-config=api/<version> flag while bringing up the API server. To turn on our specific version, we’ll need to pass --runtime-config=batch/v2alpha1=true.
For a cluster created using kops, this can be done by editing the cluster configuration using the command shown:
$ kops edit cluster --name example.cluster.k8s.local
This will open up the cluster configuration in a text editor. Update the spec attribute such that it looks like as shown:
spec:
kubeAPIServer:
runtimeConfig:
batch/v2alpha1: "true"
api:
Save the changes and exit the editor. Kubernetes cluster needs to re-read the configuration. This can be done by forcing a rolling update of the cluster using the following command:
|
Note
|
This process can easily take 30-45 minutes. Its recommended to leave the cluster without any updates during that time. |
$ kops rolling-update cluster --yes Using cluster from kubectl context: example.cluster.k8s.local
NAME STATUS NEEDUPDATE READY MIN MAX NODES master-eu-central-1a Ready 0 1 1 1 1 nodes Ready 0 2 2 2 2 I1025 20:50:51.158013 354 instancegroups.go:350] Stopping instance "i-0ba714556f0f892cc", node "ip-172-20-58-179.eu-central-1.compute.internal", in AWS ASG "master-eu-central-1a.masters.example.cluster.k8s.local". I1025 20:55:51.413506 354 instancegroups.go:350] Stopping instance "i-0265a07c3320b266b", node "ip-172-20-93-220.eu-central-1.compute.internal", in AWS ASG "nodes.example.cluster.k8s.local". I1025 20:57:52.448582 354 instancegroups.go:350] Stopping instance "i-09e2efd9f5e9ebfce", node "ip-172-20-124-144.eu-central-1.compute.internal", in AWS ASG "nodes.example.cluster.k8s.local". I1025 20:59:53.325980 354 rollingupdate.go:174] Rolling update completed!
This command will first stop one master node in the cluster, re-read the configuration information and start that master. Then it will do the same for rest of the master nodes. And then it will repeat that for each worker node in the cluster. After all the server and worker nodes have been restarted, the rolling update of the cluster is complete.
Let’s verify that the attributes are now successfully passed to the API server. Get the list of pods for the API server using the command shown:
$ kubectl get pods --all-namespaces | grep kube-apiserver kube-system kube-apiserver-ip-172-20-117-32.ec2.internal 1/1 Running 0 7m kube-system kube-apiserver-ip-172-20-62-108.ec2.internal 1/1 Running 6 16m kube-system kube-apiserver-ip-172-20-79-64.ec2.internal 1/1 Running 2 12m
The output shows three pods, one each for API server, corresponding to the three master nodes. This output is from a cluster with three master nodes. The output may be different if your cluster was created with different number of masters.
Search for the --runtime-config option as shown:
$ kubectl describe --namespace=kube-system pod <pod-name> | grep runtime
<pod-name> is name of one of the pods shown above.
A formatted output is shown below:
/usr/local/bin/kube-apiserver \ --address=127.0.0.1 \ --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota \ --allow-privileged=true \ --anonymous-auth=false \ --apiserver-count=3 \ --authorization-mode=AlwaysAllow \ --basic-auth-file=/srv/kubernetes/basic_auth.csv \ --client-ca-file=/srv/kubernetes/ca.crt \ --cloud-provider=aws \ --etcd-servers-overrides=/events#http://127.0.0.1:4002 \ --etcd-servers=http://127.0.0.1:4001 --insecure-port=8080 --kubelet-preferred-address-types=InternalIP,Hostname,ExternalIP \ --runtime-config=batch/v2alpha1=true \ --secure-port=443 \ --service-cluster-ip-range=100.64.0.0/13 \ --storage-backend=etcd2 \ --tls-cert-file=/srv/kubernetes/server.cert \ --tls-private-key-file=/srv/kubernetes/server.key \ --token-auth-file=/srv/kubernetes/known_tokens.csv \ --v=2 \ 1>>/var/log/kube-apiserver.log 2>&1
The output clearly shows that --runtime-config=batch/v2alpha1=true is passed as an option to the API server. This means the cluster is now ready for creating creating APIs with version batch/v2alpha1.
An instance group, or ig for short, is a kops concept that defines a grouping of similar nodes. In AWS, an instance group maps to an Auto Scaling Group (ASG). Instructions on how to create instance groups can be found here.
Any cluster can be deleted as shown:
$ kops delete cluster \ <cluster-name> \ --yes
<cluster-name> is the name of the cluster. For example, our example.cluster.k8s.local cluster can be deleted as:
$ kops delete cluster \ example.cluster.k8s.local \ --yes
If you leave off the --yes flag, you will get a listing of all the resources kops will delete. To confirm deletion, run the command again appending --yes.
If you created a private VPC, then an additional cleanup of resources is required as shown below:
# Find Route53 hosted zone ID from the console or via CLI and delete hosted zone aws route53 delete-hosted-zone --id $ZONEID # Delete VPC if you created earlier $ aws ec2 detach-internet-gateway --internet $IGW --vpc $VPCID aws ec2 delete-internet-gateway --internet-gateway-id $IGW aws ec2 delete-vpc --vpc-id $VPCID
To remove the state store S3 bucket:
aws s3 rb $KOPS_STATE_STORE
To create a DNS-based Kubernetes cluster you’ll need a top-level domain or subdomain that meets one of the following scenarios:
-
Domain purchased/hosted via AWS
-
A subdomain under a domain purchased/hosted via AWS
-
Setting up Route53 for a domain purchased with another registrar, transfering the domain to Route53
-
Subdomain for clusters in Route53, leaving the domain at another registrar
Then you need to follow the instructions in configure DNS. Typically, the first and the last bullets are common scenarios.
By default, create cluster command creates a single master node and two worker nodes in the specified zones.
Create a Kubernetes cluster using the following command. For the purposes of this demonstration, we will use a cluster name of example.cluster.com as our registered DNS. This will create a cluster with a single master, multi-node and multi-az configuration:
$ kops create cluster \ --name example.cluster.com \ --zones $AWS_AVAILABILITY_ZONES \ --yes
The create cluster command only creates and stores the cluster config in the S3 bucket. Adding --yes option ensures that the cluster is immediately created as well.
Alternatively, you may leave off the --yes option from the kops create cluster command. This will allow you to use kops edit cluster example.cluster.com command to view the current cluster state and make changes before actually creating the cluster.
The cluster creation, in that case, is started with the following command:
$ kops update cluster example.cluster.com --yes
Once the kops create cluster or kops update cluster command is issued with the --yes flag,, it provisions the EC2 instances, setup Auto Scaling Groups, IAM users, security groups, and install Kubernetes on each node, configures master and worker nodes. This process can take a few minutes based upon the number of master and worker nodes.
Wait for 5-8 minutes and then the cluster can be validated as shown:
$ kops validate cluster --name=example.cluster.com
Validating cluster example.cluster.com
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-central-1a Master m3.medium 1 1 eu-central-1a
nodes Node t2.medium 2 2 eu-central-1a,eu-central-1b
NODE STATUS
NAME ROLE READY
ip-172-20-51-232.ec2.internal node True
ip-172-20-60-192.ec2.internal master True
ip-172-20-91-39.ec2.internal node True
Your cluster example.cluster.com is readyVerify the client and server version:
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-12T00:45:05Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:30:51Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
It shows that Kubectl CLI version is 1.8.1 and the server version is 1.7.4. Cluster version may changed depending on kops version.
Check the list of Availability Zones that exist for your region using the following command:
$ aws --region <region> ec2 describe-availability-zones
Create a cluster with multi-master, multi-node and multi-az configuration. We can create and build the cluster in
one step by passing the --yes flag.
$ kops create cluster \ --name example.cluster.com \ --master-count 3 \ --node-count 5 \ --zones $AWS_AVAILABILITY_ZONES \ --yes
A multi-master cluster can be created by using the --master-count option and specifying the number of master nodes. An odd value is recommended. By default, the master nodes are spread across the AZs specified using the --zones option. Alternatively, --master-zones option can be used to explicitly specify the zones for the master nodes.
--zones option is also used to distribute the worker nodes. The number of workers is specified using the --node-count option.
As mentioned above, wait for 5-8 minutes for the cluster to be created. Validate the cluster:
$ kops validate cluster --name=example.cluster.com
Validating cluster example.cluster.com
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-central-1a Master m3.medium 1 1 eu-central-1a
master-eu-central-1b Master m3.medium 1 1 eu-central-1b
master-eu-central-1c Master c4.large 1 1 eu-central-1c
nodes Node t2.medium 5 5 eu-central-1a,eu-central-1b,eu-central-1c
NODE STATUS
NAME ROLE READY
ip-172-20-103-30.ec2.internal master True
ip-172-20-105-16.ec2.internal node True
ip-172-20-127-147.ec2.internal node True
ip-172-20-35-38.ec2.internal node True
ip-172-20-47-199.ec2.internal node True
ip-172-20-61-207.ec2.internal master True
ip-172-20-75-78.ec2.internal master True
ip-172-20-94-216.ec2.internal node True
Your cluster example.cluster.com is readyNote that all masters are spread across different AZs.
Your output may differ from the one shown here based up on the type of cluster you created.