Explain how integration tests can be run against any k8s cluster. One way of running such a kubernetes cluster locally which is leveraged by our CI setup is microk8s The Kubernetes resources necessary to get the test application onto the cluster are created using the Fabric8 Maven Plugin and the lifecycle of the test applications is controlled by Arquillian Cube
With FMP and Arquillian Cube setup for our project we need to configure the following things in order to properly get our test applications onto our cluster of choice:
- The
KUBECONFIG
environment variable needs to be set to the location of the Kubernetes configuration file we will use to access our cluster (this can be skipped if this file is already present in the standard locations that kubectl assumes) - The
docker.host
system property needs to be set to the URL where the docker daemon we will use to build images is listening This can be skipped when we use the default unix socket on a Linux machine - The Docker image registry were out built images will be stored needs to be set using the
image.registry
environment variable
Here we will describe the steps that are needed to setup microk8s and the maven command we will use to run the integration tests against that microk8s cluster
sudo snap install microk8s --classic --channel=1.11/stable
sleep 10
echo n | microk8s.enable dns registry istio
Ensure everything is running by inspecting the output of:
microk8s.kubectl get all --all-namespaces
It should look something like:
NAMESPACE NAME READY STATUS RESTARTS AGE
container-registry pod/registry-6bc95dfd76-274lc 1/1 Running 0 40s
kube-system pod/hostpath-provisioner-9979c7f64-f96tw 1/1 Running 0 41s
kube-system pod/kube-dns-864b8bdc77-68kmn 2/3 Running 0 41s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
container-registry service/registry NodePort 10.152.183.175 <none> 5000:32000/TCP 50s
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 1m
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP 56s
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
container-registry deployment.apps/registry 1 1 1 1 50s
kube-system deployment.apps/hostpath-provisioner 1 1 1 1 51s
kube-system deployment.apps/kube-dns 1 1 1 0 56s
NAMESPACE NAME DESIRED CURRENT READY AGE
container-registry replicaset.apps/registry-6bc95dfd76 1 1 1 41s
kube-system replicaset.apps/hostpath-provisioner-9979c7f64 1 1 1 41s
kube-system replicaset.apps/kube-dns-864b8bdc77 1 1 0 41s
Export the kube config file that will be used to access this cluster
microk8s.kubectl config view --raw > /tmp/kubeconfig
For the istio tests to work, we need a namespace that is properly configured for Istio injection.
Such a namespace called istio-test
can easily be created using:
microk8s.kubectl create -f .circleci/istio-test-namespace.yml
cd spring-cloud-kubernetes-integration-tests
KUBECONFIG=/tmp/kubeconfig mvn -Ddocker.host='unix:///var/snap/microk8s/current/docker.sock' -Dimage.registry='localhost:32000' clean package fabric8:build verify -Pfmp,it
The command above will for each test project:
- Build the ubjerjar
- Build a docker image based on that uberjar
- Launch the Arquillian Cube tests which will
- Deploy the application to the cluster
- Launch the test code
- Undeploy the application
Docker Desktop for Mac and Docker Desktop for Windows
provide a Kubernetes server runs within a Docker container on your local system. This is similar to microk8s
but is only currently available on Mac or Windows.
- Ensure you are running the Edge release of Docker Desktop
** For Mac
brew cask install docker-edge
** For Windowschoco install docker-desktop --pre
- Increase default resources as the defaults are not sufficient. Allocate 8GB of Ram
- Enable Kubernetes cluster
- Run a local insecure docker registry
**
docker run -d -p 5000:5000 --restart=always --name registry registry:2
- Configure the Docker daemon to trust the local registry
- Export the docker config
kubectl config view --raw > /tmp/kubeconfig
- Run the tests
KUBECONFIG=/tmp/kubeconfig mvn -Ddocker.host='unix:///var/run/docker.sock' -Dimage.registry='127.0.0.1:5000' clean package fabric8:build verify -Pfmp,it,spring
Each of the test modules can also be launched manually. This can be very useful for debugging purposes.
For example to launch the simple-core
application
cd spring-cloud-kubernetes-integration-tests/simple-core
KUBECONFIG=/tmp/kubeconfig mvn -Ddocker.host='unix:///var/snap/microk8s/current/docker.sock' -Dimage.registry='localhost:32000' clean package fabric8:build fabric8:deploy -Pfmp
When it's time to take down the application, simply execute:
KUBECONFIG=/tmp/kubeconfig mvn fabric8:undeploy -Pfmp