CERN Kubernetes cluster

Cluster creation via Openstack tools

Based on CERN documentation https://clouddocs.web.cern.ch/containers/quickstart.html

Log in to lxplus-cloud

> ssh [email protected]

Select the Openstack project you want to work on

export OS_PROJECT_NAME="ATLAS Harvester Kubernetes"

List the available flavors and templates

openstack flavor list
+-------+-----------+-------+------+-----------+-------+-----------+
| ID    | Name      |   RAM | Disk | Ephemeral | VCPUs | Is Public |
+-------+-----------+-------+------+-----------+-------+-----------+
| 12076 | m2.large  |  7500 |   40 |         0 |     4 | True      |
| 17895 | m2.small  |  1875 |   10 |         0 |     1 | True      |
| 21603 | m2.xlarge | 15000 |   80 |         0 |     8 | False     |
| 38242 | m2.medium |  3750 |   20 |         0 |     2 | True      |
+-------+-----------+-------+------+-----------+-------+-----------+

openstack coe cluster template list
+--------------------------------------+----------------------+
| uuid                                 | name                 |
+--------------------------------------+----------------------+
| 17760a5f-8957-4794-ab96-0d6bd8627282 | swarm-18.06-1        |
| ab08b219-3246-4995-bf76-a3123f69cb4f | swarm-1.13.1-2       |
| 6b4fc2c2-00b0-410d-a784-82b6ebdd85bc | kubernetes-1.13.10-1 |
| 8dffa2cc-8aa4-489b-a346-edc202db7673 | kubernetes-1.14.6-2  |
| f294e172-4688-48f2-8407-78874941af0a | kubernetes-1.15.3-3  |
| 4cd5f782-db78-4665-9876-20d7e08bbd6a | kubernetes-1.17.2-1  |
+--------------------------------------+----------------------+

If the cluster is large, you'll need a xlarge master to manage the cluster. The --master-count N option is not supported at CERN Choose your template and VM flavors

openstack coe cluster create CERN-EXTENSION_KUBERNETES --keypair harvester_k8s --cluster-template kubernetes-1.15.3-3 --node-count 40 --flavor m2.xlarge --master-flavor m2.xlarge --labels tiller_enabled=true --labels monitoring_enabled=true --labels grafana_admin_passwd=XXXXXX --labels influx_grafana_dashboard_enabled="true" --labels kube_csi_enabled="true" --labels kube_csi_version="cern-csi-1.0-2" --labels cloud_provider_tag="v1.15.0" --labels container_infra_prefix="gitlab-registry.cern.ch/cloud/atomic-system-containers/" --labels manila_enabled="true" --labels heat_container_agent_tag="stein-dev-2" --labels cgroup_driver="cgroupfs" --labels cephfs_csi_enabled="true" --labels cvmfs_csi_version="v1.0.0" --labels admission_control_list="NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,Priority" --labels kube_tag="v1.15.3" --labels flannel_backend="vxlan" --labels manila_version="v0.3.0" --labels cvmfs_csi_enabled="true" --labels ingress_controller="traefik" --labels autoscaler_tag="v1.15.2" --labels cephfs_csi_version="cern-csi-1.0-2"

To get all the applicable labels for a version:

openstack coe cluster template show -f json kubernetes-1.17.2-1 |     jq --raw-output '.labels | to_entries |
    map("--labels (.key)=\"(.value)\"") | join(" ")'

Wait until the cluster is ready

openstack coe cluster list
+--------------------------------------+---------------+---------------+------------+--------------+-----------------+---------------+
| uuid                                 | name          | keypair       | node_count | master_count | status          | health_status |
+--------------------------------------+---------------+---------------+------------+--------------+-----------------+---------------+
| ae17771f-9147-4fed-b8a1-8b79e67eef16 | harvester_k8s | harvester_k8s |         70 |            1 | CREATE_COMPLETE | None          |
+--------------------------------------+---------------+---------------+------------+--------------+-----------------+---------------+

Generate the config file with the master IP address and cluster certificate. You will use it in Harvester to connect to the cluster

openstack coe cluster config harvester_k8s
export KUBECONFIG=/afs/cern.ch/user/f/fbarreir/config

In order to connect to the VMs on kubernetes 1.15

ssh -i <private key file> fedora@<node name>

In order to connect to the VMs on kubernetes 1.17

ssh -i <private key file> core@<node name>

The private key file is the one associated to your Openstack keypair used to create the cluster. The node names can be retrieved e.g. through the command

kubectl get nodes

Cluster configuration via kubectl

On the harvester machine install the kubectl tool and copy the cluster config file Select the config file you just copied and just try it connects to your cluster. You should get a message like "No resources found in default namespace." since your cluster should be empty

export KUBECONFIG=/data/atlpan/k8_configs/fbarreir_cern_k8s
kubectl get pods

Create the CVMFS storage classes for the CVMFS csi driver. You can get the yaml declaration from the K8s example folder

[root@aipanda169 ~]# kubectl create -f /data/atlpan/k8_configs/cvmfs.yaml
storageclass.storage.k8s.io/csi-cvmfs-atlas created
storageclass.storage.k8s.io/csi-cvmfs-sft created
storageclass.storage.k8s.io/csi-cvmfs-grid created
storageclass.storage.k8s.io/csi-cvmfs-atlas-condb created
storageclass.storage.k8s.io/csi-cvmfs-atlas-nightlies created
persistentvolumeclaim/csi-cvmfs-atlas-pvc created
persistentvolumeclaim/csi-cvmfs-sft-pvc created
persistentvolumeclaim/csi-cvmfs-grid-pvc created
persistentvolumeclaim/csi-cvmfs-atlas-condb-pvc created
persistentvolumeclaim/csi-cvmfs-atlas-nightlies-pvc created

Now follow the general Harvester-Kubernetes instructions