This is my setup for a Kubernetes homelab that's running on some left-over computers I've collected.
The name KubeNest is just a wordplay on Google Nest and Kubernetes for being a homelab.
This project started out when I wanted to do my coding projects on my small comfortable laptop and let my other more powerful bulkier machine do the all the work.
Somehow it got out of hand and I ended up setting up a Kubernetes cluster. So now I have a homelab of a local Kubernetes cluster with Kubeflow, Docker Image Registry, NFS-server and more.
Have in mind that I had no prior knowledge about any of these systems beforehand so there most likely some things that might not completely add up.
The initial purpose for this project was to learn more about backend, servers, containers, container orcherstration and MLOps. However, one of the best ways to learn is to teach, so I figured I should document each step to have something to reflect over while setting it up. I figured it might as well share this for anyone who looking for help or inspiration.
This project is focused on running in a local network all thought some parts are exposed publicly. The "public parts" were mainly done to enable work on this project while I was not home from time to time.
The cluster is composed by the following componentes
- Kubelet 1.26.9
- Kubectl 1.26.9
- Kubeadm 1.26.9
- cri-o 1.26
- crun
- Flannel 0.21.5
- KubeFlow 1.8
- Docker image registry 2.8.3
- Prometheus
- Grafana
- NFS
OS: Ubuntu server 22.04.2
CPU: 64-bit Intel i3-2310M CPU @ 2.10GHz, 4 cores
GPU: -
RAM: 4 GB
DISK: 700 GB, HDD
TYPE: Laptop
OS: Ubuntu server 22.04.2
CPU: 64-bit Intel i7-6700HQ CPU @ 2.60GHz, 8 cores
GPU: NVIDIA GeForce GTX 960M, 2048 MB GDDR5, 640 CUDA cores (5.0)
RAM: 8 GB
DISK: 240 GB, SSD
TYPE: Laptop
OS: Ubuntu server 22.04.2
CPU: 64/bit Intel i5-4690K @ 3.50GHz, 4 cores
GPU: NVIDIA GTX 1070, 8 GB GDDR5, 1920 CUDA cores
RAM: 16 GB, DIMM DDR3 1600MHz
DISK: 250 GB, SSD
TYPE: Desktop
This project contains submodules (Kubeflow). To clone with submodules
git clone --recurse-submodules [email protected]:nklsla/KubeNest.git
- Helm
- Enable SSH on all machines
- Ubuntu 22.04
- Setup
scripts/parameters.sh - kustomize 5.0.3 (kubeflow)
- Setup SSH
- Setup Kubernetes
- Setup Firewall
- Setup NFS
- Setup Docker Image Registry
- Setup Kubeflow
- Setup GPU-operator
- Setup Extras
- Run
kub-init.shto initialize the cluster to its bare minimum, connecting workers. - Run
kub-start.shto start up all services:- sets parameters
- NFS services
- Metrics-server
- Docker Image Registry
- GPU-operator
- Kubeflow
- Prometheus
- Grafana
Issue for kubernetes 1.25.10-00:
When resetting the cluster via kubeadm reset the cni might fail giving following error message on kubectl describe pods coredns-###:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned kube-system/coredns-565d847f94-klvn6 to eva
Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-565d847f94-klvn6_kube-system_86d1ad84-5581-4164-bb9d-89b3e9a71f25_0(c6ad012ffda7373a48930d806ea2068629504240f72fb35b6eed87197fff194f): error adding pod kube-system_coredns-565d847f94-klvn6 to CNI network "cbr0": plugin type="flannel" failed (add): failed to delegate add: failed to set bridge addr: "cni0" already has an IP address different from 10.244.0.1/24
Try this
# If not installed
sudo apt install bridge-utils
sudo ip link set cni0 down
sudo brctl delbr cni0