This long read is a tutorial for deploying a Kubernetes cluster (using k3s) with 4 x OCI free-tier ARM instances and 4 x Raspberry Pi 4 (or how many do you have). Plus some applications needed for installation (Terraform and Ansible) and a lot of things installed on the cluster (Lens, MetalLB, Helm, Arkade, Longhorn, Portainer, ArgoCD, Prometheus, Grafana). I've made a series of articles on dev.to for this repo (but here there will always be the latest version).
This section is for the Oracle Cloud Infrastructure (OCI) part of the cluster.
- Obvious, an OCI account, get it from here - oracle.com/cloud. If you already have an account, be careful not to have any resources provisioned already (even for other users, or compartments), this tutorial will use all free-tier ones. Also be extra careful to pick a region not so popular, as it may have no resources available. Pick a region with enough ARM instances available. If during final steps, terraform is stuck, you can check in OCI > Compute > Instance Pools > select your own > Work requests , if there is _Failure _and in that log file there's an error Out of host capacity, then you must wait, even days until resources are freed. You can run a script from here which will try to create instances until there's something available. When that happens, go fast to your OCI, delete all that was created and then run the terraform scripts;
- I used Windows 11 with WSL2 running Ubuntu 20.04, but this will work on any Linux machine;
- Terraform installed (tested with v1.4.6 - and OCI provider v4.120.0)- how to here;
For safety we should use a separate compartment and user for our OCI configuration, not root ones. Now is a good time to create a new notes file and add some values there, you will need them later. Mostly you will add 3 things for each value (user + user_name_you_created + it's_OCID; group ... etc.). Go to Identity & Security > Compartments and click on Create Compartment. Open it and copy the OCID to your notes file. Then in Identity & Security > Users click on Create User. Open it and copy the OCID to your notes file. Then in Identity & Security > Groups click on Create Group. The same as above with the OCID. Here click on Add User to Group and add the newly created user. In Identity & Security > Policies click on Create Policy, Show manual editor and add the following
allow group <<group_you_created>> to read all-resources in compartment <<compartment_you_created>
allow group <<group_you_created>> to manage virtual-network-family in compartment <compartment_you_created>
allow group <<group_you_created>> to manage instance-family in compartment <compartment_you_created>
allow group <<group_you_created>> to manage compute-management-family in compartment <compartment_you_created>
allow group <<group_you_created>> to manage volume-family in compartment <compartment_you_created>
allow group <<group_you_created>> to manage load-balancers in compartment <compartment_you_created>
allow group <<group_you_created>> to manage network-load-balancers in compartment <compartment_you_created>
allow group <<group_you_created>> to manage dynamic-groups in compartment <compartment_you_created>
allow group <<group_you_created>> to manage policies in compartment <compartment_you_created>
allow group <<group_you_created>> to manage dynamic-groups in tenancy
Then you need access to OCI from your machine. So, create a new folder in HOME directory
mkdir ~/.oci
Generate a private key there
openssl genrsa -out ~/.oci/key.pem 2048
Change permissions for it
chmod 600 ~/.oci/key.pem
Then generate you're public key
openssl rsa -pubout -in ~/.oci/key.pem -out $HOME/.oci/key_public.pem
And then copy that public key, everything inside that file
cat ~/.oci/key_public.pem
This key has to be added to your OCI new user. Go to OCI > Identity & Security > Users select the new user and open API Keys , click on Add API Key, select Paste Public Key, and there paste all your copied key.
After you've done that you need to copy to notes the fingerprint too. Save the path to the private key too.
*note: Use ~ and not $HOME, that's the only way it worked for me.
To your notes file copy the following too:
- Tenancy OCID. Click on your avatar (from top-right), and select Tenancy.
- Region. In the top right there is the name of your region too. Now find it here and copy it's identifier (ex. eu-paris-1).
- The path to the private key. In our case -
~/.oci/key.pem
Now create a new folder to test if terraform is ok and linked with OCI. In that folder create a file main.tf and add this:
terraform {
required_providers {
oci = {
source = "oracle/oci"
version = "4.120.0"
}
}
}
# Configure the OCI provider with an API Key
provider "oci" {
tenancy_ocid = "ocid1.tenancy.oc1..aaaaaaYOURTENANCY3uzx4a"
user_ocid = "ocid1.user.oc1..aaaaaaYOURUSER4s5ga"
private_key_path = "~/.oci/oci.pem"
fingerprint = "2a:d8:YOURFINGERPRINT:a1:cd:06"
region = "eu-YOURREGION"
}
#Get a list of Availability Domains
data "oci_identity_availability_domains" "ads" {
compartment_id = "ocid1.tenancy.oc1..aaaaaaYOURTENANCYuzx4a"
}
#Output the result
output "all-availability-domains-in-your-compartment" {
value = data.oci_identity_availability_domains.ads.availability_domains
}
Now you should run terraform init to download the OCI provider and then terraform plan to see what will happen and terraform apply to receive final results. This small demo configuration file from above should return the name of the availability domains in that regions. If you receive something like "name" = "pmkj:EU-YOURREGION-1-AD-1" and no errors then everything is ok until now. This file can be deleted now.
You will need to add some new values to the notes file:
- From inside OCI, on top right corner, click on Developer Tools > Cloud Shell and there write
oci iam availability-domain list. Save the name, not id (if more than 1, pick one). This is your availability_domain variable; - Again from this console type
oci compute image list --compartment-id <YOUR-COMPARTMENT> --operating-system "Canonical Ubuntu" --operating-system-version "22.04 Minimal aarch64" --shape "VM.Standard.A1.Flex", to find the OS Image ID. Probably the first result has the latest build, in my case was Canonical-Ubuntu-22.04-Minimal-aarch64-2022.11.05-0. From here save the id. This is your os_image_id variable; - Now just google "my ip" and you will find your Public IP. Save it in CIDR format, ex. 111.222.111.99/32. This is your my_public_ip_cidr variable. I use a cheap VPS just to have a static IP. If you don't have a static IPv4 from your ISP, i don't know a quick solution for you, maybe someone can comment one. You can setup DDNS, but that can't be used in Security List afaik. Only solution every time your IP changes, go to VCN > Security List and modify the Ingress rule with the new IP (or set te Ingress to 0.0.0.0/0 and al trafiic will be permited, but bye bye security);
- Your public_key_path is your public SSH keys. If you don't have any, quickly generate them with
ssh-keygen. You should have one now in ~/.ssh/key.pub (I copied the private key, usingscpto the VPS, so I can connect to OCI from local machine and from VPS); - Last is your email address that will be used to install a certification manager. That will be your certmanager_email_address variable. I didn't setup one, as this is just a personal project for testing.
After you've cloned the repo, go to oci/terraform.tfvars and edit all values with the ones from your notes file.
This build uses the great terraform configuration files from this repo of garutilorenzo (if you have errors running all of this, you should check what changed in this repo since 12.05.23). You can read here if you want to customize your configuration and edit the main.tf file. Compared to garutilorenzo's repo, this tutorial has 1 server node + 3 worker nodes, default ingress controller set as Traefik, and it's not build by default with Longhorn and ArgoCD (they will be installed later alongside other apps).
*note - I've got some problems with clock of WSL2 not being synced to Windows clock. And provisioning didn't worked so if you receive clock errors too, verify your time with datecommand, if out of sync just run sudo hwclock -s or sudo ntpdate time.windows.com.
Now just run terraform plan and then terraform apply. If everything was ok you should have your resources created.
When the script finishes save the outputs (or you can find the values in OCI):
Outputs:
k3s_servers_ips = [
"152.x.x.115",
]
k3s_workers_ips = [
"140.x.x.158",
"140.x.x.226",
]
public_lb_ip = tolist([
{
"ip_address" = "140.x.x.159"
"ip_version" = "IPV4"
"is_public" = true
"reserved_ip" = tolist([])
},
{
"ip_address" = "10.0.1.96"
"ip_version" = "IPV4"
"is_public" = false
"reserved_ip" = tolist([])
Now you can connect to any worker or server IP using ssh -i ~/.ssh/key [email protected]. Connect to server IP and write sudo kubectl get nodes to check all nodes.
My cluster broke a lot of times. It's a testing one. Just delete all the resources in your compartment (go to Governance & Administration > Tenancy Management > Tenancy Explorer), one by one (or you could run terraform destroy, but it doesn't always work). The VCN in particular will be hard to delete, good luck. And then just restart terraform apply (if you are lucky and there are available resources).
This section is for the Raspberry PI 4 (RPI4) part of the cluster.
- At least 2 Raspberry Pi. I've got 4 of them, 3 with 4GB and 1 with 8GB. Every one needs a SD Card, a power adapter plus network cables (plus an optional switch and 4 cases);
- And the same from part 1. Windows 11 with WSL2 running Ubuntu 20.04, but this will work on any Linux & Win machine)
Installing an OS on Pi is very easy. Just insert the SD card in your PC and use Imager from official website. I choose the same OS as in the OCI cluster, that is Ubuntu Server 22.04.1 64bit. In advanced settings (bottom right of Imager) pick Set Hostname and write your own. I have rpi4-1 rpi4-2 rpi4-3 and rpi4-4. Pick Enable SSH and Set username and password, this way you can connect to Pi immediately, without a monitor and keyboard for it. Then hit Write and repeat this step for every one.
From my home router I found the devices IPs. Still there you can configure Address reservation so every time they keep their IP. Did some Port forwarding too so I can access them from everywhere using DDNS from my ISP, as I don't have static IPv4. All of these settings are configured from you home router so Google how to if you're interested.
I added every PI to local machine C:\Windows\System32\drivers\etc\hosts file to be able to control them easier.
192.168.0.201 rpi4-1
192.168.0.202 rpi4-2
192.168.0.203 rpi4-3
192.168.0.204 rpi4-4
We can check if everything is ok by running some ping rpi4-1 or ssh user@rpi4-1.
For configuration management I picked Ansible as it is agentless and not so difficult (spoiler alert, it is though). We can control all RPI4 from local machine. Install Ansible first sudo apt install ansible.
Now from your PC run the following commands (assuming you already generated ssh keys from Part 1, using ssh-keygen):
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-1
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-2
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-3
ssh-copy-id -i ~/.ssh/key.pub user@rpi4-4
This will allow Ansible to connect to every Pi without requesting the password every time.
I had to uncomment with sudo vi /etc/ansible/ansible.cfg the line with private_key_file = ~/.ssh/key. And in /etc/ansible/hosts i added the following:
[big]
rpi4-1 ansible_connection=ssh
[small]
rpi4-2 ansible_connection=ssh
rpi4-3 ansible_connection=ssh
rpi4-4 ansible_connection=ssh
[home:children]
big
small
This will add the big rpi4 in a big group and the rest of workers to an small group. Plus a home group having them all. My PC user and RPIs user is the same, but if you have a different one you have to add this to the same file:
[all:vars]
remote_user = user
Now test if everything is ok with ansible home -m ping. Green is ok.
I like to keep all my systems updated to latest version, especially this one used for testing. So we'll need to create a new file update.yml and paste below block in it:
---
- hosts: home
tasks:
- name: Update apt repo and cache
apt: update_cache=yes force_apt_get=yes cache_valid_time=3600
- name: Upgrade all packages
apt: upgrade=yes force_apt_get=yes
- name: Check if a reboot is needed
register: reboot_required_file
stat: path=/var/run/reboot-required get_md5=no
- name: Reboot the box if kernel updated
reboot:
msg: "Reboot initiated by Ansible for kernel updates"
connect_timeout: 5
reboot_timeout: 90
pre_reboot_delay: 0
post_reboot_delay: 30
test_command: uptime
when: reboot_required_file.stat.exists
This is an Ansible playbook that updates Ubuntu and then reboots the PIs. Now save it and run ansible-playbook update.yml -K -b which will ask for sudo password and run the playbook. It lasted ~10 minutes for me. On another shell you can ssh to any of the PIs and run htop to see the activity.
Now try ansible home -a "rpi-eeprom-update -a" -b -K to see if there's any firmware update for you Raspberry Pi 4.
Next step is to enable cgroups on every PI. Create a new playbook append-cmd.yml and add:
---
- hosts: home
tasks:
- name: Append cgroup to cmdline.txt
lineinfile:
path: /boot/firmware/cmdline.txt
backrefs: yes
regexp: "^(.*)$"
line: '\1 cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1'
...
This will append to end of file /boot/firmware/cmdline.txt the strings cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1. Run it with ansible-playbook append-cmd.yml -K -b.
I won't add a graphical interface and won't use Wi-Fi and Bluetooth so we can steal some memory from GPU memory, Wi-Fi and BT to be available to the Kubernetes cluster. So we need to add a few lines to /boot/firmware/config.txt using a new Ansible playbook append-cfg.yml with following content:
---
- hosts: home
tasks:
- name: Append new lines to config.txt
blockinfile:
path: /boot/firmware/config.txt
block: |
gpu_mem=16
dtoverlay=disable-bt
dtoverlay=disable-wifi
...
Run it again with ansible-playbook append-cfg.yml -K -b.
Next 2 mods must be enabled with command ansible home -a "modprobe overlay" -a "modprobe br_netfilter" -K -b.
Next playbook will create 2 files and add some lines to it. Let's call it iptable.yml and add:
---
- hosts: home
tasks:
- name: Create a file and write in it 1.
blockinfile:
path: /etc/modules-load.d/k8s.conf
block: |
overlay
br_netfilter
create: yes
- name: Create a file and write in it 2.
blockinfile:
path: /etc/sysctl.d/k8s.conf
block: |
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
create: yes
...
Run it with ansible-playbook iptable.yml -K -b. After that run ansible home -a "sysctl --system" -K -b. My PIs started to be laggy so i rebooted here with ansible home -a "reboot" -K -b.
Now the last step. Installing some apps on PIs. I'm still unsure if this is needed, but I'm pretty sure it won't harm the cluster. Create a new file install.yml and add copy this:
---
- hosts: home
tasks:
- name: Install some packages
apt:
name:
- curl
- gnupg2
- software-properties-common
- apt-transport-https
- ca-certificates
- linux-modules-extra-raspi
...
Now run it with ansible-playbook install.yml -K -b.
Now the fun part is linking the Raspberry Pi cluster to existing k3s cluster on OCI.
At this moment I added the OCI machines to the C:\Windows\System32\drivers\etc\hosts file (WSL reads this file and updates it in it's /etc/hosts file). Now my hosts looks like this:
...
192.168.0.201 rpi4-1
192.168.0.202 rpi4-2
192.168.0.203 rpi4-3
192.168.0.204 rpi4-4
140.111.111.213 oci1
140.112.112.35 oci2
152.113.113.23 oci3
140.114.114.22 oci4
And I added them to the Ansible file too (/etc/ansible/hosts). Now this file looks like this:
[big]
rpi4-1 ansible_connection=ssh
[small]
rpi4-2 ansible_connection=ssh
rpi4-3 ansible_connection=ssh
rpi4-4 ansible_connection=ssh
[home:children]
big
small
[ocis]
oci1 ansible_connection=ssh ansible_user=ubuntu
[ociw]
oci2 ansible_connection=ssh ansible_user=ubuntu
oci3 ansible_connection=ssh ansible_user=ubuntu
oci4 ansible_connection=ssh ansible_user=ubuntu
[oci:children]
ocis
ociw
[workers:children]
big
small
ociw
It is not the best naming convention, but it works. Ansible reservers the naming all so if I want to interact with all the objects I can always use ansible -m command all. Test it using ansible -a "uname -a" all. You should receive 8 responses with every Linux installed. Now you can even re-run the update ansible playbook created last part, to update OCI instances too.
K3s can work in multiple ways (here), but for our tutorial we picked High Availability with Embedded DB architecture. This one runs etcd instead of the default sqlite3 and so it's important to have an odd number of server nodes (from official documentation: "An etcd cluster needs a majority of nodes, a quorum, to agree on updates to the cluster state. For a cluster with n members, quorum is (n/2)+1.").
Initially this cluster was planned with 3 server nodes, 2 from OCI and 1 from RPi4. But after reading issues 1 and 2 on Github, there are problems with etcd being on server nodes on different networks. So this cluster will have 1 server node (this is how k3s names their master nodes): from OCI and 7 agent nodes (this is how k3s names their worker nodes): 3 from OCI and 4 from RPi4.
First we need to free some ports, so the OCI cluster can communicate with the RPi cluster. Go to VCN > Security List. You need to click on Add Ingress Rule. While I could only open the needed ports for k3s networking (listed here), I decided to open all OCI ports toward my public IP only, as there is no risk involved here. So in IP Protocol select All Protocols. Now you can test if everything is working by ssh to any RPi4 and try to ping any OCI machine or ssh to it or try another port. (this port opening part is now optional, after adding the next step).
Now to link all of them together.
We will create a VPN between all of them (and if you want to, plus local machine, plus VPS) using Wireguard. While Wireguard is not the hardest app to install and configure, there's an wonderful app that does almost everything by itself - Netmaker.
On your VPS, or your local machine (if it has a static IP) run sudo wget -qO /root/nm-quick-interactive.sh https://raw.githubusercontent.com/gravitl/netmaker/master/scripts/nm-quick-interactive.sh && sudo chmod +x /root/nm-quick-interactive.sh && sudo /root/nm-quick-interactive.sh and follow all the steps. Select Community Edition (for max 50 nodes) and for the rest pick auto.
Now you will have a dashboard at a auto-generated domain. Open that link that you received at the end of the installation in a browser and create a user and password.
It should have created for you a network. Open Network tab and then open the new network created. If you're ok with it, that's great. I changed the CIDR to something more fancier 10.20.30.0/24 and activated UDP Hole Punching for better connectivity over NAT. Now go to Access Key Tab, select your network and there you should have all your keys to connect.
Netclient, the client for every machine, needs wireguard and systemd installed. Create a new ansible playbook wireguard_install.yml and paste this:
---
- hosts: all
tasks:
- name: Install wireguard
apt:
name:
- wireguard
...
Now run ansible-playbook wireguard_install.yml -K -b. To check everything is ok until now run ansible -a "wg --version" all and then ansible -a "systemd --version" all.
Create a new file netclient_install.yml and add this:
---
- hosts: server
tasks:
- name: Add the Netmaker GPG key
shell: curl -sL 'https://apt.netmaker.org/gpg.key' | sudo tee /etc/apt/trusted.gpg.d/netclient.asc
- name: Add the Netmaker repository
shell: curl -sL 'https://apt.netmaker.org/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/netclient.list
- name: Update the package list
shell: apt update
- name: Install netclient
shell: apt install netclient
...
Now run it as usual ansible-playbook netclient_install.yml -K -b. This will install netclient on all hosts. To check, run ansible -a "netclient --version" all.
Last step is easy. Just run ansible -a "netclient join -t YOURTOKEN" -b -K. For the part in brackets, copy your Join Command from Netmaker Dashboard > Access Key. Now all hosts will share a network. This is mine, 11 machines (4 RPi4, 4 OCI instances, my VPS, my WSL and my Windows machine; last 3 are not needed).
Ssh to the OCI server and run: first sudo systemctl stop k3s, then sudo rm -rf /var/lib/rancher/k3s/server/db/etcd and then reinstall but this time with curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--flannel-iface=nm-netmaker --disable=servicelb" INSTALL_K3S_CHANNEL=latest sh -.
Note: I think it's possible just to re-run the installation, without the extra 2 commands before. Or i think it's possible just to append the 2 lines to end of file /etc/systemd/system/k3s.service (just the last 2 lines - after that run sudo systemctl daemon-reload and sudo systemctl restart k3s):
ExecStart=/usr/local/bin/k3s \
server \
'--flannel-iface=nm-netmaker' \
'--disable=servicelb' \
For agents will make an ansible playbook workers_link.yml with following content:
---
- hosts: workers
tasks:
- name: Install k3s on workers and link to server node
shell: curl -sfL https://get.k3s.io | K3S_URL=https://10.20.30.1:6443 K3S_TOKEN=MYTOKEN INSTALL_K3S_EXEC=--"flannel-iface=nm-netmaker" INSTALL_K3S_CHANNEL=latest sh -v
...
You have to paste the content from file on server sudo cat /var/lib/rancher/k3s/server/node-token as MYTOKEN, and change ip address of server if you have another. Now run it with ansible-playbook ~/ansible/link/workers_link.yml -K -b.
Finally over. Go back to server node, run sudo kubectl get nodes -owide and you should have 8 results there, 1 master node and 7 worker nodes.
I'll just install the apps and show how to access them. But you need to configure and run some demo apps to learn how to use them.
First thing installed is a beautiful dashboard - Lens. Install the desktop app on your PC, go to File > Add Cluster. You have to paste here all that you receive when running kubectl config view --minify --raw on server. Edit 127.0.0.1:6443 from that result with your server IP, in my case 10.20.30.1:6443.
Metal LB will work as our load balancer, it will give an external IP to every service type LoadBalancer. Install it with kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml. Create a file config-metallb.yaml and write this in it:
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: default
namespace: metallb-system
spec:
addresses:
- 192.168.0.150 - 192.168.0.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: default
namespace: metallb-system
spec:
ipAddressPools:
- default
Edit IP range with your values from home network. Apply it with kubectl apply -f config-metallb.yaml. Now for every time in the rest of the guide you acces a service with node ip (10.20.30.1 -10.20.30.8:port) you could only use the external IP given by metallb.
On the server it seems git is not installed. So sudo apt install git first. Plus KUBECONFIG env variable wasn't configured until now. So add this line to ~/.bashrc.
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
Helm is the package manager for Kubernetes. Installing helm is very easy, just run following commands on server:
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
Arkade is an Open Source marketplace For Kubernetes. Installation is just curl -sLS https://get.arkade.dev | sudo sh.
To run commands faster and easier you can use auto-completion for kubectl. Run apt install bash-completion and then echo 'source <(kubectl completion bash)' >>~/.bashrc. Reinitialize bash with bash command. Now after every kubectl you can hit tab and it will autocomplete for you. Try with a running pod, just write kubectl get pod and hit TAB.
First write a yaml file traefik-crd.yaml and apply it. It should have this content:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
name: traefik
namespace: kube-system
spec:
valuesContent: |-
additionalArguments:
- "--api"
- "--api.dashboard=true"
- "--api.insecure=true"
- "--log.level=DEBUG"
ports:
traefik:
expose: true
providers:
kubernetesCRD:
allowCrossNamespace: true
Dashboard should be available at http://10.20.30.4:9000/dashboard/#/ in your browser (or any ip from those 8 that are nodes).
Longhorn is a cloud native distributed block storage for Kubernetes. First create a new install-longhorn.yml file for Ansible playbook. Paste in it :
---
- hosts: all
tasks:
- name: Install some package for Longhorns
apt:
name:
- nfs-common
- open-iscsi
- util-linux
...
Run it with ansible-playbook install-longhorn.yml -K -b to install some extra components on nodes.
Move to server. Run helm repo add longhorn https://charts.longhorn.io then helm repo update
and then helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace. It will last a while, ~7 minutes. Now create a longhorn-service.yaml file and paste this:
apiVersion: v1
kind: Service
metadata:
name: longhorn-ingress-lb
namespace: longhorn-system
spec:
selector:
app: longhorn-ui
type: LoadBalancer
ports:
- name: http
protocol: TCP
port: 80
targetPort: http
Run it with kubectl apply -f longhorn-service.yaml. Now you can access the Longhorn Dashboard in your browser in 10.20.30.1:port (or any of your nodes IPs). The port you get it from running kubectl describe svc longhorn-ingress-lb -n longhorn-system | grep NodePort
Now to make Longhorn default StorageClass. Run kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}' and now you should have only one default in kubectl get storageclass.
This app is a container management platform. We'll be installed using again helm. Run helm repo add portainer https://portainer.github.io/k8s/ and then helm repo update.
Now just run helm install --create-namespace -n portainer portainer portainer/portainer. You can acces portainer UI from 10.20.30.1:30777 (or any other IP from your Wireguard network 10.20.30.1 - 10.20.30.8 :30777). It uses 10 GB, check with kubectl get pvc -n portainer (you can check in your Longhorn Dashboard the new volume created). There you will create a user and password. This is what I have after clicking on Get Started:
Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes.
Installation very easy with kubectl create namespace argocd then kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml and wait a few minutes, or check progress with kubectl get all -n argocd. Now to access the UI run kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'. Find your port with kubectl describe service/argocd-server -n argocd | grep NodePort and access the UI from 10.20.30.1:port (or another IP form your network). User is admin and password is stored in kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo.
Prometheus is my favorite metrics system from CNCF Landascape and Grafana will be used as it's dashboard. They will be installed from the ArgoCD UI using a official helm chart kube-prometheus-stack . Open Applications and click New App. Edit:
- App Name : kube-prometheus-stack
- Project Name : default
- Sync Policy : Automatic
- check Auto Create Namespace and Server Side Apply
- Repository URL : https://prometheus-community.github.io/helm-charts (select Helm)
- Chart : kube-prometheus-stack
- Cluster URL : https://kubernetes.default.svc
- Namespace : kube-prometheus-stack
- alertmanager.service.type : LoadBalancer
- prometheus.service.type : LoadBalancer
- prometheusOperator.service.type : LoadBalancer
- grafana.ingress.enabled : true.
Now hit Create. We configured the services as LoadBalancer, because by default they are ClusterIP and if you wanted to access them you have to do port-forwarding every time. You can acces the Prometheus UI from any-node:port (port you get it with kubectl describe svc kube-prometheus-stack-prometheus -n kube-prometheus-stack | grep NodePort). The same for Prometheus Alert Manager (get port with kubectl describe svc kube-prometheus-stack-alertmanager -n kube-prometheus-stack | grep NodePort). For Grafana Dashboard you need to go to any-nodeIP/kube-prometheus-stack-grafana:80. User is admin and password get it with kubectl get secret --namespace kube-prometheus-stack kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d.
k3s Documentation - here;
OCI provider documentation from Terraform - here;
OCI Oracle documentation with Tutorials - here and Guides - here;
Great GitHub repo of garutilorenzo - here. There are a few others who can help you with k8s on OCI too 1 with repo, 2, 3;
One of my old k3s install on Raspbbery PI article on LinkedIn - here - inspired from braindose.blog;
Ansible documentation - here;
Etcd documentation -here; more here why 3 server nodes;
Netmaker from here and documentation here;
A great, huge guide for running Kubernetes on Raspberry Pi - here;
For every app installed search for the official documentation;
ChatGPT helped me a lot of time, use it here.