| + | Name | +Description | +
|---|---|---|
 |
+ Ansible | +Automate OS configuration, external services installation and k3s installation and bootstrapping | +
 |
+ ArgoCD | +GitOps tool for deploying applications to Kubernetes | +
 |
+ Cloud-init | +Automate OS initial installation | +
 |
+ Ubuntu | +Cluster nodes OS | +
 |
+ K3S | +Lightweight distribution of Kubernetes | +
 |
+ containerd | +Container runtime integrated with K3S | +
 |
+ Flannel | +Kubernetes Networking (CNI) integrated with K3S | +
 |
+ CoreDNS | +Kubernetes DNS | +
 |
+ Metal LB | +Load-balancer implementation for bare metal Kubernetes clusters | +
 |
+ Traefik | +Kubernetes Ingress Controller | +
 |
+ Linkerd | +Kubernetes Service Mesh | +
 |
+ Longhorn | +Kubernetes distributed block storage | +
 |
+ Minio | +S3 Object Storage solution | +
 |
+ Cert-manager | +TLS Certificates management | +
 |
+ Hashicorp Vault | +Secrets Management solution | +
 |
+ External Secrets Operator | +Sync Kubernetes Secrets from Hashicorp Vault | +
 |
+ Velero | +Kubernetes Backup and Restore solution | +
 |
+ Restic | +OS Backup and Restore solution | +
 |
+ Prometheus | +Metrics monitoring and alerting | +
 |
+ Fluentd | +Logs forwarding and distribution | +
 |
+ Fluentbit | +Logs collection | +
 |
+ Loki | +Logs aggregation | +
 |
+ Elasticsearch | +Logs analytics | +
 |
+ Kibana | +Logs analytics Dashboards | +
 |
+ Tempo | +Distributed tracing monitoring | +
 |
+ Grafana | +Monitoring Dashboards | +
| [Letsencrypt](https://letsencrypt.org/) | TLS CA Authority | Signed valid TLS certificates |
+| 
|[IONOS](https://www.ionos.es/) | DNS | DNS and [DNS-01 challenge](https://letsencrypt.org/docs/challenge-types/#dns-01-challenge) for certificates |
+
+> **NOTE:** These resources are optional, the homelab still works without them but it won't have trusted certificates
+
+**Alternatives:**
+
+1. Use a private PKI (custom CA to sign certificates).
+
+ Currently supported. Only minor changes are required. See details in [Doc: Quick Start instructions](https://picluster.ricsanfre.com/docs/ansible).
+
+2. Use other DNS provider.
+
+ Cert-manager / Certbot used to automatically obtain certificates from Let's Encrypt can be used with other DNS providers. This will need further modifications in the way cert-manager application is deployed (new providers and/or webhooks/plugins might be required).
+
+ Currently only acme issuer (letsencytp) using IONOS as dns-01 challenge provider is configured. Check list of [supported dns01 providers](https://cert-manager.io/docs/configuration/acme/dns01/#supported-dns01-providers).
+
+### Self-hosted external services
+
+There is another list of services that I have decided to run outside the kuberentes cluster but not using any cloud service. These services currently are running on the same cluster nodes (gateway and node1), but as baremetal service.
+
+| |External Service | Resource | Purpose |
+| --- | --- | --- | --- |
+| |[Minio](https://mini.io) | S3 Object Store | Cluster Backup |
+| |[Hashicorp Vault](https://www.vaultproject.io/) | Secrets Management | Cluster secrets management |
+
+
## Cluster architecture and hardware
Home lab architecture, showed in the picture below, consist of a Kubernetes cluster of 5 nodes (1 master and 4 workers) and a firewall, built with another Raspberry PI, to isolate cluster network from your home network.
@@ -47,7 +228,7 @@ The content of this website and the source code to build it (Jekyll static based
## Usage
-Check out the documentation [Quick Start guide](http://picluster.ricsanfre.com/docs/ansible/) to know how to use and tweak cloud-init files (`/cloud-init` folder) and Ansible playbooks contained in this repository.
+Check out the documentation [Quick Start guide](http://picluster.ricsanfre.com/docs/ansible/) to know how to use and tweak cloud-init files (`/cloud-init` folder), Ansible playbooks (`/ansible` folder) and packaged Kubernetes applications ( `/argocd` folder) contained in this repository, so you can use in for your own homelab.
## About the Project
diff --git a/ansible/.vault/vault_pass.sh b/ansible/.vault/vault_pass.sh
new file mode 100755
index 00000000..82608241
--- /dev/null
+++ b/ansible/.vault/vault_pass.sh
@@ -0,0 +1,2 @@
+#!/bin/sh
+gpg --batch --use-agent --decrypt $HOME/.vault/vault_passphrase.gpg
diff --git a/.yamllint b/ansible/.yamllint
similarity index 100%
rename from .yamllint
rename to ansible/.yamllint
diff --git a/ansible.cfg b/ansible/ansible.cfg
similarity index 85%
rename from ansible.cfg
rename to ansible/ansible.cfg
index d75731c8..5f89d724 100644
--- a/ansible.cfg
+++ b/ansible/ansible.cfg
@@ -11,3 +11,5 @@ collections_path = ./
host_key_checking = false
# SSH key
private_key_file = $HOME/.ssh/ansible-ssh-key.pem
+# Vault password
+vault_password_file=./.vault/vault_pass.sh
diff --git a/ansible/backup_configuration.yml b/ansible/backup_configuration.yml
new file mode 100644
index 00000000..c5f8edd8
--- /dev/null
+++ b/ansible/backup_configuration.yml
@@ -0,0 +1,24 @@
+---
+
+- name: Configure Pi-cluster nodes backup
+ hosts: raspberrypi
+ gather_facts: true
+ tags: [backup]
+ become: true
+ pre_tasks:
+ - name: Include vault variables
+ include_vars: "vars/vault.yml"
+ # Include picluster variables
+ - name: Include picluster variables
+ include_vars: "vars/picluster.yml"
+ - name: Load CA certificate for restic
+ set_fact:
+ restic_ca_cert: "{{ lookup('file','certificates/CA.pem') }}"
+ when: not enable_letsencrypt
+ - name: Do not use CA certificate
+ set_fact:
+ restic_use_ca_cert: false
+ when: enable_letsencrypt
+ roles:
+ - role: ricsanfre.backup
+ tags: [backup]
diff --git a/ansible/create_vault_credentials.yml b/ansible/create_vault_credentials.yml
new file mode 100644
index 00000000..37a866e0
--- /dev/null
+++ b/ansible/create_vault_credentials.yml
@@ -0,0 +1,62 @@
+---
+
+- name: Generate vault variables file
+ hosts: localhost
+
+ vars_prompt:
+ - name: ionos_public_prefix
+ prompt: Enter IONOS public prefix
+ private: true
+ - name: ionos_secret
+ prompt: Enter IONOS secret
+ private: true
+
+ pre_tasks:
+ - name: Ask for SAN centralized credentials
+ when: centralized_san
+ block:
+ - name: Ask for SAN iscsi credentials 1/2
+ pause:
+ prompt: "Enter iSCSI node password: "
+ echo: false
+ register: prompt
+ - name: Set iSCSI node password variable
+ set_fact:
+ san_iscsi_node_pass: "{{ prompt.user_input }}"
+ no_log: true
+ - name: Ask for SAN iscsi credentials 2/2
+ pause:
+ prompt: "Enter iSCSI mutual password: "
+ echo: false
+ register: prompt
+ - name: Set iSCSI node password variable
+ set_fact:
+ san_iscsi_mutual_pass: "{{ prompt.user_input }}"
+ no_log: true
+
+ tasks:
+
+ - name: Create random passwords
+ ansible.builtin.set_fact:
+ "{{ item }}": "{{ lookup('ansible.builtin.password', '/dev/null chars=ascii_letters,digits' ) }}"
+ with_items:
+ - k3s_token
+ - minio_root_password
+ - minio_restic_password
+ - minio_longhorn_password
+ - minio_velero_password
+ - minio_loki_password
+ - minio_tempo_password
+ - traefik_basic_auth_password
+ - fluentd_shared_key
+ - grafana_admin_password
+ - elasticsearch_admin_password
+
+ - name: Generate vault file
+ ansible.builtin.template:
+ src: vars/vault.yml.j2
+ dest: vars/vault.yml
+
+ - name: Encryp file
+ ansible.builtin.command:
+ cmd: ansible-vault encrypt --vault-password-file=./.vault/vault-pass.sh vars/vault.yml
diff --git a/ansible/deploy_monitoring_agent.yml b/ansible/deploy_monitoring_agent.yml
new file mode 100644
index 00000000..972cc0a7
--- /dev/null
+++ b/ansible/deploy_monitoring_agent.yml
@@ -0,0 +1,19 @@
+---
+# Deploy fluentbit to get logs and prometheus metrics
+- name: Deploy fluentbit on control nodes (gateway and pimaster)
+ hosts: gateway
+ gather_facts: true
+ tags: [logging]
+ become: true
+ pre_tasks:
+ # Include vault variables
+ - name: Include vault variables
+ include_vars: "vars/vault.yml"
+ tags: ["always"]
+ # Include picluster variables
+ - name: Include picluster variables
+ include_vars: "vars/picluster.yml"
+ tags: ["always"]
+ roles:
+ - role: logging/external_node
+ tags: ['logging']
diff --git a/ansible/external_services.yml b/ansible/external_services.yml
new file mode 100644
index 00000000..1c51fcb6
--- /dev/null
+++ b/ansible/external_services.yml
@@ -0,0 +1,290 @@
+---
+## Generate TLS certificates for external services
+## Generated using certbot (letsencrypt) or selfsigned certificates
+- name: Generate external services certificates
+ hosts: localhost
+ gather_facts: true
+ tags: [certificates]
+ vars:
+ certbot_venv: "{{ playbook_dir }}/../certbot"
+ certbot_email: "{{ acme_issuer_email }}"
+ dns_ionos_prefix: "{{ vault.certmanager.ionos.public_prefix }}"
+ dns_ionos_secret: "{{ vault.certmanager.ionos.secret }}"
+ selfsigned_certificates_path: "../certificates"
+ pre_tasks:
+ # Include vault variables
+ - name: Include vault variables
+ include_vars: "vars/vault.yml"
+ # Include picluster variables
+ - name: Include picluster variables
+ include_vars: "vars/picluster.yml"
+ roles:
+ - role: certbot
+ vars:
+ - install_python_packages: false
+ when: enable_letsencrypt
+
+ tasks:
+ - name: Create Letsencrytp certificate for external services
+ command: "{{ certbot_venv }}/bin/certbot-create-cert.sh {{ item }}"
+ register: certbot_create
+ changed_when:
+ - certbot_create.rc==0
+ - '"Certificate not yet due for renewal; no action taken." not in certbot_create.stdout'
+ when: enable_letsencrypt
+ with_items:
+ - "{{ minio_hostname }}"
+ - "{{ vault_hostname }}"
+
+ - name: Create customCA-signed TLS certificate for minio
+ when: not enable_letsencrypt
+ block:
+ # Generate self-signed certificates directory
+ - name: Create certificates directory
+ file:
+ path: "{{ selfsigned_certificates_path }}"
+ state: directory
+ mode: 0750
+ # Include selfsigned certificates variables
+ - name: Include selfsigned certificates variables
+ include_vars: "vars/selfsigned-certificates.yml"
+ # Generate custom CA
+ - name: Generate custom CA
+ include_tasks: tasks/generate_custom_ca.yml
+ args:
+ apply:
+ delegate_to: localhost
+ become: false
+ # Generate selfsigned TLS certificate
+ - name: Generate customCA-signed SSL certificates
+ include_tasks: tasks/generate_ca_signed_cert.yml
+ args:
+ apply:
+ delegate_to: localhost
+ become: false
+ loop:
+ - "{{ minio_hostname }}"
+ - "{{ vault_hostname }}"
+ loop_control:
+ loop_var: server_hostname
+
+## Install Minio S3 Storage Server
+
+- name: Install Minio S3 Storage Server
+ hosts: node1
+ gather_facts: true
+ tags: [s3]
+ become: true
+ vars:
+ server_hostname: "{{ minio_hostname }}"
+ certbot_venv: "{{ playbook_dir }}/../certbot"
+ selfsigned_certificates_path: "../certificates"
+ pre_tasks:
+ # Include vault variables
+ - name: Include vault variables
+ include_vars: "vars/vault.yml"
+ # Include picluster variables
+ - name: Include picluster variables
+ include_vars: "vars/picluster.yml"
+
+ - name: Load customCA-signed TLS certificate for minio
+ set_fact:
+ minio_key: "{{ lookup('file',selfsigned_certificates_path + '/' + server_hostname + '.key') }}"
+ minio_cert: "{{ lookup('file',selfsigned_certificates_path + '/' + server_hostname + '.pem') }}"
+ when: not enable_letsencrypt
+
+ - name: Get letsencrypt TLS certificate for minio
+ block:
+ - name: check TLS certificate for minio exits
+ command: "{{ certbot_venv }}/bin/certbot-wrapper.sh certificates -d {{ minio_hostname }}"
+ register: certbot_certificates
+ delegate_to: localhost
+ become: false
+ changed_when: false
+ failed_when:
+ - '"Certificate Name: " + minio_hostname not in certbot_certificates.stdout'
+ - name: Get certificate and key paths for minio
+ set_fact:
+ cert_path: "{{ certbot_certificates.stdout | regex_search(regexp1,'\\1') }}"
+ cert_key_path: "{{ certbot_certificates.stdout | regex_search(regexp2,'\\1') }}"
+ vars:
+ regexp1: 'Certificate Path: (\S+)'
+ regexp2: 'Private Key Path: (\S+)'
+ when:
+ - certbot_certificates.rc==0
+ - '"Certificate Name: " + minio_hostname in certbot_certificates.stdout'
+
+ - name: Load tls key and cert
+ set_fact:
+ minio_key: "{{ lookup('file', cert_key_path[0] ) }}"
+ minio_cert: "{{ lookup('file', cert_path[0] ) }}"
+ when: enable_letsencrypt
+ roles:
+ - role: ricsanfre.minio
+
+## Install Hashicorp Vault Server
+
+- name: Install Vault Server
+ hosts: gateway
+ gather_facts: true
+ tags: [vault]
+ become: true
+ vars:
+ server_hostname: "{{ vault_hostname }}"
+ certbot_venv: "{{ playbook_dir }}/../certbot"
+ selfsigned_certificates_path: "../certificates"
+
+ pre_tasks:
+ # Include vault variables
+ - name: Include vault variables
+ include_vars: "vars/vault.yml"
+
+ # Include picluster variables
+ - name: Include picluster variables
+ include_vars: "vars/picluster.yml"
+
+ - name: Load customCA-signed TLS certificate for minio
+ set_fact:
+ vault_key: "{{ lookup('file',selfsigned_certificates_path + '/' + server_hostname + '.key') }}"
+ vault_cert: "{{ lookup('file',selfsigned_certificates_path + '/' + server_hostname + '.pem') }}"
+ vault_ca: "{{ lookup('file',selfsigned_certificates_path + '/CA.pem') }}"
+ when: not enable_letsencrypt
+
+ - name: Get letsencrypt TLS certificate for vault
+ block:
+ - name: check TLS certificate for vault exits
+ command: "{{ certbot_venv }}/bin/certbot-wrapper.sh certificates -d {{ vault_hostname }}"
+ register: certbot_certificates
+ delegate_to: localhost
+ become: false
+ changed_when: false
+ failed_when:
+ - '"Certificate Name: " + vault_hostname not in certbot_certificates.stdout'
+ - name: Get certificate and key paths for minio
+ set_fact:
+ cert_path: "{{ certbot_certificates.stdout | regex_search(regexp1,'\\1') }}"
+ cert_key_path: "{{ certbot_certificates.stdout | regex_search(regexp2,'\\1') }}"
+ vars:
+ regexp1: 'Certificate Path: (\S+)'
+ regexp2: 'Private Key Path: (\S+)'
+ when:
+ - certbot_certificates.rc==0
+ - '"Certificate Name: " + vault_hostname in certbot_certificates.stdout'
+
+ - name: Load tls key and cert
+ set_fact:
+ vault_key: "{{ lookup('file', cert_key_path[0] ) }}"
+ vault_cert: "{{ lookup('file', cert_path[0] ) }}"
+ when: enable_letsencrypt
+ roles:
+ - role: ricsanfre.vault
+
+ tasks:
+ # Configure ansible user profile with VAULT environement variables
+ - name: Insert http(s) export in dotfile
+ become: true
+ lineinfile:
+ path: ~{{ ansible_user }}/.bashrc
+ regexp: "^export VAULT_ADDR="
+ line: "export VAULT_ADDR='https://{{ vault_hostname }}:8200'"
+ owner: "{{ ansible_user }}"
+ create: true
+ mode: 0644
+ - name: Insert CA cert export in dotfile
+ become: true
+ lineinfile:
+ path: ~{{ ansible_user }}/.bashrc
+ regexp: "^export VAULT_CACERT="
+ line: "export VAULT_CACERT=/etc/vault/tls/vault-ca.crt"
+ owner: "{{ ansible_user }}"
+ create: true
+ mode: 0644
+ when: custom_ca
+
+ - name: Insert VAULT_TOKEN in dotfile
+ become: true
+ lineinfile:
+ path: ~{{ ansible_user }}/.bashrc
+ regexp: "^export VAULT_TOKEN="
+ line: "export VAULT_TOKEN=$(sudo jq -r '.root_token' /etc/vault/unseal.json)"
+ owner: "{{ ansible_user }}"
+ create: true
+ mode: 0644
+
+## Load all credentials into Hashicorp Vault Server
+- name: Load Vault Credentials
+ hosts: gateway
+ gather_facts: true
+ tags: [vault, credentials]
+ become: false
+ pre_tasks:
+ # Include vault variables
+ - name: Include vault variables
+ include_vars: "vars/vault.yml"
+
+ # Include picluster variables
+ - name: Include picluster variables
+ include_vars: "vars/picluster.yml"
+
+ # Install hashicorp vault collection dependencies (hvac python package) using PIP.
+ - name: Ensure PIP is installed.
+ package:
+ name:
+ - python3-pip
+ - python3-setuptools
+ state: present
+ become: true
+ - name: Ensure hvac Python library is installed.
+ pip:
+ name: hvac
+ state: present
+ become: true
+ tasks:
+
+ # Vault Login using ansible environement variables for creating token file
+ # token file will be usd in next commands
+ - name: Vault login
+ ansible.builtin.shell: bash -ilc 'vault login $VAULT_TOKEN'
+ # Interactive bash so .bashrc is loaded
+ # 'source ~/.bashrc && ![](\"https://linkerd.io/images/identity/favicon/linkerd-favicon.png\")
\n cj/$cronjob\n \n cj/$inbound\n \n cj/$outbound\n![](\"https://linkerd.io/images/identity/svg/linkerd_primary_color_white.svg\")
\n \n Cluster: $cluster\n \n replicaset/$replicaset\n \n rs/$inbound\n \n rs/$outbound\n\n \n route/$rt_route\n\n