Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: aigisuk/terraform-digitalocean-ha-k3s
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 0.1.6
Choose a base ref
...
head repository: aigisuk/terraform-digitalocean-ha-k3s
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: release
Choose a head ref
Loading
153 changes: 141 additions & 12 deletions .github/workflows/terraform.yaml
Original file line number Diff line number Diff line change
@@ -4,20 +4,149 @@ on:
pull_request:

jobs:
terraform:
name: 'Terraform Format'
runs-on: ubuntu-20.04
terraform-fmt:
name: Terraform Format
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2.3.4

- name: Setup Terraform
uses: hashicorp/setup-terraform@v1

- name: Terraform Format
id: fmt
run: terraform fmt -diff -check -no-color -recursive
continue-on-error: true

defaults:
run:
shell: bash
- uses: actions/github-script@v4
if: github.event_name == 'pull_request' && steps.fmt.outputs.exitcode != 0
env:
TF_FMT_STDOUT: "${{ steps.fmt.outputs.stdout }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Format and Style 🖌 - \`${{ steps.fmt.outcome }}\`
\`\`\`diff
${process.env.TF_FMT_STDOUT}
\`\`\`
*Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Workflow: \`${{ github.workflow }}\`*`;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
throw "failed to run `terraform fmt -check -recursive -diff`"
terraform-plan:
name: Terraform Initialize, Validate, Plan
needs: terraform-fmt
runs-on: ubuntu-latest
env:
WORKING_DIR: "examples/github_actions"
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v2.3.4

- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
with:
cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

- name: Terraform Init
id: init
run: terraform init -no-color
working-directory: ${{ env.WORKING_DIR }}
continue-on-error: true

- uses: actions/github-script@v4
if: github.event_name == 'pull_request' && steps.init.outputs.exitcode != 0
env:
TF_INIT_STDERR: "${{ steps.init.outputs.stderr }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Initialization ⚙️ - \`${{ steps.init.outcome }}\`
\`\`\`${ process.env.TF_INIT_STDERR }\`\`\`
*Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Workdir: \`${{ env.WORKING_DIR }}\`, Workflow: \`${{ github.workflow }}\`*`;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
throw "failed to run `terraform init`"
- name: Terraform Validate
id: validate
run: terraform validate -no-color
working-directory: ${{ env.WORKING_DIR }}
continue-on-error: true

- uses: actions/github-script@v4
if: github.event_name == 'pull_request' && steps.validate.outputs.exitcode != 0
env:
TF_VAL_STDERR: "${{ steps.validate.outputs.stderr }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Initialization ⚙️ - \`${{ steps.init.outcome }}\`
#### Terraform Validate 📃 - \`${{ steps.validate.outcome }}\`
\`\`\`${ process.env.TF_VAL_STDERR }\`\`\`
*Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Workdir: \`${{ env.WORKING_DIR }}\`, Workflow: \`${{ github.workflow }}\`*`;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
throw "failed to run `terraform validate`"
- name: Terraform Plan
id: plan
run: terraform plan -no-color
working-directory: ${{ env.WORKING_DIR }}
continue-on-error: true

- uses: actions/github-script@v4
if: github.event_name == 'pull_request'
env:
PLAN: "terraform\n${{ steps.plan.outputs.stdout }}"
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const output = `#### Terraform Initialization ⚙️ - \`${{ steps.init.outcome }}\`
#### Terraform Validate 📃 - \`${{ steps.validate.outcome }}\`
#### Terraform Plan 📖 - \`${{ steps.plan.outcome }}\`
<details><summary>Show Plan</summary>
\`\`\`${process.env.PLAN}\`\`\`
</details>
*Pusher: @${{ github.actor }}, Action: \`${{ github.event_name }}\`, Workdir: \`${{ env.WORKING_DIR }}\`, Workflow: \`${{ github.workflow }}\`*`;
github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: output
})
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
- name: Terraform Plan Status
if: steps.plan.outcome == 'failure'
run: exit 1

- name: Terraform Format
run: terraform fmt -check
# - name: Terraform Apply
# id: apply
# if: github.ref == 'refs/heads/release' && github.event_name == 'push'
# run: terraform apply -auto-approve
# working-directory: ${{ env.WORKING_DIR }}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -6,3 +6,4 @@ terraform.txt
.auto.tfvars
creds/
.vscode/
debug.log
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Copyright © 2021 AIGIS Services Ltd, Colin Wilson
Copyright © 2023 AIGIS UK, Colin Wilson

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

139 changes: 121 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,29 +1,36 @@
# Terraform DigitalOcean HA K3S Module
A Terraform module to provision a high availability [K3s](https://k3s.io/) cluster with external database on the DigitalOcean cloud platform.
An opinionated Terraform module to provision a high availability [K3s](https://k3s.io/) cluster with external database on the DigitalOcean cloud platform. Perfect for development or testing.

![Terraform, DigitalOcean, K3s illustration](https://res.cloudinary.com/qunux/image/upload/v1618967113/terraform-digitalocean-k3s-repo-logo_f2zyoz.svg)
![2021-10-24 05_45_35-k3s-cluster_project_DigitalOcean](https://user-images.githubusercontent.com/12916656/138615757-c73b90bc-9fe7-4214-90c7-acfad2f49222.png)

<!-- ![k3s_on_digitalocean_term_demo](examples/ha-k3s-do-demo.svg) -->

## Features
* [x] High Availability K3s Cluster provisioned on the DigitalOcean platform
* [x] Managed **PostgreSQL**/**MySQL** database provisioned. Serves as the datastore for the cluster's state (configurable options: size & node count)
* [x] The number of provisioned Servers (Masters) and Agents (Workers) is configurable
* [x] Cluster API/Servers are behind a provisioned load balancer for high availability
* [x] Flannel backend is configurable. Choose from `vxlan`, `host-gw`, `ipsec` (default) or `wireguard`
* [x] Dedicated VPC provisioned for cluster use (IP Range: `10.10.10.0/24`)
* [x] Number of provisioned Servers (Masters) and Agents (Workers) is configurable
* [x] Cluster API/Server(s) are behind a provisioned load balancer for high availability
* [x] All resources assigned to a dedicated DigitalOcean project (expect Load Balancers provisioned by app deployments)
* [x] Flannel backend is configurable. Choose from `vxlan` (default), `ipsec` or `wireguard`
* [x] DigitalOcean's CCM ([Cloud Controller Manager](https://github.com/digitalocean/digitalocean-cloud-controller-manager)) and CSI ([Container Storage Interface](https://github.com/digitalocean/csi-digitalocean)) plugins are pre-installed. Enables the cluster to leverage DigitalOcean's load balancer and volume resources
* [x] Option to make Servers (Masters) schedulable. Default is `false` i.e. `CriticalAddonsOnly=true:NoExecute`
* [x] Cluster database engine is configurable. Choose from **PostgreSQL** (v11) or **MySQL** (v8)
* [ ] Pre-install an ingress controller from **Kong**, **Nginx** or **Traefik v2** (optional)
* [ ] Pre-install the Kubernetes Dashboard (optional)
* [x] Cluster database engine is configurable. Choose between **PostgreSQL** (v11, default) or **MySQL** (v8)
* [x] Deploy [System Upgrade Controller](https://github.com/rancher/system-upgrade-controller) to manage [automated upgrades](https://rancher.com/docs/k3s/latest/en/upgrades/automated/) of the cluster [default: `false`]
* [x] Deploy the [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) [default: `false`]
* [x] Deploy Jetstack's [cert-manager](https://github.com/jetstack/cert-manager) [default: `false`]
* [x] Firewalled Nodes & Database
* [x] Deploy an ingress controller from **Kong** (Postgres or [DB-less mode](https://docs.konghq.com/gateway-oss/2.6.x/db-less-and-declarative-config/)), **Nginx** or **Traefik v2** [default: `none`]
* [ ] Generate custom `kubeconfig` file (optional)

## Compatibility/Requirements

* Requires [Terraform](https://www.terraform.io/downloads.html) 0.13 or higher.
* Requires [Terraform](https://www.terraform.io/downloads.html) 0.15 or higher.
* A DigitalOcean account and [personal access token](https://docs.digitalocean.com/reference/api/create-personal-access-token/) for accessing the DigitalOcean API - [Use this referral link for $100 free credit](https://m.do.co/c/6b3bf6d79f7d)

## Tutorial

TBC
[Deploy a HA K3s Cluster on DigitalOcean in 10 minutes using Terraform](https://colinwilson.uk/2021/04/04/deploy-a-ha-k3s-cluster-on-digitalocean-in-10-minutes-using-terraform/)

## Architecture

@@ -43,6 +50,48 @@ module "do-ha-k3s" {
ssh_key_fingerprints = ["00:11:22:33:44:55:66:77:88:99:aa:bb:cc:dd:ee:ff"]
}
```
Example output:
```
cluster_summary = {
"agents" = [
{
"id" = "246685594"
"ip_private" = "10.10.10.4"
"ip_public" = "203.0.113.10"
"name" = "k3s-agent-fra1-1a9f-1"
"price" = 10
},
]
"api_server_ip" = "198.51.100.10"
"cluster_region" = "fra1"
"servers" = [
{
"id" = "246685751"
"ip_private" = "10.10.10.5"
"ip_public" = "203.0.113.11"
"name" = "k3s-server-fra1-55b4-1"
"price" = 10
},
{
"id" = "246685808"
"ip_private" = "10.10.10.6"
"ip_public" = "203.0.113.12"
"name" = "k3s-server-fra1-d6e7-2"
"price" = 10
},
]
}
```

To manage K3s from outside the cluster, SSH into any Server node and copy the contents of `/etc/rancher/k3s/k3s.yaml` to `~/.kube/config` on an external machine with `kubectl` installed.
```
sudo scp -i .ssh/your_private_key root@203.0.113.11:/etc/rancher/k3s/k3s.yaml ~/.kube/config
```

Then replace `127.0.0.1` with the API Load Balancer IP address of your K3s Cluster (value for the `api_server_ip` key from the Terraform `cluster_summary` output).
```
sudo sed -i -e "s/127.0.0.1/198.51.100.10/g" ~/.kube/config
```

Functional examples are included in the
[examples](./examples/) directory.
@@ -54,23 +103,77 @@ Functional examples are included in the
| do_token | DigitalOcean Personal Access Token | string | N/A | yes |
| ssh_key_fingerprints | List of SSH Key fingerprints | list(string) | N/A | yes |
| region | Region in which to deploy cluster | string | `fra1` | no |
| k3s_channel | K3s release channel. `stable`, `latest`, `testing` or a specific channel or version e.g. `v1.20`, `v1.19.8+k3s1` | string | `"stable"` | no |
| vpc_network_range | Range of IP addresses for the VPC in CIDR notation | string | `10.10.10.0/24` | no |
| k3s_channel | K3s release channel. `stable`, `latest`, `testing` or a specific channel or version e.g. `v1.22`, `v1.19.8+k3s1` | string | `"stable"` | no |
| database_user | Database username | string | `"k3s_default_user"` | no |
| database_engine | Database engine. PostgreSQL (13) or MySQL (8) | string | `"postgres"` | no |
| database_engine | Database engine. `postgres` (v13) or `mysql` (v8) | string | `"postgres"` | no |
| database_size | Database Droplet size associated with the cluster e.g. `db-s-1vcpu-1gb` | string |`"db-s-1vcpu-1gb"` | no |
| database_node_count | Number of nodes that comprise the database cluster | number | `1`| no |
| flannel_backend | Flannel Backend Type. Valid options include `vxlan`, `host-gw`, `ipsec` (default) or `wireguard` | string | `ipsec`| no |
| flannel_backend | Flannel Backend Type. Valid options include `vxlan`, `ipsec` or `wireguard-native` | string | `vxlan`| no |
| server_size | Server droplet size. e.g. `s-1vcpu-2gb` | string | `s-1vcpu-2gb`| no |
| agent_size | Agent droplet size. e.g. `s-1vcpu-2gb` | string | `s-1vcpu-2gb`| no |
| server_count | Number of server (master) nodes to provision | number | `2`| no |
| agent_count | Number of agent (worker) nodes to provision | number | `1`| no |
| server_taint_criticalonly | Allow only critical addons to be scheduled on servers? (thus preventing workloads from being launched on them) | bool | `true`| no |
| server_taint_criticalonly | Allow only critical addons to be scheduled on server nodes? (thus preventing workloads from being launched on them) | bool | `true`| no |
| sys_upgrade_ctrl | Deploy the [System Upgrade Controller](https://github.com/rancher/system-upgrade-controller) | bool | `false`| no |
| ingress | Deploy an ingress controller. `none`, `traefik`, `kong`, `kong_pg` | string | `"none"`| no |
| k8s_dashboard | Deploy [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) | bool | `false`| no |
| k8s_dashboard_version | [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) version | string | `2.7.0`| no |
| cert_manager | Deploy [cert-manager](https://cert-manager.io/) | bool | `false`| no |
| cert_manager_version | [cert-manager](https://cert-manager.io/) version | string | `1.11.0`| no |
| traefik_version | [traefik](https://github.com/traefik/traefik-helm-chart) version | string | `2.9.7`| no |

## Outputs

TBC
| Name | Description |
|------|-------------|
| cluster_summary | A summary of the cluster's provisioned resources. |

## Deploy the Kubernetes Dashboard

The [Kubernetes Dashboard](https://github.com/kubernetes/dashboard) can be deployed by setting the `k8s_dashboard` input variable to `true`.

This auto-creates a Service Account named `admin-user` with admin privileges granted. The following `kubectl` command outputs the Bearer Token for the `admin-user`:

```
kubectl -n kubernetes-dashboard describe secret admin-user-token | awk '$1=="token:"{print $2}'
```
Output:
```
eyJhbGciOiJSUzI1NiI....JmL-nP-x1SPjOCNfZkg
```

Use `kubectl port-forward` to forward a local port to the dashboard:

```
kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 8080:443
```

To access the Kubernetes Dashboard go to:
```
https://localhost:8080
```
Select the `Token` option, enter the `admin-user` Bearer Token obtained earlier and click `Sign in`:

![Kubernetes-Dashboard-Login](https://user-images.githubusercontent.com/12916656/117087905-c3d99800-ad48-11eb-9245-6a73578c5e3a.png)

## Traefik Ingress

[Traefik Proxy](https://doc.traefik.io/traefik/) ingress can be deployed by setting the `ingress` input variable to `traefik`. The Traefik dashboard is enabled by default.

Use `kubectl port-forward` to forward a local port to the dashboard:

```
kubectl port-forward -n traefik $(kubectl get pods -n traefik --selector=app=traefik --output=name) 9000:9000
```

To access the Traefik Dashboard go:
```
http://localhost:9000/dashboard/
```
> Don't forget the trailing slash
## Cost
## Cost Analysis

A default deployment of this module provisions the following resources:

@@ -81,8 +184,8 @@ A default deployment of this module provisions the following resources:
| **1x** | Load Balancer | Small | 10 | **10** | **0.01488** |
| **1x** | Postgres DB Cluster | Single Basic Node | 15 | **15** | **0.022** |
| | | | **Total** | **55** |**0.082** |
##### * Prices correct at time of latest commit (check [digitalocean.com](https://www.digitalocean.com/pricing/) for current pricing)
##### **N.B.** Keep in mind, additional costs may be incurred through the provisioning of volumes and/or load balancers required by any applications deployed on the cluster.
##### * Prices correct at time of latest commit. Check [digitalocean.com/pricing](https://www.digitalocean.com/pricing/) for current pricing.
##### **N.B.** Additional costs may be incurred through the provisioning of volumes and/or load balancers required by any applications deployed to the cluster.

## Credits

15 changes: 7 additions & 8 deletions agent.tf
Original file line number Diff line number Diff line change
@@ -2,14 +2,13 @@ resource "digitalocean_droplet" "k3s_agent" {
count = var.agent_count
name = "k3s-agent-${var.region}-${random_id.agent_node_id[count.index].hex}-${count.index + 1}"

image = "ubuntu-20-04-x64"
tags = ["k3s_agent"]
region = var.region
size = var.agent_size
monitoring = true
private_networking = true
vpc_uuid = digitalocean_vpc.k3s_vpc.id
ssh_keys = var.ssh_key_fingerprints
image = "ubuntu-20-04-x64"
tags = [digitalocean_tag.agent.id]
region = var.region
size = var.agent_size
monitoring = true
vpc_uuid = digitalocean_vpc.k3s_vpc.id
ssh_keys = var.ssh_key_fingerprints
user_data = templatefile("${path.module}/user_data/k3s_agent.sh", {
k3s_channel = var.k3s_channel
k3s_token = random_password.k3s_token.result
9 changes: 9 additions & 0 deletions db_firewall.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
resource "digitalocean_database_firewall" "k3s" {

cluster_id = digitalocean_database_cluster.k3s.id

rule {
type = "tag"
value = digitalocean_tag.server.name
}
}
Loading