Skip to content

Commit

Permalink
Merge pull request #7 from midl-dev/helm
Browse files Browse the repository at this point in the history 
Helm
  • Loading branch information
@nicolasochem
nicolasochem authored Jan 11, 2022
2 parents 2832703 + 8d84aa8 commit f6207a4
Show file tree
Hide file tree
Showing 32 changed files with 556 additions and 700 deletions.
126 changes: 126 additions & 0 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
name: CI

on:
push:
# Trigger CI on all branch pushes but...
branches:
- "**"
# don't double trigger on new tag push when creating release. Should only
# trigger once for the release.
tags-ignore:
- "*.*.*"
pull_request:
release:
types: [created]

jobs:

list_containers_to_publish:
# based on
# https://stackoverflow.com/a/62953566/207209
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.gen-containers-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2

- id: gen-containers-matrix
run: |
container_list=$(jq -c -n --arg cont "$(find -name 'Dockerfile' -printf '%h\n' | sort -u | sed 's/.\///')" '{ container: $cont | split("\n")}')
echo "Dynamically generated container list based on subdirectories of the repo with a dockerfile in it. The following list will be passed to next build step:"
echo $container_list
echo "::set-output name=matrix::$container_list"
publish_containers:
# based on
# https://github.com/docker/build-push-action#usage
runs-on: ubuntu-latest
needs: list_containers_to_publish
if: github.event_name == 'release' && github.event.action == 'created'
strategy:
matrix: ${{fromJson(needs.list_containers_to_publish.outputs.matrix)}}

steps:
- name: Checkout
uses: actions/checkout@v2

- name: Login to registry
run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
- name: Docker meta
id: meta
uses: docker/metadata-action@v3
with:
images: ghcr.io/${{ github.repository_owner }}/${{ matrix.container }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=match,pattern=v(.*),group=1
- name: Push to GHCR
uses: docker/build-push-action@v2
with:
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

lint_helm_charts:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2

- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: v3.4.2

- name: Lint Helm Charts
run: helm lint charts/*

publish_helm_charts:
runs-on: ubuntu-latest
needs: [lint_helm_charts, publish_containers]
if: github.event_name == 'release' && github.event.action == 'created'
steps:
- name: Checkout
uses: actions/checkout@v2

- name: Install yq
run: |
sudo wget -q https://github.com/mikefarah/yq/releases/download/v4.2.0/yq_linux_amd64 -O /usr/bin/yq
sudo chmod +x /usr/bin/yq
- name: Get Release Version
id: get_release_version
run: echo "::set-output name=RELEASE_VERSION::${GITHUB_REF/refs\/tags\//}"

- name: Set Helm Chart and Image Versions
run: |
set -x
RELEASE_VERSION=${{ steps.get_release_version.outputs.RELEASE_VERSION }}
for chart in charts/*; do
[[ ! -d "$chart" ]] && continue
echo $chart
# Update Chart.yaml with release version
yq e ".version = \"$RELEASE_VERSION\"" -i "$chart/Chart.yaml"
# Get midl-dev/polkadot-k8s images specified in values.yaml
custom_images=$(yq e '(.polkadot_k8s_images[]) | path | .[-1]' "$chart/values.yaml")
# Update the release version of each of polkadot-k8s images
for image in $custom_images; do
image_name=$(yq e ".polkadot_k8s_images.$image" $chart/values.yaml | sed -E "s/polkadot-k8s-(.*):.*/\1/")
yq e ".polkadot_k8s_images.$image = \"midl-dev/polkadot-k8s-$image_name:$RELEASE_VERSION\"" -i $chart/values.yaml
done
done
- name: Publish Helm charts
uses: stefanprodan/helm-gh-pages@master
with:
linting: off # We already linted in a job before
token: ${{ secrets.CI_GITHUB_TOKEN }}
branch: main
owner: ${{ github.repository_owner }}
repository: charts
222 changes: 2 additions & 220 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,233 +1,15 @@
# Polkadot-k8s

This project deploys a fully featured, best practices [Polkadot](https://polkadot.network) or [Kusama](https://kusama.network) validator setup on Google Kubernetes Engine.
This project is a collection of helm charts to help you deploy [Polkadot](https://polkadot.network) or [Kusama](https://kusama.network) validation infrastructure.

Features:

* compatible with Kusama and Polkadot
* high availability and geographical distribution
* download and import a [pre-synced database](https://dotleap.com/how-to-import-a-pre-synced-kusama-database/) for faster synchronization of the node
* deploy everything in just one command - no prior knowledge of Kubernetes required

TODO:

* support for on-prem remote signer [whenever available](https://github.com/paritytech/substrate/issues/4689)
* download and import a [pre-synced database](https://polkashots.io) for faster synchronization of the node

Brought to you by MIDL.dev
--------------------------

<img src="midl-dev-logo.png" alt="MIDL.dev" height="100"/>

We help you deploy and manage a complete Polkadot or Kusama validator infrastructure for you. [Hire us](https://midl.dev).

Architecture
------------

This is a Kubernetes private cluster with two nodes located in two Google Cloud zones, in the same region.

The validator node uses a [Regional Persistent Disk](https://cloud.google.com/compute/docs/disks/#repds) so it can be respun quickly in the other node from the pool if the first node goes offline for any reason, for example base OS upgrade.

The setup is production hardened:
* usage of kubernetes secrets to store sensitive values such as node keys. They are created securely from terraform variables,
* network policies to restrict communication between pods. For example, only sentries can peer with the validator node.

## Costs

Deploying will incur Google Compute Engine charges, specifically:

* virtual machines
* regional persistent SSD storage
* network ingress
* NAT forwarding

# How to deploy

*WARNING: Polkadot/Kusama tokens have value. Use judgement and care in your network interactions, otherwise loss of funds may occur.*

## Prerequisites

1. Download and install [Terraform](https://terraform.io)

1. Download, install, and configure the [Google Cloud SDK](https://cloud.google.com/sdk/).

1. Install the [kubernetes
CLI](https://kubernetes.io/docs/tasks/tools/install-kubectl/) (aka
`kubectl`)

## Authentication

Using your Google account, active your Google Cloud access.

Login to gcloud using `gcloud auth login`

Set up [Google Default Application Credentials](https://cloud.google.com/docs/authentication/production) by issuing the command:

```
gcloud auth application-default login
```

NOTE: for production deployments, the method above is not recommended. Instead, you should use a Terraform service account following [these instructions](docs/production-hardening.md).

## Bond your tokens

Follow [these instructions](https://wiki.polkadot.network/docs/en/maintain-guides-how-to-validate-kusama#bond-ksm) to bond your KSM.

NOTE: the link above points to the official guide about [how to validate on Kusama](https://wiki.polkadot.network/docs/en/maintain-guides-how-to-validate-kusama). Not every action in this guide needs to be performed. For example, there is no need to build binaries.

## Populate terraform variables

All custom values unique to your deployment are set as terraform variables. You must populate these variables manually before deploying the setup.

A simple way is to populate a file called `terraform.tfvars` in the `terraform` folder.

NOTE: `terraform.tfvars` is not recommended for a production deployment. See [production hardening](docs/production-hardening.md).

### Network (libp2p) keys (optional)

These keys are needed for the validator to communicate to its sentries. You may pass them as a variabe, or Terraform will generate them for you.

If you want to pass them, the syntax is:

```
polkadot_node_keys = {
"polkadot-private-node-0": "b5ca09a5dccb48d5c7915f24223454fe1a557383ba0b1560cc3ed919a6e9dec5",
"polkadot-sentry-node-0": "dcf609b50868ffe379d4a992cf866deba8ad84ecca24853bacba1171ae7cdf22",
"polkadot-sentry-node-1": "ca62cb1bae8c84090f2e20dae81c79f8843fb75df065c8026e4603b85b48729f"
}
```

### Polkadot utility parameters

Set the `polkadot_version` to the desired version of polkadot container.

Set the `chain` variable to the desired network you want to launch (`polkadot` or `kusama`).

Set the `polkadot_telemetry_url` variable to the telemetry server websocket endpoint (that you would pass to polkadot's `--telemetry-url` option)

Set the `polkadot_validator_name` to your validator name as you want it to appear on telemetry (maps to polkadot's `--name` parameter).

### Archive URL (optional)

If you have an archive of the node storage, you can put the URL here. It will make the initial deployment of the nodes faster. It must be in `7z` format.

See [a resource on how to get a pre-synced archive databsae for Kusama](https://dotleap.com/how-to-import-a-pre-synced-kusama-database/).


### Payout account

Every era, you can pay your nominators automatically by calling the payoutStakers() extrinsic *from any account*. A Kubernetes cronjob will do this for you if you pass the address and its associated private key.

This should be a dust account. The private key is stored in a Kubernetes secret but should not be considered secure. Do not put meaningful amounts in this account, just enough to pay fees.

### Google Cloud project

A default Google Cloud project should have been created when you activated your account. Verify its ID with `gcloud projects list`. You may also create a dedicated project to deploy the cluster.

Set the project id in the `project` terraform variable.

NOTE: if you created a [terraform service account](docs/production-hardening.md), leave this variable empty.

### Recap : full example of terraform.tfvars file

```
project="beaming-essence-301841"
polkadot_archive_url="https://ipfs.io/ipfs/Qma3fM33cw4PGiw28SidqhFi3CXRa2tpywqLmhYveabEYQ?filename=Qma3fM33cw4PGiw28SidqhFi3CXRa2tpywqLmhYveabEYQ"
polkadot_validator_name="Hello from k8s!"
polkadot_version="v0.8.0"
chain="kusama"
polkadot_telemetry_url="wss://telemetry-backend.w3f.community/submit"
```

The example above would:
* deploy a validator setup in the Google Cloud project named `beaming-essence-301841`
* download a kusama snapshot from IPFS
* report to telemetry server `w3f.community` under the name `Hello from k8s!`

## Deploy!

1. Run the following:

```
cd terraform
terraform init
terraform plan -out plan.out
terraform apply plan.out
```

This will take time as it will:
* create a Kubernetes cluster
* build the necessary containers
* download and unzip the archives if applicable
* spin up the sentry and validator nodes
* sync the network

### Connect to the cluster

After apply is complete, your `kubectl` command should point to the correct cluster. You can then issue `kubectl get pods` and observe that your Polkadot nodes are now alive and syncing.

When you display the logs of your private node, you will see it syncing:

```
kubectl logs -f polkadot-private-node-0 --tail=10
```

### How to check your validator node is running ?

* connect to your telemetry server and search for your node by name
* set up a websocket tunnel to your local host

```
kubectl port-forward polkadot-private-node-0 9944:9944
```

Then go to the [Polkadot Js app](https://polkadot.js.org/apps/#/) and configure it to point to `localhost:9944`. You should see your node syncing.

### Inject session keys

[Follow instructions](https://wiki.polkadot.network/docs/en/maintain-guides-how-to-validate-kusama#set-session-keys) to inject session keys using the Polkadot Js app.

### Validate

[Follow instructions](https://wiki.polkadot.network/docs/en/maintain-guides-how-to-validate-kusama#validate)


## I have a kubernetes cluster already, I just want to deploy to it

[Instructions here](docs/pre-existing-cluster.md)

Apply an update
---------------

If you have pulled the most recent version of `polkadot-k8s` and wish to apply updates, issue the following commands:

```
terraform taint null_resource.push_containers && terraform taint null_resource.apply && terraform plan -out plan.out
terraform apply plan.out
```

This will rebuild the containers, then apply the kubernetes changes.

If you want to upgrade the polkadot containers version, edit the version number in `terraform.tfvars`, then issue the commands above.

The pods may or may not restart on their own, depending on what changed since last deployment. To force a restart, issue:

```
kubectl delete pod polkadot-sentry-node-0
kubectl delete pod polkadot-sentry-node-1
kubectl delete pod polkadot-private-node-0
```

Look at the logs and ensure each pod comes back online before deleting the next one.

NOTE: since these pods are part of Deployment/StatefulSet, kubernetes auto-restarts them when you delete them. Their storage is persisted so they restart where they left off.

## Wrapping up

To delete everything and terminate all the charges, issue the command:

```
terraform destroy
```

Alternatively, go to the GCP console and delete the project.
23 changes: 23 additions & 0 deletions charts/polkadot/.helmignore
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
Loading

0 comments on commit f6207a4

Please sign in to comment.