Skip to content

Commit

Permalink
Merge pull request #130 from celo-org/jcortejoso/testnet-refactor
Browse files Browse the repository at this point in the history
Testnet refactor to make it compatible with existing networks
  • Loading branch information
jcortejoso authored Aug 30, 2023
2 parents 5189773 + c77ce14 commit 711f9d7
Show file tree
Hide file tree
Showing 17 changed files with 338 additions and 67 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
**/*.tgz
**/charts/
charts/*/charts/
Chart.lock
requirements.lock
.DS_Store
Expand Down
4 changes: 2 additions & 2 deletions charts/testnet/Chart.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: testnet
apiVersion: v2
version: 0.3.2
version: 0.4.0
description: Private Celo network Helm chart for Kubernetes
home: https://clabs.co
sources:
Expand All @@ -26,4 +26,4 @@ appVersion: v1.0.0
dependencies:
- name: common
repository: oci://us-west1-docker.pkg.dev/devopsre/clabs-public-oci
version: 0.3.2
version: 0.4.0
25 changes: 20 additions & 5 deletions charts/testnet/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# testnet

![Version: 0.3.2](https://img.shields.io/badge/Version-0.3.2-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v1.0.0](https://img.shields.io/badge/AppVersion-v1.0.0-informational?style=flat-square)
![Version: 0.4.0](https://img.shields.io/badge/Version-0.4.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v1.0.0](https://img.shields.io/badge/AppVersion-v1.0.0-informational?style=flat-square)

Private Celo network Helm chart for Kubernetes

Expand All @@ -23,7 +23,7 @@ Private Celo network Helm chart for Kubernetes

| Repository | Name | Version |
|------------|------|---------|
| oci://us-west1-docker.pkg.dev/devopsre/clabs-public-oci | common | 0.3.2 |
| oci://us-west1-docker.pkg.dev/devopsre/clabs-public-oci | common | 0.4.0 |

## Values

Expand All @@ -33,12 +33,19 @@ Private Celo network Helm chart for Kubernetes
| blockscout.image.indexerTag | string | `"indexer"` | |
| blockscout.image.repository | string | `"gcr.io/celo-testnet/blockscout"` | |
| blockscout.image.webTag | string | `"web"` | |
| bootnode.bootnodeIpAddress | string | `""` | |
| bootnode.defaultBootnodeClusterIP | string | `"10.0.0.12"` | |
| bootnode.defaultClusterIP | string | `"10.0.0.12"` | |
| bootnode.image.repository | string | `"us.gcr.io/celo-testnet/geth-all"` | |
| bootnode.image.tag | string | `"21d8283af60927589566cb282ab640f1ccec6ebd"` | |
| celotool.image.repository | string | `"us.gcr.io/celo-testnet/celo-monorepo"` | |
| celotool.image.tag | string | `"celotool-dc5e5dfa07231a4ff4664816a95eae606293eae9"` | |
| dataSource.archive | object | `{}` | |
| dataSource.full | object | `{}` | |
| deletePodCronJob | object | `{"component":"tx-nodes","enabled":false,"extraFlagsPod":"","extraFlagsPvc":"","podIndex":0,"schedule":"0 10,22 * * *"}` | Enable a CronJob that will delete a pod of the statefulset to force flushing the data to disk |
| deletePodCronJob.component | string | `"tx-nodes"` | Component to delete. Valid values are validators, and tx-nodes |
| deletePodCronJob.extraFlagsPod | string | `""` | Extra cmd flags to pass to the delete pod command |
| deletePodCronJob.extraFlagsPvc | string | `""` | Extra cmd flags to pass to the delete pvc command |
| deletePodCronJob.podIndex | int | `0` | Statefulset index to delete |
| deletePodCronJob.schedule | string | `"0 10,22 * * *"` | Cron expression for the CronJob. As reference for mainnet, the sync speed is around ~2000 blocks/minute, with a blockTime of 5 seconds, 1 day are 17280 blocks (so one day of sync is around 9 minutes) |
| domain.name | string | `"celo-networks-dev"` | |
| enableBootnode | bool | `true` | |
| enableFornoIngress | bool | `true` | |
Expand Down Expand Up @@ -66,27 +73,35 @@ Private Celo network Helm chart for Kubernetes
| geth.pprof.enabled | bool | `false` | |
| geth.pprof.port | int | `6060` | |
| geth.privateTxNodediskSizeGB | int | `10` | |
| geth.proxyAffinity | object | `{}` | |
| geth.proxyExtraSnippet | string | `"echo \"Proxy\"\n"` | |
| geth.proxyNodeSelector | object | `{}` | |
| geth.proxySelector | object | `{}` | |
| geth.proxyTolerations | list | `[]` | |
| geth.resources.limits | object | `{}` | |
| geth.resources.requests.cpu | string | `"500m"` | |
| geth.resources.requests.memory | string | `"256Mi"` | |
| geth.rpc_gascap | int | `0` | |
| geth.secondaryAffinity | object | `{}` | |
| geth.secondaryNodeSelector | object | `{}` | |
| geth.secondaryTolerations | list | `[]` | |
| geth.secondayExtraSnippet | string | `"echo \"secondary-validator\"\n"` | |
| geth.static_ips | bool | `false` | |
| geth.storage | bool | `true` | |
| geth.storageClassName | string | `""` | |
| geth.txNodeAffinity | object | `{}` | |
| geth.txNodeExtraSnippet | string | `"echo \"txnode\"\n"` | |
| geth.txNodeNodeSelector | object | `{}` | |
| geth.txNodePrivateAffinity | object | `{}` | |
| geth.txNodePrivateExtraSnippet | string | `"echo \"txnode-private\"\nADDITIONAL_FLAGS=\"${ADDITIONAL_FLAGS} --http.timeout.read 600 --http.timeout.write 600 --http.timeout.idle 2400\"\n"` | |
| geth.txNodePrivateNodeSelector | object | `{}` | |
| geth.txNodePrivateTolerations | list | `[]` | |
| geth.txNodeTolerations | list | `[]` | |
| geth.txNodesIPAddressArray[0] | string | `"1.2.3.4"` | |
| geth.validatorAffinity | object | `{}` | |
| geth.validatorExtraSnippet | string | `"echo \"Validator\"\n"` | |
| geth.validatorNodeSelector | object | `{}` | |
| geth.validatorTolerations | list | `[]` | |
| geth.validatorsIPAddressArray | list | `[]` | |
| geth.verbosity | int | `1` | |
| geth.vmodule | string | `"consensus/*=2"` | |
| geth.ws_port | int | `8546` | |
Expand Down
3 changes: 2 additions & 1 deletion charts/testnet/ci/test-values.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
bootnode:
defaultBootnodeClusterIP: 10.96.12.12
defaultClusterIP: 10.96.12.12
replicas:
validators: 1
proxiesPerValidator:
Expand Down
55 changes: 47 additions & 8 deletions charts/testnet/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -120,21 +120,25 @@ spec:
{{- toYaml .pvc_annotations | nindent 8 }}
{{- end }}
spec:
{{- if $.Values.geth.storageClass }}
storageClassName: {{ $.Values.geth.storageClass }}
{{- with $.Values.geth.storageClassName }}
storageClassName: {{ . }}
{{- end }}
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
{{- $disk_size := ternary .Values.geth.privateTxNodediskSizeGB .Values.geth.diskSizeGB (eq .name "tx-nodes-private") }}
storage: {{ $disk_size }}Gi
{{- with .dataSource }}
dataSource:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
podManagementPolicy: Parallel
replicas: {{ .replicas }}
serviceName: {{ .name }}
selector:
matchLabels:
{{- include "common.standard.labels" . | nindent 6 }}
{{- include "common.standard.short_labels" . | nindent 6 }}
component: {{ .component_label }}
{{- if .proxy | default false }}
{{- $validatorProxied := printf "%s-validators-%d" .Release.Namespace .validator_index }}
Expand All @@ -143,7 +147,7 @@ spec:
template:
metadata:
labels:
{{- include "common.standard.labels" . | nindent 8 }}
{{- include "common.standard.short_labels" . | nindent 8 }}
component: {{ .component_label }}
{{- if .extraPodLabels -}}
{{- toYaml .extraPodLabels | nindent 8 }}
Expand All @@ -152,20 +156,55 @@ spec:
{{- $validatorProxied := printf "%s-validators-%d" .Release.Namespace .validator_index }}
validator-proxied: "{{ $validatorProxied }}"
{{- end }}
{{- if .Values.metrics | default false }}
annotations:
clabs.co/images: "{{ .Values.geth.image.repository }}:{{ .Values.geth.image.tag }}; {{ .Values.celotool.image.repository }}:{{ .Values.celotool.image.tag }}"
{{- include "common.prometheus-annotations" (dict "pprof" .Values.geth.pprof ) | nindent 8 }}
{{- end }}
spec:
initContainers:
{{- include "common.conditional-init-genesis-container" . | nindent 6 }}
{{- include "common.celotool-full-node-statefulset-container" (dict "Values" .Values "Release" .Release "Chart" .Chart "proxy" .proxy "mnemonic_account_type" .mnemonic_account_type "service_ip_env_var_prefix" .service_ip_env_var_prefix "ip_addresses" .ip_addresses "validator_index" .validator_index) | nindent 6 }}
{{- include "common.celotool-full-node-statefulset-container" (dict
"Values" .Values
"Release" .Release
"Chart" .Chart
"proxy" .proxy
"mnemonic_account_type" .mnemonic_account_type
"service_ip_env_var_prefix" .service_ip_env_var_prefix
"ip_addresses" .ip_addresses
"validator_index" .validator_index
) | nindent 6 }}
{{- if .unlock | default false }}
{{- include "common.import-geth-account-container" . | nindent 6 }}
{{- end }}
containers:
{{- include "common.full-node-container" (dict "Values" .Values "Release" .Release "Chart" .Chart "proxy" .proxy "proxy_allow_private_ip_flag" .proxy_allow_private_ip_flag "unlock" .unlock "rpc_apis" .rpc_apis "expose" .expose "syncmode" .syncmode "gcmode" .gcmode "ws_port" (default .Values.geth.ws_port .ws_port) "pprof" (or (.Values.metrics) (.Values.geth.pprof.enabled)) "pprof_port" (.Values.geth.pprof.port) "light_serve" .Values.geth.light.serve "light_maxpeers" .Values.geth.light.maxpeers "maxpeers" .Values.geth.maxpeers "metrics" .Values.metrics "public_ips" .public_ips "ethstats" (printf "%s-ethstats.%s" (include "common.fullname" .) .Release.Namespace) "extra_setup" .extra_setup) | nindent 6 }}
{{- include "common.full-node-container" (dict
"Values" .Values
"Release" .Release
"Chart" .Chart
"proxy" .proxy
"proxy_allow_private_ip_flag" .proxy_allow_private_ip_flag
"unlock" .unlock
"rpc_apis" .rpc_apis
"expose" .expose
"rcp_gascap" (default (int .Values.geth.rpc_gascap) (int .rcp_gascap))
"syncmode" .syncmode
"gcmode" .gcmode
"resources" .resources
"ws_port" (default .Values.geth.ws_port .ws_port)
"pprof" (or (.Values.metrics) (.Values.geth.pprof.enabled))
"pprof_port" (.Values.geth.pprof.port)
"light_serve" .Values.geth.light.serve
"light_maxpeers" .Values.geth.light.maxpeers
"maxpeers" .Values.geth.maxpeers
"metrics" .Values.metrics
"public_ips" .public_ips
"ethstats" (printf "%s-ethstats.%s" (include "common.fullname" .) .Release.Namespace)
"extra_setup" .extra_setup
) | nindent 6 }}
terminationGracePeriodSeconds: {{ .Values.geth.terminationGracePeriodSeconds | default 300 }}
{{- with .affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .node_selector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
Expand Down
10 changes: 5 additions & 5 deletions charts/testnet/templates/bootnode.deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ spec:
exec /usr/local/bin/bootnode --nodekey=/etc/bootnode/node.key --verbosity=5 ${NAT_FLAG} --networkid=${NETWORK_ID} --ping-ip-from-packet={{ .Values.geth.ping_ip_from_packet }}
env:
- name: IP_ADDRESS
value: {{ default "none" .Values.geth.bootnodeIpAddress }}
value: {{ default $.Values.bootnode.defaultClusterIP $.Values.bootnode.ipAddress }}
- name: NETWORK_ID
valueFrom:
configMapKeyRef:
Expand All @@ -52,7 +52,7 @@ spec:
initialDelaySeconds: 30
periodSeconds: 30
volumeMounts:
{{- if not .Values.geth.overwriteBootnodePrivateKey }}
{{- if not .Values.geth.overwriteprivateKey }}
- name: data
mountPath: /etc/bootnode
{{- else }}
Expand All @@ -64,7 +64,7 @@ spec:
- name: discovery
containerPort: 30301
protocol: UDP
{{- if not .Values.geth.overwriteBootnodePrivateKey }}
{{- if not .Values.geth.overwriteprivateKey }}
initContainers:
- name: generate-node-key
image: {{ .Values.celotool.image.repository }}:{{ .Values.celotool.image.tag }}
Expand All @@ -88,12 +88,12 @@ spec:
volumes:
- name: data
emptyDir: {}
{{- if .Values.geth.overwriteBootnodePrivateKey }}
{{- if .Values.bootnode.privateKey }}
- name: bootnode-pkey
secret:
secretName: {{ template "common.fullname" . }}-geth-account
items:
- key: bootnodePrivateKey
- key: privateKey
path: node.key
{{- end }}
{{- with .Values.nodeSelector }}
Expand Down
6 changes: 3 additions & 3 deletions charts/testnet/templates/bootnode.service.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@ metadata:
{{- include "common.standard.labels" . | nindent 4 }}
component: bootnode
spec:
{{- if $.Values.bootnode.bootnodeIpAddress }}
{{- if $.Values.bootnode.ipAddress }}
type: LoadBalancer
loadBalancerIP: {{ $.Values.bootnode.bootnodeIpAddress }}
loadBalancerIP: {{ $.Values.bootnode.ipAddress }}
{{- else }}
type: ClusterIP
clusterIP: {{ default $.Values.bootnode.defaultBootnodeClusterIP $.Values.bootnode.bootnodeIpAddress }}
clusterIP: {{ default $.Values.bootnode.defaultClusterIP $.Values.bootnode.ipAddress }}
{{- end }}
selector:
app: {{ template "common.name" . }}
Expand Down
110 changes: 110 additions & 0 deletions charts/testnet/templates/delete-pod-cronjob.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
{{- if .Values.deletePodCronJob.enabled -}}
{{- /*
This CronJob in intended to delete regularly a geth pod
in order to force geth to flush the data to disk, so it can
be used as an snapshot
*/}}
apiVersion: batch/v1
kind: CronJob
metadata:
labels:
{{- include "common.standard.labels" . | nindent 4 }}
component: restart-geth
name: {{ template "common.fullname" . }}-restart-geth
spec:
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 1
jobTemplate:
spec:
backoffLimit: 1
template:
metadata:
labels:
{{- include "common.standard.labels" . | nindent 12 }}
component: restart-geth
spec:
containers:
- name: restart-geth
command:
- /bin/bash
- -c
args:
- |
# Delete pvc that are not currently used
extraFlagsPvc="{{ .Values.deletePodCronJob.extraFlagsPvc }}"
unused_pvcs="$(kubectl describe pvc | grep -E "^Name:.*$|^Namespace:.*$|^Used By:.*$" | grep -B 2 "<none>" | grep -E "^Name:.*$")"
while IFS= read -r line; do
pvc=$(echo $line | awk '{print $2}')
echo "Deleting pvc $pvc"
kubectl delete pvc -n {{ .Release.Namespace }} $pvc $extraFlagsPvc
done <<< "$unused_pvcs"
# Delete the pod to force geth to flush the data to disk
extraFlagsPod="{{ .Values.deletePodCronJob.extraFlagsPod }}"
kubectl delete pod -n {{ .Release.Namespace }} {{ printf "%s-%s-%d" (include "common.fullname" .) .Values.deletePodCronJob.component (.Values.deletePodCronJob.podIndex | int) }} $extraFlagsPod
image: bitnami/kubectl:latest
imagePullPolicy: Always
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Never
schedulerName: default-scheduler
serviceAccountName: {{ template "common.fullname" . }}-restart-geth
terminationGracePeriodSeconds: 30
schedule: "{{ .Values.deletePodCronJob.schedule }}"
successfulJobsHistoryLimit: 3
suspend: false
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
{{- include "common.standard.labels" . | nindent 4 }}
component: restart-geth
name: {{ template "common.fullname" . }}-restart-geth
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
{{- include "common.standard.labels" . | nindent 4 }}
component: restart-geth
name: {{ template "common.fullname" . }}-restart-geth
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- pods
- persistentvolumes
- persistentvolumeclaims
- persistentvolumeclaims/status
verbs:
- get
- list
- watch
- delete
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
{{- include "common.standard.labels" . | nindent 4 }}
component: restart-geth
name: {{ template "common.fullname" . }}-restart-geth
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ template "common.fullname" . }}-restart-geth
subjects:
- kind: ServiceAccount
name: {{ template "common.fullname" . }}-restart-geth
namespace: {{ .Release.Namespace }}
{{- end }}
4 changes: 2 additions & 2 deletions charts/testnet/templates/geth-account.secret.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,6 @@ type: Opaque
data:
accountSecret: {{ .Values.geth.account.secret | b64enc }}
mnemonic: {{ .Values.mnemonic | b64enc }}
{{- if .Values.geth.overwriteBootnodePrivateKey }}
bootnodePrivateKey: {{ .Values.geth.bootnodePrivateKey | b64enc }}
{{- if .Values.bootnode.privateKey }}
privateKey: {{ .Values.bootnode.privateKey | b64enc }}
{{- end }}
Loading

0 comments on commit 711f9d7

Please sign in to comment.