Deploy to prod (#418)

* add janky tf * helm * actions * actions * suppress * cleanup * bug fix * testing infra updates * updates * fix cd * pin action * helm docs * pin action * wording * explain * leanr tooo speil * typo * rm extra file * rm debug stuff * typo * docs * Update infra/README.md Co-authored-by: Willard Nilges <[email protected]> * Update infra/tf/main.tf Co-authored-by: Willard Nilges <[email protected]> * simplify step 1 * updates * janky stage2 provisioner * docs * local provisioner for ssh key * Update infra/tf/gen_ssh_key.sh Co-authored-by: Willard Nilges <[email protected]> * fixes * Provision more stuff locally * delete scripts * add gitignore * Update README * Use example values files This should avoid accidentally committing secrets * Theoretically create k3s config Haven't tested this * Comment out (what ought to be) dead code * uhhhhhhhhhhhhhh * holy shit it fucking works * delete dead code * Get kubeconfig * First attempt at locally rendering * Duct tape the package installations Sleep 60... eugh. * Theoretically try to install packages for two min Haven't tested this, but this should be a little better than just sleeping the flat 60 secs * Update some stuff I don't think package management should be done through TF * Terraform can create our inventory * You can have a little jank, as a treat * Remind me to add that later * Move cluster into helm chart * oops * installed with helm lol * fuck * Add ingress * updates * rm unused * ansible * updates * updates * updates * make it work * updates * updates * make me a sandwitch * pin actions * Infra * spacing * helm values * helm values * helm values * helm values * helm values * helm values * pgadmin + temp service * pg service port * pg admin path * pg admin path * pg admin path * roll out * name * good eye willard * good eye willard * good eye willard * docs * good eye willard * examples not needed, they are here * good eye willard * good eye willard * fix * fix * import * fix url * fix * branch name * needs --------- Co-authored-by: Willard Nilges <[email protected]>
nycmeshnet · Jul 28, 2024 · 45477bb · 45477bb
1 parent 7211c3d
commit 45477bb
Show file tree

Hide file tree

Showing 27 changed files with 1,259 additions and 5 deletions.
diff --git a/.github/workflows/checkov.yaml b/.github/workflows/checkov.yaml
@@ -3,6 +3,8 @@ name: Checkov
 on:
   push:
     branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
   workflow_dispatch:
 
 permissions: read-all
@@ -24,14 +26,14 @@ jobs:
         id: checkov
         uses: bridgecrewio/checkov-action@0549dc60bddd4c55cb85c6c3a07072e3cf2ca48e
         with:
-          skip_check: CKV_DOCKER_2,CKV_DOCKER_3
+          skip_check: CKV_DOCKER_2,CKV_DOCKER_3,CKV_SECRET_6
           quiet: true
           output_format: cli,sarif
           output_file_path: console,results.sarif
           download_external_modules: true
 
       - name: Upload SARIF file
-        uses: github/codeql-action/upload-sarif@v3
+        uses: github/codeql-action/upload-sarif@d39d31e687223d841ef683f52467bd88e9b21c14 # v3
         if: success() || failure()
         with:
           sarif_file: results.sarif
diff --git a/.github/workflows/deploy_dev3.yaml b/.github/workflows/deploy_dev3.yaml
@@ -0,0 +1,63 @@
+name: Deploy Dev 3
+
+on:
+  push:
+    branches: [ main ]
+
+permissions: read-all
+
+jobs:
+    deploy_to_dev3:
+      name: Deploy to dev3
+      needs: push_to_registry
+      environment: dev3
+      runs-on: ubuntu-latest
+      if: github.ref == 'refs/heads/main'
+      steps:
+      - name: Check out the repo
+        uses: actions/checkout@v4
+
+      - name: Install SSH key
+        uses: shimataro/ssh-key-action@d4fffb50872869abe2d9a9098a6d9c5aa7d16be4 # v2
+        with:
+          key: ${{ secrets.SSH_PRIVATE_KEY }}
+          name: id_ed25519 # optional
+          known_hosts: ${{ secrets.SSH_KNOWN_HOSTS }}
+          if_key_exists: fail # replace / ignore / fail; optional (defaults to fail)
+
+      - name: Setup WireGuard
+        run:  |
+          sudo apt install wireguard
+          echo "${{ secrets.WIREGUARD_PRIVATE_KEY }}" > privatekey
+          sudo ip link add dev wg1 type wireguard
+          sudo ip address add dev wg1 ${{ secrets.WIREGUARD_OVERLAY_NETWORK_IP }} peer ${{ secrets.SSH_TARGET_IP }}
+          sudo wg set wg1 listen-port 48123 private-key privatekey peer ${{ secrets.WIREGUARD_PEER_PUBLIC_KEY }} allowed-ips 0.0.0.0/0 endpoint ${{ secrets.WIREGUARD_ENDPOINT }}
+          sudo ip link set up dev wg1
+
+      - name: Deploy Helm Chart
+        run: |
+          ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_TARGET_IP }} "\
+            sudo bash -c '\
+              cd ${{ secrets.PROJECT_PATH }} && \
+              git pull && \
+              git checkout main && \
+              cd infra/helm/meshdb && \
+              helm template . -f values.yaml \
+                --set meshdb_app_namespace=\"${{ vars.APP_NAMESPACE }}\" \
+                --set nginx.server_name=\"${{ vars.SERVER_NAME }}\" \
+                --set pg.password=\"${{ secrets.PG_PASSWORD}}\" \
+                --set aws.access_key_id=\"${{ secrets.ACCESS_KEY_ID }}\" \
+                --set aws.secret_access_key=\"${{ secrets.SECRET_ACCESS_KEY }}\" \
+                --set meshweb.backup_s3_base_folder=\"${{ vars.BACKUP_S3_BASE_FOLDER }}\" \
+                --set meshweb.backup_s3_bucket_name=\"${{ secrets.BACKUP_S3_BUCKET_NAME }}\" \
+                --set meshweb.django_secret_key=\"${{ secrets.DJANGO_SECRET_KEY }}\" \
+                --set meshweb.nn_assign_psk=\"${{ secrets.NN_ASSIGN_PSK }}\" \
+                --set meshweb.query_psk=\"${{ secrets.QUERY_PSK }}\" \
+                --set meshweb.pano_github_token=\"${{ secrets.GH_TOKEN }}\" \
+                --set pgadmin.default_email=\"${{ secrets.PGADMIN_EMAIL }}\" \
+                --set pgadmin.default_password=\"${{ secrets.PGADMIN_PASSWORD }}\" \
+                --set uisp.user=\"${{ secrets.UISP_USER }}\" \
+                --set uisp.psk=\"${{ secrets.UISP_PSK }}\" \
+                | kubectl apply -f - && \
+              kubectl -n ${{ vars.APP_NAMESPACE }} rollout restart deploy \
+            '"
diff --git a/.github/workflows/deploy_prod1.yaml b/.github/workflows/deploy_prod1.yaml
@@ -0,0 +1,64 @@
+name: Deploy Prod 1
+
+on:
+  push:
+    branches: [ main ]
+
+permissions: read-all
+
+jobs:
+    deploy_to_prod1:
+      name: Deploy to prod 1
+      needs: push_to_registry
+      environment: prod
+      runs-on: ubuntu-latest
+      if: github.ref == 'refs/heads/main'
+      steps:
+      - name: Check out the repo
+        uses: actions/checkout@v4
+
+      - name: Install SSH key
+        uses: shimataro/ssh-key-action@d4fffb50872869abe2d9a9098a6d9c5aa7d16be4 # v2
+        with:
+          key: ${{ secrets.SSH_PRIVATE_KEY }}
+          name: id_ed25519 # optional
+          known_hosts: ${{ secrets.SSH_KNOWN_HOSTS }}
+          if_key_exists: fail # replace / ignore / fail; optional (defaults to fail)
+
+      - name: Setup WireGuard
+        run:  |
+          sudo apt install wireguard
+          echo "${{ secrets.WIREGUARD_PRIVATE_KEY }}" > privatekey
+          sudo ip link add dev wg1 type wireguard
+          sudo ip address add dev wg1 ${{ secrets.WIREGUARD_OVERLAY_NETWORK_IP }} peer ${{ secrets.SSH_TARGET_IP }}
+          sudo wg set wg1 listen-port 48123 private-key privatekey peer ${{ secrets.WIREGUARD_PEER_PUBLIC_KEY }} allowed-ips 0.0.0.0/0 endpoint ${{ secrets.WIREGUARD_ENDPOINT }}
+          sudo ip link set up dev wg1
+
+      - name: Deploy Helm Chart
+        run: |
+          ssh ${{ secrets.SSH_USER }}@${{ secrets.SSH_TARGET_IP }} "\
+            sudo bash -c '\
+              cd ${{ secrets.PROJECT_PATH }} && \
+              git pull && \
+              git checkout main && \
+              cd infra/helm/meshdb && \
+              helm template . -f values.yaml \
+                --set meshdb_app_namespace=\"${{ vars.APP_NAMESPACE }}\" \
+                --set nginx.server_name=\"${{ vars.SERVER_NAME }}\" \
+                --set pg.password=\"${{ secrets.PG_PASSWORD}}\" \
+                --set aws.access_key_id=\"${{ secrets.ACCESS_KEY_ID }}\" \
+                --set aws.secret_access_key=\"${{ secrets.SECRET_ACCESS_KEY }}\" \
+                --set meshweb.backup_s3_base_folder=\"${{ vars.BACKUP_S3_BASE_FOLDER }}\" \
+                --set meshweb.backup_s3_bucket_name=\"${{ secrets.BACKUP_S3_BUCKET_NAME }}\" \
+                --set meshweb.django_secret_key=\"${{ secrets.DJANGO_SECRET_KEY }}\" \
+                --set meshweb.nn_assign_psk=\"${{ secrets.NN_ASSIGN_PSK }}\" \
+                --set meshweb.query_psk=\"${{ secrets.QUERY_PSK }}\" \
+                --set meshweb.pano_github_token=\"${{ secrets.GH_TOKEN }}\" \
+                --set pgadmin.default_email=\"${{ secrets.PGADMIN_EMAIL }}\" \
+                --set pgadmin.default_password=\"${{ secrets.PGADMIN_PASSWORD }}\" \
+                --set uisp.user=\"${{ secrets.UISP_USER }}\" \
+                --set uisp.psk=\"${{ secrets.UISP_PSK }}\" \
+                | kubectl apply -f - && \
+              kubectl -n ${{ vars.APP_NAMESPACE }} rollout restart deploy \
+            '"
+
diff --git a/.github/workflows/helm_lint.yaml b/.github/workflows/helm_lint.yaml
@@ -0,0 +1,47 @@
+name: Lint and Test Chart
+
+on: pull_request
+
+permissions: read-all
+
+jobs:
+  lint-test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+
+      - name: Set up Helm
+        uses: azure/setup-helm@20d2b4f98d41febe2bbca46408499dbb535b6258 # v3
+        with:
+          version: v3.14.0
+
+      - uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # v4
+        with:
+          python-version: '3.12'
+          check-latest: true
+
+      - name: Set up chart-testing
+        uses: helm/chart-testing-action@e6669bcd63d7cb57cb4380c33043eebe5d111992 # v2.6.1
+
+      - name: Run chart-testing (list-changed)
+        id: list-changed
+        run: |
+          changed=$(ct list-changed --target-branch ${{ github.event.repository.default_branch }})
+          if [[ -n "$changed" ]]; then
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Run chart-testing (lint)
+        if: steps.list-changed.outputs.changed == 'true'
+        run: ct lint --target-branch ${{ github.event.repository.default_branch }}
+
+      - name: Create kind cluster
+        if: steps.list-changed.outputs.changed == 'true'
+        uses: helm/[email protected]
+
+      - name: Run chart-testing (install)
+        if: steps.list-changed.outputs.changed == 'true'
+        run: ct install --target-branch ${{ github.event.repository.default_branch }}
diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml
@@ -10,7 +10,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # v4
         with:
           python-version: '3.11'
       - name: "Upgrade pip"

diff --git a/.github/workflows/no_forgoten_migrations.yaml b/.github/workflows/no_forgoten_migrations.yaml
@@ -25,7 +25,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # v4
         with:
           python-version: '3.11'
       - name: "Upgrade pip"

diff --git a/.github/workflows/run_django_tests.yaml b/.github/workflows/run_django_tests.yaml
@@ -37,7 +37,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@65d7f2d534ac1bc67fcd62888c5f4f3d2cb2b236 # v4
         with:
           python-version: '3.11'
       - name: "Upgrade pip"

diff --git a/infra/README.md b/infra/README.md
@@ -0,0 +1,41 @@
+# Meshdb Environment Setup
+
+These instructions will set up a 4 node k3s cluster on proxmox.
+- 1 "manager" node for control plane and to be used for deployments.
+- 3 "agent" nodes to run services.
+
+1. Setup a new cluster via [nycmeshnet/k8s-infra](https://github.com/nycmeshnet/k8s-infra). Get the ssh key of the mgr node via ssh-keyscan.
+
+2. Create a new "environment" in this repo and add the required secrets to the "environment":
+
+| Name    | Description |
+| -------- | ------- |
+| `ACCESS_KEY_ID` | Access key ID for s3 backups |
+| `SECRET_ACCESS_KEY` | Secret access key for s3 backups |
+| `BACKUP_S3_BUCKET_NAME` | Name of the s3 bucket to store backups |
+| `DJANGO_SECRET_KEY` | Django secret key |
+| `GH_TOKEN` | Github token for pulling down panoramas |
+| `NN_ASSIGN_PSK` | Legacy node number assign password |
+| `PGADMIN_EMAIL` | Default username for pgadmin |
+| `PGADMIN_PASSWORD` | Default password for pgadmin |
+| `PG_PASSWORD` | meshdb postgres database password |
+| `PROJECT_PATH`  |  Absolute file system path to the clone of meshdb, likely `/root/meshdb`  |
+| `QUERY_PSK` | Legacy query password |
+| `SSH_KNOWN_HOSTS`  |  Copy paste from `ssh-keyscan <mgr node IP>`  |
+| `SSH_PRIVATE_KEY`  | SSH key for the mgr node.   |
+| `SSH_TARGET_IP`  |  Mgr node IP  |
+| `SSH_USER`  | Mgr username for ssh   |
+| `UISP_PSK` | UISP readonly password |
+| `UISP_USER` | UISP readonly username |
+| `WIREGUARD_ENDPOINT`  | IP and port of the wireguard server for deployment in the format `<IP>:<Port>`   |
+| `WIREGUARD_OVERLAY_NETWORK_IP`  | Overlay network IP for wireguard server used for deployment   |
+| `WIREGUARD_PEER_PUBLIC_KEY`  | Public key of the wireguard server for deployment   |
+| `WIREGUARD_PRIVATE_KEY`  |  Private key for connecting to wireguard for deployment  |
+
+3. Create a new environment specific deployment workflow similar to `.github/workflows/deploy_prod1.yaml`
+
+4. Run the deployment.
+
+5. If you need a superuser, ssh into the mgr node and: `kubectl exec -it -n meshdbdev3 service/meshdb-meshweb bash` and then `python manage.py createsuperuser`
+
+6. If you need to import data: `cat meshdb_export.sql | kubectl exec -it --tty -n meshdbprod1 pod/meshdb-postgres-.... -- psql -U meshdb -d meshdb`
diff --git a/infra/helm/meshdb/.gitignore b/infra/helm/meshdb/.gitignore
@@ -0,0 +1,2 @@
+meshdb.yaml
+secret.values.yaml
diff --git a/infra/helm/meshdb/Chart.yaml b/infra/helm/meshdb/Chart.yaml
@@ -0,0 +1,9 @@
+apiVersion: v2
+name: meshdb
+description: A Helm chart for Kubernetes
+
+type: application
+
+version: 0.1.0
+
+appVersion: "1.16.0"
diff --git a/infra/helm/meshdb/README.md b/infra/helm/meshdb/README.md
@@ -0,0 +1,83 @@
+# meshdb
+
+A Helm chart for Kubernetes. Many meshweb variables are better described in the main README.
+
+## Values
+
+| Key | Type | Default | Description |
+|-----|------|---------|-------------|
+| autoscaling.enabled | bool | `false` |  |
+| autoscaling.maxReplicas | int | `100` |  |
+| autoscaling.minReplicas | int | `1` |  |
+| autoscaling.targetCPUUtilizationPercentage | int | `80` |  |
+| aws.access_key_id | string | `"the_real_value"` | AWS access key id for S3 |
+| aws.secret_access_key | string | `"the_real_value"` | AWS secret access key for S3 |
+| fullnameOverride | string | `"meshdb"` | App name |
+| image.pullPolicy | string | `"Always"` | pullPolicy for all images, should be `Always` |
+| map.base_url | string | `"http://admin-map.grandsvc.mesh.nycmesh.net"` | Map url |
+| meshdb_app_namespace | string | `"meshdb"` | K8s namespace used for all resources |
+| meshweb.affinity | object | `{}` |  |
+| meshweb.backup_s3_base_folder | string | `"meshdb-backups/development/"` | Base folder for django postgres backups |
+| meshweb.backup_s3_bucket_name | string | `"meshdb-data-backups"` | Bucket used for django postgres backups |
+| meshweb.disable_pano_edits | string | `"True"` | Feature flag for disabling panorama edits |
+| meshweb.disable_profiling | string | `"True"` | Disable profiling in meshweb |
+| meshweb.django_secret_key | string | `"the_real_value"` | Django secret key |
+| meshweb.enable_debug | string | `"False"` | Enable `DEBUG` in meshweb |
+| meshweb.image.repository | string | `"willnilges/meshdb"` | Docker image repo for meshweb |
+| meshweb.image.tag | string | `"main"` | Docker image tag for meshweb |
+| meshweb.liveness_probe | string | `"true"` | Enable liveness probe with `true` all other values will disable it |
+| meshweb.nn_assign_psk | string | `"the_real_value"` | Legacy NN assign form password |
+| meshweb.nodeSelector | object | `{}` |  |
+| meshweb.pano_github_token | string | `"the_real_value"` | Github token for downloading panorama |
+| meshweb.podSecurityContext | object | `{}` |  |
+| meshweb.port | int | `8081` | Port used by meshweb (internally) |
+| meshweb.query_psk | string | `"the_real_value"` | Legacy query form password |
+| meshweb.resources | object | `{}` |  |
+| meshweb.securityContext | object | `{}` |  |
+| meshweb.static_pvc_name | string | `"meshdb-static-pvc"` | Name of the PVC for static content |
+| meshweb.static_pvc_size | string | `"1Gi"` | Size of the PVC for static content |
+| meshweb.tolerations | list | `[]` |  |
+| nameOverride | string | `""` |  |
+| nginx.affinity | object | `{}` |  |
+| nginx.nodeSelector | object | `{}` |  |
+| nginx.podSecurityContext | object | `{}` |  |
+| nginx.port | int | `80` | Nginx port |
+| nginx.resources | object | `{}` |  |
+| nginx.securityContext | object | `{}` |  |
+| nginx.server_name | string | `"db.nycmesh.net"` | `server_name` used by nginx |
+| nginx.tolerations | list | `[]` |  |
+| pelias.affinity | object | `{}` |  |
+| pelias.nodeSelector | object | `{}` |  |
+| pelias.podSecurityContext | object | `{}` |  |
+| pelias.port | int | `6800` | Pelias port (internal) |
+| pelias.resources | object | `{}` |  |
+| pelias.securityContext | object | `{}` |  |
+| pelias.tolerations | list | `[]` |  |
+| pg.affinity | object | `{}` |  |
+| pg.dbname | string | `"meshdb"` | Postgres database name |
+| pg.liveness_probe | string | `"true"` | Enable liveness probe with `true` all other values will disable it |
+| pg.nodeSelector | object | `{}` |  |
+| pg.password | string | `"the_real_value"` | Password for postgres |
+| pg.podSecurityContext | object | `{}` |  |
+| pg.port | string | `"5432"` | Postgres port (internal) |
+| pg.pvc_name | string | `"meshdb-postgres-pvc"` | Name of the PVC for postgres |
+| pg.pvc_size | string | `"20Gi"` | Size of the PVC for postgres |
+| pg.resources | object | `{}` |  |
+| pg.securityContext | object | `{}` |  |
+| pg.tolerations | list | `[]` |  |
+| pg.user | string | `"meshdb"` | Postgres user |
+| podAnnotations | object | `{}` |  |
+| podLabels | object | `{}` |  |
+| redis.affinity | object | `{}` |  |
+| redis.liveness_probe | string | `"true"` | Enable liveness probe with `true` all other values will disable it |
+| redis.nodeSelector | object | `{}` |  |
+| redis.podSecurityContext | object | `{}` |  |
+| redis.port | int | `6379` | Redis port (internal) |
+| redis.resources | object | `{}` |  |
+| redis.securityContext | object | `{}` |  |
+| redis.tolerations | list | `[]` |  |
+| uisp.psk | string | `"the_real_value"` | Password for UISP |
+| uisp.url | string | `"https://uisp.mesh.nycmesh.net/nms"` | UISP url |
+| uisp.user | string | `"nycmesh_readonly"` | Username for UISP |
+
+----------------------------------------------