From 40f91db672363a14309e2ca89ff80882220875a3 Mon Sep 17 00:00:00 2001 From: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> Date: Fri, 1 Nov 2024 14:29:40 +0000 Subject: [PATCH] docs: Add manual steps to rotate encryption keys and move encryption docs to separate topic Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> --- src/data/nav.yaml | 6 +- .../ecosystem/ecosystem-installing-k8s.md | 35 ----- .../ecosystem-managing-encryption-keys.md | 131 ++++++++++++++++++ 3 files changed, 135 insertions(+), 37 deletions(-) create mode 100644 src/markdown-pages/docs/ecosystem/ecosystem-managing-encryption-keys.md diff --git a/src/data/nav.yaml b/src/data/nav.yaml index cd7012ba..d36086b5 100644 --- a/src/data/nav.yaml +++ b/src/data/nav.yaml @@ -99,9 +99,11 @@ - title: Configuring an Ecosystem using resource files path: /docs/ecosystem/resources-yaml - title: Managing credentials in an Ecosystem - path: /docs/ecosystem/ecosystem-manage-creds + path: /docs/ecosystem/ecosystem-manage-creds - title: Managing configuration properties - path: /docs/ecosystem/ecosystem-manage-cps + path: /docs/ecosystem/ecosystem-manage-cps + - title: Managing Ecosystem encryption keys + path: /docs/ecosystem/ecosystem-managing-encryption-keys - title: Managing tests in a Galasa Ecosystem path: /docs/manage-ecosystem items: diff --git a/src/markdown-pages/docs/ecosystem/ecosystem-installing-k8s.md b/src/markdown-pages/docs/ecosystem/ecosystem-installing-k8s.md index bb6d0508..5e05e052 100644 --- a/src/markdown-pages/docs/ecosystem/ecosystem-installing-k8s.md +++ b/src/markdown-pages/docs/ecosystem/ecosystem-installing-k8s.md @@ -236,41 +236,6 @@ where:
- `galasaVersion` is set to the version that you want to use and
- `` is the name that you gave to the Ecosystem during installation -## Rotating Encryption Keys - -To maintain the security of your Galasa Ecosystem, you may wish to replace the encryption key being used to encrypt credentials in the Galasa Ecosystem's credentials store with a new encryption key, and re-encrypt all your existing credentials using the new key. - -**Before proceeding to rotate encryption keys, it is highly recommended to make a backup of the existing credentials stored in your Galasa Ecosystem by running the following command using the Galasa CLI tool:** - -```console -galasactl secrets get --format yaml > /path/to/backup/file.yaml -``` - -**where `/path/to/backup/file.yaml` is either an absolute or relative path of your choice to a file where the backup will be stored.** - -For Linux and macOS, you can download and run the rotate-encryption-keys.sh script via the command-line to simplify the process of rotating encryption keys and re-encrypting credentials. - -This script requires the following command-line utilities to be installed: - -- [kubectl](https://kubernetes.io/docs/tasks/tools) (v1.30.3 or later) -- [galasactl](../cli-command-reference/installing-cli-tool.md) (0.38.0 or later) -- openssl (3.3.2 or later) - -The following flags can be supplied when running the script: -- `--release-name ` **Required**. The helm release name provided when installing the Galasa Ecosystem helm chart (see [Installing your Galasa Ecosystem](#installing-your-galasa-ecosystem)) -- `--namespace ` Optional. The Kubernetes namespace where your Galasa Ecosystem is installed -- `--bootstrap ` Optional. The bootstrap URL of the Galasa Ecosystem that is being serviced. Not required if the `GALASA_BOOTSTRAP` environment variable is set and is pointing to the correct bootstrap URL. Overrides the existing `GALASA_BOOTSTRAP` environment variable value if set - -For example: - -```console -./rotate-encryption-keys.sh --release-name example --namespace galasa-dev -``` - -The `rotate-encryption-keys.sh` script will automatically update the current encryption key with a new one, and then restart your Galasa Ecosystem's API and engine controller pods so that they can pick up the new encryption key. After rotating the encryption keys, the script will re-encrypt the existing secrets in your Galasa Ecosystem using the newly activated encryption key. - -Once the encryption keys have been rotated and the existing secrets have been re-encrypted, the script will clear the fallback decryption keys list and restart the API and engine controller pods for a final time to keep the Galasa services in sync with the contents of the encryption keys secret. - ### Troubleshooting - Check the logs by running the ```kubectl logs``` command. diff --git a/src/markdown-pages/docs/ecosystem/ecosystem-managing-encryption-keys.md b/src/markdown-pages/docs/ecosystem/ecosystem-managing-encryption-keys.md new file mode 100644 index 00000000..32d3741a --- /dev/null +++ b/src/markdown-pages/docs/ecosystem/ecosystem-managing-encryption-keys.md @@ -0,0 +1,131 @@ +--- +path: "/docs/ecosystem/ecosystem-managing-encryption-keys" +title: "Managing Ecosystem encryption keys" +--- + +To maintain the security of your Galasa Ecosystem, you may wish to replace the encryption key being used to encrypt credentials in the Galasa Ecosystem's credentials store with a new encryption key, and re-encrypt all your existing credentials using the new key. + +**Before proceeding to rotate encryption keys, it is highly recommended to make a backup of the existing credentials stored in your Galasa Ecosystem by running the following command using the Galasa CLI tool:** + +```console +galasactl secrets get --format yaml > /path/to/backup/file.yaml +``` + +**where `/path/to/backup/file.yaml` is either an absolute or relative path of your choice to a file where the backup will be stored.** + +## Prerequisites + +The following command-line utilities must be installed: + +- [kubectl](https://kubernetes.io/docs/tasks/tools) (v1.30.3 or later) +- [galasactl](../cli-command-reference/installing-cli-tool.md) (0.38.0 or later) +- openssl (3.3.2 or later) + +You must have the correct permissions to read and update Kubernetes Secrets in the Kubernetes namespace that you have installed your Galasa Ecosystem within. + +You must also have a valid personal access token for your Galasa Ecosystem set on your machine so that you can retrieve and update secrets with `galasactl`. + +## Automated steps + +For Linux and macOS, you can download and run the rotate-encryption-keys.sh script via the command-line to simplify the process of rotating encryption keys and re-encrypting credentials. + +This script requires the following command-line utilities to be installed: + +- [kubectl](https://kubernetes.io/docs/tasks/tools) (v1.30.3 or later) +- [galasactl](../cli-command-reference/installing-cli-tool.md) (0.38.0 or later) +- openssl (3.3.2 or later) + +The following flags can be supplied when running the script: +- `--release-name ` **Required**. The helm release name provided when installing the Galasa Ecosystem helm chart (see [Installing your Galasa Ecosystem](#installing-your-galasa-ecosystem)) +- `--namespace ` Optional. The Kubernetes namespace where your Galasa Ecosystem is installed +- `--bootstrap ` Optional. The bootstrap URL of the Galasa Ecosystem that is being serviced. Not required if the `GALASA_BOOTSTRAP` environment variable is set and is pointing to the correct bootstrap URL. Overrides the existing `GALASA_BOOTSTRAP` environment variable value if set + +For example: + +```console +./rotate-encryption-keys.sh --release-name example --namespace galasa-dev +``` + +The `rotate-encryption-keys.sh` script will automatically update the current encryption key with a new one, and then restart your Galasa Ecosystem's API and engine controller pods so that they can pick up the new encryption key. After rotating the encryption keys, the script will re-encrypt the existing secrets in your Galasa Ecosystem using the newly activated encryption key. + +Once the encryption keys have been rotated and the existing secrets have been re-encrypted, the script will clear the fallback decryption keys list and restart the API and engine controller pods for a final time to keep the Galasa services in sync with the contents of the encryption keys secret. + +## Manual steps + +If the `rotate-encryption-keys.sh` script fails or if you are on Windows and cannot run the script, you can manually rotate encryption keys by performing the following steps. + +This guide assumes `kubectl` is configured to manage resources in the namespace that your Galasa Ecosystem is installed within, so the `--namespace` flag is not required when running `kubectl` commands. + +This guide also assumes that the `GALASA_BOOTSTRAP` environment variable is set, so the `--bootstrap` flag is not required when running `galasactl` commands. + +1. Get all the existing secrets from your Galasa Ecosystem by running: + ``` + galasactl secrets get --format yaml + ``` + Store these secrets in a `.yaml` or `.yml` file so that you can re-encrypt them after rotating the encryption keys + +2. Find the name of the Kubernetes Secret containing your Galasa Ecosystem's encryption keys by running: + ``` + kubectl get secrets + ``` + The secret's name should be of the form `{release-name}-encryption-secret`, where `{release-name}` is the Helm release name provided when installing the Galasa Ecosystem Helm chart + +3. Get the existing encryption keys data for your Galasa Ecosystem by running: + ``` + kubectl get secret {encryption-secret-name} --output jsonpath='{ .data.encryption-keys\.yaml }' | openssl base64 -d -A + ``` + where `{encryption-secret-name}` is the name of the Kubernetes secret retrieved in step 2. The output should look like the following: + ```yaml + encryptionKey: + fallbackDecryptionKeys: [] + ``` + Place the output into a file + +4. Generate a new encryption key by running: + ``` + openssl rand -base64 32 + ``` + +5. In the file created at the end of step 3, move the existing `encryptionKey` value into the `fallbackDecryptionKeys` list and place the newly generated encryption key into the `encryptionKey` field. The file should now look like this: + ```yaml + encryptionKey: + fallbackDecryptionKeys: + - + ``` + where `` is the new encryption key generated in step 5 and `` is the old encryption key retrieved in step 3 + +6. Base64-encode the file contents by running: + ``` + openssl base64 -in + ``` + Record the base64-encoded output, making sure there are no spaces or line breaks in the recorded output + +7. Update the existing Kubernetes Secret with the rotated keys by running: + ``` + kubectl patch secret {encryption-secret-name} --type='json' -p="[{'op': 'replace', 'path': '/data/encryption-keys.yaml', 'value': ''}]" + ``` + where `` is the output recorded from step 7 + +8. Restart the Galasa Ecosystem's API server deployment by running: + ``` + kubectl rollout restart deployment {release-name}-api + kubectl rollout status deployment {release-name}-api + ``` + where `{release-name}` is the name of the Helm release provided when installing the Galasa Ecosystem Helm chart + +9. Restart the Galasa Ecosystem's engine controller deployment by running: + ``` + kubectl rollout restart deployment {release-name}-engine-controller + kubectl rollout status deployment {release-name}-engine-controller + ``` + where `{release-name}` is the name of the Helm release provided when installing the Galasa Ecosystem Helm chart + +10. Once both the API server and engine controller have been restarted successfully, you can re-encrypt your existing secrets using the YAML file you created in step 1, by running: + ``` + galasactl resources apply -f + ``` + where `` is an absolute or relative path to the YAML file created at the end of step 1 + +Your Galasa Ecosystem will now use the newly generated encryption key to encrypt and decrypt secrets until the next time it is rotated. + +To verify that your secrets can still be read correctly, you can run `galasactl secrets get --format yaml` again and compare the YAML output with the content of YAML file that you applied in step 10. If the output is the same, then the secrets have been re-encrypted successfully.