From e91083243659a502d1b40e7fde1430dd49b12827 Mon Sep 17 00:00:00 2001 From: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> Date: Mon, 4 Nov 2024 13:11:26 +0000 Subject: [PATCH] Add script to rotate encryption keys, update README with docs on rotating keys (#54) * feat: Add script to rotate encryption keys Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Add docs on how to rotate encryption keys Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * feat: Add logs to log file and more error handling Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Update encryption keys rotation in the README Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Add recommendation to backup secrets Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * fix: Change galasactl-dev to galasactl Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Remove unused flag from rotate-encryption-keys script Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> * docs: Add manual steps to rotate encryption keys Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> --------- Signed-off-by: Eamonn Mansour <47121388+eamansour@users.noreply.github.com> --- README.md | 139 +++++++++++++++- rotate-encryption-keys.sh | 339 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 477 insertions(+), 1 deletion(-) create mode 100755 rotate-encryption-keys.sh diff --git a/README.md b/README.md index b16e9af..f9ee603 100644 --- a/README.md +++ b/README.md @@ -202,9 +202,146 @@ If you want to upgrade the Galasa Ecosystem to use a newer version of Galasa, fo ```console helm repo update -helm upgrade galasa/ecosystem --reuse-values --set galasaVersion=0.33.0 --wait +helm upgrade galasa/ecosystem --reuse-values --set galasaVersion=0.38.0 --wait ``` +### Rotating Encryption Keys + +To maintain the security of your Galasa Ecosystem, you may wish to replace the encryption key being used to encrypt credentials in the Galasa Ecosystem's credentials store with a new encryption key, and re-encrypt all your existing credentials using the new key. + +When the Galasa Ecosystem Helm chart is installed, a Kubernetes Secret is created which contains the base64-encoded, 256-bit encryption keys used to encrypt credentials stored in the Galasa Ecosystem's credentials store using AES-256-GCM encryption. + +The encryption keys are stored in the following YAML structure: +```yaml +encryptionKey: +fallbackDecryptionKeys: +- +- +``` + +The `encryptionKey` key in the YAML entry represents the active base64-encoded, 256-bit encryption key used to encrypt credentials stored in the Galasa Ecosystem's credentials store. + +The `fallbackDecryptionKeys` list represents a list of base64-encoded encryption keys that are no longer in use, and allows for encryption keys to be rotated without losing previous encryption keys. This allows encrypted credentials to be decrypted with a fallback decryption key and then be encrypted using the newly activated encryption key. + +**Before proceeding to rotate encryption keys, it is highly recommended to make a backup of the existing credentials stored in your Galasa Ecosystem by running the following command using the Galasa CLI tool:** + +```console +galasactl secrets get --format yaml > /path/to/backup/file.yaml +``` + +**where `/path/to/backup/file.yaml` is either an absolute or relative path of your choice to a file where the backup will be stored.** + +#### Prerequisites + +The following command-line utilities must be installed: + +- kubectl (v1.30.3 or later) +- galasactl (0.38.0 or later) +- openssl (3.3.2 or later) + +You must have the correct permissions to read and update Kubernetes Secrets in the Kubernetes namespace that you have installed your Galasa Ecosystem within. + +You must also have a valid personal access token for your Galasa Ecosystem set on your machine so that you can retrieve and update secrets with `galasactl`. + +#### Automated steps + +For Linux and macOS, you can download and run the [`rotate-encryption-keys.sh`](./rotate-encryption-keys.sh) script via the command-line to simplify the process of rotating encryption keys and re-encrypting credentials. + +The following flags can be supplied when running the script: +- `--release-name ` **Required**. The helm release name provided when installing the Galasa Ecosystem helm chart +- `--namespace ` Optional. The Kubernetes namespace where your Galasa Ecosystem is installed +- `--bootstrap ` Optional. The bootstrap URL of the Galasa Ecosystem that is being serviced. Not required if the `GALASA_BOOTSTRAP` environment variable is set and is pointing to the correct bootstrap URL. Overrides the existing `GALASA_BOOTSTRAP` environment variable value if set + +For example: + +```console +./rotate-encryption-keys.sh --release-name example --namespace galasa-dev +``` + +The `rotate-encryption-keys.sh` script will automatically update the current encryption key with a new one, and then restart your Galasa Ecosystem's API and engine controller pods so that they can pick up the new encryption key. After rotating the encryption keys, the script will re-encrypt the existing secrets in your Galasa Ecosystem using the newly activated encryption key. + +Once the encryption keys have been rotated and the existing secrets have been re-encrypted, the script will clear the fallback decryption keys list and restart the API and engine controller pods for a final time to keep the Galasa services in sync with the contents of the encryption keys secret. + +#### Manual steps + +If the `rotate-encryption-keys.sh` script fails or if you are on Windows and cannot run the script, you can manually rotate encryption keys by performing the following steps. + +This guide assumes `kubectl` is configured to manage resources in the namespace that your Galasa Ecosystem is installed within, so the `--namespace` flag is not required when running `kubectl` commands. + +This guide also assumes that the `GALASA_BOOTSTRAP` environment variable is set, so the `--bootstrap` flag is not required when running `galasactl` commands. + +1. Get all the existing secrets from your Galasa Ecosystem by running: + ``` + galasactl secrets get --format yaml + ``` + Store these secrets in a `.yaml` or `.yml` file so that you can re-encrypt them after rotating the encryption keys + +2. Find the name of the Kubernetes Secret containing your Galasa Ecosystem's encryption keys by running: + ``` + kubectl get secrets + ``` + The secret's name should be of the form `{release-name}-encryption-secret`, where `{release-name}` is the Helm release name provided when installing the Galasa Ecosystem Helm chart + +3. Get the existing encryption keys data for your Galasa Ecosystem by running: + ``` + kubectl get secret {encryption-secret-name} --output jsonpath='{ .data.encryption-keys\.yaml }' | openssl base64 -d -A + ``` + where `{encryption-secret-name}` is the name of the Kubernetes secret retrieved in step 2. The output should look like the following: + ```yaml + encryptionKey: + fallbackDecryptionKeys: [] + ``` + Place the output into a file + +4. Generate a new encryption key by running: + ``` + openssl rand -base64 32 + ``` + +5. In the file created at the end of step 3, move the existing `encryptionKey` value into the `fallbackDecryptionKeys` list and place the newly generated encryption key into the `encryptionKey` field. The file should now look like this: + ```yaml + encryptionKey: + fallbackDecryptionKeys: + - + ``` + where `` is the new encryption key generated in step 5 and `` is the old encryption key retrieved in step 3 + +6. Base64-encode the file contents by running the following command, where `` is an absolute or relative path to the file created in step 3 (and modified in step 5): + ``` + openssl base64 -in + ``` + Record the base64-encoded output, making sure there are no spaces or line breaks in the recorded output + +7. Update the existing Kubernetes Secret with the rotated keys by running: + ``` + kubectl patch secret {encryption-secret-name} --type='json' -p="[{'op': 'replace', 'path': '/data/encryption-keys.yaml', 'value': ''}]" + ``` + where `{encryption-secret-name}` is the name of the Kubernetes secret retrieved in step 2, and `` is the output recorded from step 6 + +8. Restart the Galasa Ecosystem's API server deployment by running: + ``` + kubectl rollout restart deployment {release-name}-api + kubectl rollout status deployment {release-name}-api + ``` + where `{release-name}` is the name of the Helm release provided when installing the Galasa Ecosystem Helm chart + +9. Restart the Galasa Ecosystem's engine controller deployment by running: + ``` + kubectl rollout restart deployment {release-name}-engine-controller + kubectl rollout status deployment {release-name}-engine-controller + ``` + where `{release-name}` is the name of the Helm release provided when installing the Galasa Ecosystem Helm chart + +10. Once both the API server and engine controller have been restarted successfully, you can re-encrypt your existing secrets using the YAML file you created in step 1, by running: + ``` + galasactl resources apply -f + ``` + where `` is an absolute or relative path to the YAML file created at the end of step 1 + +Your Galasa Ecosystem will now use the newly generated encryption key to encrypt and decrypt secrets until the next time it is rotated. + +To verify that your secrets can still be read correctly, you can run `galasactl secrets get --format yaml` again and compare the YAML output with the content of YAML file that you applied in step 10. If the output is the same, then the secrets have been re-encrypted successfully. + ### Development To install the latest development version of the Galasa Ecosystem chart, clone this repository and update the following values in your [values.yaml](charts/ecosystem/values.yaml) file: diff --git a/rotate-encryption-keys.sh b/rotate-encryption-keys.sh new file mode 100755 index 0000000..ff7b614 --- /dev/null +++ b/rotate-encryption-keys.sh @@ -0,0 +1,339 @@ +#! /usr/bin/env bash + +# +# Copyright contributors to the Galasa project +# +# SPDX-License-Identifier: EPL-2.0 +# + +#----------------------------------------------------------------------------------------- +# +# Objective: Rotates the encryption key currently being used to encrypt secrets in +# the Galasa Ecosystem and re-encrypts secrets using the new encryption key +# +# Environment variable overrides: +# None +# +#----------------------------------------------------------------------------------------- + +# Where is this script executing from? +BASEDIR=$(dirname "$0");pushd $BASEDIR 2>&1 >> /dev/null ;BASEDIR=$(pwd);popd 2>&1 >> /dev/null + +cd "${BASEDIR}/.." +WORKSPACE_DIR=$(pwd) + +#----------------------------------------------------------------------------------------- +# +# Set Colors +# +#----------------------------------------------------------------------------------------- +bold=$(tput bold) +underline=$(tput sgr 0 1) +reset=$(tput sgr0) +red=$(tput setaf 1) +green=$(tput setaf 76) +white=$(tput setaf 7) +tan=$(tput setaf 202) +blue=$(tput setaf 25) + +#----------------------------------------------------------------------------------------- +# +# Headers and Logging +# +#----------------------------------------------------------------------------------------- +underline() { printf "${underline}${bold}%s${reset}\n" "$@" ;} +h1() { printf "\n${underline}${bold}${blue}%s${reset}\n" "$@" ;} +h2() { printf "\n${underline}${bold}${white}%s${reset}\n" "$@" ;} +debug() { printf "${white}%s${reset}\n" "$@" ;} +info() { printf "${white}➜ %s${reset}\n" "$@" ;} +success() { printf "${green}✔ %s${reset}\n" "$@" ;} +error() { printf "${red}✖ %s${reset}\n" "$@" ;} +warn() { printf "${tan}➜ %s${reset}\n" "$@" ;} +bold() { printf "${bold}%s${reset}\n" "$@" ;} +note() { printf "\n${underline}${bold}${blue}Note:${reset} ${blue}%s${reset}\n" "$@" ;} + +#----------------------------------------------------------------------------------------- +# Functions +#----------------------------------------------------------------------------------------- +function usage { + h1 "A helper script to rotate Galasa ecosystem encryption keys." + info "The following command-line tools must be installed before running this script:" + cat << EOF +kubectl (v1.30.3 or later) +openssl (3.3.2 or later) +galasactl (0.38.0 or later) +EOF + info "Syntax: rotate-encryption-keys.sh [OPTIONS]" + cat << EOF +Options are: +--release-name : Required. The Helm release name provided when the Galasa ecosystem Helm chart was installed. +-b | --bootstrap : Optional. The bootstrap URL of the Galasa ecosystem that is being serviced. Not required if the GALASA_BOOTSTRAP environment variable is set and is pointing to the correct bootstrap URL. +-n | --namespace : Optional. The Kubernetes namespace in which your Galasa ecosystem is running. By default, the namespace pointed to by the current Kubernetes context will be used. +EOF +} + +function check_exit_code { + # This function takes 2 parameters in the form: + # $1 an integer value of the returned exit code + # $2 an error message to display if $1 is not equal to 0 + return_code=$1 + error_msg=$2 + + if [[ "${return_code}" != "0" ]]; then + error "${error_msg}. Log file is at ${LOG_FILE}" + exit 1 + fi +} + +function check_tool_installed { + # This function takes 1 parameter: + # $1 the name of the tool to look for + tool_name=$1 + h2 "Checking ${tool_name} is installed..." + + which ${tool_name} 2>&1 > /dev/null + rc=$? + + check_exit_code ${rc} "${tool_name} is not installed. Install it and try again. rc=${rc}" + success "${tool_name} is installed OK" +} + +function check_required_tools_installed { + h1 "Checking required CLI tools are installed" + + check_tool_installed "kubectl" + check_tool_installed "openssl" + check_tool_installed "galasactl" + + success "All required tools are already installed OK" +} + +function check_galasactl_auth { + h1 "Logging into the Galasa ecosystem" + + info "Bootstrap URL is ${GALASA_BOOTSTRAP}" + + cmd="galasactl auth login" + info "Running command: ${cmd}" + + ${cmd} 2>&1 | tee -a "${LOG_FILE}" + rc=$? + check_exit_code ${rc} "Failed to log in to the Galasa ecosystem secret. Check that a personal access token has been set and try again. rc=${rc}" + success "Logged in to the Galasa ecosystem OK" +} + +function get_existing_encryption_keys { + h1 "Retrieving encryption keys" + + # Get the current secret data + decoded_secret_data=$(kubectl get secret ${ENCRYPTION_SECRET_NAME} \ + ${KUBECTL_NAMESPACE_FLAG} \ + --output jsonpath="{ .data.encryption-keys\.yaml }" | base64 --decode) + rc=$? + check_exit_code ${rc} "Failed to get encryption keys from ${ENCRYPTION_SECRET_NAME} secret. Check that the value for the --release-name flag matches an existing Helm release and try again. rc=${rc}" + + # Extract the current encryption key + export CURRENT_KEY=$(echo "${decoded_secret_data}" | grep "encryptionKey:" | awk '{print $2}') + rc=$? + check_exit_code ${rc} "Failed to get current encryption key from ${ENCRYPTION_SECRET_NAME} secret. rc=${rc}" + + # Extract the fallback decryption keys list + export FALLBACK_KEYS=$(echo "${decoded_secret_data}" | awk '/^fallbackDecryptionKeys:/,/^$/' | tail -n +2) + rc=$? + check_exit_code ${rc} "Failed to get fallback keys from ${ENCRYPTION_SECRET_NAME} secret. rc=${rc}" + + success "Existing keys retrieved OK" +} + +function rotate_encryption_keys { + h1 "Rotating encryption keys" + + # Create a new AES256 encryption key - this must be 32 characters long for 256-bit keys + info "Creating new encryption key..." + cmd="openssl rand -base64 32" + + info "Running command: ${cmd}" + new_key=$(${cmd}) + rc=$? + check_exit_code ${rc} "Failed to create a new encryption key. rc=${rc}" + success "Created new encryption key OK" + + # Add the existing encryption key to the start of the fallback keys list + updated_fallback_keys_yaml=$(cat << EOF +fallbackDecryptionKeys: +- ${CURRENT_KEY} +${FALLBACK_KEYS} +EOF +) + + # Update the Kubernetes Secret with the rotated encryption keys + patch_encryption_keys "${new_key}" "${updated_fallback_keys_yaml}" + success "Successfully rotated encryption keys" +} + +function restart_deployment { + # This function takes 1 parameter: + # $1 the deployment name to wait for + deployment_name=$1 + + kubectl rollout restart deployment ${deployment_name} ${KUBECTL_NAMESPACE_FLAG} 2>&1 | tee -a "${LOG_FILE}" + rc=$? + check_exit_code ${rc} "Failed to issue command to restart the ${deployment_name} deployment. Check that the value for the --release-name flag matches an existing Helm release and try again. rc=${rc}" + + # Wait for the rollout to complete + kubectl rollout status deployment "${deployment_name}" ${KUBECTL_NAMESPACE_FLAG} --timeout=3m 2>&1 | tee -a "${LOG_FILE}" + rc=$? + check_exit_code ${rc} "Failed to wait for ${deployment_name} to be restarted. rc=${rc}" +} + +function restart_deployments { + h1 "Restarting Galasa ecosystem deployments to use new encryption keys" + + info "Restarting API server..." + restart_deployment "${API_DEPLOYMENT_NAME}" + success "API server restarted OK" + + info "Restarting engine controller..." + restart_deployment "${ENGINE_CONTROLLER_DEPLOYMENT_NAME}" + success "Engine controller restarted OK" + + success "Restarted ecosystem deployments OK" +} + +function get_existing_secrets { + h1 "Getting existing Galasa Secrets" + + cmd="galasactl secrets get --format yaml" + info "Running command: ${cmd}" + + ${cmd} > ${SECRETS_FILE} + rc=$? + check_exit_code ${rc} "Failed to get secrets from the Galasa ecosystem. rc=${rc}" + success "Existing secrets stored at ${SECRETS_FILE}" +} + +function migrate_secrets { + h1 "Re-encrypting existing Galasa Secrets with new encryption keys" + + if [[ ! -s "${SECRETS_FILE}" ]]; then + info "No secrets found to re-encrypt" + success "OK" + else + info "Re-applying secrets" + cmd="galasactl resources apply -f ${SECRETS_FILE}" + + info "Running command: ${cmd}" + ${cmd} 2>&1 | tee -a "${LOG_FILE}" + rc=$? + check_exit_code ${rc} "Failed to re-apply secrets to the Galasa ecosystem. rc=${rc}" + success "Successfully re-encrypted existing Galasa Secrets" + fi +} + +function patch_encryption_keys { + # This function takes 2 parameters: + # $1 a base64-encoded encryption key string to be the primary encryption key + # $2 a string representing the fallbackDecryptionKeys key-value pair in YAML format + encryption_key=$1 + fallback_decryption_keys_yaml=$2 + + # Convert the keys into the expected YAML structure to be placed inside the secret + yaml=$(cat << EOF +encryptionKey: ${encryption_key} +${fallback_decryption_keys_yaml} +EOF +) + encoded_yaml=$(echo -n "${yaml}" | base64) + patch=$(cat << EOF +data: + encryption-keys.yaml: ${encoded_yaml} +EOF +) + # Update the Kubernetes Secret + echo "${patch}" | kubectl patch secret "${ENCRYPTION_SECRET_NAME}" ${KUBECTL_NAMESPACE_FLAG} --patch-file /dev/stdin 2>&1 | tee -a "${LOG_FILE}" + rc=$? + check_exit_code ${rc} "Failed to patch the encryption keys secret. rc=${rc}" +} + +function clear_fallback_keys { + h1 "Clearing fallback decryption keys" + get_existing_encryption_keys + + # Reset the fallback decryption keys list to be an empty list + updated_fallback_keys_yaml=$(cat << EOF +fallbackDecryptionKeys: [] +EOF +) + patch_encryption_keys "${CURRENT_KEY}" "${updated_fallback_keys_yaml}" + success "Successfully cleared fallback decryption keys" +} + +#----------------------------------------------------------------------------------------- +# Process parameters +#----------------------------------------------------------------------------------------- +NAMESPACE="" +RELEASE_NAME="" + +while [ "$1" != "" ]; do + case $1 in + -n | --namespace ) shift + export NAMESPACE=$1 + ;; + --release-name ) shift + export RELEASE_NAME=$1 + ;; + -b | --bootstrap ) shift + export GALASA_BOOTSTRAP=$1 + ;; + -h | --help ) usage + exit + ;; + * ) error "Unexpected argument $1" + usage + exit 1 + esac + shift +done + +if [[ -z "${RELEASE_NAME}" ]]; then + error "A release name must be provided using the --release-name flag." + usage + exit 1 +fi + +#----------------------------------------------------------------------------------------- +# Main program logic +#----------------------------------------------------------------------------------------- +set -o pipefail + +ENCRYPTION_SECRET_NAME="${RELEASE_NAME}-encryption-secret" +API_DEPLOYMENT_NAME="${RELEASE_NAME}-api" +ENGINE_CONTROLLER_DEPLOYMENT_NAME="${RELEASE_NAME}-engine-controller" + +KUBECTL_NAMESPACE_FLAG="" +if [[ -n "${NAMESPACE}" ]]; then + KUBECTL_NAMESPACE_FLAG="--namespace ${NAMESPACE}" +fi + +# Create temp directory and log file +TEMP_DIR="${BASEDIR}/temp" +mkdir -p "${TEMP_DIR}" +LOG_FILE="${TEMP_DIR}/rotate-${RELEASE_NAME}-keys.txt" +> "${LOG_FILE}" + +SECRETS_FILE="${TEMP_DIR}/secrets.yaml" + +# Before starting the rotation process, check that we have all the tools required +# and that we can authenticate with the Galasa ecosystem +check_required_tools_installed +check_galasactl_auth + +get_existing_secrets +get_existing_encryption_keys +rotate_encryption_keys +restart_deployments +migrate_secrets + +clear_fallback_keys +restart_deployments +success "Successfully rotated encryption keys and re-encrypted secrets!" \ No newline at end of file