Merge pull request #717 from aws-samples/secrets-manager-fixes

Fix some secrets manager issues

manifests/modules/security/secrets-manager/.workshop/cleanup.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+set -e
+
+kubectl delete clustersecretstore cluster-secret-store --ignore-not-found > /dev/null
+
+kubectl delete SecretProviderClass catalog-spc -n catalog --ignore-not-found > /dev/null
+
+kubectl delete ExternalSecret catalog-external-secret -n catalog --ignore-not-found > /dev/null
+
+check=$(aws secretsmanager list-secrets --filters Key="name",Values="${SECRET_NAME}" --output text)
+
+if [ ! -z "$check" ]; then
+  echo "Deleting Secrets Manager data..."
+  aws secretsmanager delete-secret --secret-id ${SECRET_NAME}
+fi

manifests/modules/security/secrets-manager/.workshop/terraform/addon.tf

@@ -82,35 +82,8 @@ resource "aws_iam_policy" "secrets_manager" {
 POLICY
 }
 
-resource "kubernetes_annotations" "catalog-sa" {
-  api_version = "v1"
-  kind        = "ServiceAccount"
-  metadata {
-    name      = "catalog"
-    namespace = "catalog"
-  }
-  annotations = {
-    "eks.amazonaws.com/role-arn" = "${module.secrets_manager_role.iam_role_arn}"
-  }
-  force = true
+output "environment" {
+  value = <<EOF
+export CATALOG_IAM_ROLE="${module.secrets_manager_role.iam_role_arn}"
+EOF
 }
-
-resource "kubectl_manifest" "cluster_secretstore" {
-  yaml_body  = <<YAML
-apiVersion: external-secrets.io/v1beta1
-kind: ClusterSecretStore
-metadata:
-  name: "cluster-secret-store"
-spec:
-  provider:
-    aws:
-      service: SecretsManager
-      region: "${data.aws_region.current.name}"
-      auth:
-        jwt:
-          serviceAccountRef:
-            name: "external-secrets-sa"
-            namespace: "external-secrets"
-YAML
-  depends_on = [module.external_secrets]
-}

manifests/modules/security/secrets-manager/external-secrets/external-secret.yaml

@@ -10,4 +10,4 @@ spec:
     kind: ClusterSecretStore
   dataFrom:
   - extract:
-      key: "eks-workshop/catalog-secret"
+      key: "$SECRET_NAME"

manifests/modules/security/secrets-manager/external-secrets/kustomization.yaml

@@ -2,7 +2,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 bases:
 - ../../../../base-application/catalog
-patches: 
-- deployment.yaml
 resources:
 - external-secret.yaml
+patches: 
+- path: deployment.yaml

manifests/modules/security/secrets-manager/mounting-secrets/kustomization.yaml

@@ -3,4 +3,5 @@ kind: Kustomization
 bases:
 - ../../../../base-application/catalog
 patches: 
-- deployment.yaml
+- path: deployment.yaml
+- path: serviceaccount.yaml

manifests/modules/security/secrets-manager/mounting-secrets/serviceaccount.yaml

@@ -0,0 +1,7 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: catalog
+  namespace: catalog
+  annotations:
+    eks.amazonaws.com/role-arn: ${CATALOG_IAM_ROLE}

manifests/modules/security/secrets-manager/secret-provider-class.yaml

@@ -7,7 +7,7 @@ spec:
   provider: aws
   parameters:
     objects: |
-        - objectName: "$EKS_CLUSTER_NAME/catalog-secret"
+        - objectName: "$SECRET_NAME"
           objectType: "secretsmanager"
           jmesPath:
             - path: username

website/docs/security/secrets-management/_category_.json

@@ -0,0 +1,3 @@
+{
+  "collapsed": false
+}

website/docs/security/secrets-management/secrets-manager/ascp.md

@@ -3,7 +3,7 @@ title: "AWS Secrets and Configuration Provider (ASCP)"
 sidebar_position: 62
 ---
 
-When you ran the `prepare-environment` script detailed in a [previous step](./index.md), it has already installed the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver that's required for this lab.
+When we ran the `prepare-environment` script detailed in a [previous step](./index.md), it has already installed the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store CSI Driver that's required for this lab.
 
 Lets then, validate if the addons deployed.
 
@@ -39,10 +39,11 @@ In order to provide access to your secrets stored in AWS Secrets Manager via CSI
 manifests/modules/security/secrets-manager/secret-provider-class.yaml
 ```
 
-In the above resource, you have two main configurations that you should be focusing. So go head and create the resource to explore those specifications.
+In the above resource, we have two main configurations that we should be focusing. So go ahead and create the resource to explore those specifications.
 
 ```bash
-$ cat eks-workshop/modules/security/secrets-manager/secret-provider-class.yaml | envsubst | kubectl apply -f -
+$ cat ~/environment/eks-workshop/modules/security/secrets-manager/secret-provider-class.yaml \
+  | envsubst | kubectl apply -f -
 ```
 
 The *objects* parameter, which is pointing to a secret named as `eks-workshop/catalog-secret` that we will store in AWS Secrets Manager in the next step. Note that we are using [jmesPath](https://jmespath.org/), to extract a specific key-value from the secret that is JSON-formatted.

website/docs/security/secrets-management/secrets-manager/create-secret.md

@@ -3,10 +3,13 @@ title: "Storing secrets in AWS Secrets Manager"
 sidebar_position: 61
 ---
 
-First, we need to store a secret in AWS Secrets Manager, lets do that using the AWS CLI, by running the command like the example below in you Cloud9 Environment.
+First, we need to store a secret in AWS Secrets Manager, lets do that using the AWS CLI:
 
 ```bash
-$ aws secretsmanager create-secret --name "$EKS_CLUSTER_NAME/catalog-secret" --secret-string '{"username":"catalog_user", "password":"default_password"}' --region $AWS_REGION
+$ export SECRET_SUFFIX=$(openssl rand -hex 4)
+$ export SECRET_NAME="$EKS_CLUSTER_NAME-catalog-secret-${SECRET_SUFFIX}"
+$ aws secretsmanager create-secret --name "$SECRET_NAME" \
+  --secret-string '{"username":"catalog_user", "password":"default_password"}' --region $AWS_REGION
 {
     "ARN": "arn:aws:secretsmanager:$AWS_REGION:$AWS_ACCOUNT_ID:secret:$EKS_CLUSTER_NAME/catalog-secret-ABCdef",
     "Name": "eks-workshop/static-secret",
@@ -16,10 +19,10 @@ $ aws secretsmanager create-secret --name "$EKS_CLUSTER_NAME/catalog-secret" --s
 
 The command above is storing a secret with a JSON encoded key/value content, for `username` and `password` credentials.
 
-Validate the new stored secret in the [AWS Secrets Manager Console](https://console.aws.amazon.com/secretsmanager/listsecrets) or run the below command in your Cloud9 Environment.
+Validate the new stored secret in the [AWS Secrets Manager Console](https://console.aws.amazon.com/secretsmanager/listsecrets) or using this command:
 
 ```bash
-$ aws secretsmanager describe-secret --secret-id "$EKS_CLUSTER_NAME/catalog-secret"
+$ aws secretsmanager describe-secret --secret-id "$SECRET_NAME"
 {
     "ARN": "arn:aws:secretsmanager:us-west-2:068535243777:secret:eks-workshop/catalog-secret-WDD8yS",
     "Name": "eks-workshop/catalog-secret",

website/docs/security/secrets-management/secrets-manager/external-secrets.md

@@ -3,9 +3,7 @@ title: "External Secrets Operator"
 sidebar_position: 64
 ---
 
-The `prepare-environment` script that you ran in a [previous step](./index.md), has already deployed the External Secrets Operator addon required for this lab.
-
-Let's validate the created addon.
+Now we can explore integrating with Secrets Managed using the External Secrets operator. This has already been installed in our EKS cluster:
 
 ```bash
 $ kubectl -n external-secrets get pods
@@ -19,24 +17,25 @@ default               0         7m
 external-secrets-sa   0         7m
 ```
 
-As you can see, you have a ServiceAccount named `external-secrets-sa`, this SA is tied to an [IRSA](../../iam-roles-for-service-accounts/), with access to AWS Secrets Manager, for retrieving secrets information.
+As we can see theres a ServiceAccount named `external-secrets-sa` which is tied to an IAM role via [IRSA](../../iam-roles-for-service-accounts/), with access to AWS Secrets Manager for retrieving secrets information.
 
 ```bash
 $ kubectl -n external-secrets describe sa external-secrets-sa | grep Annotations
 Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::068535243777:role/eks-workshop-external-secrets-sa-irsa
 ```
 
-In addition to that, you'll need to create a new cluster resource called `ClusterSecretStore` which is a cluster wide SecretStore that can be referenced by all ExternalSecrets from all namespaces.
+In addition to that, we'll need to create a new cluster resource called `ClusterSecretStore` which is a cluster-wide SecretStore that can be referenced by all ExternalSecrets from all namespaces.
 
 ```file
 manifests/modules/security/secrets-manager/cluster-secret-store.yaml
 ```
 
 ```bash
-$ cat eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml | envsubst | kubectl apply -f -
+$ cat ~/environment/eks-workshop/modules/security/secrets-manager/cluster-secret-store.yaml \
+  | envsubst | kubectl apply -f -
 ```
 
-Take a deeper look in this newly created resource specifications.
+Take a deeper look at this newly created resources specifications.
 
 ```bash
 $ kubectl get clustersecretstores.external-secrets.io 
@@ -57,7 +56,7 @@ provider:
 
 You can see here, that it's using a [JSON Web Token (jwt)](https://jwt.io/), referenced to the ServiceAccount we just checked, to sync with AWS Secrets Manager.
 
-Let's move forward and create an `ExternalSecret`, that describes what data should be fetched from AWS Secrets Manager, how the data should be transformed and saved as a Kubernetes Secret. And also patch our `catalog` Deployment to use the External Secret as source for the credentials.
+Let's move forward and create an `ExternalSecret` that describes what data should be fetched from AWS Secrets Manager, how the data should be transformed and saved as a Kubernetes Secret. Then we can patch our `catalog` Deployment to use the External Secret as source for the credentials.
 
 ```kustomization
 modules/security/secrets-manager/external-secrets/kustomization.yaml
@@ -66,18 +65,20 @@ ExternalSecret/catalog-external-secret
 ```
 
 ```bash
-$ kubectl apply -k eks-workshop/modules/security/secrets-manager/external-secrets/
+$ kubectl kustomize ~/environment/eks-workshop/modules/security/secrets-manager/external-secrets/ \
+  | envsubst | kubectl apply -f-
+$ kubectl rollout status -n catalog deployment/catalog --timeout=120s
 ```
 
-Check the newly created `ExternalSecret` resouce.
+Check the newly created `ExternalSecret` resource.
 
 ```bash
-$ kubectl -n catalog get externalsecrets.external-secrets.io                                                                                                                                                                                                                            
+$ kubectl -n catalog get externalsecrets.external-secrets.io
 NAME                      STORE                  REFRESH INTERVAL   STATUS         READY
 catalog-external-secret   cluster-secret-store   1h                 SecretSynced   True
 ```
 
-Verify that the resource has a `SecretSynced` status, which means that it was successfully syncronized from AWS Secrets Manager. Let's take a closer look to this resource specifications.
+Verify that the resource has a `SecretSynced` status, which means that it was successfully synchronized from AWS Secrets Manager. Let's take a closer look to this resources specifications.
 
 ```bash
 $ kubectl -n catalog get externalsecrets.external-secrets.io catalog-external-secret -o yaml | yq '.spec'
@@ -95,9 +96,9 @@ target:
   deletionPolicy: Retain
 ```
 
-Notice the `key` and the `secretStoreRef` parameter, pointing to the secret we stored on AWS Secrets Manager, and the `ClusterSecretStore` previously created. Also the `refreshInterval` is set to 1 hours, which means that the value from this secret will be checked and refreshed every hour.
+Notice the `key` and the `secretStoreRef` parameters pointing to the secret we stored on AWS Secrets Manager, and the `ClusterSecretStore` previously created. Also the `refreshInterval` is set to 1 hours which means that the value from this secret will be checked and refreshed every hour.
 
-But how do we use this ExternalSecret in our Pods? After we create this resouces, it automatically created a Kubernetes secret with the same name in the Namespace.
+But how do we use this ExternalSecret in our Pods? After we create this resource it automatically created a Kubernetes secret with the same name in the namespace.
 
 ```bash
 $ kubectl -n catalog get secrets
@@ -121,7 +122,7 @@ $ kubectl -n catalog get secret catalog-external-secret -o yaml | yq '.metadata.
 
 See that it has an `ownerReference` that points to External Secrets Operator.
 
-Now check that the `catalog` Pod, is already updated with the values from this new secret, and it's up and running!
+Now check that the `catalog` pod is already updated with the values from this new secret, and it's up and running!
 
 ```bash
 $ kubectl -n catalog get pods
@@ -143,6 +144,6 @@ $ kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.co
 
 ### Conclusion
 
-In conclusion, there is no best option on choosing between **AWS Secrets and Configuration Provider (ASCP)** vs. **External Secrets Operator (ESO)** in order to manage your secrets stored on **AWS Secrets Manager**. 
+In conclusion there is no best option on choosing between **AWS Secrets and Configuration Provider (ASCP)** vs. **External Secrets Operator (ESO)** in order to manage your secrets stored on **AWS Secrets Manager**. 
 
 Both tools have their specific advantages, for example, ASCP can help you avoid exposing secrets as environment variables, mounting them as volumes directly from AWS Secrets Manager into a Pod, the drawback is the need to manage those volumes. In the other hand ESO makes easier the Kubernetes Secrets lifecycle management, having also a cluster wide SecretStore, however it doesn't allow you to use Secrets as volumes. It all depends on your use case, and having both can bring you a lot more flexibility and security with Secrets Management.

website/docs/security/secrets-management/secrets-manager/mounting-secrets.md

@@ -3,7 +3,7 @@ title: "Mounting AWS Secrets Manager secret on Kubernetes Pod"
 sidebar_position: 63
 ---
 
-Now that you have a secret stored in AWS Secrets Manager and synced with a Kubernetes Secret, let's mount it inside the Pod, but first you should take a look on the `catalog` Deployment and the existing Secrets in the `catalog` Namespace.
+Now that we have a secret stored in AWS Secrets Manager and synchronized with a Kubernetes Secret let's mount it inside the Pod. First we should take a look at the `catalog` Deployment and the existing Secrets in the `catalog` namespace.
 
 The `catalog` Deployment accesses the following database credentials from the `catalog-db` secret via environment variables:
 
@@ -25,7 +25,7 @@ $ kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.co
       name: catalog-db
 ```
 
-Notice that the `catalog` Deployment don't have any `volumes` or `volumeMounts` other than an `emptyDir` mounted on `/tmp`
+Notice that the `catalog` Deployment doesn't have any `volumes` or `volumeMounts` other than an `emptyDir` mounted on `/tmp`
 
 ```bash
 $ kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.volumes'
@@ -44,15 +44,17 @@ modules/security/secrets-manager/mounting-secrets/kustomization.yaml
 Deployment/catalog
 ```
 
-Here you will be mounting the AWS Secrets Manager secret using the CSI driver with the SecretProviderClass you validated earlier on the `/etc/catalog-secret` mountPath inside the Pod. When this happens, AWS Secrets Manager will sync the content of the stored secret with Amazon EKS, and create a Kubernetes Secret with the same contents that can be consumed as Environment variables in the Pod.
+Here we will be mounting the AWS Secrets Manager secret using the CSI driver with the SecretProviderClass we validated earlier on the `/etc/catalog-secret` mountPath inside the Pod. When this happens, AWS Secrets Manager will sync the contents of the stored secret with Amazon EKS, and create a Kubernetes Secret with the same contents that can be consumed as environment variables in the Pod.
 
 ```bash
-$ kubectl apply -k ~/environment/eks-workshop/modules/security/secrets-manager/mounting-secrets/
+$ kubectl kustomize ~/environment/eks-workshop/modules/security/secrets-manager/mounting-secrets/ \
+  | envsubst | kubectl apply -f-
+$ kubectl rollout status -n catalog deployment/catalog --timeout=120s
 ```
 
 Let's validate the changes made in the `catalog` Namespace.
 
-In the Deployment, youll' be able to check that it has a new `volume` and respective `volumeMount` pointing to the CSI Secret Store Driver, and mounted on `/etc/catalog-secrets` path.
+In the Deployment, we'll be able to check that it has a new `volume` and respective `volumeMount` pointing to the CSI Secret Store Driver, and mounted on `/etc/catalog-secrets` path.
 
 ```bash
 $ kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.volumes'
@@ -73,31 +75,27 @@ $ kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.co
   name: tmp-volume
 ```
 
-Mounted Secrets are a good way to have sensitive information available as a file inside the filesystem of one or more of the Pod's containers, some benefits are not exposing the value of the secret as environment variables and when a volume contains data from a Secret, and that Secret is updated, Kubernetes tracks this and updates the data in the volume.
+Mounted Secrets are a good way to have sensitive information available as a file inside the filesystem of one or more of the Pod's containers. Some benefits are not exposing the value of the secret as environment variables and when a volume contains data from a Secret, and when that Secret is updated Kubernetes tracks it and updates the data in the volume.
 
 You can take a look on the contents of the mounted Secret inside your Pod.
 
 ```bash
-$ kubectl -n catalog exec -ti $(kubectl -n catalog get pods -l app.kubernetes.io/component=service -o name --no-headers) -- /bin/bash
-
-[appuser@catalog-76c48477ff-d9dfh ~]$ ls /etc/catalog-secret/ 
-eks-workshop_catalog-secret  password  username
-
-[appuser@catalog-76c48477ff-d9dfh ~]$ ls /etc/catalog-secret/ | while read SECRET; do cat /etc/catalog-secret/$SECRET; echo; done
+$ kubectl -n catalog exec deployment/catalog -- ls /etc/catalog-secret/ 
+eks-workshop-catalog-secret  password  username
+$ kubectl -n catalog exec deployment/catalog -- cat /etc/catalog-secret/${SECRET_NAME}
 {"username":"catalog_user", "password":"default_password"}
-default_password
+$ kubectl -n catalog exec deployment/catalog -- cat /etc/catalog-secret/username
 catalog_user
-
-[appuser@catalog-76c48477ff-d9dfh ~]$ exit
+$ kubectl -n catalog exec deployment/catalog -- cat /etc/catalog-secret/password
+default_password
 ```
 
-You could verify that there are 3 files in the mountPath `/etc/catalog-secret`. `
-1. `eks-workshop_catalog-secret`: With the unformated value of the secret.
+There are 3 files in the mountPath `/etc/catalog-secret`. `
+1. `eks-workshop-catalog-secret`: The value of the secret in JSON format.
 2. `password`: password jmesPath filtered and formatted value.
 3. `username`: username jmesPath filtered and formatted value.
 
-
-Also, the Environment Variables are now being consumed from the new Secret, `catalog-secret` that didn't exist earlier, and it was created by the SecretProviderClass via CSI Secret Store driver.
+Also, the environment variables are now being consumed from the new Secret, `catalog-secret` that didn't exist earlier, and it was created by the SecretProviderClass via CSI Secret Store driver.
 
 ```bash
 $ kubectl -n catalog get deployment catalog -o yaml | yq '.spec.template.spec.containers[] | .env'
@@ -117,4 +115,10 @@ catalog-db       Opaque   2      15h
 catalog-secret   Opaque   2      43s
 ```
 
-Now you have a Kubernetes Secrets fully integrated with AWS Secrets Manager, that can leverage secrets rotation which is a best practice for Secrets Management. So everytime a secret is rotated or updated on AWS Secrets Manager, you'll just need to rollout a new version of your Deployment for the CSI Secret Store driver can sync the Kubernetes Secrets contents.
+We can verify this by checking the environment variables in the running pod:
+
+```bash
+$ kubectl -n catalog exec -ti deployment/catalog -- env | grep DB_
+```
+
+Now we have a Kubernetes Secret fully integrated with AWS Secrets Manager that can leverage secret rotation, which is a best practice for Secrets Management. Everytime a secret is rotated or updated on AWS Secrets Manager, we can roll out a new version of your Deployment so the CSI Secret Store driver can sync the Kubernetes Secrets contents with the rotated value.