Setup of CSFLE and Queryable Encrption with AWS KMS or Azure Keyvault
SA Maintainer: Pierre Petersson
Time to setup: 2 mins
Time to execute: 10 mins
To quickly get started this example project is utilising a docker image containing tooling required to test CSFLE and Queryable Encryption with AWS KMS or Azure KeyVault. I have automated the process of setting up KMS on AWS and Azure, using terraform, and supporting scrpts to minimized manual work.
If you do not want to use the docker image and set this up on your own, follow instructions https://www.mongodb.com/docs/manual/core/csfle/tutorials/aws/aws-automatic/#use-automatic-client-side-field-level-encryption-with-aws
https://www.mongodb.com/cloud/atlas/signup
- Create Cloud Credentials
- Update credentials.env
- Run configure_kms.sh (automates configuration of creation of KMS Key)
- Run Demo Application
- A Python application that usses AWS KMS or Azure KeyVault with MongoDB Driver
- Application inserts a document with where some fields are CSFLE and Queryable Encryption enabled.
Create an AWS IAM user and add create AWS ACCESS KEYS for that user, the user must av Administrator rights to be able to run terraform to provision KMS keys, roles and policies.
We are going to use the az login to authenticate so that we can run our terraform scripts. You will need to have a valid Azure subscription.
You will only need to update the credentials.env file with you your aws/azure credentials(or other cloud provider).
Update the ONLY the following fields in the file /mongodb-fle-queryable-aws-azure-gcp/python/aws/credentials.env. DONT USE QOUTES, just = string
## FOR AWS
MONGODB_URI=mongodb+srv://user:[email protected]/?retryWrites=true&w=majority
AWS_ACCESS_KEY_ID=AXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY=QhjFqDaC9EzDP8nsp1IwwXtKByvKuo28itCoK2oW
AWS_REGION=eu-central-1
## FOR Azure
MONGODB_URI=mongodb+srv://user:[email protected]/?
# Start docker container - Initializing KMS Setup
## for AWS
docker run -it --rm --env-file=python/aws/credentials.env -v ${PWD}:/workspace piepet/iaac-aws-gcp-azure
## for Azure
docker run -it --rm --env-file=python/azure/credentials.env -v ${PWD}:/workspace piepet/iaac-aws-gcp-azure
Run configuration of KMS provider, will create KMS key in AWS or Azure.
## For AWS
cd /workspace/kms-setup/aws
./configure_kms.sh
export $(< /workspace/python/aws/credentials.env)
## For Azure
cd /workspace/kms-setup/azure
az login
./configure_kms.sh
export $(< /workspace/python/azure/credentials.env)
Python application that inserts a document with CSFLE configured. CSFLE is configured to use AWS or Azure KMS KMS provider.
# Demonstrates CSFLE with KMS Provider with client side schema.
cd /workspace/python
## For AWS
python3.8 demo_csfle_client_schema.py aws
## For Azure
python3.8 demo_csfle_client_schema.py azure
Will create a database with name DEMO-AWS-FLE or DEMO-AZURE-FLE where the keyvault collection and the user collection will be created. The CSFLE schema will be stored in database, as a validation see below.
cd /workspace/python
## For AWS
python3.8 demo_csfle.py aws
## For Azure
python3.8 demo_csfle.py azure
You should now see the following in the DEMO-AWS-FLE.users
Will create a database with name DEMO-AWS-FLE or DEMO-AZURE-QUERYABLE where the keyvault collection and the user collection will be created.
## For AWS
cd /workspace/python
python3.8 demo_queryable.py aws
## For Azure
cd /workspace/python
python3.8 demo_queryable.py azure
You should now see the following in the DEMO-AWS-FLE.users DEMO-AZURE-FLE.users
The application will automatically encrypt/decrypt the fields defined in the encryption schema thats attached to the users collection. Fields that should be shown encrypted are ssn, contact.mobile, contact.email
You should now be able to see in compass that fields that are encrypted have ****** shown as value.
Decrypt multiple Data Encryption Keys (DEK) and re-encrypts them with a new Customer Master Key (CMK). Use this method to rotate the CMK that encrypts your DEKs.
Look at field KeyMaterial in datakeys collection for respective encryptionMode (Queryable or FLE). The KeyMaterial will be replaced with new KeyMaterial.
Before running rotate:
## FLE MasterKeys and DEKS
cd /workspace/python
python3.8 rotate_fle.py aws
cd /workspace/python
python3.8 rotate_fle.py azure
## Queryable Encryption MasterKeys and DEKS
cd /workspace/python
python3.8 rotate_queryable.py aws
cd /workspace/python
python3.8 rotate_queryable.py azure
After running rotate:
If you want to rerun setup, delete vault/data folder. only the data folder. Run the following in root of this pov.
## For AWS
cd /workspace/kms-setup/aws
terraform destroy
## For Azure
cd /workspace/kms-setup/azure
terraform destroy
Blog article - How to Set Up HashiCorp Vault KMIP Secrets Engine with MongoDB CSFLE or Queryable Encryption
https://www.mongodb.com/developer/products/atlas/hashicorp-vault-kmip-secrets-engine-mongodb/
https://www.youtube.com/watch?v=KBMfaEzxNFk
https://www.mongodb.com/collateral/field-level-encryption
https://www.youtube.com/watch?v=bfyxplG1DVg&list=PL4RCxklHWZ9tKWSS21BmtFxOs51f-l1o9&index=33&t=9s
Docs (very much a WIP): https://www.mongodb.com/docs/upcoming/core/queryable-encryption/
Blog post: https://www.mongodb.com/blog/post/mongodb-releases-queryable-encryption-preview
Product page & FAQ: https://mongodb.com/products/queryable-encryption
The Docker container is published in DockerHub, no need to build. But if you would like to rebuild the image see below.
docker buildx create --name builder —use
docker buildx create --use
docker buildx build --push --platform linux/amd64,linux/arm64 --tag piepet/iaac-aws-gcp-azure .
Customer Master Key (CMK), is the encryption key used to protect(encrypt) the Data Encryption Keys, which is on the top level of the encryption hierarchy.
The Data Encryption Key (DEK) is used to encrypt the data that is plain text. Once plain text is encrypted by the DEK it will be in cipher text.
Plain text data is unencrypted information that you wish to protect,
Cipher text is Encrypted information unreadable by a human or computer without decryption.
Envelope encryption is the practice of encrypting plain text data with a data encryption key (DEK) and then encrypting the data key using the customer master key.