composer-2

Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key

This blueprint creates a Private instance of Cloud Composer version 2 on a VPC with a dedicated service account. Cloud Composer 2 is the new major version for Cloud Composer that supports:

• environment autoscaling
• workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and database.

Please consult the documentation page for an exhaustive comparison between Composer Version 1 and Version 2.

The solution will use:

• Cloud Composer
• VPC with Private Service Access to deploy resources, if no Shared VPC configuration provided.
• Google Cloud NAT to access internet resources, if no Shared VPC configuration provided.

The solution supports as inputs:

• Shared VPC
• Cloud KMS CMEK keys

This is the high level diagram:

Requirements

This blueprint will deploy all its resources into the project defined by the project_id variable. Please note that we assume this project already exists. However, if you provide the appropriate values to the project_create variable, the project will be created as part of the deployment.

If project_create is left to null, the identity performing the deployment needs the owner role on the project defined by the project_id variable. Otherwise, the identity performing the deployment needs resourcemanager.projectCreator on the resource hierarchy node specified by project_create.parent and billing.user on the billing account specified by project_create.billing_account_id.

Deployment

Run Terraform init:

terraform init

Configure the Terraform variable in your terraform.tfvars file. You need to specify at least the following variables:

project_id          = "lcaggioni-sandbox"
prefix              = "lc"
region              = "europe-west1"

You can run now:

terraform apply

You can now connect to your instance.

Customizations

VPC

If a shared VPC is not configured, a VPC will be created within the project. The following IP ranges will be used:

• Cloudsql: 10.20.10.0/24
• GKE: 10.20.11.0/28

Change the code as needed to match your needed configuration, remember that these addresses should not overlap with any other range used in network.

Shared VPC

As is often the case in real-world configurations, this blueprint accepts as input an existing Shared-VPC via the network_config variable.

Example:

network_config = {
  host_project      = "PROJECT"
  network_self_link = "projects/PROJECT/global/networks/VPC_NAME"
  region            = "europe-west3"
  subnet_self_link  = "projects/PROJECT/regions/REGION/subnetworks/VPC_NAME"
  composer_secondary_ranges = {
    pods     = "pods"
    services = "services"
  }
}
# tftest skip

Make sure that:

• The GKE API (container.googleapis.com) is enabled in the VPC host project.
• The subnet has secondary ranges configured with 2 ranges:
  • pods: /22 example: 10.10.8.0/22
  • services = /24 example: 10.10.12.0/24`
• Firewall rules are set, as described in the documentation

In order to run the example and deploy Cloud Composer on a shared VPC the identity running Terraform must have the following IAM role on the Shared VPC Host project.

• Compute Network Admin (roles/compute.networkAdmin)
• Compute Shared VPC Admin (roles/compute.xpnAdmin)

Encryption

As is often the case in real-world configurations, this blueprint accepts as input an existing Cloud KMS keys via the service_encryption_keys variable.

Example:

service_encryption_keys = {
  `europe/west1` = `projects/PROJECT/locations/REGION/keyRings/KR_NAME/cryptoKeys/KEY_NAME`
}
# tftest skip

Variables

name	description	type	required	default
prefix	Prefix used for resource names.	string	✓
project_id	Project id, references existing project if project_create is null.	string	✓
region	Region where instances will be deployed.	string	✓
composer_config	Composer environment configuration. It accepts only following attributes: environment_size, software_config and workloads_config. See attribute reference for details on settings variables.	object({…})		{…}
iam_bindings_additive	Map of Role => principal in IAM format (group:[email protected]) to be added on the project.	map(list(string))		{}
network_config	Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values.	object({…})		null
project_create	Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format.	object({…})		null
service_encryption_keys	Cloud KMS keys to use to encrypt resources. Provide a key for each region in use.	map(string)		{}

Outputs

name	description	sensitive
composer_airflow_uri	The URI of the Apache Airflow Web UI hosted within the Cloud Composer environment..
composer_dag_gcs	The Cloud Storage prefix of the DAGs for the Cloud Composer environment.
composer_service_account	Cloud Composer nodes Service Account email.

Test

module "test" {
  source     = "./fabric/blueprints/data-solutions/composer-2/"
  project_id = "composer"
  project_create = {
    billing_account_id = var.billing_account_id
    parent             = var.folder_id
  }
  prefix = var.prefix
  region = var.region
}
# tftest modules=5 resources=44 e2e