Name of the RDS Aurora cluster to deploy
| `true` | `""` | +| `username` |Username for the PostgreSQL admin user
| `true` | `""` | +| `password` |Password for the PostgreSQL admin user
| `true` | `""` | +| `vpc-id` |VPC ID to create the cluster in
| `true` | `""` | +| `subnet-ids` |List of subnet IDs to create the cluster in
| `true` | `""` | +| `cidr-blocks` |CIDR blocks to allow access from and to
| `true` | `""` | +| `availability-zones` |Array of availability zones to use for the Aurora cluster
| `true` | `""` | +| `additional-terraform-vars` |JSON object containing additional Terraform variables
| `false` | `{}` | +| `s3-backend-bucket` |Name of the S3 bucket to store Terraform state
| `true` | `""` | +| `s3-bucket-region` |Region of the bucket containing the resources states
| `false` | `""` | +| `tf-modules-revision` |Git revision of the tf modules to use
| `false` | `main` | +| `tf-modules-path` |Path where the tf Aurora modules will be cloned
| `false` | `./.action-tf-modules/aurora/` | +| `tf-cli-config-credentials-hostname` |The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file. Defaults to app.terraform.io.
The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file.
| `false` | `""` | +| `tf-terraform-version` |The version of Terraform CLI to install. Defaults to latest.
Whether or not to install a wrapper to wrap subsequent calls of the terraform binary and expose its STDOUT, STDERR, and exit code as outputs named stdout, stderr, and exitcode respectively. Defaults to true.
Version of the aws cli to use
| `false` | `2.15.52` | + + +## Outputs + +| name | description | +| --- | --- | +| `aurora-endpoint` |The endpoint of the deployed Aurora cluster
| +| `terraform-state-url` |URL of the Terraform state file in the S3 bucket
| +| `all-terraform-outputs` |All outputs from Terraform
| + + +## Runs + +This action is a `composite` action. + +## Usage + +```yaml +- uses: camunda/camunda-tf-eks-module/aurora-manage-cluster@main + with: + cluster-name: + # Name of the RDS Aurora cluster to deploy + # + # Required: true + # Default: "" + + username: + # Username for the PostgreSQL admin user + # + # Required: true + # Default: "" + + password: + # Password for the PostgreSQL admin user + # + # Required: true + # Default: "" + + vpc-id: + # VPC ID to create the cluster in + # + # Required: true + # Default: "" + + subnet-ids: + # List of subnet IDs to create the cluster in + # + # Required: true + # Default: "" + + cidr-blocks: + # CIDR blocks to allow access from and to + # + # Required: true + # Default: "" + + availability-zones: + # Array of availability zones to use for the Aurora cluster + # + # Required: true + # Default: "" + + additional-terraform-vars: + # JSON object containing additional Terraform variables + # + # Required: false + # Default: {} + + s3-backend-bucket: + # Name of the S3 bucket to store Terraform state + # + # Required: true + # Default: "" + + s3-bucket-region: + # Region of the bucket containing the resources states + # + # Required: false + # Default: "" + + tf-modules-revision: + # Git revision of the tf modules to use + # + # Required: false + # Default: main + + tf-modules-path: + # Path where the tf Aurora modules will be cloned + # + # Required: false + # Default: ./.action-tf-modules/aurora/ + + tf-cli-config-credentials-hostname: + # The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file. Defaults to `app.terraform.io`. + # + # Required: false + # Default: app.terraform.io + + tf-cli-config-credentials-token: + # The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file. + # + # Required: false + # Default: "" + + tf-terraform-version: + # The version of Terraform CLI to install. Defaults to `latest`. + # + # Required: false + # Default: latest + + tf-terraform-wrapper: + # Whether or not to install a wrapper to wrap subsequent calls of the `terraform` binary and expose its STDOUT, STDERR, and exit code as outputs named `stdout`, `stderr`, and `exitcode` respectively. Defaults to `true`. + # + # Required: false + # Default: true + + awscli-version: + # Version of the aws cli to use + # + # Required: false + # Default: 2.15.52 +``` diff --git a/.github/actions/aurora-manage-cluster/action.yml b/.github/actions/aurora-manage-cluster/action.yml new file mode 100644 index 00000000..1e79fd98 --- /dev/null +++ b/.github/actions/aurora-manage-cluster/action.yml @@ -0,0 +1,170 @@ +--- +name: Deploy RDS Aurora Cluster + +description: | + This GitHub Action automates the deployment of an RDS Aurora cluster using Terraform. + This action will also install Terraform and awscli. It will output the Aurora cluster endpoint. + +inputs: + cluster-name: + description: Name of the RDS Aurora cluster to deploy + required: true + + username: + description: Username for the PostgreSQL admin user + required: true + + password: + description: Password for the PostgreSQL admin user + required: true + + vpc-id: + description: VPC ID to create the cluster in + required: true + + subnet-ids: + description: List of subnet IDs to create the cluster in + required: true + + cidr-blocks: + description: CIDR blocks to allow access from and to + required: true + + availability-zones: + description: Array of availability zones to use for the Aurora cluster + required: true + + additional-terraform-vars: + description: JSON object containing additional Terraform variables + required: false + default: '{}' + + s3-backend-bucket: + description: Name of the S3 bucket to store Terraform state + required: true + + s3-bucket-region: + description: Region of the bucket containing the resources states + required: false + + tf-modules-revision: + description: Git revision of the tf modules to use + default: main + + tf-modules-path: + description: Path where the tf Aurora modules will be cloned + default: ./.action-tf-modules/aurora/ + + # inherited from https://github.com/hashicorp/setup-terraform/blob/main/action.yml + tf-cli-config-credentials-hostname: + description: The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration + file. Defaults to `app.terraform.io`. + default: app.terraform.io + + tf-cli-config-credentials-token: + description: The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration + file. + required: false + + tf-terraform-version: + description: The version of Terraform CLI to install. Defaults to `latest`. + default: latest + + tf-terraform-wrapper: + description: Whether or not to install a wrapper to wrap subsequent calls of the `terraform` binary and expose its STDOUT, STDERR, and exit code + as outputs named `stdout`, `stderr`, and `exitcode` respectively. Defaults to `true`. + default: 'true' + + awscli-version: + description: Version of the aws cli to use + # renovate: datasource=github-releases depName=aws/aws-cli + default: 2.15.52 + +outputs: + aurora-endpoint: + description: The endpoint of the deployed Aurora cluster + value: ${{ steps.apply.outputs.aurora_endpoint }} + + terraform-state-url: + description: URL of the Terraform state file in the S3 bucket + value: ${{ steps.utility.outputs.terraform-state-url }} + + # Add all terraform outputs dynamically + all-terraform-outputs: + description: All outputs from Terraform + value: ${{ steps.fetch_outputs.outputs.all_terraform_outputs }} + +runs: + using: composite + steps: + - name: Use Utility Actions + id: utility + # see https://github.com/orgs/community/discussions/41927 it's not possible to optimize this yet + # steps.uses cannot access the github context. + # uses: ${{ github.action_repository }}/utility-action@${{ github.action_ref }} + # TODO: hard pin this one once it's merged + #uses: "camunda/camunda-tf-eks-module/utility-action@main" + uses: ./.github/actions/utility-action + with: + awscli-version: ${{ inputs.awscli-version }} + terraform-version: ${{ inputs.terraform-version }} + + s3-backend-bucket: ${{ inputs.s3-backend-bucket }} + s3-bucket-region: ${{ inputs.s3-bucket-region }} + + tf-state-key: terraform/${{ inputs.cluster-name }}/gha/aurora/terraform.tfstate + + tf-cli-config-credentials-hostname: ${{ inputs.tf-cli-config-credentials-hostname }} + tf-cli-config-credentials-token: ${{ inputs.tf-cli-config-credentials-token }} + tf-terraform-wrapper: ${{ inputs.tf-terraform-wrapper }} + + - name: Checkout Repository Aurora modules + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + with: + repository: camunda/camunda-tf-eks-module + ref: ${{ inputs.tf-modules-revision }} + path: ${{ inputs.tf-modules-path }} + fetch-depth: 0 + + - name: Terraform Init + shell: bash + id: init + working-directory: ${{ inputs.tf-modules-path }}/modules/aurora/ + run: | + cp ../fixtures/backend.tf ./ + terraform version + terraform init -backend-config="bucket=${{ steps.utility.outputs.TFSTATE_BUCKET }}" -backend-config="key=${{ steps.utility.outputs.TFSTATE_KEY }}" -backend-config="region=${{ steps.utility.outputs.TFSTATE_REGION }}" + terraform validate -no-color + + - name: Terraform Plan + shell: bash + id: plan + working-directory: ${{ inputs.tf-modules-path }}/modules/aurora/ + run: | + echo '${{ inputs.additional-terraform-vars }}' > /tmp/var.tfvars.json + terraform plan -no-color -out aurora.plan \ + -var-file=/tmp/var.tfvars.json \ + -var "cluster_name=${{ inputs.cluster-name }}" \ + -var "username=${{ inputs.username }}" \ + -var "password=${{ inputs.password }}" \ + -var 'availability_zones=${{ inputs.availability-zones }}' \ + -var "vpc_id=${{ inputs.vpc-id }}" \ + -var 'subnet_ids=${{ inputs.subnet-ids }}' \ + -var 'cidr_blocks=${{ inputs.cidr-blocks }}' + + - name: Terraform Apply + shell: bash + id: apply + working-directory: ${{ inputs.tf-modules-path }}/modules/aurora/ + run: | + terraform apply -no-color aurora.plan + export aurora_endpoint="$(terraform output -raw aurora_endpoint)" + echo "aurora_endpoint=$aurora_endpoint" >> "$GITHUB_OUTPUT" + + - name: Fetch Terraform Outputs + shell: bash + id: fetch_outputs + working-directory: ${{ inputs.tf-modules-path }}/modules/aurora/ + run: | + all_outputs=$(terraform output -json | jq -c .) + echo "all_terraform_outputs=$all_outputs" | tee -a "$GITHUB_OUTPUT" diff --git a/.github/actions/eks-cleanup-resources/README.md b/.github/actions/eks-cleanup-resources/README.md new file mode 100644 index 00000000..99581844 --- /dev/null +++ b/.github/actions/eks-cleanup-resources/README.md @@ -0,0 +1,57 @@ +# Delete EKS resources + +## Description + +This GitHub Action automates the deletion of EKS resources using a shell script. + + +## Inputs + +| name | description | required | default | +| --- | --- | --- | --- | +| `tf-bucket` |Bucket containing the resources states
| `true` | `""` | +| `tf-bucket-region` |Region of the bucket containing the resources states, if not set, will fallback on AWS_REGION
| `false` | `""` | +| `max-age-hours` |Maximum age of resources in hours
| `false` | `20` | +| `target` |Specify an ID to destroy specific resources or "all" to destroy all resources
| `false` | `all` | +| `temp-dir` |Temporary directory prefix used for storing resource data during processing
| `false` | `./tmp/eks-cleanup/` | + + +## Runs + +This action is a `composite` action. + +## Usage + +```yaml +- uses: camunda/camunda-tf-eks-module/eks-cleanup-resources@main + with: + tf-bucket: + # Bucket containing the resources states + # + # Required: true + # Default: "" + + tf-bucket-region: + # Region of the bucket containing the resources states, if not set, will fallback on AWS_REGION + # + # Required: false + # Default: "" + + max-age-hours: + # Maximum age of resources in hours + # + # Required: false + # Default: 20 + + target: + # Specify an ID to destroy specific resources or "all" to destroy all resources + # + # Required: false + # Default: all + + temp-dir: + # Temporary directory prefix used for storing resource data during processing + # + # Required: false + # Default: ./tmp/eks-cleanup/ +``` diff --git a/.github/actions/eks-cleanup-resources/action.yml b/.github/actions/eks-cleanup-resources/action.yml index 28de6940..bbfc8700 100644 --- a/.github/actions/eks-cleanup-resources/action.yml +++ b/.github/actions/eks-cleanup-resources/action.yml @@ -1,36 +1,40 @@ +--- name: Delete EKS resources description: | - This GitHub Action automates the deletion of EKS resources using a shell script. + This GitHub Action automates the deletion of EKS resources using a shell script. + inputs: - tf-bucket: - description: 'Bucket containing the resources states' - required: true - tf-bucket-region: - description: 'Region of the bucket containing the resources states, if not set, will fallback on AWS_REGION' - max-age-hours: - description: 'Maximum age of resources in hours' - required: false - default: "20" - target: - description: 'Specify an ID to destroy specific resources or "all" to destroy all resources' - required: true - default: "all" - temp-dir: - description: 'Temporary directory prefix used for storing resource data during processing' - required: false - default: "./tmp/eks-cleanup/" + tf-bucket: + description: Bucket containing the resources states + required: true + + tf-bucket-region: + description: Region of the bucket containing the resources states, if not set, will fallback on AWS_REGION + required: false + + max-age-hours: + description: Maximum age of resources in hours + default: '20' + + target: + description: Specify an ID to destroy specific resources or "all" to destroy all resources + default: all + + temp-dir: + description: Temporary directory prefix used for storing resource data during processing + default: ./tmp/eks-cleanup/ runs: - using: 'composite' - steps: - - name: Delete resources - id: delete_resources - shell: bash - run: | - if [ -n "${{ inputs.tf-bucket-region }}" ]; then - export AWS_S3_REGION="${{ inputs.tf-bucket-region }}" - fi - - ${{ github.action_path }}/scripts/destroy.sh "${{ inputs.tf-bucket }}" ${{ github.action_path }}/../../../modules/ "${{ inputs.temp-dir }}" ${{ inputs.max-age-hours }} ${{ inputs.target }} + using: composite + steps: + - name: Delete resources + id: delete_resources + shell: bash + run: | + if [ -n "${{ inputs.tf-bucket-region }}" ]; then + export AWS_S3_REGION="${{ inputs.tf-bucket-region }}" + fi + + ${{ github.action_path }}/scripts/destroy.sh "${{ inputs.tf-bucket }}" ${{ github.action_path }}/../../../modules/ "${{ inputs.temp-dir }}" ${{ inputs.max-age-hours }} ${{ inputs.target }} diff --git a/.github/actions/eks-manage-cluster/README.md b/.github/actions/eks-manage-cluster/README.md new file mode 100644 index 00000000..bf3eaefe --- /dev/null +++ b/.github/actions/eks-manage-cluster/README.md @@ -0,0 +1,123 @@ +# Deploy an EKS Cluster + +## Description + +This GitHub Action automates the deployment of an EKS (Amazon Elastic Kubernetes Service) cluster using Terraform. +This action will also install Terraform, awscli, and kubectl. The kube context will be set on the created cluster. + + +## Inputs + +| name | description | required | default | +| --- | --- | --- | --- | +| `aws-region` |AWS region where the EKS cluster will be deployed
| `true` | `""` | +| `cluster-name` |Name of the EKS cluster to deploy
| `true` | `""` | +| `additional-terraform-vars` |JSON object containing additional Terraform variables
| `false` | `{}` | +| `s3-backend-bucket` |Name of the S3 bucket to store Terraform state
| `true` | `""` | +| `s3-bucket-region` |Region of the bucket containing the resources states, if not set, will fallback on aws-region
| `false` | `""` | +| `tf-modules-revision` |Git revision of the tf modules to use
| `false` | `main` | +| `tf-modules-path` |Path where the tf EKS modules will be cloned
| `false` | `./.action-tf-modules/eks/` | +| `login` |Authenticate the current kube context on the created cluster
| `false` | `true` | +| `tf-cli-config-credentials-hostname` |The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file. Defaults to app.terraform.io.
The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file.
| `false` | `""` | +| `tf-terraform-version` |The version of Terraform CLI to install. Instead of full version string you can also specify constraint string starting with "<" (for example <1.13.0) to install the latest version satisfying the constraint. A value of latest will install the latest version of Terraform CLI. Defaults to latest.
Whether or not to install a wrapper to wrap subsequent calls of the terraform binary and expose its STDOUT, STDERR, and exit code as outputs named stdout, stderr, and exitcode respectively. Defaults to true.
Version of the aws cli to use
| `false` | `2.15.52` | + + +## Outputs + +| name | description | +| --- | --- | +| `eks-cluster-endpoint` |The API endpoint of the deployed EKS cluster
| +| `terraform-state-url` |URL of the Terraform state file in the S3 bucket
| +| `all-terraform-outputs` |All outputs from Terraform
| + + +## Runs + +This action is a `composite` action. + +## Usage + +```yaml +- uses: camunda/camunda-tf-eks-module/eks-manage-cluster@main + with: + aws-region: + # AWS region where the EKS cluster will be deployed + # + # Required: true + # Default: "" + + cluster-name: + # Name of the EKS cluster to deploy + # + # Required: true + # Default: "" + + additional-terraform-vars: + # JSON object containing additional Terraform variables + # + # Required: false + # Default: {} + + s3-backend-bucket: + # Name of the S3 bucket to store Terraform state + # + # Required: true + # Default: "" + + s3-bucket-region: + # Region of the bucket containing the resources states, if not set, will fallback on aws-region + # + # Required: false + # Default: "" + + tf-modules-revision: + # Git revision of the tf modules to use + # + # Required: false + # Default: main + + tf-modules-path: + # Path where the tf EKS modules will be cloned + # + # Required: false + # Default: ./.action-tf-modules/eks/ + + login: + # Authenticate the current kube context on the created cluster + # + # Required: false + # Default: true + + tf-cli-config-credentials-hostname: + # The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file. Defaults to `app.terraform.io`. + # + # Required: false + # Default: app.terraform.io + + tf-cli-config-credentials-token: + # The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file. + # + # Required: false + # Default: "" + + tf-terraform-version: + # The version of Terraform CLI to install. Instead of full version string you can also specify constraint string starting with "<" (for example `<1.13.0`) to install the latest version satisfying the constraint. A value of `latest` will install the latest version of Terraform CLI. Defaults to `latest`. + # + # Required: false + # Default: latest + + tf-terraform-wrapper: + # Whether or not to install a wrapper to wrap subsequent calls of the `terraform` binary and expose its STDOUT, STDERR, and exit code as outputs named `stdout`, `stderr`, and `exitcode` respectively. Defaults to `true`. + # + # Required: false + # Default: true + + awscli-version: + # Version of the aws cli to use + # + # Required: false + # Default: 2.15.52 +``` diff --git a/.github/actions/eks-manage-cluster/action.yml b/.github/actions/eks-manage-cluster/action.yml new file mode 100644 index 00000000..15e212d1 --- /dev/null +++ b/.github/actions/eks-manage-cluster/action.yml @@ -0,0 +1,168 @@ +--- +name: Deploy an EKS Cluster + +description: | + This GitHub Action automates the deployment of an EKS (Amazon Elastic Kubernetes Service) cluster using Terraform. + This action will also install Terraform, awscli, and kubectl. The kube context will be set on the created cluster. + + +inputs: + aws-region: + description: AWS region where the EKS cluster will be deployed + required: true + + cluster-name: + description: Name of the EKS cluster to deploy + required: true + + additional-terraform-vars: + description: JSON object containing additional Terraform variables + required: false + default: '{}' + + s3-backend-bucket: + description: Name of the S3 bucket to store Terraform state + required: true + + s3-bucket-region: + description: Region of the bucket containing the resources states, if not set, will fallback on aws-region + required: false + + tf-modules-revision: + description: Git revision of the tf modules to use + default: main + + tf-modules-path: + description: Path where the tf EKS modules will be cloned + default: ./.action-tf-modules/eks/ + + login: + description: Authenticate the current kube context on the created cluster + default: 'true' + + # inherited from https://github.com/hashicorp/setup-terraform/blob/main/action.yml + tf-cli-config-credentials-hostname: + description: The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration + file. Defaults to `app.terraform.io`. + default: app.terraform.io + + tf-cli-config-credentials-token: + description: The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration + file. + required: false + + tf-terraform-version: + description: The version of Terraform CLI to install. Instead of full version string you can also specify constraint string starting with "<" (for + example `<1.13.0`) to install the latest version satisfying the constraint. A value of `latest` will install the latest version of Terraform + CLI. Defaults to `latest`. + default: latest + + tf-terraform-wrapper: + description: Whether or not to install a wrapper to wrap subsequent calls of the `terraform` binary and expose its STDOUT, STDERR, and exit code + as outputs named `stdout`, `stderr`, and `exitcode` respectively. Defaults to `true`. + default: 'true' + + awscli-version: + description: Version of the aws cli to use + # renovate: datasource=github-releases depName=aws/aws-cli + default: 2.15.52 + +outputs: + eks-cluster-endpoint: + description: The API endpoint of the deployed EKS cluster + value: ${{ steps.apply.outputs.cluster_endpoint }} + + terraform-state-url: + description: URL of the Terraform state file in the S3 bucket + value: ${{ steps.utility.outputs.terraform-state-url }} + + # Add all terraform outputs dynamically + all-terraform-outputs: + description: All outputs from Terraform + value: ${{ steps.fetch_outputs.outputs.all_terraform_outputs }} + +runs: + using: composite + steps: + - name: Use Utility Actions + id: utility + # seehttps://github.com/orgs/community/discussions/41927 it's not possible to optimize this yet + # steps.uses cannot access the github context. + # uses: ${{ github.action_repository }}/utility-action@${{ github.action_ref }} + # TODO: hard pin this one once it's merged + #uses: "camunda/camunda-tf-eks-module/utility-action@main" + uses: ./.github/actions/utility-action + with: + awscli-version: ${{ inputs.awscli-version }} + terraform-version: ${{ inputs.terraform-version }} + + aws-region: ${{ inputs.aws-region }} + s3-backend-bucket: ${{ inputs.s3-backend-bucket }} + s3-bucket-region: ${{ inputs.s3-bucket-region }} + + tf-state-key: terraform/${{ inputs.cluster-name }}/gha/eks-cluster/terraform.tfstate + + tf-cli-config-credentials-hostname: ${{ inputs.tf-cli-config-credentials-hostname }} + tf-cli-config-credentials-token: ${{ inputs.tf-cli-config-credentials-token }} + tf-terraform-wrapper: ${{ inputs.tf-terraform-wrapper }} + + - name: Checkout Repository EKS modules + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + with: + repository: camunda/camunda-tf-eks-module + ref: ${{ inputs.tf-modules-revision }} + path: ${{ inputs.tf-modules-path }} + fetch-depth: 0 + + - name: Terraform Init + shell: bash + id: init + working-directory: ${{ inputs.tf-modules-path }}/modules/eks-cluster/ + run: | + cp ../fixtures/backend.tf ./ + terraform version + terraform init -backend-config="bucket=${{ steps.utility.outputs.TFSTATE_BUCKET }}" -backend-config="key=${{ steps.utility.outputs.TFSTATE_KEY }}" -backend-config="region=${{ steps.utility.outputs.TFSTATE_REGION }}" + terraform validate -no-color + + - name: Terraform Plan + shell: bash + id: plan + working-directory: ${{ inputs.tf-modules-path }}/modules/eks-cluster/ + run: | + echo '${{ inputs.additional-terraform-vars }}' > /tmp/var.tfvars.json + terraform plan -no-color -out eks.plan \ + -var-file=/tmp/var.tfvars.json \ + -var "name=${{ inputs.cluster-name }}" \ + -var "region=${{ inputs.aws-region }}" \ + -var "name=${{ inputs.cluster-name }}" + + - name: Terraform Apply + shell: bash + id: apply + working-directory: ${{ inputs.tf-modules-path }}/modules/eks-cluster/ + run: | + terraform apply -no-color eks.plan + export cluster_endpoint="$(terraform output -raw cluster_endpoint)" + echo "cluster_endpoint=$cluster_endpoint" >> "$GITHUB_OUTPUT" + + - name: Fetch Terraform Outputs + shell: bash + id: fetch_outputs + working-directory: ${{ inputs.tf-modules-path }}/modules/eks-cluster/ + run: | + all_outputs=$(terraform output -json | jq -c .) + echo "all_terraform_outputs=$all_outputs" | tee -a "$GITHUB_OUTPUT" + + - name: Configure kubectl + shell: bash + id: kube_config + if: inputs.login == 'true' + run: | + aws eks --region ${{ inputs.aws-region }} update-kubeconfig --name ${{ inputs.cluster-name }} + + - name: Output Kube Config + shell: bash + if: inputs.login == 'true' + run: | + kubectl config view + kubectl get ns diff --git a/.github/actions/utility-action/README.md b/.github/actions/utility-action/README.md new file mode 100644 index 00000000..84efd8f8 --- /dev/null +++ b/.github/actions/utility-action/README.md @@ -0,0 +1,99 @@ +# Utility Actions + +## Description + +A set of utility steps to be used across different workflows, including: +- Installing Terraform +- Installing AWS CLI +- Setting Terraform variables +- Checking/Creating an S3 bucket + + +## Inputs + +| name | description | required | default | +| --- | --- | --- | --- | +| `awscli-version` |Version of the AWS CLI to install
| `false` | `2.15.52` | +| `terraform-version` |Version of Terraform to install
| `false` | `latest` | +| `s3-backend-bucket` |Name of the S3 bucket to store Terraform state
| `true` | `""` | +| `s3-bucket-region` |Region of the bucket containing the resources states, if not set, will fallback on aws-region
| `false` | `""` | +| `aws-region` |AWS region to use for S3 bucket operations
| `true` | `""` | +| `tf-state-key` |Key use to store the tfstate file (e.g.: /tfstates/terraform.tfstate)
| `true` | `""` | +| `tf-cli-config-credentials-hostname` |The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file
| `false` | `app.terraform.io` | +| `tf-cli-config-credentials-token` |The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file
| `false` | `""` | +| `tf-terraform-wrapper` |Whether or not to install a wrapper for Terraform CLI
| `false` | `true` | + + +## Outputs + +| name | description | +| --- | --- | +| `terraform-state-url` |URL of the Terraform state file in the S3 bucket
| +| `TFSTATE_BUCKET` |S3 bucket name for Terraform state
| +| `TFSTATE_REGION` |Region of the S3 bucket for Terraform state
| +| `TFSTATE_KEY` |Key of the Terraform state file in the S3 bucket
| + + +## Runs + +This action is a `composite` action. + +## Usage + +```yaml +- uses: camunda/camunda-tf-eks-module/utility-action@main + with: + awscli-version: + # Version of the AWS CLI to install + # + # Required: false + # Default: 2.15.52 + + terraform-version: + # Version of Terraform to install + # + # Required: false + # Default: latest + + s3-backend-bucket: + # Name of the S3 bucket to store Terraform state + # + # Required: true + # Default: "" + + s3-bucket-region: + # Region of the bucket containing the resources states, if not set, will fallback on aws-region + # + # Required: false + # Default: "" + + aws-region: + # AWS region to use for S3 bucket operations + # + # Required: true + # Default: "" + + tf-state-key: + # Key use to store the tfstate file (e.g.: /tfstates/terraform.tfstate) + # + # Required: true + # Default: "" + + tf-cli-config-credentials-hostname: + # The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file + # + # Required: false + # Default: app.terraform.io + + tf-cli-config-credentials-token: + # The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration file + # + # Required: false + # Default: "" + + tf-terraform-wrapper: + # Whether or not to install a wrapper for Terraform CLI + # + # Required: false + # Default: true +``` diff --git a/.github/actions/utility-action/action.yml b/.github/actions/utility-action/action.yml new file mode 100644 index 00000000..c85de862 --- /dev/null +++ b/.github/actions/utility-action/action.yml @@ -0,0 +1,123 @@ +--- +name: Utility Actions + +# TODO: this action should be extracted and made available as a global reusable action (see https://github.com/camunda/camunda-tf-eks-module/pull/112#discussion_r1761274718) + +description: | + A set of utility steps to be used across different workflows, including: + - Installing Terraform + - Installing AWS CLI + - Setting Terraform variables + - Checking/Creating an S3 bucket + + +inputs: + awscli-version: + description: Version of the AWS CLI to install + # renovate: datasource=github-releases depName=aws/aws-cli + default: 2.15.52 + + terraform-version: + description: Version of Terraform to install + default: latest + + s3-backend-bucket: + description: Name of the S3 bucket to store Terraform state + required: true + + s3-bucket-region: + description: Region of the bucket containing the resources states, if not set, will fallback on aws-region + required: false + + aws-region: + description: AWS region to use for S3 bucket operations + required: true + + tf-state-key: + description: 'Key use to store the tfstate file (e.g.: /tfstates/terraform.tfstate)' + required: true + + tf-cli-config-credentials-hostname: + description: The hostname of a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration + file + default: app.terraform.io + + tf-cli-config-credentials-token: + description: The API token for a HCP Terraform/Terraform Enterprise instance to place within the credentials block of the Terraform CLI configuration + file + required: false + + tf-terraform-wrapper: + description: Whether or not to install a wrapper for Terraform CLI + default: 'true' + +outputs: + terraform-state-url: + description: URL of the Terraform state file in the S3 bucket + value: ${{ steps.set-terraform-variables.outputs.terraform-state-url }} + TFSTATE_BUCKET: + description: S3 bucket name for Terraform state + value: ${{ steps.set-terraform-variables.outputs.TFSTATE_BUCKET }} + TFSTATE_REGION: + description: Region of the S3 bucket for Terraform state + value: ${{ steps.set-terraform-variables.outputs.TFSTATE_REGION }} + TFSTATE_KEY: + description: Key of the Terraform state file in the S3 bucket + value: ${{ steps.set-terraform-variables.outputs.TFSTATE_KEY }} + +runs: + using: composite + steps: + - name: Install Terraform + uses: hashicorp/setup-terraform@b9cd54a3c349d3f38e8881555d616ced269862dd # v3 + with: + cli_config_credentials_hostname: ${{ inputs.tf-cli-config-credentials-hostname }} + cli_config_credentials_token: ${{ inputs.tf-cli-config-credentials-token }} + terraform_version: ${{ inputs.terraform-version }} + terraform_wrapper: ${{ inputs.tf-terraform-wrapper }} + + - name: Install AWS CLI + shell: bash + run: | + if ! command -v aws &> /dev/null; then + echo "AWS CLI not found, installing..." + curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${{ inputs.awscli-version }}.zip" -o "awscliv2.zip" + unzip -qq awscliv2.zip + sudo ./aws/install + rm -f awscliv2.zip + else + echo "Warning: AWS CLI is already installed." + fi + + - name: Set Terraform variables + shell: bash + id: set-terraform-variables + run: | + export TFSTATE_BUCKET="${{ inputs.s3-backend-bucket }}" + export TFSTATE_KEY="${{ inputs.tf-state-key }}" + + if [ -z "${{ inputs.s3-bucket-region }}" ]; then + export TFSTATE_REGION="${{ inputs.aws-region }}" + else + export TFSTATE_REGION="${{ inputs.s3-bucket-region }}" + fi + + echo "TFSTATE_BUCKET=${TFSTATE_BUCKET}" >> "$GITHUB_OUTPUT" + echo "TFSTATE_REGION=${TFSTATE_REGION}" >> "$GITHUB_OUTPUT" + echo "TFSTATE_KEY=${TFSTATE_KEY}" >> "$GITHUB_OUTPUT" + + terraform_state_url="s3://${TFSTATE_BUCKET}/${TFSTATE_KEY}" + echo "terraform-state-url=${terraform_state_url}" >> "$GITHUB_OUTPUT" + + - name: Check if S3 bucket exists + id: create-s3-bucket + shell: bash + run: | + if aws s3api head-bucket --bucket ${{ inputs.s3-backend-bucket }} --region ${{ steps.set-terraform-variables.outputs.TFSTATE_REGION }} 2>/dev/null; then + echo "Bucket already exists" + else + echo "Bucket does not exist, creating..." + aws s3api create-bucket --bucket ${{ inputs.s3-backend-bucket }} --region ${{ steps.set-terraform-variables.outputs.TFSTATE_REGION }} --create-bucket-configuration LocationConstraint=${{ steps.set-terraform-variables.outputs.TFSTATE_REGION }} + fi + + aws s3api put-public-access-block --bucket ${{ inputs.s3-backend-bucket }} --region ${{ steps.set-terraform-variables.outputs.TFSTATE_REGION }} --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" diff --git a/.github/labeler.yml b/.github/labeler.yml index c0978e34..92c89942 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -1,17 +1,18 @@ +--- # Add 'feature' label to any PR where the head branch name starts with `feature` or has a `feature` section in the name feature: - - head-branch: ['^feature', 'feature'] + - head-branch: [^feature, feature] # Add 'test' label to any changes within 'test' folder or any subfolders test: - - changed-files: - - any-glob-to-any-file: - - test/**.go - - test/**/go.mod - - modules/fixtures/** + - changed-files: + - any-glob-to-any-file: + - test/**.go + - test/**/go.mod + - modules/fixtures/** terraform: - - changed-files: - - any-glob-to-any-file: - - modules/**.tf - - .tool-versions + - changed-files: + - any-glob-to-any-file: + - modules/**.tf + - .tool-versions diff --git a/.github/workflows/daily-cleanup.yml b/.github/workflows/daily-cleanup.yml index 280d8773..09190cae 100644 --- a/.github/workflows/daily-cleanup.yml +++ b/.github/workflows/daily-cleanup.yml @@ -2,82 +2,82 @@ name: Daily cleanup resources on: - workflow_dispatch: - inputs: - max_age_hours: - description: "Maximum age of resources in hours" - required: true - default: "20" - pull_request: - # the paths should be synced with ../labeler.yml - paths: - - .github/workflows/daily-cleanup.yml - - .github/actions/eks-cleanup-resources/** + workflow_dispatch: + inputs: + max_age_hours: + description: Maximum age of resources in hours + required: true + default: '20' + pull_request: + # the paths should be synced with ../labeler.yml + paths: + - .github/workflows/daily-cleanup.yml + - .github/actions/eks-cleanup-resources/** - schedule: - - cron: '0 1 * * *' # At 01:00 everyday. + schedule: + - cron: 0 1 * * * # At 01:00 everyday. env: - MAX_AGE_HOURS: "${{ github.event.inputs.max_age_hours || '20' }}" - AWS_PROFILE: "infex" + MAX_AGE_HOURS: ${{ github.event.inputs.max_age_hours || '20' }} + AWS_PROFILE: infex - # please keep those variables synced with tests.yml - AWS_REGION: "eu-west-2" - TF_STATE_BUCKET: "tests-eks-tf-state-eu-central-1" - TF_STATE_BUCKET_REGION: "eu-central-1" + # please keep those variables synced with tests.yml + AWS_REGION: eu-west-2 + TF_STATE_BUCKET: tests-eks-tf-state-eu-central-1 + TF_STATE_BUCKET_REGION: eu-central-1 jobs: - cleanup-clusters: - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 - with: - ref: ${{ github.head_ref }} - fetch-depth: 0 + cleanup-clusters: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + with: + ref: ${{ github.head_ref }} + fetch-depth: 0 - - name: Install tooling using asdf - uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 + - name: Install tooling using asdf + uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 - - name: Use repo .tool-version as global version - run: cp .tool-versions ~/.tool-versions + - name: Use repo .tool-version as global version + run: cp .tool-versions ~/.tool-versions - - name: Import Secrets - id: secrets - uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 - with: - url: ${{ secrets.VAULT_ADDR }} - method: approle - roleId: ${{ secrets.VAULT_ROLE_ID }} - secretId: ${{ secrets.VAULT_SECRET_ID }} - exportEnv: false - secrets: | - secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; - secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; + - name: Import Secrets + id: secrets + uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 + with: + url: ${{ secrets.VAULT_ADDR }} + method: approle + roleId: ${{ secrets.VAULT_ROLE_ID }} + secretId: ${{ secrets.VAULT_SECRET_ID }} + exportEnv: false + secrets: | + secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; + secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; - # Official action does not support profiles - - name: Add profile credentials to ~/.aws/credentials - run: | - aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} - aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} - aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} + # Official action does not support profiles + - name: Add profile credentials to ~/.aws/credentials + run: | + aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} - - name: Delete orphans resources - timeout-minutes: 360 - if: always() - uses: ./.github/actions/eks-cleanup-resources - with: - tf-bucket: "${{ env.TF_STATE_BUCKET }}" - tf-bucket-region: "${{ env.TF_STATE_BUCKET_REGION }}" - max-age-hours: "${{ env.MAX_AGE_HOURS }}" - target: "all" + - name: Delete orphans resources + timeout-minutes: 360 + if: always() + uses: ./.github/actions/eks-cleanup-resources + with: + tf-bucket: ${{ env.TF_STATE_BUCKET }} + tf-bucket-region: ${{ env.TF_STATE_BUCKET_REGION }} + max-age-hours: ${{ env.MAX_AGE_HOURS }} + target: all - - name: Notify in Slack in case of failure - id: slack-notification - if: failure() && github.event_name == 'schedule' - uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main - with: - vault_addr: ${{ secrets.VAULT_ADDR }} - vault_role_id: ${{ secrets.VAULT_ROLE_ID }} - vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} + - name: Notify in Slack in case of failure + id: slack-notification + if: failure() && github.event_name == 'schedule' + uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main + with: + vault_addr: ${{ secrets.VAULT_ADDR }} + vault_role_id: ${{ secrets.VAULT_ROLE_ID }} + vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} diff --git a/.github/workflows/labeler.yml b/.github/workflows/labeler.yml index 0af34642..86b93f3b 100644 --- a/.github/workflows/labeler.yml +++ b/.github/workflows/labeler.yml @@ -1,26 +1,27 @@ -name: "Pull Request Labeler" +--- +name: Pull Request Labeler on: - pull_request_target: - schedule: - - cron: "0 1 * * 1" - pull_request: - paths: - - .github/workflows/labeler.yml + pull_request_target: + schedule: + - cron: 0 1 * * 1 + pull_request: + paths: + - .github/workflows/labeler.yml jobs: - labeler: - permissions: - contents: read - pull-requests: write - runs-on: ubuntu-latest - steps: - - uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5 + labeler: + permissions: + contents: read + pull-requests: write + runs-on: ubuntu-latest + steps: + - uses: actions/labeler@8558fd74291d67161a8a78ce36a881fa63b766a9 # v5 - - name: Notify in Slack in case of failure - id: slack-notification - if: failure() && github.event_name == 'schedule' - uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main - with: - vault_addr: ${{ secrets.VAULT_ADDR }} - vault_role_id: ${{ secrets.VAULT_ROLE_ID }} - vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} + - name: Notify in Slack in case of failure + id: slack-notification + if: failure() && github.event_name == 'schedule' + uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main + with: + vault_addr: ${{ secrets.VAULT_ADDR }} + vault_role_id: ${{ secrets.VAULT_ROLE_ID }} + vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} diff --git a/.github/workflows/links.yml b/.github/workflows/links.yml index f9617703..67316bb0 100644 --- a/.github/workflows/links.yml +++ b/.github/workflows/links.yml @@ -1,52 +1,53 @@ +--- name: Check external links on: - push: - workflow_dispatch: - schedule: - - cron: "0 3 1 * *" - pull_request: - paths: - - .github/workflows/links.yml + push: + workflow_dispatch: + schedule: + - cron: 0 3 1 * * + pull_request: + paths: + - .github/workflows/links.yml jobs: - lint: - name: links-check - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + lint: + name: links-check + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 - - name: Get Current Timestamp - id: timestamp - run: echo "TIMESTAMP=$(date +%s)" >> "$GITHUB_ENV" + - name: Get Current Timestamp + id: timestamp + run: echo "TIMESTAMP=$(date +%s)" >> "$GITHUB_ENV" - - name: Restore lychee cache - uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4 - with: - path: .lycheecache - key: "cache-lychee-${{ env.TIMESTAMP }}" - restore-keys: cache-lychee- + - name: Restore lychee cache + uses: actions/cache@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9 # v4 + with: + path: .lycheecache + key: cache-lychee-${{ env.TIMESTAMP }} + restore-keys: cache-lychee- - - name: Link Checker - uses: lycheeverse/lychee-action@2b973e86fc7b1f6b36a93795fe2c9c6ae1118621 # v1.10.0 - with: - fail: true - args: -c ./lychee-links.toml --base . --cache --max-cache-age 1d . --verbose --no-progress '*.md' './**/*.md' - token: "${{ secrets.GITHUB_TOKEN }}" + - name: Link Checker + uses: lycheeverse/lychee-action@2b973e86fc7b1f6b36a93795fe2c9c6ae1118621 # v1.10.0 + with: + fail: true + args: -c ./lychee-links.toml --base . --cache --max-cache-age 1d . --verbose --no-progress '*.md' './**/*.md' + token: ${{ secrets.GITHUB_TOKEN }} - - name: Create Issue From File - if: failure() && github.event_name == 'schedule' - uses: peter-evans/create-issue-from-file@24452a72d85239eacf1468b0f1982a9f3fec4c94 # v5 - with: - title: Link Checker Report - content-filepath: ./lychee/out.md - labels: report, automated issue + - name: Create Issue From File + if: failure() && github.event_name == 'schedule' + uses: peter-evans/create-issue-from-file@24452a72d85239eacf1468b0f1982a9f3fec4c94 # v5 + with: + title: Link Checker Report + content-filepath: ./lychee/out.md + labels: report, automated issue - - name: Notify in Slack in case of failure - id: slack-notification - if: failure() && github.event_name == 'schedule' - uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main - with: - vault_addr: ${{ secrets.VAULT_ADDR }} - vault_role_id: ${{ secrets.VAULT_ROLE_ID }} - vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} + - name: Notify in Slack in case of failure + id: slack-notification + if: failure() && github.event_name == 'schedule' + uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main + with: + vault_addr: ${{ secrets.VAULT_ADDR }} + vault_role_id: ${{ secrets.VAULT_ROLE_ID }} + vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 93393ba5..6efeb5d3 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -3,17 +3,17 @@ name: lint on: - push: - workflow_dispatch: + push: + workflow_dispatch: jobs: - lint: - name: pre-commit - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 - - name: Install tooling using asdf - uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 + lint: + name: pre-commit + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + - name: Install tooling using asdf + uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 - - uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1 + - uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1 diff --git a/.github/workflows/test-gha-eks.yml b/.github/workflows/test-gha-eks.yml new file mode 100644 index 00000000..59303d48 --- /dev/null +++ b/.github/workflows/test-gha-eks.yml @@ -0,0 +1,184 @@ +--- +name: EKS Cluster with an AuroraDB creation and destruction test + +on: + schedule: + - cron: 0 1 * * 2 # At 01:00 on Tuesday. + + workflow_dispatch: + inputs: + cluster_name: + description: Cluster name. + required: false + type: string + create_db: + description: Should the aurora db be created + default: 'true' + delete_cluster: + description: Whether to delete the cluster. + default: 'true' + db_username: + description: Database username. + required: false + type: string + db_password: + description: Database password. + required: false + type: string + + pull_request: + # the paths should be synced with ../labeler.yml + paths: + - modules/fixtures/backend.tf + - modules/fixtures/fixtures.default.eks.tfvars + - modules/fixtures/fixtures.default.aurora.tfvars + - modules/eks-cluster/**.tf + - modules/aurora/**.tf + - .tool-versions + - .github/workflows/test-gha-eks-manage-cluster.yml + - .github/actions/eks-manage-cluster/*.yml + +# limit to a single execution per actor of this workflow +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +env: + + AWS_PROFILE: infex + AWS_REGION: eu-west-2 + # /!\ always use one of the available test region https://github.com/camunda/infraex-common-config + + # please keep those synced with tests.yml + TF_STATE_BUCKET: tests-eks-tf-state-eu-central-1 + TF_STATE_BUCKET_REGION: eu-central-1 + + CREATE_DB: ${{ github.event.inputs.create_db || 'true' }} + +jobs: + action-test: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + with: + ref: ${{ github.head_ref }} + fetch-depth: 0 + + - name: Install tooling using asdf + uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 + + - name: Get Cluster Info + id: commit_info + run: | + if [[ -n "${{ inputs.cluster_name }}" ]]; then + cluster_name="${{ inputs.cluster_name }}" + else + cluster_name="cl-$(git rev-parse --short HEAD)" + fi + echo "cluster_name=$cluster_name" | tee -a "$GITHUB_OUTPUT" + + if [[ -n "${{ inputs.db_username }}" ]]; then + db_username="${{ inputs.db_username }}" + else + db_username="user$(openssl rand -hex 4 | tr -d '/@" ')" + fi + echo "db_username=$db_username" | tee -a "$GITHUB_OUTPUT" + + if [[ -n "${{ inputs.db_password }}" ]]; then + db_password="${{ inputs.db_password }}" + else + db_password="$(openssl rand -base64 12 | tr -d '/@" ')" + fi + echo "db_password=$db_password" | tee -a "$GITHUB_OUTPUT" + + # Get the current commit hash for the modules revision + tf_modules_revision=$(git rev-parse HEAD) + echo "tf_modules_revision=$tf_modules_revision" | tee -a "$GITHUB_OUTPUT" + + - name: Import Secrets + id: secrets + uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 + with: + url: ${{ secrets.VAULT_ADDR }} + method: approle + roleId: ${{ secrets.VAULT_ROLE_ID }} + secretId: ${{ secrets.VAULT_SECRET_ID }} + exportEnv: false + secrets: | + secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; + secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; + + - name: Add profile credentials to ~/.aws/credentials + run: | + aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} + + - name: Create EKS Cluster + timeout-minutes: 45 + uses: ./.github/actions/eks-manage-cluster + id: create_eks_cluster + with: + cluster-name: ${{ steps.commit_info.outputs.cluster_name }} + aws-region: ${{ env.AWS_REGION }} + + additional-terraform-vars: '{"np_capacity_type": "SPOT", "np_instance_types": ["t2.medium"]}' + + s3-backend-bucket: ${{ env.TF_STATE_BUCKET }} + s3-bucket-region: ${{ env.TF_STATE_BUCKET_REGION }} + tf-modules-revision: ${{ steps.commit_info.outputs.tf_modules_revision }} + + - name: Prepare Aurora Cluster + id: prepare_aurora_cluster + run: | + vpc_id=$(echo '${{ steps.create_eks_cluster.outputs.all-terraform-outputs }}' | jq -c -r '.vpc_id.value') + echo "vpc_id=$vpc_id" | tee -a "$GITHUB_OUTPUT" + + private_subnet_ids=$(echo '${{ steps.create_eks_cluster.outputs.all-terraform-outputs }}' | jq -c -r '.private_subnet_ids.value') + echo "private_subnet_ids=$private_subnet_ids" | tee -a "$GITHUB_OUTPUT" + + private_vpc_cidr_blocks=$(echo '${{ steps.create_eks_cluster.outputs.all-terraform-outputs }}' | jq -c -r '.private_vpc_cidr_blocks.value') + echo "private_vpc_cidr_blocks=$private_vpc_cidr_blocks" | tee -a "$GITHUB_OUTPUT" + + availability_zones=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=${vpc_id}" --query 'Subnets[].AvailabilityZone' --output json | jq 'unique' -c) + echo "availability_zones=$availability_zones" | tee -a "$GITHUB_OUTPUT" + + - name: Create Aurora Cluster + timeout-minutes: 20 + uses: ./.github/actions/aurora-manage-cluster + id: create_aurora_cluster + if: env.CREATE_DB == 'true' + with: + cluster-name: ${{ steps.commit_info.outputs.cluster_name }} + username: ${{ steps.commit_info.outputs.db_username }} + password: ${{ steps.commit_info.outputs.db_password }} + + s3-backend-bucket: ${{ env.TF_STATE_BUCKET }} + s3-bucket-region: ${{ env.TF_STATE_BUCKET_REGION }} + tf-modules-revision: ${{ steps.commit_info.outputs.tf_modules_revision }} + + vpc-id: ${{ steps.prepare_aurora_cluster.outputs.vpc_id }} + subnet-ids: ${{ steps.prepare_aurora_cluster.outputs.private_subnet_ids }} + cidr-blocks: ${{ steps.prepare_aurora_cluster.outputs.private_vpc_cidr_blocks }} + + availability-zones: ${{ steps.prepare_aurora_cluster.outputs.availability_zones }} + + - name: Delete Clusters + timeout-minutes: 60 + if: always() && !(github.event_name == 'workflow_dispatch' && inputs.delete_cluster == 'false') + uses: ./.github/actions/eks-cleanup-resources + with: + tf-bucket: ${{ env.TF_STATE_BUCKET }} + tf-bucket-region: ${{ env.TF_STATE_BUCKET_REGION }} + max-age-hours: 0 + target: ${{ steps.commit_info.outputs.cluster_name }} + + - name: Notify in Slack in case of failure + id: slack-notification + if: failure() && github.event_name == 'schedule' + uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main + with: + vault_addr: ${{ secrets.VAULT_ADDR }} + vault_role_id: ${{ secrets.VAULT_ROLE_ID }} + vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index b19dae18..9bffc803 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -2,247 +2,248 @@ name: Terraform modules tests on: - schedule: - - cron: '0 1 * * 2' - workflow_dispatch: - pull_request: - # the paths should be synced with ../labeler.yml - paths: - - test/**.go - - test/**/go.mod - - modules/fixtures/** - - modules/**.tf - - .tool-versions - - .github/workflows/tests.yml - - justfile + schedule: + - cron: 0 1 * * 2 + workflow_dispatch: + pull_request: + # the paths should be synced with ../labeler.yml + paths: + - test/**.go + - test/**/go.mod + - modules/fixtures/** + - modules/**.tf + - .tool-versions + - .github/workflows/tests.yml + - justfile # limit to a single execution per ref of this workflow concurrency: - group: "${{ github.workflow }}-${{ github.ref }}" - cancel-in-progress: true + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true env: - # please keep those variables synced with daily-cleanup.yml - AWS_PROFILE: "infex" - AWS_REGION: "eu-west-2" # /!\ always use one of the available test region https://github.com/camunda/infraex-common-config - TESTS_TF_BINARY_NAME: "terraform" + # please keep those variables synced with daily-cleanup.yml + AWS_PROFILE: infex + AWS_REGION: eu-west-2 # /!\ always use one of the available test region https://github.com/camunda/infraex-common-config + TESTS_TF_BINARY_NAME: terraform - TF_STATE_BUCKET: "tests-eks-tf-state-eu-central-1" - TF_STATE_BUCKET_REGION: "eu-central-1" + # please keep test-gha*.yml synced + TF_STATE_BUCKET: tests-eks-tf-state-eu-central-1 + TF_STATE_BUCKET_REGION: eu-central-1 jobs: - # We can skip some tests using the commit description (skip-tests:NameOfTest1,NameOfTest2) or all tests (skip-tests:all) (see `DEVELOPER.md`) - # If all tests are skipped, the result of this workflow will be `failed` on purpose - # If you want to skip tests and have no error, you need to use `testing-ci-not-necessary` as a label on the PR - configure-tests: - runs-on: ubuntu-latest - if: >- - github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || ( - github.event_name == 'pull_request' && - !contains(github.event.pull_request.labels.*.name, 'testing-ci-not-necessary') - ) - outputs: - test_functions: ${{ steps.extract_test_functions.outputs.test_functions }} - cluster_id: ${{ steps.short_git_sha.outputs.short_git_sha }} - steps: - - name: Checkout repository - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 - with: - ref: ${{ github.head_ref }} - fetch-depth: 0 - - - name: Get Short GitHub SHA - id: short_git_sha - run: echo "short_git_sha=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT" - - - name: Extract Test Functions - id: extract_test_functions - run: | - test_functions=$(grep -rho 'func \(Test[^ ]*\)' ./test/src/ | sed 's/func \(Test[^ ]*\)(t/\1/' | tr '\n' ',' | sed 's/,$//') - echo "test_functions=$test_functions" - - : # Extract test names marked to be skipped from the commit message description - commit_message=$(git log -1 --pretty=format:"%B") - echo "commit_message=$commit_message" - - skipped_tests=$(echo "$commit_message" | grep 'skip-tests' | sed 's/skip-tests://') - echo "skipped_tests=$skipped_tests" - - : # If all tests are marked to be skipped, then clear the test_functions list completely - if [ "$skipped_tests" == "all" ]; then - test_functions="" - echo "Skipping all tests (skip-tests:all found), this workflow will fail. If you want to skip-tests for a PR, please use the label 'testing-ci-not-necessary'" - else - : # Otherwise, remove the tests marked to be skipped from the test_functions list - if [ -n "$skipped_tests" ]; then - for test in $(echo "$skipped_tests" | tr ',' '\n'); do - echo "Skipping test: $test" - test_functions=$(echo "$test_functions" | sed "s/$test//g" | sed 's/,,/,/g' | sed 's/^,//' | sed 's/,$//') - echo "test_functions=$test_functions" - done - fi - fi - - : # to json array - IFS=',' read -ra array <<< "$test_functions" - json_array="[" - for element in "${array[@]}" - do - json_array+="\"$element\"," - done - test_functions="${json_array%,}]" - - echo "test_functions=${test_functions}" >> "$GITHUB_OUTPUT" - echo "test_functions=${test_functions}" - - integration-tests: - runs-on: ubuntu-latest - needs: - - configure-tests - strategy: - fail-fast: false # don't propagate failing jobs - matrix: - test_function: ${{ fromJson(needs.configure-tests.outputs.test_functions) }} - steps: - - name: Checkout repository - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 - - - name: Install tooling using asdf - uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 - - - name: Import Secrets - id: secrets - uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 - with: - url: ${{ secrets.VAULT_ADDR }} - method: approle - roleId: ${{ secrets.VAULT_ROLE_ID }} - secretId: ${{ secrets.VAULT_SECRET_ID }} - exportEnv: false - secrets: | - secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; - secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; - - # Official action does not support profiles - - name: Add profile credentials to ~/.aws/credentials - run: | - aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} - aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} - aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} - - - name: Get go.mod details - uses: Eun/go-mod-details@b719cd324463e2037cf3a0dd1dd6091bdc2730f4 # v1 - id: go-mod-details - with: - modfile: ${{ github.workspace }}/test/src/go.mod - - - name: Launch test - timeout-minutes: 125 - run: | - export TESTS_CLUSTER_ID="${{ needs.configure-tests.outputs.cluster_id }}" - export TESTS_CLUSTER_REGION="${{ env.AWS_REGION }}" - export TESTS_TF_BINARY_NAME="${{ env.TESTS_TF_BINARY_NAME }}" - just test ${{ matrix.test_function }} "--junitfile ${{ matrix.test_function }}_unit-tests.xml" - - # this is a workaround for test report not working as expected due to https://github.com/test-summary/action/issues/5 - - name: Filter logger.go from the test report (too large) + # We can skip some tests using the commit description (skip-tests:NameOfTest1,NameOfTest2) or all tests (skip-tests:all) (see `DEVELOPER.md`) + # If all tests are skipped, the result of this workflow will be `failed` on purpose + # If you want to skip tests and have no error, you need to use `testing-ci-not-necessary` as a label on the PR + configure-tests: + runs-on: ubuntu-latest + if: >- + github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || ( + github.event_name == 'pull_request' && + !contains(github.event.pull_request.labels.*.name, 'testing-ci-not-necessary') + ) + outputs: + test_functions: ${{ steps.extract_test_functions.outputs.test_functions }} + cluster_id: ${{ steps.short_git_sha.outputs.short_git_sha }} + steps: + - name: Checkout repository + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + with: + ref: ${{ github.head_ref }} + fetch-depth: 0 + + - name: Get Short GitHub SHA + id: short_git_sha + run: echo "short_git_sha=$(git rev-parse --short HEAD)" >> "$GITHUB_OUTPUT" + + - name: Extract Test Functions + id: extract_test_functions + run: | + test_functions=$(grep -rho 'func \(Test[^ ]*\)' ./test/src/ | sed 's/func \(Test[^ ]*\)(t/\1/' | tr '\n' ',' | sed 's/,$//') + echo "test_functions=$test_functions" + + : # Extract test names marked to be skipped from the commit message description + commit_message=$(git log -1 --pretty=format:"%B") + echo "commit_message=$commit_message" + + skipped_tests=$(echo "$commit_message" | grep 'skip-tests' | sed 's/skip-tests://') + echo "skipped_tests=$skipped_tests" + + : # If all tests are marked to be skipped, then clear the test_functions list completely + if [ "$skipped_tests" == "all" ]; then + test_functions="" + echo "Skipping all tests (skip-tests:all found), this workflow will fail. If you want to skip-tests for a PR, please use the label 'testing-ci-not-necessary'" + else + : # Otherwise, remove the tests marked to be skipped from the test_functions list + if [ -n "$skipped_tests" ]; then + for test in $(echo "$skipped_tests" | tr ',' '\n'); do + echo "Skipping test: $test" + test_functions=$(echo "$test_functions" | sed "s/$test//g" | sed 's/,,/,/g' | sed 's/^,//' | sed 's/,$//') + echo "test_functions=$test_functions" + done + fi + fi + + : # to json array + IFS=',' read -ra array <<< "$test_functions" + json_array="[" + for element in "${array[@]}" + do + json_array+="\"$element\"," + done + test_functions="${json_array%,}]" + + echo "test_functions=${test_functions}" >> "$GITHUB_OUTPUT" + echo "test_functions=${test_functions}" + + integration-tests: + runs-on: ubuntu-latest + needs: + - configure-tests + strategy: + fail-fast: false # don't propagate failing jobs + matrix: + test_function: ${{ fromJson(needs.configure-tests.outputs.test_functions) }} + steps: + - name: Checkout repository + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + + - name: Install tooling using asdf + uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 + + - name: Import Secrets + id: secrets + uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 + with: + url: ${{ secrets.VAULT_ADDR }} + method: approle + roleId: ${{ secrets.VAULT_ROLE_ID }} + secretId: ${{ secrets.VAULT_SECRET_ID }} + exportEnv: false + secrets: | + secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; + secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; + + # Official action does not support profiles + - name: Add profile credentials to ~/.aws/credentials + run: | + aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} + + - name: Get go.mod details + uses: Eun/go-mod-details@b719cd324463e2037cf3a0dd1dd6091bdc2730f4 # v1 + id: go-mod-details + with: + modfile: ${{ github.workspace }}/test/src/go.mod + + - name: Launch test + timeout-minutes: 125 + run: | + export TESTS_CLUSTER_ID="${{ needs.configure-tests.outputs.cluster_id }}" + export TESTS_CLUSTER_REGION="${{ env.AWS_REGION }}" + export TESTS_TF_BINARY_NAME="${{ env.TESTS_TF_BINARY_NAME }}" + just test ${{ matrix.test_function }} "--junitfile ${{ matrix.test_function }}_unit-tests.xml" + + # this is a workaround for test report not working as expected due to https://github.com/test-summary/action/issues/5 + - name: Filter logger.go from the test report (too large) + if: always() + run: | + sed 's/ /\n/g' < "./test/src/${{ matrix.test_function }}_unit-tests.xml" | grep -E -v '^.*logger\.go.*$' | sed 's/\n/ /g' > "./test/src/${{ matrix.test_function }}_unit-tests_filtered.xml" + + - name: Upload test reports + if: always() + uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4 + with: + name: test-reports-${{ matrix.test_function }} + path: ./test/src/${{ matrix.test_function }}_unit-tests_filtered.xml + retention-days: 1 + + - name: Remove profile credentials from ~/.aws/credentials + if: always() + run: | + rm -rf ~/.aws/credentials + + test-report: + runs-on: ubuntu-latest + if: ${{ always() && needs.configure-tests.result == 'success' }} + needs: + - configure-tests + - integration-tests + steps: + - name: Download artifacts + uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4 + with: + pattern: test-reports-* + path: /tmp/testreports + merge-multiple: true + + - name: Run test-summary + uses: test-summary/action@31493c76ec9e7aa675f1585d3ed6f1da69269a86 # v2 + with: + paths: /tmp/testreports/**/*.xml + + cleanup-resources: + runs-on: ubuntu-latest if: always() - run: | - sed 's/ /\n/g' < "./test/src/${{ matrix.test_function }}_unit-tests.xml" | grep -E -v '^.*logger\.go.*$' | sed 's/\n/ /g' > "./test/src/${{ matrix.test_function }}_unit-tests_filtered.xml" - - - name: Upload test reports - if: always() - uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4 - with: - name: test-reports-${{ matrix.test_function }} - path: "./test/src/${{ matrix.test_function }}_unit-tests_filtered.xml" - retention-days: 1 - - - name: Remove profile credentials from ~/.aws/credentials - if: always() - run: | - rm -rf ~/.aws/credentials - - test-report: - runs-on: ubuntu-latest - if: ${{ always() && needs.configure-tests.result == 'success' }} - needs: - - configure-tests - - integration-tests - steps: - - name: Download artifacts - uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4 - with: - pattern: test-reports-* - path: /tmp/testreports - merge-multiple: true - - - name: Run test-summary - uses: test-summary/action@31493c76ec9e7aa675f1585d3ed6f1da69269a86 # v2 - with: - paths: /tmp/testreports/**/*.xml - - cleanup-resources: - runs-on: ubuntu-latest - if: always() - needs: - - configure-tests - - integration-tests - steps: - - name: Checkout repository - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 - with: - ref: ${{ github.head_ref }} - fetch-depth: 0 - - - name: Install tooling using asdf - uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 - - - name: Import Secrets - id: secrets - uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 - with: - url: ${{ secrets.VAULT_ADDR }} - method: approle - roleId: ${{ secrets.VAULT_ROLE_ID }} - secretId: ${{ secrets.VAULT_SECRET_ID }} - exportEnv: false - secrets: | - secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; - secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; - - # Official action does not support profiles - - name: Add profile credentials to ~/.aws/credentials - run: | - aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} - aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} - aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} - - - name: Delete resources of this run - timeout-minutes: 125 - if: always() - uses: ./.github/actions/eks-cleanup-resources - with: - tf-bucket: "${{ env.TF_STATE_BUCKET }}" - tf-bucket-region: "${{ env.TF_STATE_BUCKET_REGION }}" - max-age-hours: "0" - target: "${{ needs.configure-tests.outputs.cluster_id }}" - - notify-on-failure: - runs-on: ubuntu-latest - if: failure() - needs: - - configure-tests - - integration-tests - - test-report - - cleanup-resources - steps: - - name: Notify in Slack in case of failure - id: slack-notification - if: github.event_name == 'schedule' - uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main - with: - vault_addr: ${{ secrets.VAULT_ADDR }} - vault_role_id: ${{ secrets.VAULT_ROLE_ID }} - vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} + needs: + - configure-tests + - integration-tests + steps: + - name: Checkout repository + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4 + with: + ref: ${{ github.head_ref }} + fetch-depth: 0 + + - name: Install tooling using asdf + uses: asdf-vm/actions/install@05e0d2ed97b598bfce82fd30daf324ae0c4570e6 # v3 + + - name: Import Secrets + id: secrets + uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c # v3 + with: + url: ${{ secrets.VAULT_ADDR }} + method: approle + roleId: ${{ secrets.VAULT_ROLE_ID }} + secretId: ${{ secrets.VAULT_SECRET_ID }} + exportEnv: false + secrets: | + secret/data/products/infrastructure-experience/ci/common AWS_ACCESS_KEY; + secret/data/products/infrastructure-experience/ci/common AWS_SECRET_KEY; + + # Official action does not support profiles + - name: Add profile credentials to ~/.aws/credentials + run: | + aws configure set aws_access_key_id ${{ steps.secrets.outputs.AWS_ACCESS_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set aws_secret_access_key ${{ steps.secrets.outputs.AWS_SECRET_KEY }} --profile ${{ env.AWS_PROFILE }} + aws configure set region ${{ env.AWS_REGION }} --profile ${{ env.AWS_PROFILE }} + + - name: Delete resources of this run + timeout-minutes: 125 + if: always() + uses: ./.github/actions/eks-cleanup-resources + with: + tf-bucket: ${{ env.TF_STATE_BUCKET }} + tf-bucket-region: ${{ env.TF_STATE_BUCKET_REGION }} + max-age-hours: '0' + target: ${{ needs.configure-tests.outputs.cluster_id }} + + notify-on-failure: + runs-on: ubuntu-latest + if: failure() + needs: + - configure-tests + - integration-tests + - test-report + - cleanup-resources + steps: + - name: Notify in Slack in case of failure + id: slack-notification + if: github.event_name == 'schedule' + uses: camunda/infraex-common-config/.github/actions/report-failure-on-slack@fe25bf36edcc03717275a4e05969cdb5483388df # main + with: + vault_addr: ${{ secrets.VAULT_ADDR }} + vault_role_id: ${{ secrets.VAULT_ROLE_ID }} + vault_secret_id: ${{ secrets.VAULT_SECRET_ID }} diff --git a/.lint/terraform_docs/.terraform-docs.yml b/.lint/terraform_docs/.terraform-docs.yml index edd51d8f..7cfe2d36 100644 --- a/.lint/terraform_docs/.terraform-docs.yml +++ b/.lint/terraform_docs/.terraform-docs.yml @@ -1,42 +1,42 @@ --- # Configuration documentation https://terraform-docs.io/user-guide/configuration/ -formatter: "markdown table" # Required -version: "" # version is managed by asdf +formatter: markdown table # Required +version: '' # version is managed by asdf header-from: main.tf -footer-from: "" +footer-from: '' recursive: - enabled: false + enabled: false content: |- - {{ .Modules }} - {{ .Resources }} - {{ .Inputs }} - {{ .Outputs }} + {{ .Modules }} + {{ .Resources }} + {{ .Inputs }} + {{ .Outputs }} sections: - hide: [] - show: [] + hide: [] + show: [] output-values: - enabled: false - from: "" + enabled: false + from: '' sort: - enabled: true - by: name + enabled: true + by: name settings: - anchor: true - color: true - default: true - description: true - escape: true - hide-empty: false - html: true - indent: 2 - lockfile: false - read-comments: true - required: true - sensitive: true - type: true + anchor: true + color: true + default: true + description: true + escape: true + hide-empty: false + html: true + indent: 2 + lockfile: false + read-comments: true + required: true + sensitive: true + type: true diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 337f94fb..f5b6eed5 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -2,56 +2,64 @@ # See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: + - repo: https://github.com/camunda/infraex-common-config + rev: 5a264b0 # TODO: replace with main onced it's merged + hooks: + - id: update-action-readmes-docker -- repo: https://github.com/pre-commit/pre-commit-hooks - rev: v4.6.0 - hooks: - - id: check-added-large-files - - id: end-of-file-fixer - - id: trailing-whitespace - - id: check-yaml - args: [--allow-multiple-documents] - - id: check-json - - id: check-symlinks - - id: check-shebang-scripts-are-executable - - id: detect-private-key + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v4.6.0 + hooks: + - id: check-added-large-files + - id: end-of-file-fixer + - id: trailing-whitespace + - id: check-yaml + args: [--allow-multiple-documents] + - id: check-json + - id: check-symlinks + - id: check-shebang-scripts-are-executable + - id: detect-private-key -- repo: https://github.com/rhysd/actionlint - rev: v1.7.1 - hooks: - - id: actionlint-docker + - repo: https://github.com/rhysd/actionlint + rev: v1.7.1 + hooks: + - id: actionlint-docker -- repo: https://github.com/renovatebot/pre-commit-hooks - rev: 38.80.0 - hooks: - - id: renovate-config-validator - args: ["--strict"] + - repo: https://github.com/renovatebot/pre-commit-hooks + rev: 38.80.0 + hooks: + - id: renovate-config-validator + args: [--strict] -- repo: https://github.com/compilerla/conventional-pre-commit - rev: v3.4.0 # use tags until renovate supports sha: https://github.com/renovatebot/renovate/issues/22567 - hooks: - - id: conventional-pre-commit - stages: [commit-msg] - args: ["--strict" , "--force-scope"] + - repo: https://github.com/compilerla/conventional-pre-commit + rev: v3.4.0 # use tags until renovate supports sha: https://github.com/renovatebot/renovate/issues/22567 + hooks: + - id: conventional-pre-commit + stages: [commit-msg] + args: [--strict, --force-scope] -- repo: https://github.com/antonbabenko/pre-commit-terraform - rev: v1.95.0 - hooks: - - id: terraform_fmt - - id: terraform_tflint - args: - - --args=--config=__GIT_WORKING_DIR__/.lint/tflint/.tflint.hcl - - id: terraform_docs - args: - - --hook-config=--path-to-file=README.md - - --hook-config=--create-file-if-not-exist=true - - --hook-config=--add-to-existing-file=true - - --args=--config=.lint/terraform_docs/.terraform-docs.yml + - repo: https://github.com/antonbabenko/pre-commit-terraform + rev: v1.95.0 + hooks: + - id: terraform_fmt + - id: terraform_tflint + args: + - --args=--config=__GIT_WORKING_DIR__/.lint/tflint/.tflint.hcl + - id: terraform_docs + args: + - --hook-config=--path-to-file=README.md + - --hook-config=--create-file-if-not-exist=true + - --hook-config=--add-to-existing-file=true + - --args=--config=.lint/terraform_docs/.terraform-docs.yml -- repo: https://github.com/dnephin/pre-commit-golang - rev: v0.5.1 - hooks: - - id: go-fmt - - id: no-go-testing - - id: go-mod-tidy -... + - repo: https://github.com/dnephin/pre-commit-golang + rev: v0.5.1 + hooks: + - id: go-fmt + - id: no-go-testing + - id: go-mod-tidy + + - repo: https://github.com/jumanjihouse/pre-commit-hook-yamlfmt + rev: 0.2.3 + hooks: + - id: yamlfmt diff --git a/modules/aurora/README.md b/modules/aurora/README.md index 39e8c7f2..26c72c90 100644 --- a/modules/aurora/README.md +++ b/modules/aurora/README.md @@ -55,7 +55,7 @@ No modules. | [engine](#input\_engine) | The engine type e.g. aurora, aurora-mysql, aurora-postgresql, ... | `string` | `"aurora-postgresql"` | no | | [engine\_version](#input\_engine\_version) | The DB engine version for Postgres to use. | `string` | `"15.4"` | no | | [iam\_auth\_enabled](#input\_iam\_auth\_enabled) | Determines whether IAM auth should be activated for IRSA usage | `bool` | `false` | no | -| [iam\_roles](#input\_iam\_roles) | Allows propagating additional IAM roles to the Aurora cluster to allow e.g. access to S3 | `list` | `[]` | no | +| [iam\_roles](#input\_iam\_roles) | Allows propagating additional IAM roles to the Aurora cluster to allow e.g. access to S3 | `list(string)` | `[]` | no | | [instance\_class](#input\_instance\_class) | The instance type of the Aurora instances | `string` | `"db.t3.medium"` | no | | [num\_instances](#input\_num\_instances) | Number of instances | `string` | `"1"` | no | | [password](#input\_password) | The password for the postgres admin user. Important: secret value! | `string` | n/a | yes | diff --git a/modules/aurora/variables.tf b/modules/aurora/variables.tf index f208f256..565b7a5f 100644 --- a/modules/aurora/variables.tf +++ b/modules/aurora/variables.tf @@ -9,7 +9,8 @@ variable "engine" { } variable "engine_version" { - type = string + type = string + # renovate: datasource=endoflife-date depName=amazon-rds-postgresql versioning=semver default = "15.4" description = "The DB engine version for Postgres to use." } @@ -69,6 +70,7 @@ variable "vpc_id" { # Allows adding additional iam roles to grant access from Aurora to e.g. S3 variable "iam_roles" { + type = list(string) default = [] description = "Allows propagating additional IAM roles to the Aurora cluster to allow e.g. access to S3" } diff --git a/modules/eks-cluster/README.md b/modules/eks-cluster/README.md index 32d4fd17..90bda043 100644 --- a/modules/eks-cluster/README.md +++ b/modules/eks-cluster/README.md @@ -51,8 +51,8 @@ module "eks_cluster" { |------|-------------|------|---------|:--------:| | [access\_entries](#input\_access\_entries) | Map of access entries to add to the cluster. | `any` | `{}` | no | | [authentication\_mode](#input\_authentication\_mode) | The authentication mode for the cluster. | `string` | `"API"` | no | -| [cluster\_node\_ipv4\_cidr](#input\_cluster\_node\_ipv4\_cidr) | The CIDR block for public and private subnets of loadbalancers and nodes. Between /28 and /16. | `string` | n/a | yes | -| [cluster\_service\_ipv4\_cidr](#input\_cluster\_service\_ipv4\_cidr) | The CIDR block to assign Kubernetes service IP addresses from. Between /24 and /12. | `string` | n/a | yes | +| [cluster\_node\_ipv4\_cidr](#input\_cluster\_node\_ipv4\_cidr) | The CIDR block for public and private subnets of loadbalancers and nodes. Between /28 and /16. | `string` | `"10.192.0.0/16"` | no | +| [cluster\_service\_ipv4\_cidr](#input\_cluster\_service\_ipv4\_cidr) | The CIDR block to assign Kubernetes service IP addresses from. Between /24 and /12. | `string` | `"10.190.0.0/16"` | no | | [enable\_cluster\_creator\_admin\_permissions](#input\_enable\_cluster\_creator\_admin\_permissions) | Indicates whether or not to add the cluster creator (the identity used by Terraform) as an administrator via access entry. | `bool` | `true` | no | | [kubernetes\_version](#input\_kubernetes\_version) | Kubernetes version to be used by EKS | `string` | `"1.30"` | no | | [name](#input\_name) | Name being used for relevant resources - including EKS cluster name | `string` | n/a | yes | diff --git a/modules/eks-cluster/variables.tf b/modules/eks-cluster/variables.tf index 90193e29..5c23ab90 100644 --- a/modules/eks-cluster/variables.tf +++ b/modules/eks-cluster/variables.tf @@ -62,11 +62,13 @@ variable "np_capacity_type" { variable "cluster_service_ipv4_cidr" { description = "The CIDR block to assign Kubernetes service IP addresses from. Between /24 and /12." type = string + default = "10.190.0.0/16" } variable "cluster_node_ipv4_cidr" { description = "The CIDR block for public and private subnets of loadbalancers and nodes. Between /28 and /16." type = string + default = "10.192.0.0/16" } variable "authentication_mode" { diff --git a/modules/fixtures/postgres-client-irsa.yml b/modules/fixtures/postgres-client-irsa.yml index 5fabb4bc..c5aed9c1 100644 --- a/modules/fixtures/postgres-client-irsa.yml +++ b/modules/fixtures/postgres-client-irsa.yml @@ -1,76 +1,77 @@ +--- # this manifest contains a version with the IRSA connection check, it is currently listed as a TODO # it may be implemented or dropped depending on if it's relevant or not to test IRSA connection for the db apiVersion: batch/v1 kind: Job metadata: - name: postgres-client - labels: - app: postgres-client + name: postgres-client + labels: + app: postgres-client spec: - backoffLimit: 0 - template: - spec: - restartPolicy: Never - containers: - - name: postgres-client - image: ubuntu:latest - command: - - sh - - "-c" - - | - /bin/bash <<'EOF' - set -o pipefail && \ - apt-get update && \ - apt-get install -y python3 python3-pip build-essential postgresql-client && \ - echo "Creating IRSA db user" && \ - mkdir -p /tmp/scripts && cp /scripts/create_aurora_pg_db.sh /tmp/scripts/create_aurora_pg_db.sh && chmod +x /tmp/scripts/create_aurora_pg_db.sh && /tmp/scripts/create_aurora_pg_db.sh && \ - echo "Testing connection using IRSA" && \ - python3 -m pip install awscli && \ - AWS_PG_PASSWORD=$(aws rds generate-db-auth-token --hostname $AURORA_ENDPOINT --port $AURORA_PORT --region $AWS_REGION --username $AURORA_USERNAME_IRSA) && \ - psql -h $AURORA_ENDPOINT -p $AURORA_PORT "dbname=$AURORA_DB_NAME user=$AURORA_USERNAME_IRSA password=$AWS_PG_PASSWORD" -c 'SELECT version();' + backoffLimit: 0 + template: + spec: + restartPolicy: Never + containers: + - name: postgres-client + image: ubuntu:latest + command: + - sh + - -c + - | + /bin/bash <<'EOF' + set -o pipefail && \ + apt-get update && \ + apt-get install -y python3 python3-pip build-essential postgresql-client && \ + echo "Creating IRSA db user" && \ + mkdir -p /tmp/scripts && cp /scripts/create_aurora_pg_db.sh /tmp/scripts/create_aurora_pg_db.sh && chmod +x /tmp/scripts/create_aurora_pg_db.sh && /tmp/scripts/create_aurora_pg_db.sh && \ + echo "Testing connection using IRSA" && \ + python3 -m pip install awscli && \ + AWS_PG_PASSWORD=$(aws rds generate-db-auth-token --hostname $AURORA_ENDPOINT --port $AURORA_PORT --region $AWS_REGION --username $AURORA_USERNAME_IRSA) && \ + psql -h $AURORA_ENDPOINT -p $AURORA_PORT "dbname=$AURORA_DB_NAME user=$AURORA_USERNAME_IRSA password=$AWS_PG_PASSWORD" -c 'SELECT version();' - EOF - volumeMounts: - - name: scripts - mountPath: /scripts - readOnly: true - env: - - name: AURORA_ENDPOINT - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_endpoint - - name: AURORA_USERNAME - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_username - - name: AURORA_USERNAME_IRSA - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_username_irsa - - name: AURORA_PASSWORD - valueFrom: - secretKeyRef: - name: aurora-secret - key: aurora_password - - name: AURORA_PORT - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_port - - name: AWS_REGION - valueFrom: - configMapKeyRef: - name: aurora-config - key: aws_region - - name: AURORA_DB_NAME - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_db_name - volumes: - - name: scripts - configMap: - name: postgres-scripts + EOF + volumeMounts: + - name: scripts + mountPath: /scripts + readOnly: true + env: + - name: AURORA_ENDPOINT + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_endpoint + - name: AURORA_USERNAME + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_username + - name: AURORA_USERNAME_IRSA + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_username_irsa + - name: AURORA_PASSWORD + valueFrom: + secretKeyRef: + name: aurora-secret + key: aurora_password + - name: AURORA_PORT + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_port + - name: AWS_REGION + valueFrom: + configMapKeyRef: + name: aurora-config + key: aws_region + - name: AURORA_DB_NAME + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_db_name + volumes: + - name: scripts + configMap: + name: postgres-scripts diff --git a/modules/fixtures/postgres-client.yml b/modules/fixtures/postgres-client.yml index 95edbb70..eb809559 100644 --- a/modules/fixtures/postgres-client.yml +++ b/modules/fixtures/postgres-client.yml @@ -1,70 +1,71 @@ +--- apiVersion: batch/v1 kind: Job metadata: - name: postgres-client - labels: - app: postgres-client + name: postgres-client + labels: + app: postgres-client spec: - backoffLimit: 0 - template: - spec: - restartPolicy: Never - containers: - - name: postgres-client - image: ubuntu:latest - command: - - sh - - "-c" - - | - /bin/bash <<'EOF' - set -o pipefail && \ - apt-get update && \ - apt-get install -y python3 python3-pip build-essential postgresql-client && \ - echo "Creating IRSA db user" && \ - mkdir -p /tmp/scripts && cp /scripts/create_aurora_pg_db.sh /tmp/scripts/create_aurora_pg_db.sh && chmod +x /tmp/scripts/create_aurora_pg_db.sh && /tmp/scripts/create_aurora_pg_db.sh + backoffLimit: 0 + template: + spec: + restartPolicy: Never + containers: + - name: postgres-client + image: ubuntu:latest + command: + - sh + - -c + - | + /bin/bash <<'EOF' + set -o pipefail && \ + apt-get update && \ + apt-get install -y python3 python3-pip build-essential postgresql-client && \ + echo "Creating IRSA db user" && \ + mkdir -p /tmp/scripts && cp /scripts/create_aurora_pg_db.sh /tmp/scripts/create_aurora_pg_db.sh && chmod +x /tmp/scripts/create_aurora_pg_db.sh && /tmp/scripts/create_aurora_pg_db.sh - EOF - volumeMounts: - - name: scripts - mountPath: /scripts - readOnly: true - env: - - name: AURORA_ENDPOINT - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_endpoint - - name: AURORA_USERNAME - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_username - - name: AURORA_USERNAME_IRSA - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_username_irsa - - name: AURORA_PASSWORD - valueFrom: - secretKeyRef: - name: aurora-secret - key: aurora_password - - name: AURORA_PORT - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_port - - name: AWS_REGION - valueFrom: - configMapKeyRef: - name: aurora-config - key: aws_region - - name: AURORA_DB_NAME - valueFrom: - configMapKeyRef: - name: aurora-config - key: aurora_db_name - volumes: - - name: scripts - configMap: - name: postgres-scripts + EOF + volumeMounts: + - name: scripts + mountPath: /scripts + readOnly: true + env: + - name: AURORA_ENDPOINT + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_endpoint + - name: AURORA_USERNAME + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_username + - name: AURORA_USERNAME_IRSA + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_username_irsa + - name: AURORA_PASSWORD + valueFrom: + secretKeyRef: + name: aurora-secret + key: aurora_password + - name: AURORA_PORT + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_port + - name: AWS_REGION + valueFrom: + configMapKeyRef: + name: aurora-config + key: aws_region + - name: AURORA_DB_NAME + valueFrom: + configMapKeyRef: + name: aurora-config + key: aurora_db_name + volumes: + - name: scripts + configMap: + name: postgres-scripts diff --git a/modules/fixtures/whoami-deployment.yml b/modules/fixtures/whoami-deployment.yml index 2eb1abf7..4bb9e2eb 100644 --- a/modules/fixtures/whoami-deployment.yml +++ b/modules/fixtures/whoami-deployment.yml @@ -2,40 +2,39 @@ apiVersion: apps/v1 kind: Deployment metadata: - name: whoami - namespace: example + name: whoami + namespace: example spec: - replicas: 1 - selector: - matchLabels: - app: whoami - template: - metadata: - labels: - app: whoami - spec: - containers: - - name: whoami - image: containous/whoami - ports: - - containerPort: 80 - readinessProbe: - httpGet: - path: /health - port: 80 - initialDelaySeconds: 10 - periodSeconds: 5 + replicas: 1 + selector: + matchLabels: + app: whoami + template: + metadata: + labels: + app: whoami + spec: + containers: + - name: whoami + image: containous/whoami + ports: + - containerPort: 80 + readinessProbe: + httpGet: + path: /health + port: 80 + initialDelaySeconds: 10 + periodSeconds: 5 --- apiVersion: v1 kind: Service metadata: - name: whoami-service - namespace: example + name: whoami-service + namespace: example spec: - selector: - app: whoami - ports: - - protocol: TCP - port: 80 - targetPort: 80 ---- + selector: + app: whoami + ports: + - protocol: TCP + port: 80 + targetPort: 80