Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/GitHub AKS Provisioning #160

Open
wants to merge 54 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
318f1f8
updated with AKS provision workflow in github action
prathibhapadma Jul 1, 2022
069c94a
updated with AKS provision workflow in github action
prathibhapadma Jul 1, 2022
7d28403
automatic commit dor docs
prathibhapadma Jul 1, 2022
5ea4d24
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 1, 2022
f70b423
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 1, 2022
ec02564
automatic commit dor docs
prathibhapadma Jul 1, 2022
81cb809
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 1, 2022
412c2a7
Merge branch 'master' into feature/github_aks_provision
prathibhapadma Jul 1, 2022
9008087
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 1, 2022
be22db0
Automatic generation of documentation
prathibhapadma Jul 1, 2022
d81e5ab
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 1, 2022
22f0704
Automatic generation of documentation
prathibhapadma Jul 1, 2022
5c30817
Create setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 1, 2022
74f6b58
Delete documentation/src directory
prathibhapadma Jul 1, 2022
2488464
Merge branch 'devonfw:master' into feature/github_aks_provision
prathibhapadma Jul 5, 2022
8fc8de7
updating aks provision document
prathibhapadma Jul 5, 2022
62f4add
Automatic generation of documentation
prathibhapadma Jul 5, 2022
1a486b6
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
2df5177
Automatic generation of documentation
prathibhapadma Jul 5, 2022
4002c10
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
9927dd8
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
6008fd0
Automatic generation of documentation
prathibhapadma Jul 5, 2022
b87f5e7
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
43d0a9c
Automatic generation of documentation
prathibhapadma Jul 5, 2022
dd01418
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
895e452
Automatic generation of documentation
prathibhapadma Jul 5, 2022
4ac04d0
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
1250980
Automatic generation of documentation
prathibhapadma Jul 5, 2022
24d98ed
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 5, 2022
c5554a1
Automatic generation of documentation
prathibhapadma Jul 5, 2022
e95872b
Merge branch 'devonfw:master' into feature/github_aks_provision
prathibhapadma Jul 6, 2022
e27a753
updated cluster name in azure devops pipeline
prathibhapadma Jul 7, 2022
5a20455
Automatic generation of documentation
prathibhapadma Jul 7, 2022
f2b19b1
Update aks-provisioning.yml.template
prathibhapadma Jul 7, 2022
26b457f
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 7, 2022
e62c914
Automatic generation of documentation
prathibhapadma Jul 7, 2022
21868f9
Update terraform.tfvars
prathibhapadma Jul 7, 2022
821c228
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Jul 7, 2022
42b78f3
Automatic generation of documentation
prathibhapadma Jul 7, 2022
be8d44d
Merge branch 'devonfw:master' into feature/github_aks_provision
prathibhapadma Jul 7, 2022
ae6d73a
Update aks-provisioning.yml.template
prathibhapadma Jul 9, 2022
aa8a072
Update aks-provisioning.yml.template
prathibhapadma Jul 12, 2022
efbd4df
Merge branch 'master' into feature/github_aks_provision
prathibhapadma Jul 12, 2022
c05d1a2
updated with common variable group
prathibhapadma Jul 18, 2022
30f9517
Update aks-provisioning.yml.template
prathibhapadma Jul 18, 2022
89cd9b5
fixed end of the lines
prathibhapadma Jul 20, 2022
58929fd
cluster name in help for aks
ultymatom Aug 4, 2022
15e22f9
Merge branch 'master' into feature/github_aks_provision
ultymatom Aug 4, 2022
8f5029a
Update aks-provisioning.yml.template
prathibhapadma Sep 8, 2022
d97e86a
Update setup-aks-provisioning-pipeline.asciidoc
prathibhapadma Sep 8, 2022
3843096
Automatic generation of documentation
prathibhapadma Sep 8, 2022
8d551bb
Merge branch 'devonfw:master' into feature/github_aks_provision
prathibhapadma Sep 23, 2022
6e8cb73
Merge branch 'master' into feature/github_aks_provision
prathibhapadma Oct 18, 2022
1f8e2e0
Automatic generation of documentation
prathibhapadma Oct 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 19 additions & 11 deletions documentation/azure-devops/setup-aks-provisioning-pipeline.asciidoc
Original file line number Diff line number Diff line change
@@ -1,17 +1,25 @@
= Setting up a Azure AKS provisioning pipeline on Azure DevOps
:provider: Azure Devops
:pipeline_type: pipeline
:trigger_sentence_azure:
:pipeline_type2: pipeline
:path_provider: azure-devops
:aks_variables_path: Azure DevOps > Pipelines > Library > `aks-variables`
= Setting up a Azure AKS provisioning {pipeline_type} on {provider}

In this section we will create a pipeline which will provision an Azure AKS cluster. This pipeline will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details.
In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details.

The creation of the pipeline will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the pipeline and the terraform files for creating the cluster will be pushed to it.
The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it.

Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag.

The script located at `/scripts/pipelines/azure-devops/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning pipeline based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch.
The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch.

== Prerequisites

* Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.
* Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.
* Install the https://marketplace.visualstudio.com/items?itemName=ms-devlabs.custom-terraform-tasks[Terraform extension] for Azure DevOps.
* Create a https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml#create-a-service-connection[service connection] to Azure Resource Manager and name it `aks-connection`. If you already have a service connection available or you need a specific connection name, please update `aks-pipeline.cfg` accordingly.


* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command:

```
Expand All @@ -32,7 +40,7 @@ az storage container create -n <storage container name> --account-name <storage

* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`).

== Creating the pipeline using provided script
== Creating the {pipeline_type} using provided script

Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example:

Expand All @@ -55,7 +63,7 @@ pipeline_generator.sh \
[-w]
```

NOTE: The config file for the AKS provisioning pipeline is located at `/scripts/pipelines/azure-devops/templates/aks/aks-pipeline.cfg`.
NOTE: The config file for the AKS provisioning {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`.

=== Flags
```
Expand Down Expand Up @@ -83,7 +91,7 @@ NOTE: Rancher is installed on the cluster after provisioning when using the abov

NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed.

In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning pipeline on the location it is expected by default (`~/.kube/config`) or either:
In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning {pipeline_type} on the location it is expected by default (`~/.kube/config`) or either:

```
# via environment variable (you can add this on your profile)
Expand All @@ -94,7 +102,7 @@ kubectl <command>
kubectl <command> --kubeconfig=<path to downloaded kubeconfig>
```

To get the DNS name of the NGINX Ingress controller on the EKS cluster, go into Azure DevOps > Pipelines > Library > `aks-variables`.
To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables_path}.

Rancher, if installed, will be available on `https://<ingress controller domain>/dashboard`. You will be asked for an initial password, which can be retrieved with:

Expand All @@ -104,4 +112,4 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{

=== Appendix: Destroying the cluster

To destroy the provisioned resources, set `operation` pipeline variable value to `destroy` and run the pipeline.
To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}.
114 changes: 114 additions & 0 deletions documentation/github/setup-aks-provisioning-pipeline.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
:provider: Github
:pipeline_type: workflow
:trigger_sentence_github:
:pipeline_type2: Github action
:path_provider: github
:aks_variables_path: .github > vars > `deployment-env-variables.env`
= Setting up a Azure AKS provisioning {pipeline_type} on {provider}

In this section we will create a {pipeline_type} which will provision an Azure AKS cluster. This {pipeline_type} will be configured to be manually triggered by the user. As part of AKS cluster provisioning, a NGINX Ingress controller is deployed and a variable group with the name `aks-variables` is created, which contains, among others, the DNS name of the Ingress controller, that you you will need to add as CNAME record on the domains used in your application Ingress manifest files. Refer to the appendix for more details.

The creation of the {pipeline_type} will follow the project workflow, so a new branch named `feature/aks-provisioning` will be created, the YAML file for the {pipeline_type} and the terraform files for creating the cluster will be pushed to it.

Then, a Pull Request (PR) will be created in order to merge the new branch into the appropiate branch (provided in `-b` flag). The PR will be automatically merged if the repository policies are met. If the merge is not possible, either the PR URL will be shown as output, or it will be opened in your web browser if using `-w` flag.

The script located at `/scripts/pipelines/{path_provider}/pipeline_generator.sh` will automatically create this new branch, create the AKS provisioning {pipeline_type} based on the YAML template, create the Pull Request and, if it is possible, merge this new branch into the specified branch.

== Prerequisites


* Add AZURE credentials as https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository[Github Secrets] in your repository and name it `AZURE_USERNAME`, `AZURE_PASSWORD`. If you already have a available credentials or you need a specific credentials connection, please update `aks-provisioning.yml` accordingly.

* An Azure resource group in the desired cluster location (e.g. `westeurope`). You can use an existing one or create a new one with the following command:

```
az group create -n <resource group name> -l <location>
```

* An Azure storage account within the previous resource group. You can use an existing one or create a new one with the following command:

```
az storage account create -n <storage account name> -g <resource group name> -l <location>
```

* An Azure storage container in Azure within the previous storage account. You can use an existing one or create a new one with the following command:

```
az storage container create -n <storage container name> --account-name <storage account name>
```

* This script will commit and push the corresponding YAML template into your repository, so please be sure your local repository is up-to-date (i.e you have pulled the latest changes with `git pull`).

== Creating the {pipeline_type} using provided script

Before executing the script you will need to customize some input variables about the environment. To do so, you can either edit `terraform.tfvars` file or take advantage of the `set-terraform-variables.sh` script located at `/scripts/environment-provisioning/azure/aks`, which allows you to create or update values for the required variables, passing them as flags. As a full example:

```
./set-terraform-variables.sh --location <location> --resource_group_name <resource group name> --instance_type <worker instance type> --worker_node_count <number of worker nodes> --dns_prefix <dns prefix>
```

=== Usage
```
pipeline_generator.sh \
-c <config file path> \
-n <pipeline name> \
-d <project local path> \
--cluster-name <cluster name> \
--resource-group <resource group name> \
--storage-account <storage account name> \
--storage-container <storage container name> \
[--rancher] \
[-b <branch>] \
[-w]
```

NOTE: The config file for the AKS provisioning {pipeline_type} is located at `/scripts/pipelines/{path_provider}/templates/aks/aks-pipeline.cfg`.

=== Flags
```
-c, --config-file [Required] Configuration file containing pipeline definition.
-n, --pipeline-name [Required] Name that will be set to the pipeline.
-d, --local-directory [Required] Local directory of your project (the path should always be using '/' and not '\').
--cluster-name [Required] Name for the cluster.
--resource-group [Required] Name of the resource group for the cluster.
--storage--account [Required] Name of the storage account for the cluster.
--storage-container [Required] Name of the storage container where the Terraform state of the cluster will be stored.
--rancher Install Rancher to manage the cluster.
-b, --target-branch Name of the branch to which the Pull Request will target. PR is not created if the flag is not provided.
-w Open the Pull Request on the web browser if it cannot be automatically merged. Requires -b flag.
```

=== Example

```
./pipeline_generator.sh -c ./templates/aks/aks-pipeline.cfg -n aks-provisioning -d C:/Users/$USERNAME/Desktop/quarkus-project --cluster-name devon-hangar --resource-group devonfw --storage-account hangar --storage-container aks-state --rancher -b develop -w
```

NOTE: Rancher is installed on the cluster after provisioning when using the above command.

=== Appendix: Interacting with the cluster

NOTE: Make sure you have https://kubernetes.io/docs/tasks/tools/#kubectl[kubectl] installed.

In order to interact with your cluster you will need to download the artifact `kubeconfig` generated by the cluster provisioning {pipeline_type} on the location it is expected by default (`~/.kube/config`) or either:

```
# via environment variable (you can add this on your profile)
export KUBECONFIG=<path to downloaded kubeconfig>
kubectl <command>

# via command-line flag
kubectl <command> --kubeconfig=<path to downloaded kubeconfig>
```

To get the DNS name of the NGINX Ingress controller on the AKS cluster, go into {aks_variables_path}.

Rancher, if installed, will be available on `https://<ingress controller domain>/dashboard`. You will be asked for an initial password, which can be retrieved with:

```
kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'
```

=== Appendix: Destroying the cluster

To destroy the provisioned resources, set `operation` {pipeline_type} variable value to `destroy` and run the {pipeline_type}.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
:provider: Azure Devops
:pipeline_type: pipeline
:trigger_sentence_azure:
:pipeline_type2: pipeline
:path_provider: azure-devops
:aks_variables_path: Azure DevOps > Pipelines > Library > `aks-variables`
include::../common_templates/setup-aks-provisioning-pipeline.asciidoc[]
20 changes: 10 additions & 10 deletions documentation/src/azure-devops/setup-package-pipeline.asciidoc
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
:provider: Azure DevOps
:pipeline_type: pipeline
:trigger_sentence: This pipeline will be configured in order to be triggered every time quality pipeline is executed successfully on a commit for release/* and develop branches, requiring manual launch for other branches but still enforcing that quality pipeline has passed
:pipeline_type2: pipeline
:path_provider: azure-devops
:build-pipeline:
:quality-pipeline:
:extra_args_quarkus: --build-pipeline-name quarkus-project-build --quality-pipeline-name quarkus-project-quality
:extra_args_node: --build-pipeline-name node-project-build --quality-pipeline-name node-project-quality
:extra_args_angular: --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality
:provider: Azure DevOps
:pipeline_type: pipeline
:trigger_sentence: This pipeline will be configured in order to be triggered every time quality pipeline is executed successfully on a commit for release/* and develop branches, requiring manual launch for other branches but still enforcing that quality pipeline has passed
:pipeline_type2: pipeline
:path_provider: azure-devops
:build-pipeline:
:quality-pipeline:
:extra_args_quarkus: --build-pipeline-name quarkus-project-build --quality-pipeline-name quarkus-project-quality
:extra_args_node: --build-pipeline-name node-project-build --quality-pipeline-name node-project-quality
:extra_args_angular: --build-pipeline-name angular-project-build --quality-pipeline-name angular-project-quality
include::../common_templates/setup-package-pipeline.asciidoc[]
20 changes: 10 additions & 10 deletions documentation/src/azure-devops/setup-quality-pipeline.asciidoc
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
:provider: Azure DevOps
:pipeline_type: pipeline
:trigger_sentence: This pipeline will be configured in order to be triggered every time the test pipeline is executed successfully on a commit
:pipeline_type2: pipeline
:path_provider: azure-devops
:build-pipeline:
:test-pipeline:
:extra_args_quarkus: --build-pipeline-name quarkus-project-build --test-pipeline-name quarkus-project-test
:extra_args_node: --build-pipeline-name node-project-build --test-pipeline-name node-project-test
:extra_args_angular: --build-pipeline-name angular-project-build --test-pipeline-name angular-project-test
:provider: Azure DevOps
:pipeline_type: pipeline
:trigger_sentence: This pipeline will be configured in order to be triggered every time the test pipeline is executed successfully on a commit
:pipeline_type2: pipeline
:path_provider: azure-devops
:build-pipeline:
:test-pipeline:
:extra_args_quarkus: --build-pipeline-name quarkus-project-build --test-pipeline-name quarkus-project-test
:extra_args_node: --build-pipeline-name node-project-build --test-pipeline-name node-project-test
:extra_args_angular: --build-pipeline-name angular-project-build --test-pipeline-name angular-project-test
include::../common_templates/setup-quality-pipeline.asciidoc[]
Loading