Skip to content

Commit

Permalink
Mcdaniel 202301 (#28)
Browse files Browse the repository at this point in the history
* scaffold refactored k8s modules

* create an admin domain

* documentation

* documentation

* update actions for github, aws, tutor

* bump terraform versions

* restore ci_actions_amazon_ecr_login_version

* restore ci_deploy_tutor_plugin_credentials_version

* restore and update ci_actions_setup_build_action_version

* revert to enabling mfe with tutor from command line

* add root_domain

* consolidate to stack_namespace

* consolidate to stack_namespace

* consolidate to stack_namespace

* add mock outputs

* consolidate to stack_namespace

* consolidate to stack_namespace

* documentation

* revert to fork

* should use namespace instead of environment_namespace

* disable all mfe's by default. set ENABLE_WEB_PROXY=false

* parameterize the aws account_id

* add environment_domain

* parameterize and standardize cert issuer manifests

* fix local initializations

* parameterize the admin hosted zone

* standardize secret names

* standardize the cert used for all admin subdomains

* standardize the cert used for all admin subdomains

* refactor cluster-issuer manifest into a template

* move environment certs and ingresses to stack

* lint namespace

* lint namespace

* lint namespace

* lint namespace

* lint namespace

* add kubectl provider

* a

* fix namespace

* add k8s dns records

* refactor cert-manager and bump to v1.10

* lint cert-manager

* testing

* documentation

* move nginx controller to kubernetes_ingress_clb

* move all vpa resources to kubernetes_monitoring

* add cert_manager_namespace

* lint

* refactor cert-manager into its own module

* rename to stack-ingress-nginx-controller

* rename to common-ingress-nginx-controller

* add mock outputs

* testing

* add namespace

* add namespace

* add namespace

* add data.aws_route53_zone.admin_domain

* add stack_namespace

* rename to common-ingress-nginx-controller

* add admin_domain

* testing

* documentation

* refactor karpenter into its own module

* lint

* parameterize whether to install karpenter, kubeapps, prometheus, dashboard

* refactor rm_directory()

* refactor post_gen_project.py

* refactor post_gen_project.py

* documentation
  • Loading branch information
lpm0073 authored Jan 9, 2023
1 parent 6e81a2e commit 9998001
Show file tree
Hide file tree
Showing 23 changed files with 375 additions and 75 deletions.
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/).

## [1.0.8] (2023-1-9)

- refactor karpenter into its own module.
- parameterize helm chart version of vertical-pod-autoscaler
- add Cookiecutter Y/N install parameters to toggle optional Kubernetes features: Karpenter, Prometheus, Dashboard, Kubeapps

## [1.0.7] (2023-1-8)

- refactor cert-manager into its own module and move from environment to stack.
Expand Down
16 changes: 13 additions & 3 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,16 @@ Builds a fully functional Docker-based, horizontally-scaled Open edX installatio
:width: 100%
:alt: Bastion Welcome Screen

Complete Kubernetes Auto scaling
--------------------------------

Your new Kubernetes infrastructure platform leverages `Prometheus <https://prometheus.io/>`_ and `metrics-server <https://github.com/kubernetes-sigs/metrics-server>`_ to provide you with fully self-maintained auto-scaling features:

- `Vertical Pod Auto-scaling <https://github.com/kubernetes/autoscaler>`_: a set of Kubernetes components that automatically adjust the amount of CPU and memory requested by your running pods based on performance metrics collected real-time by prometheus and metrics-server.
- `Horizontal Pod Auto-scaling <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/>`_: a built-in Kubernetes feature that automatically updates the pod count of your application deployments with the aim of automatically scaling the workload to match demand.
- `Karpenter <https://karpenter.sh/>`_: automatically launches just the right number of AWS EC2 compute node (ie application server instance) resources to handle your cluster's applications. It is designed to let you take full advantage of the cloud with fast and simple compute provisioning for Kubernetes clusters.


Terraform-based AWS infrastructure management
---------------------------------------------

Expand All @@ -68,11 +78,11 @@ This configuration scales automatically, reliably supporting anywhere from a few

**NEW IN VERSION 1.0.2: SPOT PRICING FOR EC2 INSTANCES** Save up to 75% off the cost of on-demand EC2 instances by using AWS' flexible `spot-pricing <https://aws.amazon.com/ec2/spot/pricing/>`_ .

**NEW IN VERSION 1.0.3: an optional fully-configured remote MongoDB server running on an EC2 instance. Set cookiecutter.stack_add_remote_mongodb=Y to choose this option.**
**NEW IN VERSION 1.0.3:** an optional fully-configured remote MongoDB server running on an EC2 instance. Set cookiecutter.stack_add_remote_mongodb=Y to choose this option.

**NEW IN VERSION 1.0.5: Kubernetes upgrade to 1.24, plus a new adminstrative server with all of the preinstalled software that you'll need to administer your Open edX platform. Set cookiecutter.stack_add_bastion=Y to choose this option.**
**NEW IN VERSION 1.0.5:** Kubernetes upgrade to 1.24, plus a new adminstrative server with all of the preinstalled software that you'll need to administer your Open edX platform. Set cookiecutter.stack_add_bastion=Y to choose this option.

**NEW IN VERSION 1.0.6: `Kubernetes Dashboard <https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/>`_ and `Kubeapps <https://kubeapps.dev/>`_ web applications.
**NEW IN VERSION 1.0.8:** `Kubernetes Dashboard <https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/>`_ and `Kubeapps <https://kubeapps.dev/>`_ web applications.

Github Workflows for Build and Deploy
-------------------------------------
Expand Down
5 changes: 5 additions & 0 deletions cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@
"global_aws_route53_hosted_zone_id": "Z1234567ABCDE1U23DEF",
"global_aws_region": "us-east-1",
"global_account_id": "123456789012",
"stack_install_k8s_dashboard": ["Y", "N"],
"stack_install_k8s_kubeapps": ["Y", "N"],
"stack_install_k8s_karpenter": ["Y", "N"],
"stack_install_k8s_prometheus": ["Y", "N"],
"ci_build_tutor_version": "14.2.3",
"ci_build_open_edx_version": "nutmeg.2",
"ci_build_theme_repository": "edx-theme-example",
Expand Down Expand Up @@ -121,6 +125,7 @@
"terraform_aws_modules_vpc": "~> 3.18",
"terraform_helm_cert_manager": "~> 1.10",
"terraform_helm_ingress_nginx_controller": "~> 4.4",
"terraform_helm_vertical_pod_autoscaler": "~> 6.0",
"terraform_provider_kubernetes_version": "~> 2.16",
"terraform_provider_hashicorp_aws_version": "~> 4.48",
"terraform_provider_hashicorp_local_version": "~> 2.2",
Expand Down
44 changes: 40 additions & 4 deletions hooks/post_gen_project.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,45 @@
"""
"""

import os
import shutil

def rm_directory(dir_path: str):
if os.path.exists(dir_path):
shutil.rmtree(dir_path)

TERMINATOR = "\x1b[0m"
WARNING = "\x1b[1;33m [WARNING]: "
INFO = "\x1b[1;33m [INFO]: "
HINT = "\x1b[3;33m"
SUCCESS = "\x1b[1;32m [SUCCESS]: "

def remove_k8s_dashboard():
dir_path = os.path.join("terraform", "stacks", "modules", "kubernetes_dashboard")
rm_directory(dir_path)

def rm_directory(dir_path: str):
if os.path.exists(dir_path):
shutil.rmtree(dir_path)
dir_path = os.path.join("terraform", "environments", "modules", "kubernetes_dashboard")
rm_directory(dir_path)

def remove_k8s_kubeapps():
dir_path = os.path.join("terraform", "stacks", "modules", "kubernetes_kubeapps")
rm_directory(dir_path)

dir_path = os.path.join("terraform", "environments", "modules", "kubernetes_kubeapps")
rm_directory(dir_path)

def remove_k8s_karpenter():
dir_path = os.path.join("terraform", "stacks", "modules", "kubernetes_karpenter")
rm_directory(dir_path)

dir_path = os.path.join("terraform", "environments", "modules", "kubernetes_karpenter")
rm_directory(dir_path)

def remove_k8s_prometheus():
dir_path = os.path.join("terraform", "stacks", "modules", "kubernetes_prometheus")
rm_directory(dir_path)

dir_path = os.path.join("terraform", "environments", "modules", "kubernetes_prometheus")
rm_directory(dir_path)

def remove_bastion():
dir_path = os.path.join("terraform", "stacks", "modules", "ec2_bastion")
Expand Down Expand Up @@ -77,6 +101,18 @@ def main():
if "{{ cookiecutter.stack_add_remote_mongodb }}".upper() != "Y":
remove_mongodb()

if "{{ cookiecutter.stack_install_k8s_dashboard }}".upper() != "Y":
remove_k8s_dashboard()

if "{{ cookiecutter.stack_install_k8s_kubeapps }}".upper() != "Y":
remove_k8s_kubeapps()

if "{{ cookiecutter.stack_install_k8s_karpenter }}".upper() != "Y":
remove_k8s_karpenter()

if "{{ cookiecutter.stack_install_k8s_prometheus }}".upper() != "Y":
remove_k8s_prometheus()

print(SUCCESS + "Your Open edX devops repo has been initialized." + TERMINATOR)


Expand Down
47 changes: 47 additions & 0 deletions hooks/pre_gen_project.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
"""
"""
import os
import shutil

def rm_directory(dir_path: str):
if os.path.exists(dir_path):
shutil.rmtree(dir_path)

TERMINATOR = "\x1b[0m"
WARNING = "\x1b[1;33m [WARNING]: "
Expand All @@ -16,3 +22,44 @@
assert (
github_repo_name == github_repo_name.lower()
), "'{}' project slug should be all lowercase".format(github_repo_name)

def reinitialize(repo_path):
if os.path.exists(repo_path):
print(INFO + "reinitializing the repository folder {folder}" + TERMINATOR.format(
folder=repo_path
))
rm_directory(repo_path)

def main():
"""
If we find an existing repository then we should preemptively delete the existing
Cookiecutter output in order to avoid situations where deleted, deprecated, or relocated
template files would otherwise remain as 'residue' of the Cookiecutter output.
FIX NOTE: this runs out of the /tmp folder and therefore none of these paths are found.
need to find a code snippet that references the correct destination path
"""
print(INFO + "Open edX devops Cookiecutter" + TERMINATOR)

repo_path = os.path.join("{{ cookiecutter.github_repo_name }}")
if not os.path.exists(repo_path):
return

print(INFO + "Open edX devops found an existing repository {{ cookiecutter.github_repo_name }}." + TERMINATOR)

terraform_path = os.path.join("{{ cookiecutter.github_repo_name }}", "terraform")
reinitialize(terraform_path)

github_path = os.path.join("{{ cookiecutter.github_repo_name }}", ".github")
reinitialize(github_path)

ci_path = os.path.join("{{ cookiecutter.github_repo_name }}", "ci")
reinitialize(ci_path)

scripts_path = os.path.join("{{ cookiecutter.github_repo_name }}", "scripts")
reinitialize(scripts_path)

print(INFO + "Open edX devops has reinitialized the repository cookiecutter.github_repo_name" + TERMINATOR)

if __name__ == "__main__":
main()
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
data "template_file" "cluster-issuer" {
template = file("${path.module}/manifests/cluster-issuer.yml.tpl")
vars = {
namespace = var.environment_namespace
aws_region = var.aws_region
namespace = var.environment_namespace
aws_region = var.aws_region
hosted_zone_id = data.aws_route53_zone.environment_domain.id
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,8 @@ resource "kubectl_manifest" "certificate" {
data "template_file" "cluster-issuer" {
template = file("${path.module}/manifests/cluster-issuer.yml.tpl")
vars = {
namespace = var.namespace
aws_region = var.aws_region
namespace = var.namespace
aws_region = var.aws_region
hosted_zone_id = data.aws_route53_zone.admin_domain.id
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,5 @@

output "cert_manager_policy" {
description = "the cert-manager IAM policy ARN"
value = aws_iam_policy.cert_manager_policy.arn
value = aws_iam_policy.cert_manager_policy.arn
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
data "template_file" "vpa-cert-manager-cainjector" {
template = file("${path.module}/manifests/verticalpodautoscalers/vpa-openedx-cert-manager-cainjector.yaml.tpl")
vars = {
environment_namespace = var.namespace
}
}

data "template_file" "vpa-cert-manager-webhook" {
template = file("${path.module}/manifests/verticalpodautoscalers/vpa-openedx-cert-manager-webhook.yaml.tpl")
vars = {
environment_namespace = var.namespace
}
}

data "template_file" "vpa-cert-manager" {
template = file("${path.module}/manifests/verticalpodautoscalers/vpa-openedx-cert-manager.yaml.tpl")
vars = {
environment_namespace = var.namespace
}
}


resource "kubectl_manifest" "vpa-cert-manager" {
yaml_body = data.template_file.vpa-cert-manager.rendered
}

resource "kubectl_manifest" "vpa-cert-manager-cainjector" {
yaml_body = data.template_file.vpa-cert-manager-cainjector.rendered
}

resource "kubectl_manifest" "vpa-cert-manager-webhook" {
yaml_body = data.template_file.vpa-cert-manager-webhook.rendered
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
# FIXED. but see note below about version.
#
# see: https://registry.terraform.io/modules/terraform-aws-modules/iam/aws/latest/submodules/iam-role-for-service-accounts-eks

module "karpenter_controller_irsa_role" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
# mcdaniel aug-2022: specifying an explicit version causes this module to create
Expand Down Expand Up @@ -79,11 +80,6 @@ resource "helm_release" "karpenter" {
value = aws_iam_instance_profile.karpenter.name
}

depends_on = [
helm_release.metrics_server,
helm_release.prometheus,
helm_release.vpa
]
}

resource "random_pet" "this" {
Expand Down Expand Up @@ -164,7 +160,6 @@ resource "kubectl_manifest" "vpa-karpenter" {
yaml_body = file("${path.module}/yml/verticalpodautoscalers/vpa-karpenter.yaml")

depends_on = [
helm_release.vpa,
helm_release.karpenter
]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#------------------------------------------------------------------------------
# written by: Lawrence McDaniel
# https://lawrencemcdaniel.com/
#
# date: Mar-2022
#
# usage: create an EKS cluster
#------------------------------------------------------------------------------
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#------------------------------------------------------------------------------
# written by: Lawrence McDaniel
# https://lawrencemcdaniel.com/
#
# date: Aug-2022
#
# usage: all providers for Kubernetes and its sub-systems. The general strategy
# is to manage authentications via aws cli where possible, simply to limit
# the environment requirements in order to get this module to work.
#
# another alternative for each of the providers would be to rely on
# the local kubeconfig file.
#------------------------------------------------------------------------------

data "aws_eks_cluster" "eks" {
name = var.stack_namespace
}

data "aws_eks_cluster_auth" "eks" {
name = var.stack_namespace
}

provider "kubernetes" {
host = data.aws_eks_cluster.eks.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
token = data.aws_eks_cluster_auth.eks.token
}

provider "kubectl" {
host = data.aws_eks_cluster.eks.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.eks.token
}


provider "helm" {
kubernetes {
host = data.aws_eks_cluster.eks.endpoint
cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority.0.data)
token = data.aws_eks_cluster_auth.eks.token
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#------------------------------------------------------------------------------
# written by: Lawrence McDaniel
# https://lawrencemcdaniel.com/
#
# date: Mar-2022
#
# usage: create an EKS cluster
#------------------------------------------------------------------------------
variable "stack_namespace" {
type = string
}

variable "karpenter_node_group_iam_role_name" {
type = string
}

variable "karpenter_node_group_iam_role_arn" {
type = string
}

variable "oidc_provider_arn" {
type = string
}

variable "tags" {
description = "A map of tags to add to all resources. Tags added to launch configuration or templates override these values for ASG Tags only."
type = map(string)
default = {}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#------------------------------------------------------------------------------
# written by: Lawrence McDaniel
# https://lawrencemcdaniel.com/
#
# date: Mar-2022
#
# usage: create an EKS cluster
#------------------------------------------------------------------------------
terraform {
required_version = "{{ cookiecutter.terraform_required_version }}"

required_providers {
local = "{{ cookiecutter.terraform_provider_hashicorp_local_version }}"
random = {
source = "hashicorp/random"
version = "{{ cookiecutter.terraform_provider_hashicorp_random_version }}"
}

aws = {
source = "hashicorp/aws"
version = "{{ cookiecutter.terraform_provider_hashicorp_aws_version }}"
}
kubectl = {
source = "gavinbunney/kubectl"
version = "{{ cookiecutter.terraform_provider_hashicorp_kubectl_version }}"
}
helm = {
source = "hashicorp/helm"
version = "{{ cookiecutter.terraform_provider_hashicorp_helm_version }}"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "{{ cookiecutter.terraform_provider_kubernetes_version }}"
}
}
}
Loading

0 comments on commit 9998001

Please sign in to comment.