Skip to content

Commit

Permalink
feat: New Bottlerocket pattern (#1869)
Browse files Browse the repository at this point in the history
  • Loading branch information
@rodrigobersa
rodrigobersa authored Aug 4, 2024
1 parent 0921b88 commit ff0c8ab
Show file tree
Hide file tree
Showing 13 changed files with 602 additions and 0 deletions.
7 changes: 7 additions & 0 deletions docs/patterns/bottlerocket.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
title: Bottlerocket
---

{%
include-markdown "../../patterns/bottlerocket/README.md"
%}
151 changes: 151 additions & 0 deletions patterns/bottlerocket/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# Bottlerocket with Bottlerocket Update Operator

This pattern demostrates how to deploy Amazon EKS Clusters using [Bottlerocket OS](https://aws.amazon.com/bottlerocket/) for Managed Node Groups (MNG) and [Karpenter](https://karpenter.sh/), using [Bottlerocket Update Operator (BRUPOP)](https://github.com/bottlerocket-os/bottlerocket-update-operator) to manage CVE patches automatically at the Node OS level. The BROPUP doesn't work with minor or major upgrades, just patch level.

## Deploy

See [here](https://aws-ia.github.io/terraform-aws-eks-blueprints/getting-started/#prerequisites) for the prerequisites and steps to deploy this pattern.

## Validate

1. List all Nodes in the cluster. You should see three Nodes that belongs to the defined MNG, and should be in the `v1.30.0-eks-fff26e3` version since we are using a specific AMI ID to test the BRUPOP.

```sh
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-2-29.us-west-2.compute.internal Ready <none> 7m24s v1.30.0-eks-fff26e3
ip-10-0-26-48.us-west-2.compute.internal Ready <none> 7m23s v1.30.0-eks-fff26e3
ip-10-0-43-187.us-west-2.compute.internal Ready <none> 7m19s v1.30.0-eks-fff26e3
```

2. Check for the Label `"bottlerocket.aws/updater-interface-version"="2.0.0"` that is set to all the Nodes in the MNG. This Label is responsible to mark the Nodes that will have updates managed by BRUPOP.

```sh
$ kubectl get nodes -L bottlerocket.aws/updater-interface-version
NAME STATUS ROLES AGE VERSION UPDATER-INTERFACE-VERSION
ip-10-0-2-29.us-west-2.compute.internal Ready <none> 79m v1.30.0-eks-fff26e3 2.0.0
ip-10-0-26-48.us-west-2.compute.internal Ready <none> 79m v1.30.0-eks-fff26e3 2.0.0
ip-10-0-43-187.us-west-2.compute.internal Ready <none> 79m v1.30.0-eks-fff26e3 2.0.0
```

3. Validate if all the Pods are in Running status, and Ready.

```sh
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
brupop-bottlerocket-aws brupop-agent-2msn5 1/1 Running 0 3m20s
brupop-bottlerocket-aws brupop-agent-7kvx5 1/1 Running 0 3m20s
brupop-bottlerocket-aws brupop-agent-8d8n8 1/1 Running 0 3m20s
brupop-bottlerocket-aws brupop-apiserver-7b45c5546f-dzwqz 1/1 Running 0 3m20s
brupop-bottlerocket-aws brupop-apiserver-7b45c5546f-lvnt4 1/1 Running 0 3m20s
brupop-bottlerocket-aws brupop-apiserver-7b45c5546f-xmvx2 1/1 Running 0 3m20s
brupop-bottlerocket-aws brupop-controller-deployment-7fcfc69978-rwkml 1/1 Running 0 3m20s
cert-manager cert-manager-5b44f85959-zc5zc 1/1 Running 0 4m2s
cert-manager cert-manager-cainjector-7f97f54fd-kjnq5 1/1 Running 0 4m3s
cert-manager cert-manager-webhook-c59f66876-jkwhj 1/1 Running 0 4m3s
karpenter karpenter-7b7958bbf5-647n2 1/1 Running 0 11m
karpenter karpenter-7b7958bbf5-s8475 1/1 Running 0 11m
kube-system aws-node-5n496 2/2 Running 0 10m
kube-system aws-node-krz6q 2/2 Running 0 10m
kube-system aws-node-tx76l 2/2 Running 0 10m
kube-system coredns-544fd9dfb5-6l6nt 1/1 Running 0 9m27s
kube-system coredns-544fd9dfb5-hcq84 1/1 Running 0 9m27s
kube-system kube-proxy-9sh2s 1/1 Running 0 9m19s
kube-system kube-proxy-gl5g9 1/1 Running 0 9m24s
kube-system kube-proxy-jwcqp 1/1 Running 0 9m15s
```

4. Test the Bottlerocket Update Operator. By default in this pattern, it's set to check for updates every hour.

```hcl
set = [{
name = "scheduler_cron_expression"
value = "0 * * * * * *" # Default Unix Cron syntax, set to check every hour. Example "0 0 23 * * Sat *" Perform update checks every Saturday at 23H / 11PM
}]
```

Describe any Node with the `v1.30.0-eks-fff26e3` version.

```sh
$ kubectl describe node ip-10-0-43-187.us-west-2.compute.internal | grep Image
OS Image: Bottlerocket OS 1.20.0 (aws-k8s-1.30)
```

1. Wait a couple of minutes and check that one or more Nodes were updated to a newer version, in this example, `v1.30.1-eks-e564799`.

```sh
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-2-29.us-west-2.compute.internal Ready <none> 83m v1.30.1-eks-e564799
ip-10-0-26-48.us-west-2.compute.internal Ready <none> 83m v1.30.0-eks-fff26e3
ip-10-0-43-187.us-west-2.compute.internal Ready <none> 83m v1.30.0-eks-fff26e3
```

6. Describe the Node with the `v1.30.1-eks-e564799` version.

```sh
$ kubectl describe node ip-10-0-2-29.us-west-2.compute.internal | grep Image
OS Image: Bottlerocket OS 1.20.4 (aws-k8s-1.30)
```

7. In the Karpenter's EC2NodeClass configuration, the default OS is also set to Bottlerocket, but in it's latest version, and the label to perform automated updates is not set, since Karpenter is configured to expire the Nodes every 24 hours.

```sh
kubectl describe ec2nodeclasses.karpenter.k8s.aws default | grep Status -A50 | egrep 'Amis|Id|Name'
Amis:
Id: ami-079c06d95540e8c92
Name: bottlerocket-aws-k8s-1.30-nvidia-x86_64-v1.20.4-b6163b2a
Id: ami-079c06d95540e8c92
Name: bottlerocket-aws-k8s-1.30-nvidia-x86_64-v1.20.4-b6163b2a
Id: ami-05206fcf92965fc27
Name: bottlerocket-aws-k8s-1.30-nvidia-aarch64-v1.20.4-b6163b2a
Id: ami-05206fcf92965fc27
Name: bottlerocket-aws-k8s-1.30-nvidia-aarch64-v1.20.4-b6163b2a
Id: ami-03b7010797ca2bf91
Name: bottlerocket-aws-k8s-1.30-x86_64-v1.20.4-b6163b2a
Id: ami-0c37339cac90815d6
Name: bottlerocket-aws-k8s-1.30-aarch64-v1.20.4-b6163b2a
```

8. To validate that, use the `kubectl` command to create an example deployment, and scale it to any desired amount of replicas. Karpenter should provision a new Node in with the latest available version for Bottlerocket.

```sh
$ kubectl scale deployment inflate --replicas 10
deployment.apps/inflate scaled

$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
inflate-7849c696cd-2668t 1/1 Running 0 49s 10.0.34.254 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-5wffm 1/1 Running 0 49s 10.0.46.13 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-8x5ws 1/1 Running 0 49s 10.0.35.190 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-9nhvr 1/1 Running 0 49s 10.0.42.99 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-cbr5q 1/1 Running 0 49s 10.0.35.195 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-jcr7r 1/1 Running 0 49s 10.0.33.41 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-nhjt4 1/1 Running 0 49s 10.0.35.213 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-p9j7x 1/1 Running 0 49s 10.0.43.102 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-qr7th 1/1 Running 0 49s 10.0.37.221 ip-10-0-45-41.us-west-2.compute.internal <none> <none>
inflate-7849c696cd-rzjzr 1/1 Running 0 49s 10.0.33.210 ip-10-0-45-41.us-west-2.compute.internal <none> <none>

$ kubect get nodes
NAME STATUS ROLES AGE VERSION
ip-10-0-2-29.us-west-2.compute.internal Ready <none> 90m v1.30.1-eks-e564799
ip-10-0-26-48.us-west-2.compute.internal Ready <none> 90m v1.30.0-eks-fff26e3
ip-10-0-43-187.us-west-2.compute.internal Ready <none> 90m v1.30.0-eks-fff26e3
ip-10-0-45-41.us-west-2.compute.internal Ready <none> 60s v1.30.1-eks-e564799

$ kubectl describe node ip-10-0-45-41.us-west-2.compute.internal | grep Image
OS Image: Bottlerocket OS 1.20.4 (aws-k8s-1.30)
```

## Destroy

Scale down the example application to de-provision Karpenter Nodes.

```sh
$ kubectl delete -f example.yaml
deployment.apps "inflate" deleted
```

{%
include-markdown "../../docs/_partials/destroy.md"
%}
108 changes: 108 additions & 0 deletions patterns/bottlerocket/addons.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
################################################################################
# EKS Blueprints Addons
################################################################################
data "aws_ecrpublic_authorization_token" "token" {
provider = aws.virginia
}

module "eks_blueprints_addons" {
source = "aws-ia/eks-blueprints-addons/aws"
version = "~> 1.16"

cluster_name = module.eks.cluster_name
cluster_endpoint = module.eks.cluster_endpoint
cluster_version = module.eks.cluster_version
oidc_provider_arn = module.eks.oidc_provider_arn

enable_cert_manager = true
cert_manager = {
wait = true
wait_for_jobs = true
values = [<<-EOT
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
cainjector:
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
webhook:
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
startupapicheck:
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
EOT
]
}

enable_karpenter = true
karpenter = {
repository_username = data.aws_ecrpublic_authorization_token.token.user_name
repository_password = data.aws_ecrpublic_authorization_token.token.password
version = "v0.36"
}

enable_bottlerocket_update_operator = true
bottlerocket_update_operator = {
values = [<<-EOT
scheduler_cron_expression: "* * * * * * *"
placement:
agent:
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
controller:
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
apiserver:
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
EOT
]
}

tags = local.tags
}

################################################################################
# Karpenter resources
################################################################################
resource "aws_eks_access_entry" "karpenter" {

cluster_name = module.eks.cluster_name
principal_arn = module.eks_blueprints_addons.karpenter.node_iam_role_arn
type = "EC2_LINUX"

tags = local.tags

depends_on = [module.eks_blueprints_addons]
}

resource "helm_release" "karpenter_resources" {
name = "karpenter-resources"
chart = "./karpenter-resources"
set_list {
name = "nodepool.zone"
value = local.azs
}
values = [<<-EOT
ec2nodeclass:
securityGroupSelectorTerms:
tags: ${module.eks.cluster_name}
subnetSelectorTerms:
tags: ${module.eks.cluster_name}
tags: ${module.eks.cluster_name}
role: ${split("/", module.eks_blueprints_addons.karpenter.node_iam_role_arn)[1]}
blockDeviceMappings:
ebs:
kmsKeyID: ${module.ebs_kms_key.key_id}
EOT
]

depends_on = [module.eks_blueprints_addons]
}
113 changes: 113 additions & 0 deletions patterns/bottlerocket/eks.tf
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
################################################################################
# EKS Cluster
################################################################################
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.20"

cluster_name = local.name
cluster_version = "1.30"
cluster_endpoint_public_access = true

cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets

cluster_addons = {
coredns = {
most_recent = true
}
kube-proxy = {
most_recent = true
}
vpc-cni = {
most_recent = true
before_compute = true
}
}
# Give the Terraform identity admin access to the cluster
# which will allow resources to be deployed into the cluster
enable_cluster_creator_admin_permissions = true
authentication_mode = "API"

eks_managed_node_groups = {
bottlerocket = {
ami_type = "BOTTLEROCKET_x86_64"
instance_types = ["m5.large", "m5a.large"]

iam_role_attach_cni_policy = true
# The below AMI release version is for testing purposes in order to validate Botterocket Update Operator.
ami_release_version = "1.20.0-fcf71a47"

min_size = 1
max_size = 5
desired_size = 3

ebs_optimized = true
enable_monitoring = true
block_device_mappings = {
xvda = {
device_name = "/dev/xvda"
ebs = {
encrypted = true
kms_key_id = module.ebs_kms_key.key_arn
delete_on_termination = true
}
}
xvdb = {
device_name = "/dev/xvdb"
ebs = {
encrypted = true
kms_key_id = module.ebs_kms_key.key_arn
delete_on_termination = true
}
}
}

# The following block customize your Bottlerocket instances, including kubernetes tags, host and kernel parameters, and user-data.
bootstrap_extra_args = <<-EOT
[settings.host-containers.admin]
enabled = false
[settings.host-containers.control]
enabled = true
[settings.kernel]
lockdown = "integrity"
[settings.kubernetes.node-labels]
"bottlerocket.aws/updater-interface-version" = "2.0.0"
[settings.kubernetes.node-taints]
"CriticalAddonsOnly" = "true:NoSchedule"
EOT
}
}

tags = merge(local.tags, {
"karpenter.sh/discovery" = local.name
})
}

module "ebs_kms_key" {
source = "terraform-aws-modules/kms/aws"
version = "~> 1.5"

description = "Customer managed key to encrypt EKS managed node group volumes"

# Policy
key_administrators = [
data.aws_caller_identity.current.arn
]

key_service_roles_for_autoscaling = [
"arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
module.eks.cluster_iam_role_arn,
module.eks_blueprints_addons.karpenter.iam_role_arn
]

aliases = ["eks/${local.name}/ebs"]

tags = local.tags
}
Loading

0 comments on commit ff0c8ab

Please sign in to comment.