Skip to content

Latest commit

 

History

History
453 lines (358 loc) · 22.5 KB

README.md

File metadata and controls

453 lines (358 loc) · 22.5 KB
Raw

App Mesh Controller

⚠️ This controller is published in multiple repos: Contributions to this Helm chart must be written to aws/aws-app-mesh-controller-for-k8s Github repo. PRs to other repos like aws/eks-charts may be closed or overwritten upon next controller release.

App Mesh controller Helm chart for Kubernetes

Note: If you wish to use App Mesh preview features, please refer to our preview version instructions.

Prerequisites

  • Kubernetes >= 1.14
  • IAM permissions (see below)

Installing the Chart

Note: AppMesh controller v1.0.0+ is backwards incompatible with old versions(e.g. v0.5.0). If you're running an older version of App Mesh controller, please go to the upgrade section below before you proceed. If you are unsure, please run the appmesh-controller/upgrade/pre_upgrade_check.sh script to check if your cluster can be upgraded

Add the EKS repository to Helm:

helm repo add eks https://aws.github.io/eks-charts

Install the App Mesh CRDs:

kubectl apply -k "github.com/aws/eks-charts/stable/appmesh-controller//crds?ref=master"

Create namespace

kubectl create ns appmesh-system

The controller runs on the worker nodes, so it needs access to the AWS App Mesh / Cloud Map resources via IAM permissions. The IAM permissions can either be setup via IAM roles for service account or can be attached directly to the worker node IAM roles.

Setup IAM Role for Service Account

export CLUSTER_NAME=<eks-cluster-name>
export AWS_REGION=<aws-region e.g. us-east-1>
export AWS_ACCOUNT_ID=<AWS account ID>

Enable IAM OIDC provider

eksctl utils associate-iam-oidc-provider --region=$AWS_REGION \
    --cluster=$CLUSTER_NAME \
    --approve

Download the IAM policy for AWS App Mesh Kubernetes Controller

curl -o controller-iam-policy.json https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/master/config/iam/controller-iam-policy.json

Create an IAM policy called AWSAppMeshK8sControllerIAMPolicy

aws iam create-policy \
    --policy-name AWSAppMeshK8sControllerIAMPolicy \
    --policy-document file://controller-iam-policy.json

Take note of the policy ARN that is returned

Create an IAM role for service account for the App Mesh Kubernetes controller, use the ARN from the step above

Note: if you deleted serviceaccount in the appmesh-system namespace, you will need to delete and re-create iamserviceaccount. eksctl does not override the iamserviceaccount correctly (see this issue)

eksctl create iamserviceaccount --cluster $CLUSTER_NAME \
    --namespace appmesh-system \
    --name appmesh-controller \
    --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSAppMeshK8sControllerIAMPolicy  \
    --override-existing-serviceaccounts \
    --approve

Deploy appmesh-controller

Note: To enable mTLS via SDS(SPIRE), please set "sds.enabled=true".

helm upgrade -i appmesh-controller eks/appmesh-controller \
    --namespace appmesh-system \
    --set region=$AWS_REGION \
    --set serviceAccount.create=false \
    --set serviceAccount.name=appmesh-controller

The configuration section lists the parameters that can be configured during installation.

Note If you want to start the controller in the EKS private cluster, enable the app mesh and service discovery VPC endpoints to the linked private subnet first. Also accountId is a required field now as --set accountId=$AWS_ACCOUNT_ID.
If you want to enable X-ray tracing in private cluster, enable the X-ray VPC endpoint. Also, ECR VPC endpoint does not support public repository. Controller uses public.ecr.aws/xray/aws-xray-daemon:latest by default, so you need to pull this image to local and push it into your personal ECR repository. Set it when deploying the controller like:

helm upgrade -i appmesh-controller eks/appmesh-controller \
    --namespace appmesh-system \
    --set region=$AWS_REGION \
    --set serviceAccount.create=false \
    --set serviceAccount.name=appmesh-controller \
    --set accountId=$AWS_ACCOUNT_ID \
    --set log.level=debug \
    --set tracing.enabled=true \
    --set tracing.provider=x-ray \
    --set xray.image.repository={your-account-id}.dkr.ecr.{your-region}.amazonaws.com/{your-repository} \
    --set xray.image.tag={your-xray-daemon-image-tag}

Verify if the X-ray daemon being injected successfully when binding application deployment with virtual node/gateway.
More troubleshooting please see: https://docs.aws.amazon.com/eks/latest/userguide/private-clusters.html

Note Make sure that the Envoy proxies have the following IAM policies attached for the Envoy to authenticate with AWS App Mesh and fetch it's configuration

There are 2 ways you can attach the above policy to the Envoy Pod

With IRSA

Download the Envoy IAM policy

curl -o envoy-iam-policy.json https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/master/config/iam/envoy-iam-policy.json

Create an IAM policy called AWSAppMeshEnvoyIAMPolicy

aws iam create-policy \
    --policy-name AWSAppMeshEnvoyIAMPolicy \
    --policy-document file://envoy-iam-policy.json

Take note of the policy ARN that is returned

If your Mesh enabled applications are already using IRSA then you can attach the above policy to the role belonging to the existing IRSA or you can edit the Trust Relationship of the existing iam role which has this envoy policy so that some other service account in your mesh can also assume this role.

If not then you can create a service account for your application namespace and use the ARN from the step above. Ensure that Application Namespace already exists

eksctl create iamserviceaccount --cluster $CLUSTER_NAME \
    --namespace <ApplicationNamespaceName to which Envoy gets Injected> \
    --name envoy-proxy \
    --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSAppMeshEnvoyIAMPolicy  \
    --override-existing-serviceaccounts \
    --approve

Reference this Service Account in your application pod spec. This should be the pod which would get injected with the Envoy. Refer below example:

https://github.com/aws/aws-app-mesh-examples/blob/5a2d04227593d292d52e5e2ca638d808ebed5e70/walkthroughs/howto-k8s-fargate/v1beta2/manifest.yaml.template#L220

Without IRSA

Find the Node Instance IAM Role from your worker nodes and attach below policies to it. Note If you created service account for the controller as indicated above then you can skip attaching the Controller IAM policy to worker nodes. Instead attach only the Envoy IAM policy.

Controller IAM policy

curl -o controller-iam-policy.json https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/master/config/iam/controller-iam-policy.json

Envoy IAM policy Attach the below envoy policy to your Worker Nodes (Node Instance IAM Role)

curl -o envoy-iam-policy.json https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/master/config/iam/envoy-iam-policy.json

Apply the IAM policy directly to the worker nodes by replacing the <NODE_INSTANCE_IAM_ROLE_NAME>, <policy-name>, and <policy-filename> in below command:

aws iam put-role-policy --role-name <NODE_INSTANCE_IAM_ROLE_NAME> --policy-name <policy-name> --policy-document file://<policy-filename>

Deploy appmesh-controller

helm upgrade -i appmesh-controller eks/appmesh-controller \
    --namespace appmesh-system

The configuration section lists the parameters that can be configured during installation.

Installation on EKS with Fargate

export CLUSTER_NAME=<eks-cluster-name>
export AWS_REGION=<aws-region e.g. us-east-1>
export AWS_ACCOUNT_ID=<AWS account ID>

Create namespace

kubectl create ns appmesh-system

Setup EKS Fargate profile

eksctl create fargateprofile --cluster $CLUSTER_NAME --namespace appmesh-system

Enable IAM OIDC provider

eksctl utils associate-iam-oidc-provider --region=$AWS_REGION --cluster=$CLUSTER_NAME --approve

Download the IAM policy for AWS App Mesh Kubernetes Controller

curl -o controller-iam-policy.json https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/master/config/iam/controller-iam-policy.json

Create an IAM policy called AWSAppMeshK8sControllerIAMPolicy

aws iam create-policy \
    --policy-name AWSAppMeshK8sControllerIAMPolicy \
    --policy-document file://controller-iam-policy.json

Take note of the policy ARN that is returned

Create an IAM role for service account for the App Mesh Kubernetes controller, use the ARN from the step above

Note: if you deleted serviceaccount in the appmesh-system namespace, you will need to delete and re-create iamserviceaccount. eksctl does not override the iamserviceaccount correctly (see this issue)

eksctl create iamserviceaccount --cluster $CLUSTER_NAME \
    --namespace appmesh-system \
    --name appmesh-controller \
    --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/AWSAppMeshK8sControllerIAMPolicy  \
    --override-existing-serviceaccounts \
    --approve

Deploy appmesh-controller

Note: mTLS via SDS(SPIRE) is not supported on Fargate.

helm upgrade -i appmesh-controller eks/appmesh-controller \
    --namespace appmesh-system \
    --set region=$AWS_REGION \
    --set serviceAccount.create=false \
    --set serviceAccount.name=appmesh-controller

Upgrade

This section will assist you in upgrading the appmesh-controller from <=v0.5.0 version to >=v1.0.0 version.

You can either build new CRDs from scratch or migrate existing CRDs to the new schema. Please refer to the documentation here for the new API spec. Also, you can find several examples here with v1beta2 spec to help you get started.

Starting v1.0.0, Mesh resource supports namespaceSelectors, where you can either select namespace based on labels (recommended option) or select all namespaces. To select a namespace in a Mesh, you will need to define namespaceSelector:

apiVersion: appmesh.k8s.aws/v1beta2
kind: Mesh
metadata:
  name: <mesh-name>
spec:
  namespaceSelector:
    matchLabels:
      mesh: <mesh-name> // any string value

Note: If you set namespaceSelector: {}, mesh will select all the namespace in your cluster. Labels on your namespace spec is a no-op when selecting all namespaces.

In the namespace spec, you will need to add a label mesh: <mesh-name>. Here's a sample namespace spec:

apiVersion: v1
kind: Namespace
metadata:
  name: ns
  labels:
    mesh: <mesh-name>
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled

For more examples, please refer to the walkthroughs here. If you don't find an example that fits your use-case, please read the API spec here. If you find an issue in the documentation or the examples, please open an issue and we'll help resolve it.

Upgrade without preserving old App Mesh resources

# Keep old App Mesh controller running, it is responsible to cleanup App Mesh resources in AWS
# Delete all existing App Mesh custom resources (CRs)
kubectl delete virtualservices --all --all-namespaces
kubectl delete virtualnodes --all --all-namespaces
kubectl delete meshes --all --all-namespaces

# Delete all existing App Mesh CRDs
kubectl delete customresourcedefinition/virtualservices.appmesh.k8s.aws
kubectl delete customresourcedefinition/virtualnodes.appmesh.k8s.aws
kubectl delete customresourcedefinition/meshes.appmesh.k8s.aws
# Note: If a CRD stuck in deletion, it means there still exists some App Mesh custom resources, please check and delete them.

# Delete App Mesh controller
helm delete appmesh-controller -n appmesh-system

# Delete App Mesh injector
helm delete appmesh-inject -n appmesh-system

Run the appmesh-controller/upgrade/pre_upgrade_check.sh script and make sure it passes before you proceed

Now you can proceed with the installation steps described above

Upgrade preserving old App Mesh resources

# Save manifests of all existing App Mesh custom resources
kubectl get virtualservices --all-namespaces -o yaml > virtualservices.yaml
kubectl get virtualnodes --all-namespaces -o yaml > virtualnodes.yaml
kubectl get meshes --all-namespaces -o yaml > meshes.yaml

# Delete App Mesh controller, so it won’t clean up App Mesh resources in AWS while we deleting App Mesh CRs later.
helm delete appmesh-controller -n appmesh-system

# Delete App Mesh injector.
helm delete appmesh-inject -n appmesh-system

# Remove finalizers from all existing App Mesh CRs. Otherwise, you won’t be able to delete them

# To remove the finalizers, you could kubectl edit resource, and delete the finalizers attribute from the spec or run the following command to override finalizers. e.g for virtualnodes
# kubectl get virtualnodes --all-namespaces -o=jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\n"}{end}' | xargs -n2 sh -c 'kubectl patch virtualnode/$1 -n $0 -p '\''{"metadata":{"finalizers":null}}'\'' --type=merge'

# Alternatively, you could modify one resource at a time using
# kubectl get <RESOURCE_TYPE> <RESOURCE_NAME> -n <NAMESPACE> -o=json | jq '.metadata.finalizers = null' | kubectl apply -f -

# Delete all existing App Mesh CRs:
kubectl delete virtualservices --all --all-namespaces
kubectl delete virtualnodes --all --all-namespaces
kubectl delete meshes --all --all-namespaces

# Delete all existing App Mesh CRDs.
kubectl delete customresourcedefinition/virtualservices.appmesh.k8s.aws
kubectl delete customresourcedefinition/virtualnodes.appmesh.k8s.aws
kubectl delete customresourcedefinition/meshes.appmesh.k8s.aws
# Note: If CRDs are stuck in deletion, it means there still exists some App Mesh CRs, please check and delete them.

Run the appmesh-controller/upgrade/pre_upgrade_check.sh script and make sure it passes before you proceed

Translate the saved old YAML manifests using v1beta1 App Mesh CRD into v1beta2 App Mesh CRD format. Please refer to CRD types ( https://github.com/aws/aws-app-mesh-controller-for-k8s/tree/master/config/crd/bases) and Go types (https://github.com/aws/aws-app-mesh-controller-for-k8s/tree/master/apis/appmesh/v1beta2) for the CRD Documentation. Samples applications are in the repo https://github.com/aws/aws-app-mesh-examples for reference.

Note: Please specify the current appmesh resource names in the awsName field of the translated specs.

Install the appmesh-controller, label the namespace with values that mesh is selecting on and apply the translated manifest

Upgrade from prior script installation

If you've installed the App Mesh controllers with scripts, you can remove the controllers with the steps below.

# remove injector objects
kubectl delete ns appmesh-inject
kubectl delete ClusterRoleBinding aws-app-mesh-inject-binding
kubectl delete ClusterRole aws-app-mesh-inject-cr
kubectl delete  MutatingWebhookConfiguration aws-app-mesh-inject

# remove controller objects
kubectl delete ns appmesh-system
kubectl delete ClusterRoleBinding app-mesh-controller-binding
kubectl delete ClusterRole app-mesh-controller

Run the appmesh-controller/upgrade/pre_upgrade_check.sh script and make sure it passes before you proceed

For handling the existing custom resources and the CRDs please refer to either of the previous upgrade sections as relevant.

Running on ARM Based Instances

The controller can run on ARM based instances. To do this you need to specify the ARM64 controller image when deploying the Helm chart. You can specify the ARM64 image by setting the image.tag Helm parameter to <VERSION>-linux_arm64

For example, to run controller version 1.9.0 on ARM instances you could run the following:

helm upgrade -i appmesh-controller eks/appmesh-controller \
    --namespace appmesh-system \
    --set region=$AWS_REGION \
    --set image.tag=v1.9.0-linux_arm64

Uninstalling the Chart

To uninstall/delete the appmesh-controller deployment:

$ helm delete appmesh-controller -n appmesh-system

The command removes all the Kubernetes components associated with the chart and deletes the release.

Configuration

The following tables lists the configurable parameters of the chart and their default values.

Parameter Description Default
image.repository image repository 840364872350.dkr.ecr.us-west-2.amazonaws.com/amazon/appmesh-controller
image.tag image tag <VERSION>
image.pullPolicy image pull policy IfNotPresent
log.level controller log level, possible values are info and debug info
sds.enabled If true, SDS will be enabled in Envoy false
sds.udsPath Unix Domain Socket Path of the SDS Provider(SPIRE in the current release) /run/spire/sockets/agent.sock
resources.requests/cpu pod CPU request 100m
resources.requests/memory pod memory request 64Mi
resources.limits/cpu pod CPU limit 2000m
resources.limits/memory pod memory limit 1Gi
affinity node/pod affinities None
nodeSelector node labels for pod assignment {}
podAnnotations annotations to add to each pod {}
podLabels labels to add to each pod {}
tolerations list of node taints to tolerate []
rbac.create if true, create and use RBAC resources true
rbac.pspEnabled If true, create and use a restricted pod security policy false
serviceAccount.annotations optional annotations to add to service account {}
serviceAccount.create If true, create a new service account true
serviceAccount.name Service account to be used None
sidecar.image.repository Envoy image repository. If you override with non-Amazon built Envoy image, you will need to test/ensure it works with the App Mesh 840364872350.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-envoy
sidecar.image.tag Envoy image tag <VERSION>
sidecar.logLevel Envoy log level info
sidecar.envoyAdminAccessPort Envoy Admin Access Port 9901
sidecar.envoyAdminAccessLogFile Envoy Admin Access Log File /tmp/envoy_admin_access.log
sidecar.resources.requests Envoy container resource requests requests: cpu 10m memory 32Mi
sidecar.resources.limits Envoy container resource limits limits: cpu "" memory ""
sidecar.lifecycleHooks.preStopDelay Envoy container PreStop Hook Delay Value 20s
sidecar.lifecycleHooks.postStartInterval Envoy container PostStart Hook Interval Value 5s
sidecar.lifecycleHooks.postStartTimeout Envoy container PostStart Hook Timeout Value 180s
sidecar.probes.readinessProbeInitialDelay Envoy container Readiness Probe Initial Delay 1s
sidecar.probes.readinessProbePeriod Envoy container Readiness Probe Period 10s
sidecar.waitUntilProxyReady Enable pod postStart hook to delay application startup until proxy is ready to accept traffic false
init.image.repository Route manager image repository 840364872350.dkr.ecr.us-west-2.amazonaws.com/aws-appmesh-proxy-route-manager
init.image.tag Route manager image tag <VERSION>
stats.tagsEnabled If true, Envoy should include app-mesh tags false
stats.statsdEnabled If true, Envoy should publish stats to statsd endpoint @ 127.0.0.1:8125 false
stats.statsdAddress DogStatsD daemon IP address. This will be overridden if stats.statsdSocketPath is specified 127.0.0.1
stats.statsdPort DogStatsD daemon port. This will be overridden if stats.statsdSocketPath is specified 8125
stats.statsdSocketPath DogStatsD Unix domain socket path. If statsd is enabled but this value is not specified then we will use combination of statsAddress:statsPort as the default None
cloudMapCustomHealthCheck.enabled If true, CustomHealthCheck will be enabled for CloudMap Services false
cloudMapDNS.ttl Sets CloudMap DNS TTL. Will set value for new CloudMap services, but will not update existing CloudMap services. Existing CloudMap services can be updated using the AWS CloudMap API 300
tracing.enabled If true, Envoy will be configured with tracing false
tracing.provider The tracing provider can be x-ray, jaeger or datadog x-ray
tracing.address Jaeger or Datadog agent server address (ignored for X-Ray) appmesh-jaeger.appmesh-system
tracing.port Jaeger or Datadog agent port (ignored for X-Ray) 9411
tracing.samplingRate X-Ray tracer sampling rate. Value can be a decimal number between 0 and 1.00 (100%) 0.05
tracing.logLevel X-Ray agent log level, from most verbose to least: dev, debug, info, prod(default), warn, error. prod
tracing.role X-Ray agent assume the specified IAM role to upload segments to a different account None
enableCertManager Enable Cert-Manager false
xray.image.repository X-Ray image repository public.ecr.aws/xray/aws-xray-daemon
xray.image.tag X-Ray image tag latest
accountId AWS Account ID for the Kubernetes cluster None
env environment variables to be injected into the appmesh-controller pod {}
livenessProbe Liveness probe settings for the controller (see values.yaml)
podDisruptionBudget PodDisruptionBudget {}
tlsMinVersion Minimum TLS version for the controller webhook server as shown in here VersionTLS12
tlsCipherSuite Comma delimited TLS cipher suites for the controller webhook server as shown here None