Run Airbyte on EKS

[fatchat]

https://docs.airbyte.com/deploying-airbyte/on-kubernetes-via-helm

Set up a new Airbyte machine to experiment on
We want workers to run on EKS
The Airbyte API should work the same

[fatchat]

Plural is a tool called which can centrally manage an AWS Kubernetes cluster

1. Install the command-line EKS manager eksctl

2. Create an IAM role having the following permissions

AWS Service	Access Level
CloudFormation	Full Access
EC2	Full: Tagging ___ Limited: List, Read, Write
EC2 Auto Scaling	Limited: List, Write
EKS	Full Access
IAM	Limited: List, Read, Write, Permissions Management
Systems Manager	Limited: List, Read

3. Set up eksctl to use this role

4. Run

eksctl create cluster \
    --name=airbyte \
    --nodes-max 4 \
    --node-volume-size=50 \
    --node-volume-type=io1 \
    --tags tag=airbyte-eks \
    --ssh-access \
    --ssh-public-key=secrets/ddp.pub \
    --region=ap-south-1 

This takes a loooooong time

3. Install kubectl

4. Run

eksctl utils write-kubeconfig --cluster=<CLUSTER NAME>

kubectl config get-contexts
kubectl config use-context <eks context>

5. Install helm

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

6. helm repo add airbyte https://airbytehq.github.io/helm-charts

7. helm repo update

[fatchat]

8. helm install april-9 airbyte/airbyte. This command needs to be run with --values so we can pass RDS and GSM parameters in

[fatchat]

helm install did not complete successfully

the pod april-9-airbyte-bootloader's logs show that airbyte-db never started

kubectl exec airbyte-db-0 -- bash tells us that pod airbyte-db-0 does not have a host assigned