Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates and setup for ala dev #1

Open
patkyn opened this issue Dec 12, 2024 · 1 comment
Open

Updates and setup for ala dev #1

patkyn opened this issue Dec 12, 2024 · 1 comment

Comments

@patkyn
Copy link
Contributor

patkyn commented Dec 12, 2024 •
edited
Loading

Current installation in EKS dev with fargate pods

  1. ala-storage
  2. Airflow with local postgres db
  3. Solr
  4. Zookeeper
  5. Biocache cassandra
  6. Biocache service
  7. Ala Hub
  8. Collectory mysql
  9. Collectory
patkyn added a commit that referenced this issue Dec 12, 2024
@patkyn
#1 - Use latest version of airflow helm chart 8.9.0 with airflow 2.9
b8e7ed4
patkyn added a commit that referenced this issue Dec 12, 2024
@patkyn
#1 - Changes to the airflow helm values configuration for current ver…
42d2f83 
…sion of helm chart
patkyn added a commit that referenced this issue Dec 12, 2024
@patkyn
#1 - Add zookeeper persistence volume
167372b
patkyn added a commit that referenced this issue Dec 12, 2024
@patkyn
#1 - Zookeeper version upgrade, set persistence claim and network loa…
7005ce6 
…d balancer config
patkyn added a commit that referenced this issue Dec 12, 2024
@patkyn
#1 - Solr config changes for the mountPath and ingress for internal alb
06e28ba
patkyn added a commit that referenced this issue Dec 12, 2024
@patkyn
#1 - biocache cassandra, service, collectory and publishing service
5075863
@patkyn
Copy link
Contributor Author

patkyn commented Dec 12, 2024 •
edited
Loading

Findings:

  1. As most of our DAGs contain many smaller tasks, a better suited airflow executer would be CeleryExecuter. With Kubernetes executor, every task launches a pod, hence this slows down the airflow execution

  2. At this stage, Dynamic Provisioning is not supported for EKS Pods running on Fargate. https://docs.aws.amazon.com/eks/latest/userguide/fargate.html

image

Hence, a manual minor tweak needed to make the mounted path to work for StatefulSets zookeeper (to support more replicas) and this is not supported in the bitnami zookeeper helm chart.
Extra configurations needed for volumeMounts for replicas path
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@patkyn