Skip to content

Latest commit

 

History

History
44 lines (40 loc) · 2.41 KB

README.md

File metadata and controls

44 lines (40 loc) · 2.41 KB
Raw

bridgeAI-airflow-DAGs

Testing the DAGs on local KIND cluster

  1. Delete the kind-cluster bridgeai-gitops-infra if it already exists
kind delete cluster --name bridgeai-gitops-infra
  1. Clone the bridgeAI infra repo, bridgeai-gitops-infra
git clone https://github.com/digicatapult/bridgeAI-gitops-infra.git
  1. Modify bridgeai-gitops-infra repo - flux/clusters/kind-cluster/config/airflow-values.yaml with the branch of this DAGs repo that you want to test; for example branch: feature/data-ingestion
  2. Modify bridgeai-gitops-infra repo - flux/clusters/kind-cluster/base/flux-system/gotk-sync.yaml with branch of the infra repo that you are using to test; for example test/data-ingestion
  3. Add the changes in above files to the branch test/data-ingestion
  4. Commit and push the change to bridgeai-gitops-infra test/data-ingestion branch
  5. Update the feature/data-ingestion branch of bridgeAI-airflow-DAGs repo with the new dag and relevant changes that you want to test
  6. From the bridgeai-gitops-infra repo, test/data_ingestion branch, run the following to bring up a local kind cluster. Check the readme for more details.
./flux/scripts/add-kind-cluster.sh
  1. From the bridgeai-gitops-infra repo, test/data_ingestion branch, run the following to install and start the infra on the above created cluster. Check the readme for more details.
lux/scripts/install-flux.sh -b test/data_ingestion

You can check the logs by running kubectl events --watch from a terminal 10. Inspect the configmaps if needed

kubectl get configmaps
kubectl describe configmap <config-map-name>
  1. Ensure you have the PVCs needed up and ready
kubectl get pvc
  1. Access the Airflow UI - http://localhost:3080/airflow
  2. You can update the airflow variable from the airflow UI; Admin -> Variables
  3. Open the DAG and trigger it. You can inspect the logs in the airflow UI as well as kubectl events.

Some commands are;
kubectl get pods - to get list of pods in the default namespace
kubectl events --watch - to monitor the events from kubernetes
kubectl log <pod> - to get the logs from a specific pod\