ETL for loading weather data from open-meteo API to database
System diagram is located in the readme of this repo: (https://github.com/kgmcquate/lake-freeze-frontend)
TODO:
- Create publicly-available job monitoring dashboard from CloudWatch logs
- Create publicly-available dashboard for weather data
- Add Data Quality step and alert
- Evaluate cost of running this in Azure instead of AWS EMR Serverless
- Run airflow on EC2 (with nat gateway) and make the UI publcly available
- Add unit tests