Airflow Unit Tests and Integration Tests

Gerador de DAGs no Apache Airflow para fazer clipping do Diário Oficial da União.

Classwork projects and home works done through Udacity data engineering nano degree

Data Engineering examples for Airflow, Prefect, and Mage.ai; dbt for BigQuery, Redshift, ClickHouse, PostgreSQL; Spark/PySpark for Batch processing; and Kafka for Stream processing

Sync DAG changes from Git to Airflow

Zero configuration Airflow plugin that let you manage your DAG files.

Orchestration of data science and earth observation models in Apache Airflow, scale-up with Celery Executor, experiment with jupyter notebook using a docker containers composition

My self-learning about Apache Airflow

This is my Apache Airflow Local development setup on Windows 10 WSL2/Mac using docker-compose. It will also include some sample DAGs and workflows.

🎵 LyricWave - Your AI Music Composer 🎶 Compose Unique MP4 Songs Effortlessly! LyricWave uses AI to create personalized music by harmonizing lyrics with captivating melodies and synthetic vocals. Unleash your musical creativity today! 🚀🎶

Apache Airflow Guide

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows

Loan Default Prediction using PySpark, with jobs scheduled by Apache Airflow and Integration with Spark using Apache Livy

An end-to-end Twitter Data Pipeline that extracts data from Twitter and loads it into AWS S3.

WebApp to bring together Text Summarization and Sentiment Analysis of the stock related news to better understand the stock price trends.

Here I added 9 projects which have been made by me during my apprenticeship in Yandex.Practicum as data engineer.

DAGs, Python Scripts, and SQL scripts for creating and updating the Saint Louis Regional Entity Database

A starting point for a data stack using Python, Apache Airflow and Metabase.

This project creates a basic web service for solving image-based CAPTCHAs. Using the Flask framework, it allows users to upload CAPTCHA images and employs an Optical Character Recognition (OCR) pipeline to extract the embedded text.