Data Engineering Project with HDFS and Kafka

A project to create a data pipeline with data taken from Hepsiburada data engineering case study.

• docker-compose.yml

• config-hadoop

• Producer

  • Dockerfile

  • HB data

  • Kafka producer

  • requirements

• Consumer

  • Dockerfile

  • Kafka consumer

  • requirements

  • HDFS

Steps

Open an Ubuntu machine via AWS EC2 for the project.

Open the necessary ports on the machine through the firewall.

You also need to open the necessary ports with the operating system.

sudo ufw allow 9870
sudo ufw allow 8080
sudo ufw allow 8088

Then, stand up the docker images.

docker-compose up --build

One minute after the images stand up, data begins to be written to the Kafka topic and activity begins in the data pipeline.

Data from Kafka topic. IP:8080 or 0.0.0.0:8080

Hadoop HDFS interface. IP:9870 or 0.0.0.0:9870

Data from HDFS. IP:9870 or 0.0.0.0:9870

Hadoop cluster interface. IP:8088 or 0.0.0.0:8088

Ahmet Furkan Demir