Skip to content

Latest commit

 

History

History
43 lines (30 loc) · 1.64 KB

README.md

File metadata and controls

43 lines (30 loc) · 1.64 KB
Raw

README

apache-beam-kafka-example

Apache Kafka Integration with Apache Beam

Requirements

STEP 1: GET KAFKA

  • Download the latest Kafka release and extract it:

  • $ tar -xzf kafka_2.13-3.2.0.tgz

  • $ cd kafka_2.13-3.2.0

STEP 2: START THE KAFKA ENVIRONMENT

NOTE: Your local environment must have Java 8+ installed.

Run the following commands in order to start all services in the correct order:

Start the ZooKeeper service

Note: Soon, ZooKeeper will no longer be required by Apache Kafka.
$ bin/zookeeper-server-start.sh config/zookeeper.properties
  • Open another terminal session and run:

Start the Kafka broker service

$ bin/kafka-server-start.sh config/server.properties
  • Once all services have successfully launched, you will have a basic Kafka environment running and ready to use.

STEP 3: CREATE A TOPIC TO STORE YOUR EVENTS

$ bin/kafka-topics.sh --create --topic <tpoic_name> --bootstrap-server localhost:9092

3. Create a Kafka Console Producer.

Run the following command to start a Kafka Producer, using console interface, writing to sampleTopic.

$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic new_topic

4. Create a Kafka Console Consumer.

Run the following command to start a Kafka Consumer, using console interface, subscribed to sampleTopic.

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic new_topic --from-beginning

How to run the starter project?

  1. To build the service: Open the Kafka-Integration-With-Apache-Beam
  2. folder in terminal and run mvn clean install.
  3. To run the test cases: run mvn clean test.