CEPModeMon - Complex Event Processing Modelling & Monitoring

CEPModeMon is a Prototype for a combined webapplication for modelling and monitoring Complex Event Processing Pipelines.

Requirements

Tested on Ubuntu 20.04 and Chrome 83
nodejs & npm
dockercompose

Installation

In the root of CEPModeMon:

npm install

How To Run

Option 1: Modeling Only

Start a simple kafka container

cd testenvironment
docker-compose up --force-recreate kafka zookeeper 

Proceed with the Start CEPModeMon

For Testing Purposes:

The Pre-Processor can be started without Data Collector after all Event Streams are labeled within the model.

cd testenvironment
docker-compose up cepmodemon

---

Option 2: Modeling & Monitoring Of CEPTA

Clone CEPTA Monitoring Version and configure your Mongo Database connection (TrainData) for the Replayer.

git clone --depth 1 --branch CEPModeMon [email protected]:bptlab/cepta.git
cd cepta

Start your prepared Mongo DB Container as State (optional) with Replay collection with station data:

./deployment/dev/devenv.sh up mongo 

Build CEPTA Images and Run CEPTA Core with Pre-Processor Service (cepmodemon) in Docker:

BUILD=1 ./deployment/dev/devenv.sh up --force-recreate kafka zookeeper kafdrop cepmodemon

When Kafdrop Broker is reachable on http://localhost:9001/ proceed with the Start Of CEPModeMon and continue here

2. In CEPModeMon click "Load".
3. Choose: ./examples/CEPTA.json
4. In CEPModeMon click "Start Monitoring".
5. Start the CEPTA Replayer (replace --mongo-port option with your Datasource for TrainData ).

bazel run -- //auxiliary/producers/replayer:replayer --port 8083 --replay-log debug --pause 100 --mode proportional --no-repeat --mongodb-port 27018 --immediate --include-sources PLANNED_TRAIN_DATA,LIVE_TRAIN_DATA

Now you should see after a while that the line charts display the aggregated Event Count (per 5 second windows) on the Event Streams and individual Events along the Event Streams, like this:

Note

If not all Streams get aggregate information the Pre-Processor might have missed the others on start up: Close everything and restart but Run Core and Pre-Processor separately:

./deployment/dev/devenv.sh up --force-recreate kafka zookeeper kafdrop core

After Kafdrop is reachable:

./deployment/dev/devenv.sh up cepmodemon

Proceed with Start of CEPModeMon.

Start CEPModeMon

In the root of CEPModeMon:

npm run start

In Chrome head to http://localhost:3000/

enjoy!

Requirements For Development

For Hot-Reload with nodemon

npm install -g nodemon

For rebuilding protobuf schemas you need to install protoc to run:

npm run build_proto

Known Issues

• F5 refresh in Client will reset CEPModeMon Client but not the Webserver.
• Actions like rename of Stream labels or Load cannot be canceled.
• Continue Monitoring after a Stop will not reset the Old Monitoring Data.
• Zooming may influence insertion of new Elements, if not on neutral zoom level.
• Pre-Processor might start too early.
• Chartjs labels might not work correctly.
• No Security precautions in any way.

Example Architecture

CEPModeMons monitoring capabilities require an architecture like this:

This repository contains the Modelling and Monitoring Tool. It serves as a high level Consumer in the necessary Monitoring CEP Pipeline. Other components are necessary to get the Monitoring-Data and to pre-process the data before visualisation in CEPModeMon.

CEPTA Data Collector

Serves as high level Producer.
The modified CEPTA version is located here: CEPTA-with-DataCollector

Pre-Processor

Serves as high level Event Processing Agent.
The Pre-Processor is located here: PreProcessor or per Docker Container cepmodemon/prepro:v0.1.9

Available Modelling Components

Drag'n'drop-able atomic conceptional Consumers, Producers and mostly Event Processing Agentss.

For scale toggling or element deletion use the + on the upper left.