Trucking Data Simulator

About the Simulator

The trucking data simulator allows you to do the following:

• Generate streaming events for different sensors on a truck
• Control the number of trucks/drivers on the road
• Control the number of events generated by each truck
• Use real trucking routes with real lat/long locations for the truck
• Control the output of the event (csv, json)
• Control what metadata goes into the event (schema metadata)
• Control where the event is generated (in a file, as an event into Kafka)

Out of the box, it supports two event types representing 2 different sesnsors that would be on a truck:

• Truck Geo Event

  • Sample Raw Event:

2016-10-19 14:54:38.18|truck_geo_event|40|23|Jeff Markham|1090292248|Peoria to Ceder Rapids Route 2|Normal|40.7|-89.52|1| 

• Truck Speed Event

2016-10-19 14:54:38.197|truck_speed_event|40|23|Jeff Markham|1090292248|Peoria to Ceder Rapids Route 2|73|

Building the Simulator

Pre-requisites

1. Java 8
2. setup your JAVA_HOME to point to java 8
3. Put JAVA_HOME on your PATH
4. Install Maven
5. Put Maven on your PATH
6. Git

Build the Simulator

• Clone the repo and the build the libraries

git clone https://github.com/georgevetticaden/hdp.git
cd hdp/app-utils/hdp-app-utils/
mvn clean install -DskipTests=true
cd ../hdp-app-utils/../../reference-apps/iot-trucking-app/
mvn clean install -DskipTests=true
cd trucking-data-simulator
mvn clean compile assembly:single
cd target

Running the Simulator

1. Download the Data-Loader.zip.
2. Unzip it to the location where you want to run the simulator. Lets call the directory you unzip it to as: $DATA_LOADER_HOME.
3. untar the route.tar.gz file (tar -zxvf $DATA_LOADER_HOME/routes.tar.gz)

The simulator has a number of different runners based on the type of output you want generated. The below walks through some of the scenarios

Example 1: Generate Trucking Events with SchemaId Embedded in Event

• Command:

nohup java -cp \
data-loader-jar-with-dependencies.jar \
hortonworks.hdf.sam.refapp.simulator.SimulationRegistrySerializerRunnerApp \
20000 \
hortonworks.hdf.sam.refapp.simulator.impl.domain.transport.Truck \
hortonworks.hdf.sam.refapp.simulator.impl.collectors.FileEventWithSchemaIdCollector \
1 \
$DATA_LOADER_HOME/routes/midwest/ \
10000 \
/tmp/truck-sensor-data/all-streams-with-schemaid-embedded.txt \
$REST_URL_TO_SCHEMA_REGISTRY \
ALL_STREAMS & 

• Details

  • Each Truck (13 trucks total) will generate 20000 events.
  • Each event will be prepended with schema-id and schema-version
  • Each truck will wait 10 seconds before its route is finished before going on the road again.
  • All of these events will be streamed into the following file: /tmp/truck-sensor-data/all-streams-with-schemaid-embedded.txt

• Sample Output

<schema-id>9<schema-id><schema-version>1<schema-version>|2017-05-01 12:30:42.086|truck_geo_event|69|13|Suresh Srinivas|6|Des Moines to Chicago|Lane Departure|41.62|-93.58|1|

<schema-id>10<schema-id><schema-version>1<schema-version>|2017-05-01 12:30:42.099|truck_speed_event|69|13|Suresh Srinivas|6|Des Moines to Chicago|104|

Example 2: Generate Trucking Events in CSV Format

• Command:

nohup java -cp \
data-loader-jar-with-dependencies.jar \
hortonworks.hdf.sam.refapp.simulator.SimulationRunnerApp \
20000 \
hortonworks.hdf.sam.refapp.simulator.impl.domain.transport.Truck \
hortonworks.hdf.sam.refapp.simulator.impl.collectors.FileEventCollector \
1 \
$DATA_LOADER_HOME/routes/midwest/ \
10000 \
/tmp/truck-sensor-data/all-streams.txt \
ALL_STREAMS &

• Details

  • Each Truck (13 trucks total) will generate 20000 events.
  • Each truck will wait 10 seconds before its route is finished before going on the road again.
  • All of these events will be streamed into the following file: /tmp/truck-sensor-data/all-streams.txt

• Sample Output

2017-05-01 12:38:42.747|truck_geo_event|101|13|Suresh Srinivas|12|Des Moines to Chicago|Unsafe tail distance|41.62|-93.58|1|

2017-05-01 12:38:42.799|truck_speed_event|101|13|Suresh Srinivas|12|Des Moines to Chicago|80|

2017-05-01 12:38:42.8|truck_geo_event|102|10|George Vetticaden|4|Saint Louis to Tulsa|Normal|38.64|-90.18|1|

2017-05-01 12:38:42.801|truck_speed_event|102|10|George Vetticaden|4|Saint Louis to Tulsa|65|

Example 3: Generate Trucking Events Serialized by HWX Schema Registry

nohup java -cp \
data-loader-jar-with-dependencies.jar \
hortonworks.hdf.sam.refapp.simulator.SimulationRegistrySerializerRunnerApp \
20000 \
hortonworks.hdf.sam.refapp.simulator.impl.domain.transport.Truck \
hortonworks.hdf.sam.refapp.simulator.impl.collectors.FileEventSerializedWithRegistryCollector \
1 \
$DATA_LOADER_HOME/routes/midwest/ \
10000 \
/tmp/truck-sensor-data/all-streams-serialized-with-hwx-registry.txt \
$REST_URL_TO_SCHEMA_REGISTRY \
ALL_STREAMS &

• Details

  • Each Truck (13 trucks total) will generate 20000 events.
  • Each event is serialized into Avro using the HWX Schema Registry
  • Each truck will wait 10 seconds before its route is finished before going on the road again.
  • All of these events will be streamed into the following file: /tmp/truck-sensor-data/all-streams-serialized-with-hwx-registry.txt

• Sample Output

^A^@^@^@^@^@^@^@        ^@^@^@^A^@.2017-05-01 13:13:16.499^^truck_geo_eventL^X^PJoe Witt^B,Saint Louis to Memphis^\Lane Departure<8f>Âõ(\OC@<9a><99><99><99><99><89>VÀ^B^A^@^@^@^@^@^@^@
^@^@^@^A^@.2017-05-01 13:13:17.119"truck_speed_eventL^X^PJoe Witt^B,Saint Louis to Memphis¦^A^A^@^@^@^@^@^@^@   ^@^@^@^A^@