How to execute

Before you execute zkndb, make sure you have successfully build the code as explained in How to build section.

Storage setup

Current zkndb has implementation of three storage types: HDFS, ZooKeeper and MySQL Cluster (NDB). We will highlight what are the specific setup for each storage in order to make zkndb's storage implementation works.

HDFS Setup

We used Hadoop 3.0.0 snapshot to perform our experiment. Alternatively, you can get and build the hadoop from our Hadoop fork.

In our sample deployment, we deployed 1 namenode and 3 datanodes. One of these datanodes was deployed in the same machine as the namenode. You can download the sample configuration files from here. Extract it and place the respective namenode/datanode configuration files in the correct $hadoop_path/etc for each machine. Note that each of the extracted folders contains a README file that explains what you need to replace (e.g. machine addresses).

To keep things simple, you can set the HDFS permission to allow any user to read and write data into the root directory.

ZooKeeper Setup

We used ZooKeeper 3.4.5. No zkndb-specific setup in our ZooKeeper cluster.

MySQL Cluster Setup

We used MySQL Cluster 7.2.8. After setup your MySQL cluster, you need to create zkndb database and applicationstate table inside zkndb database. The scripts to setup the database can be downloaded from here. There are three scripts available:

1. setupZkndbDb.sh. This script creates zkndb database and applicationstate table.
2. countApplicationState.sh. This script counts the number of persisted applicationstate in the table.
3. truncateApplicationState.sh. This script clear persisted applicationstate by using truncate table.

Run each script without any argument to see how it uses.

Execution

Assuming you already uploaded the precompiled executables to the machine you want to execute zkndb at, you should place the previously downloaded mysql lib folder content inside $zkndb_project_root/dist/mysql/lib. Now that everything is ready, you can download this script, place it in $zkndb_project_root/dist to facilitate the execution. Don't forget to change the $zkndb_project_root in the script.

The script starts by running zkndb benchmarking for Zookeeper, NDB and HDFS and generating the respective output csv files. These generated csv files are parsed and aggregated into a file called $zkndb_project_root/dist/results/plotThis. This file is used to generate a gnuplot graph into $zkndb_project_root/dist/histo.jpeg.

Take into consideration that this script receives four arguments : number of threads, metric period, execution time and time per cycle (e.g. 12 1000 60000 0 = 12 threads, logging each second for a whole minute at full throttle). The generated graph should look something like this:

Feel free to add new storage implementations and metrics! Please refer back to Getting-started for details.