Connector Setup

Building the Aerospike Hadooop Connector requires maven and git.

Install Maven

Note: Do not use sudo apt-get install maven ! It may not build the packages properly. Instead, do the following. As hduser on ztg-client, in /home/hduser:

hduser@ztg-client:~$ wget http://mirror.cc.columbia.edu/pub/software/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
... ‘apache-maven-3.0.5-bin.tar.gz’ saved 
hduser@ztg-client:~$ sudo tar xzf apache-maven-3.0.5-bin.tar.gz -C /usr/local
[sudo] password for hduser: 
hduser@ztg-client:~$ cd /usr/local
hduser@ztg-client:/usr/local$ sudo ln -s apache-maven-3.0.5 maven
hduser@ztg-client:/usr/local$ ls
apache-maven-3.0.5  bin  etc  games  include  lib  man  maven  sbin  share  src

Next, set up Maven path system-wide. Add the two lines below in maven.sh:

hduser@ztg-client:/usr/local$ sudo vi /etc/profile.d/maven.sh
export M2_HOME=/usr/local/maven
export PATH=${M2_HOME}/bin:${PATH}

Login again or do:

hduser@ztg-client:/usr/local$ source /etc/profile
hduser@ztg-client:/usr/local$ cd ~

Check maven install.

hduser@ztg-client:~$ mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 05:51:28-0800)
Maven home: /usr/local/maven
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-45-generic", arch: "amd64", family: "unix"
hduser@ztg-client:~$

Install Git

Install git to get Aerospike Hadoop Connector. As hduser on ztg-client, in /home/hduser:

hduser@ztg-client:~$ pwd
/home/hduser
hduser@ztg-client:~$ sudo apt-get install git

Check if hdclient can also access git. su to hdclient and,

hdclient@ztg-client:~$ git --version
git version 1.9.1

Install Aerospike Hadoop Connector

Note: We will install the Aerospike Hadoop Connector in ztg-client only since hdclient will compile its jars and send to the Apache Hadoop cluster for running jobs. Start in hdclient’s home directory. As hdclient on ztg-client, in /home/hdclient:

hdclient@ztg-client:~$ cd ~
hdclient@ztg-client:~$

Git - steps to download Aerospike Hadoop Connector

Aerospike has provided the Aerospike Hadoop Connector via a public repo on github. You must first fork this repo into your github account and then clone it. We walk through this example using organization acmegroup and user johndoe on github.com.

(Hint: To search Aerospike’s public repositories on github, use the search string “user:aerospike” in the search entry box for Repositories at github.com.)

johndoe has forked this repo at acmegroup/aerospike-hadoop. User johndoe has access to acmegroup organization. In user johndoe’s account settings, we can add as many id_rsa.pub keys as we want. Users with those public keys can then clone repos in acmegroup. So, we added hdclient’s public key to johndoe and then cloned the fork as follows:

hdclient@ztg-client:~$ more ~/.ssh/id_rsa.pub
ssh-rsa AAAAB3Nza[ . . . . . . . ]u2qD hdclient@ztg-client

Copy and paste this key to johndoe -> Account -> Settings -> Add SSH Keys on github’s web interface.

Now we can clone the fork and pull Aerospike’s Hadoop Connector code with examples into hdclient@ztg-client.

hdclient@ztg-client:~$ git clone [email protected]:zintecgroup/aerospike-hadoop.git
Cloning into 'aerospike-hadoop'...
Warning: Permanently added the RSA host key for IP address '192.30.252.129' to the list of known hosts.
remote: Counting objects: 1291, done.
remote: Total 1291 (delta 0), reused 0 (delta 0), pack-reused 1291
Receiving objects: 100% (1291/1291), 206.54 KiB | 0 bytes/s, done.
Resolving deltas: 100% (356/356), done.
Checking connectivity... done.
hdclient@ztg-client:~$

We now have the ~/aerospike-hadoop with the connector code in hdclient at ztg-client.

Build using Maven

hdclient@ztg-client:~$ cd ~/aerospike-hadoop
hdclient@ztg-client:~/aerospike-hadoop$ mvn clean package
[INFO] Reactor Summary:
[INFO] 
[INFO] aerospike-mapreduce ............................... SUCCESS [9.316s]
[INFO] sampledata ........................................ SUCCESS [8.747s]
[INFO] word_count_input .................................. SUCCESS [9.411s]
[INFO] aggregate_int_input ............................... SUCCESS [7.161s]
[INFO] word_count_output ................................. SUCCESS [7.741s]
[INFO] session_rollup .................................... SUCCESS [7.102s]
[INFO] generate_profiles ................................. SUCCESS [7.740s]
[INFO] external_join ..................................... SUCCESS [7.260s]
[INFO] spark_session_rollup .............................. SUCCESS [20.529s]
[INFO] aerospike-hadoop-examples ......................... SUCCESS [0.001s]
[INFO] aerospike-hadoop-parent ........................... SUCCESS [0.001s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:25.184s
[INFO] Finished at: Mon Jun 29 04:51:09 PDT 2015
[INFO] Final Memory: 148M/502M

Alternatively you can use gradle that runs on top of maven to build individual packages.

hdclient@ztg-client:~/aerospike-hadoop$ ls
build.gradle  gradlew      mapreduce  sampledata       WORLDCUP_FILELIST
examples      gradlew.bat  pom.xml    settings.gradle
gradle        LICENSE      README.md  TODO.md

hdclient@ztg-client:~/aerospike-hadoop$ ./gradlew :mapreduce:jar

Downloading http://services.gradle.org/distributions/gradle-1.11-bin.zip
.....
Unzipping /home/hdclient/.gradle/wrapper/dists/gradle-1.11-bin/4h5v8877arc3jhuqbm3osbr7o7/gradle-1.11-bin.zip to /home/hdclient/.gradle/wrapper/dists/gradle-1.11-bin/4h5v8877arc3jhuqbm3osbr7o7
Set executable permissions for: /home/hdclient/.gradle/wrapper/dists/gradle-1.11-bin/4h5v8877arc3jhuqbm3osbr7o7/gradle-1.11/bin/gradle
Download http://repo1.maven.org/maven2/org/sonatype/sisu/inject/cglib/2.2.1-v20090111/cglib-2.2.1-v20090111.pom
Download http://repo1.maven.org/maven2/org/sonatype/sisu/inject/cglib/2.2.1-v20090111/cglib-2.2.1-v20090111.jar
:mapreduce:compileJava
:mapreduce:processResources UP-TO-DATE
:mapreduce:classes
:mapreduce:jar
BUILD SUCCESSFUL
Total time: 2 mins 51.503 secs
hdclient@ztg-client:~/aerospike-hadoop$

We are now ready to test the examples on our Apache Hadoop cluster.