running a spark application fails on EC2 with warcbase dependecy

[dportabella]

I have a simple spark application, and it runs ok on my laptop using spark-submit.
However, I get this runtime error when I run it using spark-submit on a Amazon EC2 cluster:

$ /root/spark/bin/spark-submit --class application.Example --master spark://ec2-54-227-170-20.compute-1.amazonaws.com:7077 /root/example-assembly-0.1-SNAPSHOT.jar 

java.lang.IncompatibleClassChangeError: Found class org.apache.hadoop.mapreduce.TaskAttemptContext, but interface was expected
    at org.warcbase.mapreduce.WacGenericInputFormat$GenericArchiveRecordReader.initialize(WacGenericInputFormat.java:71)

I made sure that /root/example-assembly-0.1-SNAPSHOT.jar does not contain org/apache/spark nor org/apache/hadoop.

I am using spark 1.6.1 and hadoop 2.6.0. I see that warcbase depends on hadoop 2.6.0-cdh5.7.1.

What can be the problem?