java.lang.AbstractMethodError

[rhinempi]

Error log reported by Chu Wang:

17/02/08 18:44:20 ERROR Utils: Aborting task
java.lang.AbstractMethodError: uni.bielefeld.cmg.sparkhit.pipeline.SparkPipe$2SparkBatchAlign.call(Ljava/lang/Object;)Ljava/util/Iterator;
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:124)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply$mcV$sp(PairRDDFunctions.scala:1203)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1203)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13$$anonfun$apply$7.apply(PairRDDFunctions.scala:1203)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1325)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1211)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$13.apply(PairRDDFunctions.scala:1190)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:85)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
17/02/08 18:44:20 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)

[rhinempi]

Spark 2.0.0 version API changed return type of call function to "iterator".

Fixed

[tom-dyar]

Hi! I ran into this while just getting set up -- is there a workaround??

[rhinempi]

Hi Tom,
which Spark version are you using? There is a major change on the interface of the Spark 2.0.0 version. If you are using Sparkhit on the Spark 2.0.0+, choose the Sparkhit 1.0 version. If you have set up a Spark cluster with 2.0.0- version (say 1.6.0), you can still use the Sparkhit 0.8 version by changing the sparkhit executable shell :
name="sparkhit"
version="1.0" # change to 0.8 if you are using Spark version under 2.0.0
spark_version="2.0.0" # only for auto downloading Spark package
Let me know if you have further questions.

[tom-dyar]

Liren, That fixed it , thanks! I was also having trouble running the example in which it would complain about the input and reference files not existng so I had to copyFromLocal to to hdfs (running on a cluster backed by hdfs not standalone). Seems to be some confusion between local and hdfs files? I will look at the code if I get a chance, it might be helpful to check for the input and reference files as well as non-empty result directory at the beginning of a run since that was disappointing to get through the run only to have it fail at the end when it couldn’t write to the result directory... Tom From: Liren Huang <[email protected]> Sent: Saturday, February 24, 2018 7:23:04 AM To: rhinempi/sparkhit Cc: Thomas Dyar (EXTERNAL); Comment Subject: Re: [rhinempi/sparkhit] java.lang.AbstractMethodError (#1) Hi Tom, which Spark version are you using? There is a major change on the interface of the Spark 2.0.0 version. If you are using Sparkhit on the Spark 2.0.0+, choose the Sparkhit 1.0 version. If you have set up a Spark cluster with 2.0.0- version (say 1.6.0), you can still use the Sparkhit 0.8 version by changing the sparkhit executable shell : name="sparkhit" version="1.0" # change to 0.8 if you are using Spark version under 2.0.0 spark_version="2.0.0" # only for auto downloading Spark package Let me know if you have further questions. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rhinempi_sparkhit_issues_1-23issuecomment-2D368224827&d=DwMFaQ&c=n7UHtw8cUfEZZQ61ciL2BA&r=Wpxr3yZIgJUDc4CsPhUpuiAtTKDn_lya4DWla3Q21iI&m=oOTbRAUg9ZSSE-2ZdtH2NOM8aT7_1vlWxv-JOuiGhHw&s=bWZbVFSJ6XVFE3zjI28BYaT0wAZZdd9InpAiYVfChEA&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIfLuRHIDBwsbXiYRGTYuARhVPhdRKNcks5tX-5F8ogaJpZM4L775e&d=DwMFaQ&c=n7UHtw8cUfEZZQ61ciL2BA&r=Wpxr3yZIgJUDc4CsPhUpuiAtTKDn_lya4DWla3Q21iI&m=oOTbRAUg9ZSSE-2ZdtH2NOM8aT7_1vlWxv-JOuiGhHw&s=KySptUO-y4HAmQ3PDNud_JwMagOf7qG8k2ljpxvj8I8&e=>.

[rhinempi]

Hi Tom,

Thank you for your advise and comments.
I will update the functions in the next release.

Liren