Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting started task #117

Open
ELMERWANG opened this issue Mar 10, 2021 · 9 comments
Open

Getting started task #117

ELMERWANG opened this issue Mar 10, 2021 · 9 comments
Assignees
@zhangjiajin

Comments

@ELMERWANG
Copy link

Hi,

I am quite new to stream mining and I am trying to get the streamDM running. However when I finished setting up all the dependencies I got some issues reported below.

Expected behavior

I want to obtain the confusion matrix by using the example commands.

Observed behavior

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/03/09 18:40:50 INFO SparkContext: Running Spark version 2.1.0
21/03/09 18:40:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/09 18:40:50 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0)
21/03/09 18:40:50 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/03/09 18:40:50 INFO SecurityManager: Changing view acls to: shunyao
21/03/09 18:40:50 INFO SecurityManager: Changing modify acls to: shunyao
21/03/09 18:40:50 INFO SecurityManager: Changing view acls groups to:
21/03/09 18:40:50 INFO SecurityManager: Changing modify acls groups to:
21/03/09 18:40:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shunyao); groups with view permissions: Set(); users with modify permissions: Set(shunyao); groups with modify permissions: Set()
21/03/09 18:40:50 INFO Utils: Successfully started service 'sparkDriver' on port 41123.
21/03/09 18:40:50 INFO SparkEnv: Registering MapOutputTracker
21/03/09 18:40:50 INFO SparkEnv: Registering BlockManagerMaster
21/03/09 18:40:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/09 18:40:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/09 18:40:50 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-8d18da96-7b2c-48c9-bbd2-b49a29b16274
21/03/09 18:40:50 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
21/03/09 18:40:50 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/09 18:40:51 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/09 18:40:51 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.122.1:4040
21/03/09 18:40:51 INFO SparkContext: Added JAR file:/home/shunyao/Downloads/streamDM-0.2/scripts/../target/scala-2.10/streamdm-spark-streaming-_2.10-0.2.jar at spark://192.168.122.1:41123/jars/streamdm-spark-streaming-_2.10-0.2.jar with timestamp 1615333251155
21/03/09 18:40:51 INFO Executor: Starting executor ID driver on host localhost
21/03/09 18:40:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45261.
21/03/09 18:40:51 INFO NettyBlockTransferService: Server created on 192.168.122.1:45261
21/03/09 18:40:51 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/09 18:40:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.122.1, 45261, None)
21/03/09 18:40:51 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.122.1:45261 with 413.9 MB RAM, BlockManagerId(driver, 192.168.122.1, 45261, None)
21/03/09 18:40:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.122.1, 45261, None)
21/03/09 18:40:51 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.122.1, 45261, None)
Exception in thread "main" java.lang.Exception: Problem creating instance of class: EvaluatePrequential
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:139)
at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:43)
at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: Problems with option: streamReader
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:60)
at com.github.javacliparser.AbstractOption.resetToDefault(AbstractOption.java:90)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:84)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:63)
at com.github.javacliparser.ClassOption.(ClassOption.java:34)
at org.apache.spark.streamdm.tasks.EvaluatePrequential.(EvaluatePrequential.scala:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:137)
... 11 more
Caused by: java.lang.Exception: Class not found: FileReader
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:57)
... 22 more
21/03/09 18:40:51 INFO SparkContext: Invoking stop() from shutdown hook
21/03/09 18:40:51 INFO SparkUI: Stopped Spark web UI at http://192.168.122.1:4040
21/03/09 18:40:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/09 18:40:51 INFO MemoryStore: MemoryStore cleared
21/03/09 18:40:51 INFO BlockManager: BlockManager stopped
21/03/09 18:40:51 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/09 18:40:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/09 18:40:51 INFO SparkContext: Successfully stopped SparkContext
21/03/09 18:40:51 INFO ShutdownHookManager: Shutdown hook called
21/03/09 18:40:51 INFO ShutdownHookManager: Deleting directory /tmp/spark-6f0a2908-b3f5-477c-bccc-6302be0e18f8

Steps to reproduce the issue

I tried different OS: macOS, Kali, Ubuntu and CentOS 7.

These steps are what I did and followed:
Step 1: Download & install the Java openjdk8.
Step 2: Download & install the Spark (I tried 2.3.2 and 2.1.0 with Hadoop or without Hadoop)
Step 3: Download & install Scala (from https://www.scala-lang.org/files/archive/scala-2.11.8.deb)
Step 4: Download & install SBT (from https://sbt.bintray.com/native-packages/sbt/0.13.15/sbt-0.13.15.tgz)
Step 5: Download StreamDM from Github.
Step 6: Change the SPARK_HOME to the Spark path.
Step 7: In the streamDM directory, run sbt package
Step 8: In the /scripts, run ./spark.sh or ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)"
Result: Errors occurred.

Command line
sbt package
./spark.sh
./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)"

Infrastructure details (Contains the versions that I tried)

  • Java Version: 8 | 11
  • Scala Version: 2.10.2 | 2.11.8 | 2.11.11
  • Spark version: 2.1.0 | 2.3.2
  • OS version: macOS, Ubuntu 20.04, Kali, CentOS 7
  • Cluster mode or local mode? Local

For the Spark, do I need to download with Hadoop or without Hadoop or the file name only contains spark-x.x.x.tgz?
Such as:

  • spark-2.3.2-bin-hadoop2.7.tgz
  • spark-2.3.2-bin-without-hadoop.tgz
  • spark-2.3.2.tgz

Is anything wrong with my setup? I produced the same error on different OS.
Please help, thanks in advance :)
@zhangjiajin
Copy link
Member

@abifet
@hmgomes

@hmgomes
Copy link
Collaborator

hmgomes commented Mar 19, 2021

Hi @ELMERWANG
It looks like a problem with SGDLearner. Can you run it using a Hoeffding tree?

./spark.sh "EvaluatePrequential -l (trees.HoeffdingTree) -s (FileReader -f ../data/mydata.arff -k 450 -i 45500)" 1> results_ht_mydata.csv 2> log_ht_mydata.log

Btw, if you are new to stream mining, I suggest this open book to get started

@ELMERWANG
Copy link
Author

Hi @hmgomes,

Thank you for your reply, I have tried the command that you provide, however the error now changed to:

Error message:
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/03/21 19:46:27 INFO SparkContext: Running Spark version 2.1.0
21/03/21 19:46:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/21 19:46:27 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.122.1 instead (on interface virbr0)
21/03/21 19:46:27 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
21/03/21 19:46:27 INFO SecurityManager: Changing view acls to: shunyao
21/03/21 19:46:27 INFO SecurityManager: Changing modify acls to: shunyao
21/03/21 19:46:27 INFO SecurityManager: Changing view acls groups to:
21/03/21 19:46:27 INFO SecurityManager: Changing modify acls groups to:
21/03/21 19:46:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(shunyao); groups with view permissions: Set(); users with modify permissions: Set(shunyao); groups with modify permissions: Set()
21/03/21 19:46:28 INFO Utils: Successfully started service 'sparkDriver' on port 45035.
21/03/21 19:46:28 INFO SparkEnv: Registering MapOutputTracker
21/03/21 19:46:28 INFO SparkEnv: Registering BlockManagerMaster
21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/03/21 19:46:28 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-deb8bb4e-c0e0-450e-b68d-3f96548bece1
21/03/21 19:46:28 INFO MemoryStore: MemoryStore started with capacity 413.9 MB
21/03/21 19:46:28 INFO SparkEnv: Registering OutputCommitCoordinator
21/03/21 19:46:28 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/03/21 19:46:28 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.122.1:4040
21/03/21 19:46:28 INFO SparkContext: Added JAR file:/home/shunyao/Downloads/streamDM-0.2/scripts/../target/scala-2.10/streamdm-spark-streaming-_2.10-0.2.jar at spark://192.168.122.1:45035/jars/streamdm-spark-streaming-_2.10-0.2.jar with timestamp 1616370388511
21/03/21 19:46:28 INFO Executor: Starting executor ID driver on host localhost
21/03/21 19:46:28 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38860.
21/03/21 19:46:28 INFO NettyBlockTransferService: Server created on 192.168.122.1:38860
21/03/21 19:46:28 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/21 19:46:28 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.122.1, 38860, None)
21/03/21 19:46:28 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.122.1:38860 with 413.9 MB RAM, BlockManagerId(driver, 192.168.122.1, 38860, None)
21/03/21 19:46:28 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.122.1, 38860, None)
21/03/21 19:46:28 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.122.1, 38860, None)
Exception in thread "main" java.lang.Exception: Problem creating instance of class: EvaluatePrequential
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:139)
at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:43)
at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalArgumentException: Problems with option: streamReader
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:60)
at com.github.javacliparser.AbstractOption.resetToDefault(AbstractOption.java:90)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:84)
at com.github.javacliparser.AbstractClassOption.(AbstractClassOption.java:63)
at com.github.javacliparser.ClassOption.(ClassOption.java:34)
at org.apache.spark.streamdm.tasks.EvaluatePrequential.(EvaluatePrequential.scala:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:137)
... 11 more
Caused by: java.lang.Exception: Class not found: FileReader
at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
at com.github.javacliparser.ClassOption.setValueViaCLIString(ClassOption.java:57)
... 22 more
21/03/21 19:46:28 INFO SparkContext: Invoking stop() from shutdown hook
21/03/21 19:46:28 INFO SparkUI: Stopped Spark web UI at http://192.168.122.1:4040
21/03/21 19:46:28 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/03/21 19:46:28 INFO MemoryStore: MemoryStore cleared
21/03/21 19:46:28 INFO BlockManager: BlockManager stopped
21/03/21 19:46:28 INFO BlockManagerMaster: BlockManagerMaster stopped
21/03/21 19:46:28 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/03/21 19:46:28 INFO SparkContext: Successfully stopped SparkContext
21/03/21 19:46:28 INFO ShutdownHookManager: Shutdown hook called
21/03/21 19:46:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-d340aa5a-9e9b-4d68-a6fd-b24a0276338d

And the result file is empty...

@hmgomes
Copy link
Collaborator

hmgomes commented Mar 22, 2021

Hi @ELMERWANG

That is interesting. Can you try using Spark 2.4.5?

Cheers,
Heitor

@ELMERWANG
Copy link
Author

Hi @hmgomes,

Thanks, may I ask which version should I download? With or without hadoop?

[ ] spark-2.4.5-bin-hadoop2.6.tgz 2020-02-02 20:27 220M

[ ] spark-2.4.5-bin-hadoop2.7.tgz 2020-02-02 20:27 222M

[ ] spark-2.4.5-bin-without-hadoop-scala-2.12.tgz 2020-02-02 20:27 139M

[ ] spark-2.4.5-bin-without-hadoop.tgz 2020-02-02 20:27 160M

[ ] spark-2.4.5.tgz 2020-02-02 20:27 15M

@LYZ1125
Copy link

LYZ1125 commented Mar 31, 2021

@ELMERWANG Hi,I met the same problem as yours. Do you solve your problem?I want to get some suggestions to solve these problems. Thank you!

Infrastructure details
Java Version: 8
Scala Version: 2.11.8
Spark version: 2.3.2
OS version: Ubuntu
sbt version:0.13.18

When I run these Command lines
sbt package
./spark.sh

I get some errors:

java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-30 06:25:41 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

after I ran ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)",I got the follewing result:

/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: syntax error near unexpected token `newline'
/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: `<!--主节点IP-->'
2021-03-30 06:28:28 WARN  Utils:66 - Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.65.128 instead (on interface ens33)
2021-03-30 06:28:28 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-03-30 06:28:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-30 06:28:32 INFO  SparkContext:54 - Running Spark version 2.3.2
2021-03-30 06:28:32 INFO  SparkContext:54 - Submitted application: streamDM
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuanzhanglin); groups with view permissions: Set(); users  with modify permissions: Set(yuanzhanglin); groups with modify permissions: Set()
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44865.
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering MapOutputTracker
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering BlockManagerMaster
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2021-03-30 06:28:33 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-beb224af-932d-4d9f-9695-42a4a64c2287
2021-03-30 06:28:33 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2021-03-30 06:28:33 INFO  log:192 - Logging initialized @6306ms
2021-03-30 06:28:33 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2021-03-30 06:28:33 INFO  Server:419 - Started @6448ms
2021-03-30 06:28:33 INFO  AbstractConnector:278 - Started ServerConnector@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42a9a63e{/jobs,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bd1ceca{/jobs/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30c31dd7{/jobs/job,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596df867{/jobs/job/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1fca1e{/stages,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241a53ef{/stages/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@344344fa{/stages/stage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@615f972{/stages/stage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@285f09de{/stages/pool,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73393584{/stages/pool/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31500940{/storage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1827a871{/storage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e64352{/storage/rdd,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7249dadf{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4362d7df{/environment,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66238be2{/environment/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c25b8a7{/executors,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@200606de{/executors/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@750fe12e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f8908f6{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e587920{/static,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b7ffa4{/,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ed190be{/api,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41382722{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dac3fd8{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  SparkContext:54 - Added JAR file:/home/yuanzhanglin/streamDM/scripts/../target/scala-2.11/streamdm-spark-streaming-_2.11-0.2.jar at spark://192.168.65.128:44865/jars/streamdm-spark-streaming-_2.11-0.2.jar with timestamp 1617110914021
2021-03-30 06:28:34 INFO  Executor:54 - Starting executor ID driver on host localhost
2021-03-30 06:28:34 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37021.
2021-03-30 06:28:34 INFO  NettyBlockTransferService:54 - Server created on 192.168.65.128:37021
2021-03-30 06:28:34 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.65.128:37021 with 366.3 MB RAM, BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3703bf3c{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.Exception: Class not found: or
	at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
	at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:55)
	at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021-03-30 06:28:34 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2021-03-30 06:28:34 INFO  AbstractConnector:318 - Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:34 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2021-03-30 06:28:34 INFO  MemoryStore:54 - MemoryStore cleared
2021-03-30 06:28:34 INFO  BlockManager:54 - BlockManager stopped
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2021-03-30 06:28:34 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2021-03-30 06:28:34 INFO  SparkContext:54 - Successfully stopped SparkContext
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Shutdown hook called
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-27030bd4-3077-4318-8fec-a42efb3c4dbb
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-d29803f8-d90b-491c-82e5-cc2599ac2e0f

@ELMERWANG
Copy link
Author

@ELMERWANG Hi,I met the same problem as yours. Do you solve your problem?I want to get some suggestions to solve these problems. Thank you!

Infrastructure details
Java Version: 8
Scala Version: 2.11.8
Spark version: 2.3.2
OS version: Ubuntu
sbt version:0.13.18

When I run these Command lines
sbt package
./spark.sh

I get some errors:

java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-30 06:25:41 ERROR Executor:91 - Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:84)
	at org.apache.spark.streamdm.core.DenseInstance.add(DenseInstance.scala:26)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:91)
	at org.apache.spark.streamdm.classifiers.SGDLearner$$anonfun$train$1$$anonfun$3.apply(SGDLearner.scala:85)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
	at org.apache.spark.InterruptibleIterator.foldLeft(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
	at org.apache.spark.InterruptibleIterator.aggregate(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.rdd.RDD$$anonfun$aggregate$1$$anonfun$21.apply(RDD.scala:1122)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2130)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

after I ran ./spark.sh "EvaluatePrequential -l (SGDLearner -l 0.01 -o LogisticLoss -r ZeroRegularizer) –s (FileReader –k 100 –d 60 –f ../data/mydata)",I got the follewing result:

/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: syntax error near unexpected token `newline'
/home/yuanzhanglin/local/spark-2.3.2/conf/spark-env.sh: line 71: `<!--主节点IP-->'
2021-03-30 06:28:28 WARN  Utils:66 - Your hostname, ubuntu resolves to a loopback address: 127.0.1.1; using 192.168.65.128 instead (on interface ens33)
2021-03-30 06:28:28 WARN  Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address
2021-03-30 06:28:32 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2021-03-30 06:28:32 INFO  SparkContext:54 - Running Spark version 2.3.2
2021-03-30 06:28:32 INFO  SparkContext:54 - Submitted application: streamDM
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls to: yuanzhanglin
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing view acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - Changing modify acls groups to: 
2021-03-30 06:28:32 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuanzhanglin); groups with view permissions: Set(); users  with modify permissions: Set(yuanzhanglin); groups with modify permissions: Set()
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 44865.
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering MapOutputTracker
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering BlockManagerMaster
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2021-03-30 06:28:33 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2021-03-30 06:28:33 INFO  DiskBlockManager:54 - Created local directory at /tmp/blockmgr-beb224af-932d-4d9f-9695-42a4a64c2287
2021-03-30 06:28:33 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2021-03-30 06:28:33 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2021-03-30 06:28:33 INFO  log:192 - Logging initialized @6306ms
2021-03-30 06:28:33 INFO  Server:351 - jetty-9.3.z-SNAPSHOT, build timestamp: unknown, git hash: unknown
2021-03-30 06:28:33 INFO  Server:419 - Started @6448ms
2021-03-30 06:28:33 INFO  AbstractConnector:278 - Started ServerConnector@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:33 INFO  Utils:54 - Successfully started service 'SparkUI' on port 4040.
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@42a9a63e{/jobs,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5bd1ceca{/jobs/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@30c31dd7{/jobs/job,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@596df867{/jobs/job/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@c1fca1e{/stages,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@241a53ef{/stages/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@344344fa{/stages/stage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@615f972{/stages/stage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@285f09de{/stages/pool,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@73393584{/stages/pool/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@31500940{/storage,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1827a871{/storage/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@48e64352{/storage/rdd,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7249dadf{/storage/rdd/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4362d7df{/environment,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@66238be2{/environment/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c25b8a7{/executors,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@200606de{/executors/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@750fe12e{/executors/threadDump,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@f8908f6{/executors/threadDump/json,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3e587920{/static,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@77b7ffa4{/,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5ed190be{/api,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@41382722{/jobs/job/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7dac3fd8{/stages/stage/kill,null,AVAILABLE,@Spark}
2021-03-30 06:28:33 INFO  SparkUI:54 - Bound SparkUI to 0.0.0.0, and started at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  SparkContext:54 - Added JAR file:/home/yuanzhanglin/streamDM/scripts/../target/scala-2.11/streamdm-spark-streaming-_2.11-0.2.jar at spark://192.168.65.128:44865/jars/streamdm-spark-streaming-_2.11-0.2.jar with timestamp 1617110914021
2021-03-30 06:28:34 INFO  Executor:54 - Starting executor ID driver on host localhost
2021-03-30 06:28:34 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37021.
2021-03-30 06:28:34 INFO  NettyBlockTransferService:54 - Server created on 192.168.65.128:37021
2021-03-30 06:28:34 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMasterEndpoint:54 - Registering block manager 192.168.65.128:37021 with 366.3 MB RAM, BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, 192.168.65.128, 37021, None)
2021-03-30 06:28:34 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3703bf3c{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.Exception: Class not found: or
	at com.github.javacliparser.ClassOption.cliStringToObject(ClassOption.java:132)
	at org.apache.spark.streamdm.streamDMJob$.main(streamDMJob.scala:55)
	at org.apache.spark.streamdm.streamDMJob.main(streamDMJob.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2021-03-30 06:28:34 INFO  SparkContext:54 - Invoking stop() from shutdown hook
2021-03-30 06:28:34 INFO  AbstractConnector:318 - Stopped Spark@21a5fd96{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2021-03-30 06:28:34 INFO  SparkUI:54 - Stopped Spark web UI at http://192.168.65.128:4040
2021-03-30 06:28:34 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2021-03-30 06:28:34 INFO  MemoryStore:54 - MemoryStore cleared
2021-03-30 06:28:34 INFO  BlockManager:54 - BlockManager stopped
2021-03-30 06:28:34 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2021-03-30 06:28:34 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2021-03-30 06:28:34 INFO  SparkContext:54 - Successfully stopped SparkContext
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Shutdown hook called
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-27030bd4-3077-4318-8fec-a42efb3c4dbb
2021-03-30 06:28:34 INFO  ShutdownHookManager:54 - Deleting directory /tmp/spark-d29803f8-d90b-491c-82e5-cc2599ac2e0f

Unfortunately, no luck on fixing it, sorry..

@hmgomes
Copy link
Collaborator

hmgomes commented Apr 9, 2021

Hi @hmgomes,

Thanks, may I ask which version should I download? With or without hadoop?

[ ] spark-2.4.5-bin-hadoop2.6.tgz 2020-02-02 20:27 220M

[ ] spark-2.4.5-bin-hadoop2.7.tgz 2020-02-02 20:27 222M

[ ] spark-2.4.5-bin-without-hadoop-scala-2.12.tgz 2020-02-02 20:27 139M

[ ] spark-2.4.5-bin-without-hadoop.tgz 2020-02-02 20:27 160M

[ ] spark-2.4.5.tgz 2020-02-02 20:27 15M

Hi @ELMERWANG

I have the version with Hadoop
"spark-2.4.5-bin-hadoop2.7"

Regards,
Heitor

@hmgomes
Copy link
Collaborator

hmgomes commented Apr 9, 2021

Hi @LYZ1125

Spark version: 2.3.2
Try spark 2.4.5

Regards,
Heitor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhangjiajin zhangjiajin

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hmgomes @zhangjiajin @LYZ1125 @ELMERWANG