Skip to content

Commit

Permalink
added possibility to run spark shell directly out of the project dire…
Browse files Browse the repository at this point in the history
…ctory
  • Loading branch information
matthiasbertsch committed Jan 12, 2018
1 parent 56c498e commit 83bf607
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 0 deletions.
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@ mvn clean package

Building DESQ with `mvn clean package -Pprovided` builds the `desq-<branch>-<releasedate>-no-spark.jar` with the excluded Apache Spark dependency.

To start a Spark shell directly out of the project directory, use

```
mvn compile && mvn exec:exec -Dexec.executable="java" -Dexec.args="-cp \"%classpath\" -Dscala.usejavacp=true org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main spark-shell"
```

## DESQ Examples

DESQ supports a variety of pattern mining tasks, some of which are highlighted
Expand Down
21 changes: 21 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,16 @@
<artifactId>maven-shade-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<relocations>
<relocation>
<pattern>com.google.common</pattern>
Expand Down Expand Up @@ -297,6 +307,11 @@
<artifactId>spark-core_2.11</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_2.11</artifactId>
<version>2.1.0</version>
</dependency>
</dependencies>
<properties>
<suffix>full</suffix>
Expand All @@ -313,6 +328,12 @@
<version>2.1.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-repl_2.11</artifactId>
<version>2.1.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
<properties>
<suffix>no-spark</suffix>
Expand Down

0 comments on commit 83bf607

Please sign in to comment.