Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build errors due to dependencies #189

Open
mpkmtv opened this issue Jun 18, 2020 · 2 comments
Open

build errors due to dependencies #189

mpkmtv opened this issue Jun 18, 2020 · 2 comments

Comments

@mpkmtv
Copy link

mpkmtv commented Jun 18, 2020

I'm new to scala and spark. Got overwhelmed with build errors due to unresolved dependencies like :

[warn] Note: Unresolved dependencies path:
[error] sbt.librarymanagement.ResolveException: Error downloading com.databricks:sbt-databricks;sbtVersion=1.0;scalaVersion=2.12:0.1.3
[error] Not found
[error] Not found
[error] not found: https://repo1.maven.org/maven2/com/databricks/sbt-databricks_2.12_1.0/0.1.3/sbt-databricks-0.1.3.pom
[error] not found: https://dl.bintray.com/spark-packages/maven/com/databricks/sbt-databricks_2.12_1.0/0.1.3/sbt-databricks-0.1.3.pom
[error] not found: https://oss.sonatype.org/content/repositories/releases/com/databricks/sbt-databricks_2.12_1.0/0.1.3/sbt-databricks-0.1.3.pom
[error] not found: https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.databricks/sbt-databricks/scala_2.12/sbt_1.0/0.1.3/ivys/ivy.xml
[error] not found: https://repo.typesafe.com/typesafe/ivy-releases/com.databricks/sbt-databricks/scala_2.12/sbt_1.0/0.1.3/ivys/ivy.xml
[error] Error downloading com.jsuereth:sbt-pgp;sbtVersion=1.0;scalaVersion=2.12:1.0.0
[`

Any help ?

@hitachi-ben
Copy link

@mpkmtv, did you ever figure it out? Having the same issue.

@wangyum
Copy link
Contributor

wangyum commented Nov 1, 2024

You can use pre-build jars.
spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar.tgz

tar -zxf spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar.tgz

java --add-opens=java.base/sun.nio.ch=ALL-UNNAMED -Djava.io.tmpdir=/hadoop1/tmp,/hadoop2/tmp -Dspark.local.dir=/hadoop1/tmp,/hadoop2/tmp -cp /home/hadoop/spark-3.5.3/spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar:/home/hadoop/spark-3.5.3/spark-3.5.3-bin-hadoop3/jars/*:/home/hadoop/spark-3.5.3/scopt_2.12-4.0.1.jar  com.databricks.spark.sql.perf.tpcds.GenTPCDSData --dsdgenDir /home/hadoop/spark-3.5.3/tpcds-kit/tools -s 5000 -l hdfs://myhdfs:9000/user/hive/warehouse/tpcds5t -f parquet

After generating the data, you can run benchmark through:

git clone https://github.com/wangyum/spark.git -b run-benchmark && cd spark
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location hdfs://myhdfs:9000/user/hive/warehouse/tpcds5t --cbo"

wangyum added a commit to wangyum/spark that referenced this issue Nov 1, 2024
…perf.

Run command:
```
SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location hdfs://myhdfs:9000/user/hive/warehouse/tpcds5t --cbo"
```

Please see more details: databricks/spark-sql-perf#189
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants