Skip to content

Commit

Permalink
[SPARK-43104][BUILD] Set shadeTestJar of protobuf module to false
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This pr aims to set `shadeTestJar` of `protobuf` module to `false` to skip shade `spark-protobuf_**-tests.jar` process.

### Why are the changes needed?

When `shadeTestJar` is true,  `maven-shade-plugin` always try to find(sometime is downloading) some non-existent jars:

```
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-client-api/3.3.5/hadoop-client-api-3.3.5-tests.jar
[WARNING] Could not get tests for org.apache.hadoop:hadoop-client-api:jar:3.3.5:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/tukaani/xz/1.9/xz-1.9-tests.jar
[WARNING] Could not get tests for org.tukaani:xz:jar:1.9:compile
Downloading from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/slf4j/slf4j-api/2.0.7/slf4j-api-2.0.7-tests.jar
Downloaded from gcs-maven-central-mirror: https://maven-central.storage-download.googleapis.com/maven2/org/slf4j/slf4j-api/2.0.7/slf4j-api-2.0.7-tests.jar (0 B at 0 B/s)
[WARNING] Could not get tests for com.google.protobuf:protobuf-java:jar:3.22.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/curator/curator-framework/2.13.0/curator-framework-2.13.0-tests.jar
[WARNING] Could not get tests for org.apache.curator:curator-framework:jar:2.13.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-slf4j2-impl/2.20.0/log4j-slf4j2-impl-2.20.0-tests.jar
[WARNING] Could not get tests for org.apache.logging.log4j:log4j-slf4j2-impl:jar:2.20.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/jetbrains/annotations/17.0.0/annotations-17.0.0-tests.jar
[WARNING] Could not get tests for org.jetbrains:annotations:jar:17.0.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-1.2-api/2.20.0/log4j-1.2-api-2.20.0-tests.jar
[WARNING] Could not get tests for org.apache.logging.log4j:log4j-1.2-api:jar:2.20.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/threeten/threeten-extra/1.7.1/threeten-extra-1.7.1-tests.jar
[WARNING] Could not get tests for org.threeten:threeten-extra:jar:1.7.1:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/yetus/audience-annotations/0.13.0/audience-annotations-0.13.0-tests.jar
[WARNING] Could not get tests for org.apache.yetus:audience-annotations:jar:0.13.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.14.2/jackson-databind-2.14.2-tests.jar
[WARNING] Could not get tests for com.fasterxml.jackson.core:jackson-databind:jar:2.14.2:compile
Downloading from central: https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0-tests.jar
[WARNING] Could not get tests for com.google.code.findbugs:jsr305:jar:3.0.0:runtime
Downloading from central: https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.9.1/snappy-java-1.1.9.1-tests.jar
[WARNING] Could not get tests for org.xerial.snappy:snappy-java:jar:1.1.9.1:compile
Downloading from central: https://repo.maven.apache.org/maven2/javax/activation/activation/1.1.1/activation-1.1.1-tests.jar
[WARNING] Could not get tests for javax.activation:activation:jar:1.1.1:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.8.1/hive-storage-api-2.8.1-tests.jar
[WARNING] Could not get tests for org.apache.hive:hive-storage-api:jar:2.8.1:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/spark-project/spark/unused/1.0.0/unused-1.0.0-tests.jar
[WARNING] Could not get tests for org.spark-project.spark:unused:jar:1.0.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/2.12.17/scala-library-2.12.17-tests.jar
[WARNING] Could not get tests for org.scala-lang:scala-library:jar:2.12.17:compile
Downloading from central: https://repo.maven.apache.org/maven2/com/github/luben/zstd-jni/1.5.4-2/zstd-jni-1.5.4-2-tests.jar
[WARNING] Could not get tests for com.github.luben:zstd-jni:jar:1.5.4-2:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-core/2.20.0/log4j-core-2.20.0-tests.jar
[WARNING] Could not get tests for org.apache.logging.log4j:log4j-core:jar:2.20.0:runtime
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/curator/curator-recipes/2.13.0/curator-recipes-2.13.0-tests.jar
[WARNING] Could not get tests for org.apache.curator:curator-recipes:jar:2.13.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.14.2/jackson-annotations-2.14.2-tests.jar
[WARNING] Could not get tests for com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-client-runtime/3.3.5/hadoop-client-runtime-3.3.5-tests.jar
[WARNING] Could not get tests for org.apache.hadoop:hadoop-client-runtime:jar:3.3.5:compile
Downloading from central: https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-core/2.14.2/jackson-core-2.14.2-tests.jar
[WARNING] Could not get tests for com.fasterxml.jackson.core:jackson-core:jar:2.14.2:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/logging/log4j/log4j-api/2.20.0/log4j-api-2.20.0-tests.jar
[WARNING] Could not get tests for org.apache.logging.log4j:log4j-api:jar:2.20.0:compile
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/curator/curator-client/2.13.0/curator-client-2.13.0-tests.jar
[WARNING] Could not get tests for org.apache.curator:curator-client:jar:2.13.0:compile
```

But the shaded `spark-protobuf_**-tests.jar` was not used now and sbt assembly didn't do this.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Actions
- Check ` build/mvn clean install -pl connector/protobuf` and all test passed

Closes apache#40753 from LuciferYang/protobuf-shadeTestJar-false.

Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Chao Sun <[email protected]>
  • Loading branch information
LuciferYang authored and sunchao committed Apr 14, 2023
1 parent 4c938d6 commit 70094da
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion connector/protobuf/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@
<artifactId>maven-shade-plugin</artifactId>
<configuration>
<shadedArtifactAttached>false</shadedArtifactAttached>
<shadeTestJar>true</shadeTestJar>
<shadeTestJar>false</shadeTestJar>
<artifactSet>
<includes>
<include>com.google.protobuf:*</include>
Expand Down

0 comments on commit 70094da

Please sign in to comment.