Skip to content

Commit

Permalink
[SPARK-42422][BUILD] Upgrade maven-shade-plugin to 3.4.1
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?
This pr aims upgrade `maven-shade-plugin` from 3.2.4 to 3.4.1

### Why are the changes needed?
The `maven-shade-plugin` was [built by Java 8](apache/maven-shade-plugin@3327341) from 3.4.1, all other changes as follows:

- https://github.com/apache/maven-shade-plugin/releases/tag/maven-shade-plugin-3.3.0
- apache/maven-shade-plugin@maven-shade-plugin-3.3.0...maven-shade-plugin-3.4.1

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?

- Pass GitHub Actions
- Manual check:
There are 6 modules actually use shade function, checked the maven compilation logs manually:

1. spark-core

Before

```
[INFO] --- maven-shade-plugin:3.2.4:shade (default)  spark-core_2.12 ---
[INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```

After

```
[INFO] --- maven-shade-plugin:3.4.1:shade (default)  spark-core_2.12 ---
[INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.50.v20221201 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```

2.  spark-network-yarn

Before

```
[INFO] --- maven-shade-plugin:3.2.4:shade (default)  spark-network-yarn_2.12 ---
[INFO] Including org.apache.spark:spark-network-shuffle_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.spark:spark-network-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including io.netty:netty-all:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-socks:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-unix-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler-proxy:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-epoll:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar.
[INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar.
[INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar.
[INFO] Including com.google.crypto.tink:tink:jar:1.7.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.8.9 in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:4.2.15 in the shaded jar.
[INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.39 in the shaded jar.
[INFO] Including org.roaringbitmap:shims:jar:0.9.39 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```

After

```
[INFO] --- maven-shade-plugin:3.4.1:shade (default)  spark-network-yarn_2.12 ---
[INFO] Including org.apache.spark:spark-network-shuffle_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.spark:spark-network-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including io.netty:netty-all:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-buffer:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-http2:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-codec-socks:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-unix-common:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-handler-proxy:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-resolver:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-epoll:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final in the shaded jar.
[INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.87.Final in the shaded jar.
[INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar.
[INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar.
[INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar.
[INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar.
[INFO] Including com.google.crypto.tink:tink:jar:1.7.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.8.9 in the shaded jar.
[INFO] Including io.dropwizard.metrics:metrics-core:jar:4.2.15 in the shaded jar.
[INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.39 in the shaded jar.
[INFO] Including org.roaringbitmap:shims:jar:0.9.39 in the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
```

3. spark-protobuf

Before

```
[INFO] --- maven-shade-plugin:3.2.4:shade (default)  spark-protobuf_2.12 ---
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
```

After

```
[INFO] --- maven-shade-plugin:3.4.1:shade (default)  spark-protobuf_2.12 ---
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
```

4. spark-connect-common

Before

```
[INFO] --- maven-shade-plugin:3.2.4:shade (default)  spark-connect-common_2.12 ---
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
```

After

```
[INFO] --- maven-shade-plugin:3.4.1:shade (default)  spark-connect-common_2.12 ---
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
```

5. spark-connect

Before

```
[INFO] --- maven-shade-plugin:3.2.4:shade (default)  spark-connect_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including org.checkerframework:checker-qual:jar:3.12.0 in the shaded jar.
[INFO] Including com.google.errorprone:error_prone_annotations:jar:2.7.1 in the shaded jar.
[INFO] Including com.google.j2objc:j2objc-annotations:jar:1.3 in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.9.0 in the shaded jar.
[INFO] Including com.google.android:annotations:jar:4.1.1.4 in the shaded jar.
[INFO] Including org.codehaus.mojo:animal-sniffer-annotations:jar:1.19 in the shaded jar.
[INFO] Including io.perfmark:perfmark-api:jar:0.25.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.api.grpc:proto-google-common-protos:jar:2.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
```

After

```
[INFO] --- maven-shade-plugin:3.4.1:shade (default)  spark-connect_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
[INFO] Including org.checkerframework:checker-qual:jar:3.12.0 in the shaded jar.
[INFO] Including com.google.errorprone:error_prone_annotations:jar:2.7.1 in the shaded jar.
[INFO] Including com.google.j2objc:j2objc-annotations:jar:1.3 in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.code.gson:gson:jar:2.9.0 in the shaded jar.
[INFO] Including com.google.android:annotations:jar:4.1.1.4 in the shaded jar.
[INFO] Including org.codehaus.mojo:animal-sniffer-annotations:jar:1.19 in the shaded jar.
[INFO] Including io.perfmark:perfmark-api:jar:0.25.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.api.grpc:proto-google-common-protos:jar:2.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
```

6. spark-connect-client-jvm

Before

```
[INFO] --- maven-shade-plugin:3.2.4:shade (default)  spark-connect-client-jvm_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
```

After

```
[INFO] --- maven-shade-plugin:3.4.1:shade (default)  spark-connect-client-jvm_2.12 ---
[INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar.
[INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar.
[INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar.
[INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar.
[INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar.
[INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar.
[INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar.
```

Also checked the  file numbers and file names in jars after decompression,  the result is the same.

Closes apache#39994 from LuciferYang/SPARK-42422.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
(cherry picked from commit 9b625fd)
Signed-off-by: Sean Owen <srowen@gmail.com>
  • Loading branch information
LuciferYang authored and dongjoon-hyun committed Feb 13, 2023
1 parent f8471e8 commit ee29fe1
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3178,7 +3178,7 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<version>3.4.1</version>
<dependencies>
<dependency>
<groupId>org.ow2.asm</groupId>
Expand Down

0 comments on commit ee29fe1

Please # to comment.