Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-42422][BUILD] Upgrade
maven-shade-plugin
to 3.4.1
### What changes were proposed in this pull request? This pr aims upgrade `maven-shade-plugin` from 3.2.4 to 3.4.1 ### Why are the changes needed? The `maven-shade-plugin` was [built by Java 8](apache/maven-shade-plugin@3327341) from 3.4.1, all other changes as follows: - https://github.com/apache/maven-shade-plugin/releases/tag/maven-shade-plugin-3.3.0 - apache/maven-shade-plugin@maven-shade-plugin-3.3.0...maven-shade-plugin-3.4.1 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manual check: There are 6 modules actually use shade function, checked the maven compilation logs manually: 1. spark-core Before ``` [INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-core_2.12 --- [INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. ``` After ``` [INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-core_2.12 --- [INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.50.v20221201 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. ``` 2. spark-network-yarn Before ``` [INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-network-yarn_2.12 --- [INFO] Including org.apache.spark:spark-network-shuffle_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including org.apache.spark:spark-network-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including io.netty:netty-all:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-buffer:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec-http:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec-http2:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec-socks:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-common:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-handler:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-unix-common:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-handler-proxy:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-resolver:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-classes-epoll:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.87.Final in the shaded jar. [INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar. [INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar. [INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar. [INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar. [INFO] Including com.google.crypto.tink:tink:jar:1.7.0 in the shaded jar. [INFO] Including com.google.code.gson:gson:jar:2.8.9 in the shaded jar. [INFO] Including io.dropwizard.metrics:metrics-core:jar:4.2.15 in the shaded jar. [INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.39 in the shaded jar. [INFO] Including org.roaringbitmap:shims:jar:0.9.39 in the shaded jar. [INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. ``` After ``` [INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-network-yarn_2.12 --- [INFO] Including org.apache.spark:spark-network-shuffle_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including org.apache.spark:spark-network-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including io.netty:netty-all:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-buffer:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec-http:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec-http2:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-codec-socks:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-common:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-handler:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-unix-common:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-handler-proxy:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-resolver:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-classes-epoll:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-classes-kqueue:jar:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.87.Final in the shaded jar. [INFO] Including io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.87.Final in the shaded jar. [INFO] Including org.apache.commons:commons-lang3:jar:3.12.0 in the shaded jar. [INFO] Including org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 in the shaded jar. [INFO] Including org.rocksdb:rocksdbjni:jar:7.9.2 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar. [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar. [INFO] Including org.apache.commons:commons-crypto:jar:1.1.0 in the shaded jar. [INFO] Including com.google.crypto.tink:tink:jar:1.7.0 in the shaded jar. [INFO] Including com.google.code.gson:gson:jar:2.8.9 in the shaded jar. [INFO] Including io.dropwizard.metrics:metrics-core:jar:4.2.15 in the shaded jar. [INFO] Including org.roaringbitmap:RoaringBitmap:jar:0.9.39 in the shaded jar. [INFO] Including org.roaringbitmap:shims:jar:0.9.39 in the shaded jar. [INFO] Including com.google.code.findbugs:jsr305:jar:3.0.0 in the shaded jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. ``` 3. spark-protobuf Before ``` [INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-protobuf_2.12 --- [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. ``` After ``` [INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-protobuf_2.12 --- [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. ``` 4. spark-connect-common Before ``` [INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-connect-common_2.12 --- [INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar. [INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar. [INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar. ``` After ``` [INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-connect-common_2.12 --- [INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar. [INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar. [INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar. ``` 5. spark-connect Before ``` [INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-connect_2.12 --- [INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar. [INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar. [INFO] Including org.checkerframework:checker-qual:jar:3.12.0 in the shaded jar. [INFO] Including com.google.errorprone:error_prone_annotations:jar:2.7.1 in the shaded jar. [INFO] Including com.google.j2objc:j2objc-annotations:jar:1.3 in the shaded jar. [INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. [INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar. [INFO] Including com.google.code.gson:gson:jar:2.9.0 in the shaded jar. [INFO] Including com.google.android:annotations:jar:4.1.1.4 in the shaded jar. [INFO] Including org.codehaus.mojo:animal-sniffer-annotations:jar:1.19 in the shaded jar. [INFO] Including io.perfmark:perfmark-api:jar:0.25.0 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar. [INFO] Including com.google.api.grpc:proto-google-common-protos:jar:2.0.1 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar. [INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar. ``` After ``` [INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-connect_2.12 --- [INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar. [INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar. [INFO] Including org.checkerframework:checker-qual:jar:3.12.0 in the shaded jar. [INFO] Including com.google.errorprone:error_prone_annotations:jar:2.7.1 in the shaded jar. [INFO] Including com.google.j2objc:j2objc-annotations:jar:1.3 in the shaded jar. [INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. [INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar. [INFO] Including com.google.code.gson:gson:jar:2.9.0 in the shaded jar. [INFO] Including com.google.android:annotations:jar:4.1.1.4 in the shaded jar. [INFO] Including org.codehaus.mojo:animal-sniffer-annotations:jar:1.19 in the shaded jar. [INFO] Including io.perfmark:perfmark-api:jar:0.25.0 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar. [INFO] Including com.google.api.grpc:proto-google-common-protos:jar:2.0.1 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar. [INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar. ``` 6. spark-connect-client-jvm Before ``` [INFO] --- maven-shade-plugin:3.2.4:shade (default) spark-connect-client-jvm_2.12 --- [INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar. [INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar. [INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. [INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar. [INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar. ``` After ``` [INFO] --- maven-shade-plugin:3.4.1:shade (default) spark-connect-client-jvm_2.12 --- [INFO] Including org.apache.spark:spark-connect-common_2.12:jar:3.5.0-SNAPSHOT in the shaded jar. [INFO] Including com.google.guava:failureaccess:jar:1.0.1 in the shaded jar. [INFO] Including io.grpc:grpc-netty:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-core:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-api:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-context:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-protobuf-lite:jar:1.47.0 in the shaded jar. [INFO] Including io.grpc:grpc-services:jar:1.47.0 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java-util:jar:3.19.2 in the shaded jar. [INFO] Including io.grpc:grpc-stub:jar:1.47.0 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.12 in the shaded jar. [INFO] Including com.google.guava:guava:jar:31.0.1-jre in the shaded jar. [INFO] Including com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava in the shaded jar. ``` Also checked the file numbers and file names in jars after decompression, the result is the same. Closes apache#39994 from LuciferYang/SPARK-42422. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Sean Owen <srowen@gmail.com> (cherry picked from commit 9b625fd) Signed-off-by: Sean Owen <srowen@gmail.com>
- Loading branch information