Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

YARN-11657. Remove protobuf-2.5 as dependency of hadoop-yarn modules #6575

Merged
merged 3 commits into from
Feb 23, 2024
Merged

YARN-11657. Remove protobuf-2.5 as dependency of hadoop-yarn modules #6575

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
18223bf
YARN-11657. Remove protobuf-2.5 as dependency of hadoop-yarn modules
steveloughran Feb 22, 2024
a726eb4
YARN-11657. update LICENSE-binary
steveloughran Feb 22, 2024
24633bc
YARN-11657. Remove protobuf-2.5 as dependency of hadoop-yarn-api
steveloughran Feb 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions BUILDING.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,12 +315,12 @@ Controlling the redistribution of the protobuf-2.5 dependency

The protobuf 2.5.0 library is used at compile time to compile the class
org.apache.hadoop.ipc.ProtobufHelper; this class known to have been used by
external projects in the past. Protobuf 2.5 is not used elsewhere in
external projects in the past. Protobuf 2.5 is not used directly in
the Hadoop codebase; alongside the move to Protobuf 3.x a private successor
class, org.apache.hadoop.ipc.internal.ShadedProtobufHelper is now used.

The hadoop-common module no longer exports its compile-time dependency on
protobuf-2.5. Hadoop distributions no longer include it.
protobuf-java-2.5.
Any application declaring a dependency on hadoop-commmon will no longer get
the artifact added to their classpath.
If is still required, then they must explicitly declare it:
Expand All @@ -337,10 +337,14 @@ Controlling the redistribution of the protobuf-2.5 dependency

-Dcommon.protobuf2.scope=compile

If this is done then protobuf-2.5.0.jar will again be exported as a
If this is done then protobuf-java-2.5.0.jar will again be exported as a
hadoop-common dependency, and included in the share/hadoop/common/lib/
directory of any Hadoop distribution built.

Note that protobuf-java-2.5.0.jar is still placed in
share/hadoop/yarn/timelineservice/lib; this is needed by the hbase client
library.

----------------------------------------------------------------------------------
Building components separately

Expand Down
3 changes: 2 additions & 1 deletion LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,9 +395,10 @@ hadoop-tools/hadoop-sls/src/main/html/js/thirdparty/d3.v3.js
hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/static/d3-3.5.17.min.js
leveldb v1.13

com.google.protobuf:protobuf-java:2.5.0
com.google.protobuf:protobuf-java:3.6.1
com.google.re2j:re2j:1.1
com.jcraft:jsch:0.1.54
com.jcraft:jsch:0.1.55
com.thoughtworks.paranamer:paranamer:2.3
jakarta.activation:jakarta.activation-api:1.2.1
org.fusesource.leveldbjni:leveldbjni-all:1.8
Expand Down
5 changes: 0 additions & 5 deletions hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,11 +62,6 @@
<artifactId>hadoop-annotations</artifactId>
</dependency>

<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>

<dependency>
<groupId>org.apache.hadoop.thirdparty</groupId>
<artifactId>hadoop-shaded-protobuf_3_21</artifactId>
Expand Down
1 change: 1 addition & 0 deletions ...oject/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<scope>${transient.protobuf2.scope}</scope>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any pointers why are we putting scope in every declarition rather than just putting it up in the parent pom?
c373d3f

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The dependent poms (hadoop-common...) aren't publishing it any more, just using it for compilation.
  2. I don't trust the base declaration to do the right thing.

</dependency>
<dependency>
<groupId>junit</groupId>
Expand Down
4 changes: 0 additions & 4 deletions hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,10 +112,6 @@
<groupId>com.google.inject.extensions</groupId>
<artifactId>guice-servlet</artifactId>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
Expand Down