HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

HarshitGupta11 · 2024-02-21T13:54:29Z

Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module.

How was this patch tested?

Integration Test using mvn clean verify -Dparallel-tests -DtestsThreadCount=16 -Dscale -fae

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

steveloughran · 2024-02-21T14:15:24Z

you will need to update LICENSE-binary
and run the hadoop-aws tests -where?
and run the "qualifying an SDK update" steps -where and what problems surfaced?

Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module.

hadoop-yetus · 2024-02-21T15:58:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	xmllint	0m 1s		xmllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 50s		trunk passed
+1 💚	compile	0m 21s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	0m 21s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	0m 26s		trunk passed
+1 💚	javadoc	0m 26s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 23s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	80m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 14s		the patch passed
+1 💚	compile	0m 13s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	0m 13s		the patch passed
+1 💚	compile	0m 13s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	0m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	0m 14s		the patch passed
+1 💚	javadoc	0m 13s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	0m 13s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	36m 6s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	0m 17s		hadoop-project in the patch passed.
+1 💚	asflicense	0m 34s		The patch does not generate ASF License warnings.
		122m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/artifact/out/Dockerfile
GITHUB PR	#6568
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint
uname	Linux 187f99e9288d 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `6bde82e`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/testReport/
Max. process+thread count	702 (vs. ulimit of 5500)
modules	C: hadoop-project U: hadoop-project
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/1/console
versions	git=2.25.1 maven=3.6.3
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2024-02-22T17:03:53Z

+1 pending you state which s3 endpoint you ran the tests and what the parameters to your test run were.

ahmarsuhail · 2024-02-22T19:08:02Z

@HarshitGupta11 could you upgrade to the latest version, or at least 2.24.6 instead? We found an issue with the S3 async client not implementing abort which is fixed in 2.24.6. This doesn't really affect us currently, but if someone is using Client side encryption, then it might cause issues. Related to #6164

steveloughran · 2024-02-23T15:55:51Z

@HarshitGupta11 need to know where you qualified the SDK...

hadoop-yetus · 2024-02-24T06:25:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	xmllint	0m 0s		xmllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 1s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 42s		Maven dependency ordering for branch
+1 💚	mvninstall	31m 41s		trunk passed
+1 💚	compile	17m 4s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	compile	15m 42s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	mvnsite	20m 45s		trunk passed
+1 💚	javadoc	8m 37s		trunk passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 42s		trunk passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	48m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	1m 7s		Maven dependency ordering for patch
+1 💚	mvninstall	29m 25s		the patch passed
+1 💚	compile	16m 39s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javac	16m 39s		the patch passed
+1 💚	compile	15m 42s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	javac	15m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	15m 29s		the patch passed
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	javadoc	8m 26s		the patch passed with JDK Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
+1 💚	javadoc	7m 43s		the patch passed with JDK Private Build-1.8.0_392-8u392-ga-1~20.04-b08
+1 💚	shadedclient	49m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	733m 15s	/patch-unit-root.txt	root in the patch failed.
+1 💚	asflicense	1m 38s		The patch does not generate ASF License warnings.
		1018m 4s

Reason	Tests
Failed junit tests	hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
	hadoop.hdfs.server.datanode.TestDirectoryScanner
	hadoop.hdfs.protocol.TestBlockListAsLongs

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/3/artifact/out/Dockerfile
GITHUB PR	#6568
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs
uname	Linux 8ba8950c227c 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `752b1f3`
Default Java	Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/3/testReport/
Max. process+thread count	4275 (vs. ulimit of 5500)
modules	C: hadoop-project . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6568/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

HarshitGupta11 · 2024-02-26T07:44:28Z

@HarshitGupta11 could you upgrade to the latest version, or at least 2.24.6 instead? We found an issue with the S3 async client not implementing abort which is fixed in 2.24.6. This doesn't really affect us currently, but if someone is using Client side encryption, then it might cause issues. Related to #6164

Hi yeah, I have updated the ticket and ran the hadoop test suite again a normal and directory bucket in us-west-2

HarshitGupta11 · 2024-02-26T07:49:26Z

@HarshitGupta11 need to know where you qualified the SDK...

Still running all the tests as mentioned, I have run the test suite under normal conditions in both kind of buckets but there are a couple of changes needed with CSE-KMS and access point buckets, trying to figure those out.

steveloughran · 2024-02-26T16:09:33Z

ok, the CSE stuff is going to fail, don't worry

HarshitGupta11 · 2024-02-29T08:36:44Z

Ran all the tests as per the Qualifying sdk update, all seems to be fine.

steveloughran

+1

welcome to the "qualifying an aws release" work

Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging w.r.t. IMDS module. Contributed by Harshit Gupta

### What changes were proposed in this pull request? This PR aims to upgrade AWS SDK v2 to 2.24.6. ### Why are the changes needed? Like HADOOP-19082 (Apache Hadoop 3.4.1), Apache Spark 4.0.0 had better use the latest one. - apache/hadoop#6568 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45821 from dongjoon-hyun/SPARK-47695. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

github-actions bot added build trunk labels Feb 21, 2024

HADOOP-19082: Update AWS SDK V2 to 2.24.1

cf11a93

Update the AWS SDK to 2.24.1 from 2.23.5 for latest updates in packaging w.r.t. imds module.

HarshitGupta11 force-pushed the HADOOP-19082 branch from 6bde82e to cf11a93 Compare February 21, 2024 14:17

Updated to 2.24.6

752b1f3

steveloughran approved these changes Mar 5, 2024

View reviewed changes

steveloughran changed the title ~~HADOOP-19082: Update AWS SDK V2 to 2.24.1~~ HADOOP-19082: Update AWS SDK V2 to 2.24.6 Mar 5, 2024

steveloughran merged commit d974a12 into apache:trunk Mar 5, 2024

asfgit pushed a commit that referenced this pull request Mar 5, 2024

HADOOP-19082: S3A: Update AWS SDK V2 to 2.24.6 (#6568)

397c10a

Update the AWS SDK to 2.24.6 from 2.23.5 for latest updates in packaging w.r.t. IMDS module. Contributed by Harshit Gupta

dongjoon-hyun mentioned this pull request Apr 2, 2024

[SPARK-47695][BUILD] Upgrade AWS SDK v2 to 2.24.6 apache/spark#45821

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

Uh oh!

HarshitGupta11 commented Feb 21, 2024

Uh oh!

steveloughran commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

steveloughran commented Feb 22, 2024

Uh oh!

ahmarsuhail commented Feb 22, 2024 •

edited

Loading

Uh oh!

steveloughran commented Feb 23, 2024

Uh oh!

hadoop-yetus commented Feb 24, 2024

Uh oh!

HarshitGupta11 commented Feb 26, 2024

Uh oh!

HarshitGupta11 commented Feb 26, 2024

Uh oh!

steveloughran commented Feb 26, 2024

Uh oh!

HarshitGupta11 commented Feb 29, 2024

Uh oh!

steveloughran left a comment

Uh oh!

Uh oh!

HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

HADOOP-19082: Update AWS SDK V2 to 2.24.6 #6568

Uh oh!

Conversation

HarshitGupta11 commented Feb 21, 2024

How was this patch tested?

For code changes:

Uh oh!

steveloughran commented Feb 21, 2024

Uh oh!

hadoop-yetus commented Feb 21, 2024

Uh oh!

steveloughran commented Feb 22, 2024

Uh oh!

ahmarsuhail commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Feb 23, 2024

Uh oh!

hadoop-yetus commented Feb 24, 2024

Uh oh!

HarshitGupta11 commented Feb 26, 2024

Uh oh!

HarshitGupta11 commented Feb 26, 2024

Uh oh!

steveloughran commented Feb 26, 2024

Uh oh!

HarshitGupta11 commented Feb 29, 2024

Uh oh!

steveloughran left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ahmarsuhail commented Feb 22, 2024 •

edited

Loading