HADOOP-19120. ApacheHttpClient adaptation in ABFS.

[saxenapranav]

JIRA: https://issues.apache.org/jira/browse/HADOOP-19120

trunk pr: #6633

Apache httpclient 4.5.x is the new default implementation of http connections;
this supports a large configurable pool of connections along with
the ability to limit their lifespan.

The networking library can be chosen using the configuration
option fs.azure.networking.library

The supported values are

• APACHE_HTTP_CLIENT : Use Apache HttpClient [Default]
• JDK_HTTP_URL_CONNECTION : Use JDK networking library

[saxenapranav]

---

:::: AGGREGATED TEST RESULT ::::

============================================================
HNS-OAuth

[WARNING] Tests run: 153, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 644, Failures: 0, Errors: 0, Skipped: 82
[WARNING] Tests run: 424, Failures: 0, Errors: 0, Skipped: 57

============================================================
HNS-SharedKey

[WARNING] Tests run: 153, Failures: 0, Errors: 0, Skipped: 3
[WARNING] Tests run: 644, Failures: 0, Errors: 0, Skipped: 34
[WARNING] Tests run: 424, Failures: 0, Errors: 0, Skipped: 44

============================================================
NonHNS-SharedKey

[WARNING] Tests run: 153, Failures: 0, Errors: 0, Skipped: 9
[WARNING] Tests run: 628, Failures: 0, Errors: 0, Skipped: 274
[WARNING] Tests run: 424, Failures: 0, Errors: 0, Skipped: 47

============================================================
AppendBlob-HNS-OAuth

[WARNING] Tests run: 153, Failures: 0, Errors: 0, Skipped: 2
[WARNING] Tests run: 644, Failures: 0, Errors: 0, Skipped: 84
[WARNING] Tests run: 424, Failures: 0, Errors: 0, Skipped: 81

Time taken: 28 mins 30 secs.
azureuser@pranav-ind-vm:~/hadoop/hadoop-tools/hadoop-azure$ git log
commit 99c6095 (HEAD -> saxenapranav/abfs-apachehttpclient-3.4, origin/saxenapranav/abfs-apachehttpclient-3.4)
Author: Pranav Saxena <>
Date: Tue Jul 23 21:42:37 2024 -0700

cherrypick of b60497ff41e1dc149d1610f4cc6ea4e0609f9946 : https://github.com/apache/hadoop/commit/b60497ff41e1dc149d1610f4cc6ea4e0609f9946 :  ApacheHttpClient adaptation in ABFS.
https://github.com/apache/hadoop/pull/6633

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	17m 47s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 21 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	14m 33s		Maven dependency ordering for branch
+1 💚	mvninstall	35m 42s		branch-3.4 passed
+1 💚	compile	19m 39s		branch-3.4 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	18m 12s		branch-3.4 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	4m 43s		branch-3.4 passed
+1 💚	mvnsite	2m 31s		branch-3.4 passed
+1 💚	javadoc	2m 0s		branch-3.4 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 34s		branch-3.4 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 44s		branch-3.4 passed
+1 💚	shadedclient	39m 28s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 23s		the patch passed
+1 💚	compile	18m 38s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	18m 38s		the patch passed
+1 💚	compile	18m 17s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	18m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 40s	/results-checkstyle-root.txt	root: The patch generated 4 new + 18 unchanged - 0 fixed = 22 total (was 18)
+1 💚	mvnsite	2m 29s		the patch passed
+1 💚	javadoc	1m 53s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 35s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	4m 6s		the patch passed
+1 💚	shadedclient	39m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 1s		hadoop-common in the patch passed.
+1 💚	unit	2m 33s		hadoop-azure in the patch passed.
+1 💚	asflicense	1m 1s		The patch does not generate ASF License warnings.
		280m 56s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/1/artifact/out/Dockerfile
GITHUB PR	#6959
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux d1f7dd27beb9 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 99c6095
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/1/testReport/
Max. process+thread count	1258 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 53s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 21 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	14m 7s		Maven dependency ordering for branch
+1 💚	mvninstall	34m 48s		branch-3.4 passed
+1 💚	compile	26m 39s		branch-3.4 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	23m 40s		branch-3.4 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	4m 54s		branch-3.4 passed
+1 💚	mvnsite	2m 35s		branch-3.4 passed
+1 💚	javadoc	1m 58s		branch-3.4 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 27s		branch-3.4 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 59s		branch-3.4 passed
+1 💚	shadedclient	37m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 33s		the patch passed
+1 💚	compile	24m 29s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	24m 29s		the patch passed
+1 💚	compile	22m 13s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	22m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 49s	/results-checkstyle-root.txt	root: The patch generated 2 new + 18 unchanged - 0 fixed = 20 total (was 18)
+1 💚	mvnsite	2m 30s		the patch passed
+1 💚	javadoc	1m 53s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 28s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	4m 33s		the patch passed
+1 💚	shadedclient	38m 1s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 44s		hadoop-common in the patch passed.
+1 💚	unit	2m 35s		hadoop-azure in the patch passed.
+1 💚	asflicense	1m 0s		The patch does not generate ASF License warnings.
		282m 8s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/3/artifact/out/Dockerfile
GITHUB PR	#6959
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux ac10b97b1a59 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 48ba0ad
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/3/testReport/
Max. process+thread count	1260 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	1m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	markdownlint	0m 1s		markdownlint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 21 new or modified test files.
			_ branch-3.4 Compile Tests _
+0 🆗	mvndep	14m 12s		Maven dependency ordering for branch
+1 💚	mvninstall	34m 25s		branch-3.4 passed
+1 💚	compile	27m 11s		branch-3.4 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	compile	21m 35s		branch-3.4 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	checkstyle	5m 8s		branch-3.4 passed
+1 💚	mvnsite	2m 27s		branch-3.4 passed
+1 💚	javadoc	2m 0s		branch-3.4 passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 27s		branch-3.4 passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	3m 59s		branch-3.4 passed
+1 💚	shadedclient	38m 35s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for patch
+1 💚	mvninstall	1m 27s		the patch passed
+1 💚	compile	24m 49s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javac	24m 49s		the patch passed
+1 💚	compile	22m 57s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	javac	22m 57s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	4m 58s	/results-checkstyle-root.txt	root: The patch generated 2 new + 18 unchanged - 0 fixed = 20 total (was 18)
+1 💚	mvnsite	2m 31s		the patch passed
+1 💚	javadoc	1m 54s		the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
+1 💚	spotbugs	4m 34s		the patch passed
+1 💚	shadedclient	38m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	20m 51s		hadoop-common in the patch passed.
+1 💚	unit	2m 33s		hadoop-azure in the patch passed.
+1 💚	asflicense	0m 58s		The patch does not generate ASF License warnings.
		284m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/2/artifact/out/Dockerfile
GITHUB PR	#6959
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
uname	Linux a5589c6342ac 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	branch-3.4 / 48ba0ad
Default Java	Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/2/testReport/
Max. process+thread count	1258 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-azure U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6959/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

[saxenapranav]

Hi @steveloughran , This is backport of trunk PR #6633. Requesting your kind review please. Thank you very much!

[steveloughran]

thanks. just testing the trunk code locally, to see if there are any problems first

[steveloughran]

LGTM
+1

I did see lots of failures on trunk, but that seems to be because my account has been deleted. oops.