HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks. #7414

LiuGuH · 2025-02-20T12:45:55Z

Description of PR

JIRA: HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks.

Now when a block commit , it should add expectedStorages into PendingReconstructionBlocks if datanode ibr is not sent from expected locations. And for a ec block group , only blkStriped.getRealTotalBlockNum() == expectedStorages.length will be added into PendingReconstructionBlocks.

The problem is that when an ec file size <= ecPolicy.getCellSize *(ecPolicy.getNumDataUnits()-1), it always will be blkStriped.getRealTotalBlockNum() < expectedStorages.length. This will lead to namenode generation reconstruction task first if other ibrs comme later.

For example:

(1) A EC file with XOR-2-1-1024k，file size <=1024k
(2) The file has one ec block and 3 replics (blk0,blk1,blk2) , and only blk0,blk2 has data because of file size <=1024k
(3) Namenode recived IBR0
(4) Client close file, now the file reach MinStorage and ec block commit
(5) Now blkStriped.getRealTotalBlockNum()=1 expectedStorages.length=3 ,so blk1,blk2 cannot add into PendingReconstructionBlocks
(6) Ec block complete and file close, because it has missing block so add into neededReconstruction
(7) Namenode generation ec reconstruction task and deliver to the datanode DN2
(8) Namenode recived IBR2 (DN2 blk2)
(9) Namenode recived new block (DN2 ec reconstruction task blk2')
(10) For ecPolicy, DN2 has two block blk2 ,blk2' , this increases the risk of data loss
(11) For other ecPolicy, on a same datanode may has two different replicas on one ec block group , and namenode will only record the later replica.

So the right way is:
(1) addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks
(2) In order to reduce PendingReconstructionBlocks timeout, when a file close or ibr commes, we should remove block which has enough replicas from PendingReconstructionBlocks

hadoop-yetus · 2025-02-20T14:58:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	34m 57s		trunk passed
+1 💚	compile	1m 19s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 11s		trunk passed
+1 💚	mvnsite	1m 18s		trunk passed
+1 💚	javadoc	1m 9s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 8s		trunk passed
+1 💚	shadedclient	35m 59s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 6s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 13s		the patch passed
+1 💚	compile	1m 1s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 1s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 58s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 13 new + 136 unchanged - 0 fixed = 149 total (was 136)
+1 💚	mvnsite	1m 10s		the patch passed
+1 💚	javadoc	0m 56s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 30s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 7s		the patch passed
+1 💚	shadedclient	37m 19s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 33s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		131m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/1/artifact/out/Dockerfile
GITHUB PR	#7414
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux cd9e02dea34a 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `b21d58f`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/1/testReport/
Max. process+thread count	721 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2025-02-20T22:54:51Z

@zhangshuyan0 Could you please review this PR? Thank you very much!

hadoop-yetus · 2025-02-24T15:09:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	39m 53s		trunk passed
+1 💚	compile	1m 21s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 10s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 14s		trunk passed
+1 💚	mvnsite	1m 20s		trunk passed
+1 💚	javadoc	1m 14s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 38s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 14s		trunk passed
+1 💚	shadedclient	41m 6s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 11s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 22s		the patch passed
+1 💚	compile	1m 9s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 9s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 3s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 136 unchanged - 0 fixed = 144 total (was 136)
+1 💚	mvnsite	1m 13s		the patch passed
+1 💚	javadoc	1m 3s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 34s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 13s		the patch passed
+1 💚	shadedclient	41m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 34s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		146m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/2/artifact/out/Dockerfile
GITHUB PR	#7414
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux a3b9c298ab9a 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a2ed6a6`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/2/testReport/
Max. process+thread count	572 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-02-25T06:15:56Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 51s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	43m 41s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 12s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 12s		trunk passed
+1 💚	mvnsite	1m 21s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 16s		trunk passed
+1 💚	shadedclient	40m 46s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 8s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 15s		the patch passed
+1 💚	compile	1m 3s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 12s		the patch passed
+1 💚	javadoc	1m 1s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 11s		the patch passed
+1 💚	shadedclient	40m 48s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 34s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		149m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/3/artifact/out/Dockerfile
GITHUB PR	#7414
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 0efda37b9691 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3168657`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/3/testReport/
Max. process+thread count	537 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-02-26T09:52:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 50s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 59s		trunk passed
+1 💚	compile	1m 23s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 13s		trunk passed
+1 💚	mvnsite	1m 18s		trunk passed
+1 💚	javadoc	1m 13s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 42s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 11s		trunk passed
+1 💚	shadedclient	41m 5s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 7s		the patch passed
+1 💚	compile	1m 15s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 15s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 4s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 2s		the patch passed
+1 💚	mvnsite	1m 9s		the patch passed
+1 💚	javadoc	1m 1s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 11s		the patch passed
+1 💚	shadedclient	40m 59s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 34s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		146m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/4/artifact/out/Dockerfile
GITHUB PR	#7414
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 33a64cef7303 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `747576c`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/4/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

…Storages in PendingReconstructionBlocks

hadoop-yetus · 2025-02-28T08:46:32Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	42m 46s		trunk passed
+1 💚	compile	1m 20s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 11s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 13s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
-1 ❌	javadoc	1m 14s	/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚	javadoc	1m 41s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 13s		trunk passed
+1 💚	shadedclient	41m 22s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 8s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 4s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 4s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 11s		the patch passed
-1 ❌	javadoc	1m 1s	/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.txt	hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04.
+1 💚	javadoc	1m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 11s		the patch passed
+1 💚	shadedclient	40m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 36s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		148m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/5/artifact/out/Dockerfile
GITHUB PR	#7414
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 85efbca13939 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `85b6dec`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/5/testReport/
Max. process+thread count	530 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

LiuGuH · 2025-03-04T07:40:36Z

@Hexiaoqiao @ZanderXu @tomscut , Hello Sir , please reivew this PR if you have time . Thanks very much

slfan1989 · 2025-03-07T07:11:10Z

@Hexiaoqiao @ZanderXu @tomscut , Hello Sir , please reivew this PR if you have time . Thanks very much

@LiuGuH Thanks for the contribution! I'm not good at EC, but I asked for help from other members.

hadoop-yetus · 2025-03-07T08:33:10Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 55s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 9s		trunk passed
+1 💚	compile	1m 26s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 12s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	1m 13s		trunk passed
+1 💚	mvnsite	1m 19s		trunk passed
+1 💚	javadoc	1m 14s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 39s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 15s		trunk passed
+1 💚	shadedclient	41m 11s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	1m 7s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 13s		the patch passed
+1 💚	compile	1m 3s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 3s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	1m 1s		the patch passed
+1 💚	mvnsite	1m 10s		the patch passed
+1 💚	javadoc	1m 1s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 31s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	3m 12s		the patch passed
+1 💚	shadedclient	41m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	1m 37s		hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		150m 37s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/6/artifact/out/Dockerfile
GITHUB PR	#7414
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux bf0c0f849f1b 5.15.0-131-generic #141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fcd8fad`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/6/testReport/
Max. process+thread count	578 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7414/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

LiuGuH · 2025-03-07T10:39:40Z

@LiuGuH Thanks for the contribution! I'm not good at EC, but I asked for help from other members.

Thanks

github-actions bot added HDFS trunk labels Feb 20, 2025

slfan1989 changed the title ~~HDFS-17732. addExpectedReplicasToPending should add EC block expected…~~ HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks. Feb 20, 2025

LiuGuH force-pushed the HDFS-17732 branch from a2ed6a6 to 3168657 Compare February 25, 2025 03:45

LiuGuH force-pushed the HDFS-17732 branch from 3168657 to 747576c Compare February 26, 2025 07:24

HDFS-17732. addExpectedReplicasToPending should add EC block expected…

85b6dec

…Storages in PendingReconstructionBlocks

LiuGuH force-pushed the HDFS-17732 branch from 747576c to 85b6dec Compare February 28, 2025 06:16

triggle compile

fcd8fad

slfan1989 requested review from Hexiaoqiao, haiyang1987, ZanderXu and zhangshuyan0 March 7, 2025 07:10

slfan1989 mentioned this pull request Mar 9, 2025

[DISCUSS] [JDK17] Upgrade JUnit / JavaDoc from 4 to 5. #7337

Draft

74 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks. #7414

HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks. #7414

LiuGuH commented Feb 20, 2025 •

edited

Loading

hadoop-yetus commented Feb 20, 2025

slfan1989 commented Feb 20, 2025

hadoop-yetus commented Feb 24, 2025

hadoop-yetus commented Feb 25, 2025

hadoop-yetus commented Feb 26, 2025

hadoop-yetus commented Feb 28, 2025

LiuGuH commented Mar 4, 2025

slfan1989 commented Mar 7, 2025

hadoop-yetus commented Mar 7, 2025

LiuGuH commented Mar 7, 2025

HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks. #7414

Are you sure you want to change the base?

HDFS-17732. addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks. #7414

Conversation

LiuGuH commented Feb 20, 2025 • edited Loading

Description of PR

hadoop-yetus commented Feb 20, 2025

slfan1989 commented Feb 20, 2025

hadoop-yetus commented Feb 24, 2025

hadoop-yetus commented Feb 25, 2025

hadoop-yetus commented Feb 26, 2025

hadoop-yetus commented Feb 28, 2025

LiuGuH commented Mar 4, 2025

slfan1989 commented Mar 7, 2025

hadoop-yetus commented Mar 7, 2025

LiuGuH commented Mar 7, 2025

LiuGuH commented Feb 20, 2025 •

edited

Loading