bug: HDFS Storage flush error

[yufansong]

Describe the bug

When trying to use HDFS as the storage object and flush content, hummock will meet some problems. And also, when selecting HDFS as the storage, some e2e tests will fail.

In my environment, when launched by ./risedev d, flush is ok, e2e test is also ok. When opening HDFS feature, build, ./risedev d hdfs, flush and e2e test will meet the problem above.

When I check the HDFS, there are some data already storage successfully.

To Reproduce

1. Use the branch wcy/push_for_yufan which already enables the hdfs feature.
2. Launch the EC2 machine and try to build the branch and then ./risedev d hdfs
3. Connect the database, write data, and flush it.

Expected behavior

No response

Additional context

No response

[wcy-fdu]

I believe this is some hdfs config issue, will check it later.

[Xuanwo]

Hi, can this issue be reproduced consistently or is it random?

---

Just a quick note, OpenDAL has built-in support for retrying operations in case of temporary errors. You can find more information about it at this link: https://docs.rs/opendal/latest/opendal/layers/struct.RetryLayer.html.

[wcy-fdu]

May due to the hdfs classpath is incorrect.

loadFileSystems error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
hdfsBuilderConnect(forceNewInstance=0, nn=127.0.0.1:9000, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
(unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.)
thread 'risingwave-main' panicked at 'called `Result::unwrap()` on an `Err` value: Unexpected (temporary) at  => uncategorized error, source: Unknown error 255 (os error 255)

[yufansong]

I think in the test case I mentioned, the database has already flushed part of the data into the system, which means the program should have already found the correct HDFS classpath.

[Xuanwo]

I think the problem has been resolved with apache/opendal#2114. Kindly upgrade to opendal v0.33.1 and try again.

[wcy-fdu]

I think the problem has been resolved with apache/incubator-opendal#2114. Kindly upgrade to opendal v0.33.1 and try again.

Thanks, let me try it later.

[wcy-fdu]

After a quick verification, I think hdfs works now. I will recheck this later, thanks for your work~ @Xuanwo