Tracking issues of 0.1.0 version for Apache Paimon Rust

[Xuanwo]

Hello everyone, this is the tracking update for the paimon rust 0.1.0 release.

Goal

Before we outline the tasks for the 0.1.0 release, let me clarify the project's goal:

Developing a complete implementation of Paimon in pure Rust.

• Users can read/write paimon table like they do in java API.
• Users can read/write paimon table in arrow format.
• Native support for DataFusion
  • But also enable users to implement their own query engines based on this project.
• Native WASM support (a.k.a paimon-wasm)
• Native Python binding based on rust core (a.k.a paimon-py)
• Hive catalog support

Tasks

This will be our initial release, and I aim to include basic read support in it.

• Spec: Implement types that needed by paimon.
  • feat(spec): Add schema types #5
  • Datatypes
  • spec: Implement Snapshot #6
  • spec: Implement Manifest List #14
  • spec: Implement Manifest #7
  • spec: Implement Datafile #10
  • Global Index
  • Data File Index
• Catalog
  • feat(catalog): add catalog API #62
  • catalog: Implement the filesystem catalog #70 (leave hive catalog in next release)
• Arrow Integration:
  • reader: Allow reading paimon data into arrow format. #71
• Basic Read Process: No schema evolution, no merging, no deletion.
  • scan: Implement scan to generate data split from the Snapshot #72 (filter & project push down are not required in v0.1)
• IO: Integrate with Apache OpenDAL for IO.
• Release Utils: Impelment scripts to help generate and verify ASF releases.

After all those tasks, I expect users can read an existing paimon table from storage services.

[QuakeWang]

@Xuanwo I will do the Snapshot task.

[Xuanwo]

@Xuanwo I will do the Snapshot task.

Welcome, have fun!

[dharanad]

hello @Xuanwo , I would like to work on implementing Manifest.

[Xuanwo]

hello @Xuanwo , I would like to work on implementing Manifest.

Great, have fun!

[crrow]

Hey, I'm new around here and this project sounds interesting. Can I jump in and help out?

[Xuanwo]

Manifest

Thanks a lot, welcome to pick up a part. I'm willing to update tracking issues for you.

[crrow]

Manifest

Thanks a lot, welcome to pick up a part. I'm willing to update tracking issues for you.

okay, let me try data file

[Asura7969]

I want to try the manifest list

[thexiay]

Cool, it's good project to be familiar with rust and big data infrastructure😆

[QuakeWang]

I will do the Changelog task, and the corresponding issue have been created #16.

[Xuanwo]

Cool, it's good project to be familiar with rust and big data infrastructure😆

Welcome!

I will do the Changelog task, and the corresponding issue have been created #16.

Updated.

[suxiaogang223]

Hi, I am a big fan of rust, and I usually do data lake-related development. Be happy to join the development of paimon-rust in the future.

[Aitozi]

Does the FileSystemCatalog rely on the Integrate with Apache OpenDAL ? There are some IO operation in the FileSystemCatalog. @Xuanwo

[Xuanwo]

Does the FileSystemCatalog rely on the Integrate with Apache OpenDAL ?

Yes, my plan is using opendal for those IO operations.

[devillove084]

I will try to implement the File index part and try to follow the implementation logic of the Java version and use the Rust style as much as possible. @Xuanwo

[Xuanwo]

I will try to implement the File index part and try to follow the implementation logic of the Java version and use the Rust style as much as possible. @Xuanwo

Thanks!

[ForeverAngry]

Hi all, I'd like to help out with the project. I'm a bit new to rust. Let me know where I can lend a hand at!

[Xuanwo]

CC @Aitozi, do you have any suggestions regarding our current focus (reviewing the tables)? Which part do you think is the most important? Is it okay for us to ignore all the Index entries?