Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: support metadata tables #823

Open
xxchan opened this issue Dec 18, 2024 · 7 comments
Open

feat: support metadata tables #823

xxchan opened this issue Dec 18, 2024 · 7 comments
Labels
enhancement New feature or request

Comments

@xxchan
Copy link
Member

xxchan commented Dec 18, 2024

In Iceberg Spark/Flink there are metadata tables that provide information around the table: https://iceberg.apache.org/docs/latest/spark-queries/#inspecting-tables

Supporting this in iceberg-rust allows other engines (like RisingWave) to support these "standard" metadata tables.

reference implementation:
- https://py.iceberg.apache.org/api/#inspecting-tables
- https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/inspect.py#L58-L90

List of all metadata tables: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MetadataTableType.java#L23

@Fokko
Copy link
Contributor

Fokko commented Dec 18, 2024

Thanks for working on this @xxchan. Metadata tables are very important when it comes to the metadata operations. For example, the snapshots table in #822 will make it easy to perform operations like expiring snapshots.

@flaneur2020
Copy link
Contributor

i'd like to contribute to some of the implementation of these metadata tables, similar to the work done by @rshkv based on @xxchan's branch.

I plan to start with the MANIFESTS table first. could you kindly let me know if anyone else is currently working on this or has plans to do so? I want to avoid any potential conflicts. thanks.

@rshkv
Copy link
Contributor

rshkv commented Dec 28, 2024

@flaneur2020, apologies, I was actually just working on manifests and I should be able to put that up in a few days. If you want to go ahead with entries or files there'll be some overlap and code to reuse but that's fine - happy to rebase.

Ignore the above. I'm looking at entries now. Not working on manifests.

@DeaconDesperado
Copy link
Contributor

I'd like to attempt the FILES metadata table if not already being worked on. Thanks!

@liurenjie1024
Copy link
Contributor

I'd like to attempt the FILES metadata table if not already being worked on. Thanks!

Welcome to contribute!

@felixscherz
Copy link

Hi, I'm going to have a look at the PARTITIONS table:)

@xxchan
Copy link
Member Author

xxchan commented Feb 2, 2025

Hi, feel free to take subtasks. But kindly reminder that we need to resolve #868 before proceeding

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

7 participants