Support for Record Key and Precombine Key While Converting Iceberg Table to Hudi Table

[amnchauhan]

Is there any way to add record key and precombine key while converting iceberg to hudi table, this is important to start streaming job using spark in converted Hudi table.

[the-other-tim-brown]

There is a concept of a primary key in Iceberg that is copied over to the Hudi table as the record key but we don't have anything specifying the precombine key right now. We would need to add an option to the Hudi Target to accept these manually provided configs.

[amnchauhan]

Hi @the-other-tim-brown Thank you for the clarification!
It's good to know that Iceberg's primary key can already be mapped to Hudi's record key during conversion.I need to test if a streaming job will run on the converted hudi table by specifying primary key while creating Iceberg table through Trino. As far as i know primary key in iceberg is only supported by few engines, i will test and update.
However, the lack of support for specifying a precombine key limits certain use cases, especially when handling duplicate records or updates in Hudi.
This would be very helpful for scenarios involving high-throughput updates or upserts in streaming pipelines.

Looking forward to your thoughts!

[the-other-tim-brown]

Yes it is definitely limiting and should be fixed