Refactor hoodie-hive registration #181
Conversation
| @@ -21,30 +21,45 @@ | |||
| import com.beust.jcommander.Parameter; | |||
|
|
|||
| import java.io.Serializable; | |||
| import java.util.ArrayList; | |||
| import java.util.List; | |||
|
|
|||
| /** | |||
| * Configs needed to sync data into Hive. | |||
| */ | |||
There was a problem hiding this comment.
A lot of the files have no license.. Can we add it in
There was a problem hiding this comment.
Github PR is doing this fancy thing where it is hiding the lines - you will find a small blue icon on the left which when expanded shows the license. Just checked - All these files have valid licenses in them.
| @@ -21,30 +21,45 @@ | |||
| import com.beust.jcommander.Parameter; | |||
|
|
|||
| import java.io.Serializable; | |||
| import java.util.ArrayList; | |||
| import java.util.List; | |||
|
|
|||
| /** | |||
| * Configs needed to sync data into Hive. | |||
| */ | |||
| public class HiveSyncConfig implements Serializable { | |||
There was a problem hiding this comment.
A lot of the files have no copy right?
There was a problem hiding this comment.
Same thing as above
| * Tool to sync new data from commits, into Hive in terms of | ||
| * Tool to sync a hoodie HDFS dataset with a hive metastore table. | ||
| * Either use it as a api HiveSyncTool.syncHoodieTable(HiveSyncConfig) | ||
| * or as a command line java -cp hoodie-hive.jar HiveSyncTool [args] |
There was a problem hiding this comment.
Can you also make sure alll the docs are upto date? quickstart, deltastreamer etc..
There was a problem hiding this comment.
Good point. I will update the PR with documentation changes after i test this on the staging pipeline.
| hoodieHiveClient.createTable(schema, HoodieInputFormat.class.getName(), | ||
| MapredParquetOutputFormat.class.getName(), ParquetHiveSerDe.class.getName()); | ||
| break; | ||
| case MERGE_ON_READ: |
There was a problem hiding this comment.
for MERGE_ON_READ storage, we need to create both the tables.. One backed by HoodieInputFormat and one backed by HoodieRealtimeInputFormat.. You may want to factor this into the tool cli also
There was a problem hiding this comment.
Yeah left it as a TODO comment for now. We need to figure out some details before we register the RO table. Mostly around major compaction (we only do minor IO bound compactions). Will take this up seperately
| // Get the last time we successfully synced partitions | ||
| Optional<String> lastCommitTimeSynced = Optional.empty(); | ||
| if (tableExists) { | ||
| lastCommitTimeSynced = hoodieHiveClient.getLastCommitTimeSynced(); |
There was a problem hiding this comment.
👍 x 💯 . (if only we can make the cleaner also do this & every other meta operation, HDFS can scale a ton more)
| // read from the log file wrote | ||
| commitMetadata = HoodieCommitMetadata | ||
| .fromBytes(activeTimeline.getInstantDetails(lastDeltaCommit).get()); | ||
| filePath = commitMetadata.getFileIdAndFullPaths().values().stream().filter(s -> s.contains( |
There was a problem hiding this comment.
Note to self: we do log the schema with each commit, so we are guaranteed to get a schema from whichever log we read.
prazanna
force-pushed
the
hive-sync-refactor
branch
from
1d04504 to
f46a26a
Compare
|
Okay cool. please open a ticket for tracking double registering both RO and RT tables.. Otherwise gtg. Approve! |
|
Opened #193 for that. |
Refactor to