Key by paneindex and reshuffle before loading files. #34324
Conversation
|
Assigning reviewers. If you would like to opt out of this review, comment R: @liferoad for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| self, | ||
| element, | ||
| load_job_name_prefix, | ||
| pane_info=beam.DoFn.PaneInfoParam, |
There was a problem hiding this comment.
is this safe to remove pane_info?
There was a problem hiding this comment.
We are still using pane info, but just adding it earlier as part of the key that gets processed by TriggerLoadJob
| @@ -1104,6 +1099,8 @@ def _load_data( | |||
| # Load data using temp tables | |||
| trigger_loads_outputs = ( | |||
| partitions_using_temp_tables | |||
| | "KeyByPaneIndexWithTempTables" >> beam.ParDo(KeyByPaneIndex()) | |||
There was a problem hiding this comment.
do we need to worry about the update compatibility?
There was a problem hiding this comment.
Yes, I will make those changes before merging this
|
Thanks @claudevdm. I know there is context somewhere, but could you briefly explain why we need this change here? |
The Reshuffle implementation in python does not preserve pane index (see #28219) If we keep fetching pane info in the original DoFn then ALL load jobs will have pane info 0 (see linked bug) and only the very first load job will succeed. So another option is to fix the Reshuffle implementation (I have tested this too), but that is a more intrusive change that probably needs more discussion. Note #28219 was for Java that had a similar issue, but Python never had a Reshuffle before actually loading the files so it didn't have any effect |
| class KeyByPaneIndex(beam.DoFn): | ||
| def process(self, element, pane_info=beam.DoFn.PaneInfoParam): | ||
| destination, (partition_key, files) = element | ||
| return [(destination, (partition_key, files, pane_info.index))] |
There was a problem hiding this comment.
This is very similar to a first attempt to fix the same issue in Java - #28272 add pane info in "WritePartition" so it is survived in ReShuffle. In earlier discussion it was decided to go with fixing ReShuffle itself (and that PR was closed). Shall we go for same decision here?
There was a problem hiding this comment.
One of the original concern was upgrade compatibility. However because a ReShuffle is added here, pipeline graph will be changed anyways
There was a problem hiding this comment.
Does it make sense to do this keying by pane in the upstream PartitionFiles DoFn? i.e. instead of creating a new dedicated step.
I think that'll make it easier to revert this quick fix when Reshuffle gets fixed (which should be the longterm solution)
Add Reshuffle before triggering load jobs to ensure determinism.
The Reshuffle implementation in python does not preserve pane index (see #28219)
If we keep fetching pane info in the original DoFn then ALL load jobs will have pane info 0 (see linked bug) and only the very first load job will succeed.
So another option is to fix the Reshuffle implementation (I have tested this too), but that is a more intrusive change that probably needs more discussion.
Note #28219 was for Java that had a similar issue, but Python never had a Reshuffle before actually loading the files so it didn't have any effect
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.