Skip to content

Commit

Permalink
Cherry pick (#32202)
Browse files Browse the repository at this point in the history
Co-authored-by: scwhittle <scwhittle@users.noreply.github.com>
  • Loading branch information
damccorm and scwhittle authored Aug 15, 2024
1 parent 5e69723 commit 57e5a46
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 3 deletions.
4 changes: 3 additions & 1 deletion CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,9 @@

## Known Issues

* ([#X](https://github.com/apache/beam/issues/X)).
* Large Dataflow graphs using runner v2, or pipelines explicitly enabling the `upload_graph` experiment, will fail at construction time ([#32159](https://github.com/apache/beam/issues/32159)).
* Python pipelines that run with 2.53.0-2.58.0 SDKs and read data from GCS might be affected by a data corruption issue ([#32169](https://github.com/apache/beam/issues/32169)). The issue will be fixed in 2.59.0 ([#32135](https://github.com/apache/beam/pull/32135)). To work around this, update the google-cloud-storage package to version 2.18.2 or newer.
* [KafkaIO] Records read with `ReadFromKafkaViaSDF` are redistributed and may contain duplicates regardless of the configuration. This affects Java pipelines with Dataflow v2 runner and xlang pipelines reading from Kafka, ([#32196](https://github.com/apache/beam/issues/32196))

# [2.57.0] - 2024-06-26

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1730,8 +1730,6 @@ public PCollection<KafkaRecord<K, V>> expand(PBegin input) {
.withKeyDeserializerProvider(kafkaRead.getKeyDeserializerProvider())
.withValueDeserializerProvider(kafkaRead.getValueDeserializerProvider())
.withManualWatermarkEstimator()
.withRedistribute()
.withAllowDuplicates() // must be set with withRedistribute option.
.withTimestampPolicyFactory(kafkaRead.getTimestampPolicyFactory())
.withCheckStopReadingFn(kafkaRead.getCheckStopReadingFn())
.withConsumerPollingTimeout(kafkaRead.getConsumerPollingTimeout());
Expand Down

0 comments on commit 57e5a46

Please # to comment.