Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Nice to have configurable batching in Sink when reading from Kafka #8

Open
ryancrawcour opened this issue Mar 20, 2019 · 8 comments
Open

Nice to have configurable batching in Sink when reading from Kafka #8

ryancrawcour opened this issue Mar 20, 2019 · 8 comments
Labels
blocked Needing somebody else to do something somewhere else first feature-request New feature or request

Comments

@ryancrawcour
Copy link
Contributor

Will read as batch, by default.
Batch size should default to a chosen value, and be configurable.
@ryancrawcour ryancrawcour added this to the P0 milestone Mar 20, 2019
@ryancrawcour
Copy link
Contributor Author

ryancrawcour commented Mar 20, 2019 •
edited
Loading

Config -

batch.size
Batch size between 1 (dedicated PutItemRequest for each record) and 25 (which is the maximum number of items in a BatchWriteItemRequest)

Type: int
Default: 1
Importance: high

@ryancrawcour
Copy link
Contributor Author

Look to leverage BulkExecutor for bulk write operations to Cosmos DB

@ryancrawcour ryancrawcour mentioned this issue Jun 17, 2019
Closed
@yorek yorek self-assigned this Jun 17, 2019
@yorek
Copy link
Member

yorek commented Jun 17, 2019

BulkExecutorAPI has a lot of overhead which doesn't make it suitable for just 25 rows

@yorek yorek removed their assignment Jun 17, 2019
@jyotsnaravikumar

This comment has been minimized.

# to view
ryancrawcour pushed a commit that referenced this issue Jun 24, 2019
@marcelaldecoa
Merge pull request #8 from marcelaldecoa/marcelaldecoa/issue-70-incor…
a82536c 
…rect-continuation-token

Fix #70 Incorrect logic for the continuation token in the Source Connector.
@ryancrawcour ryancrawcour changed the title Must have support for batching when reading from Kafka Must have support configurable batching when reading from Kafka Apr 20, 2020
@ryancrawcour ryancrawcour reopened this Apr 20, 2020
@ryancrawcour
Copy link
Contributor Author

must ensure new java version has this support for configurable batch size.

@ryancrawcour ryancrawcour modified the milestones: P0, P1 Apr 20, 2020
@ryancrawcour
Copy link
Contributor Author

related to #148

@ryancrawcour ryancrawcour mentioned this issue Nov 23, 2020
Closed
@brandynbrown brandynbrown removed this from the P1 milestone Nov 23, 2020
@ryancrawcour ryancrawcour self-assigned this Dec 3, 2020
@brandynbrown brandynbrown added blocked Needing somebody else to do something somewhere else first refine Issues needing refinement and removed investigate labels Dec 7, 2020
@ryancrawcour
Copy link
Contributor Author

adding #blocked to this because the Cosmos DB Java driver does not yet support BULK / BATCH operations. They are there, but only in preview and are likely to change.

will wait until the implementation has been finalized by Cosmos DB and will then adopt

@ryancrawcour ryancrawcour removed refine Issues needing refinement batching labels Dec 8, 2020
@brandynbrown brandynbrown added this to the M3 milestone Jan 7, 2021
@brandynbrown brandynbrown removed this from the M3 milestone Jan 25, 2021
@brandynbrown brandynbrown added feature-request New feature or request e-handoff labels Jan 25, 2021
@brandynbrown brandynbrown mentioned this issue Feb 4, 2021
Closed
1 task
@brandynbrown brandynbrown changed the title Must have support configurable batching when reading from Kafka Sink: Must have support configurable batching when reading from Kafka Feb 18, 2021
@brandynbrown
Copy link
Contributor

BulkExecutorAPI is going to be deprecated;
Blocked waiting for GA of v4 SDK

@ryancrawcour ryancrawcour changed the title Sink: Must have support configurable batching when reading from Kafka Nice to have support configurable batching in Sink when reading from Kafka Feb 28, 2021
@ryancrawcour ryancrawcour changed the title Nice to have support configurable batching in Sink when reading from Kafka Nice to have support for configurable batching in Sink when reading from Kafka Feb 28, 2021
@ryancrawcour ryancrawcour changed the title Nice to have support for configurable batching in Sink when reading from Kafka Nice to have configurable batching in Sink when reading from Kafka Feb 28, 2021
@kushagraThapar kushagraThapar mentioned this issue Aug 12, 2022
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
blocked Needing somebody else to do something somewhere else first feature-request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yorek @jyotsnaravikumar @brandynbrown @ryancrawcour