Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: make utilities for creating record batches from Stream available to Sink classes #1026

Closed
kgpayne opened this issue Oct 3, 2022 · 2 comments
Labels

Comments

@kgpayne
Copy link
Contributor

kgpayne commented Oct 3, 2022

Feature scope

Targets (data type handling, batching, SQL object generation, etc.)

Description

Currently the utilities for creating batch files from lists of records is only available on Stream classes (and descendants). In implementing target-snowflake, I wished to overload bulk_insert_records (as is expected for database specific optimisations) to use the same mechanism as process_batch_files for bulk loading Snowflake via an internal stage. However bulk_insert_records receives a list of Record payloads, and process_batch_files expects a list of file URI's. Therefore, to create the necessary URI's, I reached for the helper methods on the Stream class implemented to support the creation of BATCH messages in the Tap.

I propose we:

  1. Refactor to make the utilities for serialising records into files reusable in both Taps and Targets
  2. Modify bulk_insert_records on the Sink class to use process_batch_files (or visa versa) so that developers need only implement 1 bulk insert method that is used regardless of whether the Target receives records or batches
@stale
Copy link

stale bot commented Jul 18, 2023

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@stale stale bot added the stale label Jul 18, 2023
@stale stale bot closed this as completed Aug 8, 2023
@stale stale bot removed the stale label Aug 8, 2023
Copy link

stale bot commented Aug 7, 2024

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

@stale stale bot added the stale label Aug 7, 2024
@stale stale bot closed this as completed Aug 28, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants