You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Targets (data type handling, batching, SQL object generation, etc.)
Description
Currently the utilities for creating batch files from lists of records is only available on Stream classes (and descendants). In implementing target-snowflake, I wished to overload bulk_insert_records (as is expected for database specific optimisations) to use the same mechanism as process_batch_files for bulk loading Snowflake via an internal stage. However bulk_insert_records receives a list of Record payloads, and process_batch_files expects a list of file URI's. Therefore, to create the necessary URI's, I reached for the helper methods on the Stream class implemented to support the creation of BATCH messages in the Tap.
I propose we:
Refactor to make the utilities for serialising records into files reusable in both Taps and Targets
Modify bulk_insert_records on the Sink class to use process_batch_files (or visa versa) so that developers need only implement 1 bulk insert method that is used regardless of whether the Target receives records or batches
The text was updated successfully, but these errors were encountered:
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.
Feature scope
Targets (data type handling, batching, SQL object generation, etc.)
Description
Currently the utilities for creating batch files from lists of records is only available on
Stream
classes (and descendants). In implementingtarget-snowflake
, I wished to overloadbulk_insert_records
(as is expected for database specific optimisations) to use the same mechanism asprocess_batch_files
for bulk loading Snowflake via an internal stage. Howeverbulk_insert_records
receives a list of Record payloads, andprocess_batch_files
expects a list of file URI's. Therefore, to create the necessary URI's, I reached for the helper methods on theStream
class implemented to support the creation of BATCH messages in the Tap.I propose we:
bulk_insert_records
on theSink
class to useprocess_batch_files
(or visa versa) so that developers need only implement 1 bulk insert method that is used regardless of whether the Target receives records or batchesThe text was updated successfully, but these errors were encountered: