You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple dbt run of specific incremental(replace_where) model in which we pass the date (or country) as var argument.
For example, we have a model we run every day using Airflow for which we pass the a date relative to the Airflow scheduler.
FYI https://github.com/dbt-labs/dbt-athena/pull/650/files
If we want to process by batch of N days in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.
Who will this benefit?
For those who uses repalce_where as incremental strategy.
Example Use Case: Run the same incremental model concurrently with different --vars in order to parallelly insert multiple data partitions
Are you interested in contributing this feature?
I am interested in contributing to this feature if needed.
The text was updated successfully, but these errors were encountered:
This issue is going to solved more comprehensively with dbt-labs/dbt-core#10672; however, I'll take a look at your PR, and assuming it doesn't hurt any other use case, I'm not against taking this in the short term.
Describe the feature
For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple
dbt run
of specific incremental(replace_where
) model in which we pass the date (or country) as var argument.For example, we have a model we run every day using Airflow for which we pass the a date relative to the Airflow scheduler.
FYI
https://github.com/dbt-labs/dbt-athena/pull/650/files
If we want to process by batch of N days in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.
Who will this benefit?
For those who uses
repalce_where
as incremental strategy.Example Use Case: Run the same incremental model concurrently with different
--vars
in order to parallelly insert multiple data partitionsAre you interested in contributing this feature?
I am interested in contributing to this feature if needed.
The text was updated successfully, but these errors were encountered: