Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Set unique table suffix to allow parallel incremental execution #810

Closed
huangxingyi-git opened this issue Sep 26, 2024 · 1 comment
Closed
Labels
enhancement New feature or request

Comments

@huangxingyi-git
Copy link
Contributor

huangxingyi-git commented Sep 26, 2024

Describe the feature

For some specific cases (eg. backfill very large amount of data), we need to execute parallel multiple dbt run of specific incremental(replace_where) model in which we pass the date (or country) as var argument.
For example, we have a model we run every day using Airflow for which we pass the a date relative to the Airflow scheduler.
FYI
https://github.com/dbt-labs/dbt-athena/pull/650/files

If we want to process by batch of N days in parallel using Airflow concurrency, we need the tmp table create by each of the dbt run to be unique. Else, you are going to end up with N insert attempting to run with the same __dbt_tmp name, creating conflict and ultimately creating failure.

Who will this benefit?

For those who uses repalce_where as incremental strategy.
Example Use Case: Run the same incremental model concurrently with different --vars in order to parallelly insert multiple data partitions

Are you interested in contributing this feature?

I am interested in contributing to this feature if needed.

@benc-db
Copy link
Collaborator

benc-db commented Sep 27, 2024

This issue is going to solved more comprehensively with dbt-labs/dbt-core#10672; however, I'll take a look at your PR, and assuming it doesn't hurt any other use case, I'm not against taking this in the short term.

@benc-db benc-db closed this as completed Oct 11, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants