Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FEATURE] apply_source_filter config added to ALL appropriate macros #222

Open
philmaddocks opened this issue Jan 8, 2024 · 0 comments
Open
Assignees
Labels
feature This is is requesting a new feature

Comments

@philmaddocks
Copy link

philmaddocks commented Jan 8, 2024

Is your feature request related to a problem? Please describe.
AutomateDV assumes that the data being pointed at the macros is always deltas. In our architecture, we have a data lake where we create an event base history for each source table. We'd like to point macros at this (and can for the most part) but apply_source_filter is not across all of the macros.

Describe the solution you'd like
I'd like to have the apply_source_filter config available on all macros. This would allow idempotent builds over event base history tables from source where we can simply restart jobs, or full-refresh the data vault should issues arise. I would expect this to be available on all macros excluding possibly hubs and links.

Describe alternatives you've considered
The config is already there for the main sat macro, but I don't see any harm adding it to more as an option so there is the flexibility to limit the data considered by the macro rather than having to create a delta layer.

AB#5352

@philmaddocks philmaddocks added the feature This is is requesting a new feature label Jan 8, 2024
@DVAlexHiggs DVAlexHiggs added feature This is is requesting a new feature and removed feature This is is requesting a new feature labels May 15, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
feature This is is requesting a new feature
Projects
None yet
Development

No branches or pull requests

2 participants