Execute setup only if task is ready to run

[bjoernpollex-sc]

I'm trying to design a DAG that builds two tables. For each of the build-tasks, I have to check some prerequisites before running them. Additionally, there is a shared setup-task that needs to run. The basic DAG looks something like this:

This works fine, except for one issue. When I clear the DAG, the setup is executed immediately. However, if both of the prerequisite checks fail, then the actual build-tasks can't run, so the setup was not necessary after all. Now, to fix this, I could add a task-dependency from the prerequisite checks to the setup-task, like this:

That solves the issue described above, but introduces an unwanted side-effect, because now build_table1 is a downstream task of check_table2_prerequisites. This means that e.g. clearing all downstream tasks of check_table2_prerequesites would trigger a build of table1.

Is there any way to solve this in Airflow? shared_setup is a task that needs to run before either of table1 or table2 is built, ideally only after the prerequisite check is green, without introducing a transitive dependency among unrelated tasks.

[bjoernpollex-sc]

Bumping this, it would be really great to get some feedback on this. One possible solution would be to turn this into a feature request for Airflow - add an option to schedule setup tasks only if at least one task within the scope of that setup task is ready to run.