Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add note about using dag_run.conf in BashOperator #9143

Merged
merged 3 commits into from
Jun 8, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion airflow/example_dags/example_trigger_target_dag.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ def run_this_func(**context):

bash_task = BashOperator(
task_id="bash_task",
bash_command='echo "Here is the message: \'{{ dag_run.conf["message"] if dag_run else "" }}\'"',
bash_command='echo "Here is the message: $message"',
env={'message': '{{ dag_run.conf["message"] if dag_run else "" }}'},
dag=dag,
)
31 changes: 31 additions & 0 deletions airflow/operators/bash.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,37 @@ class BashOperator(BaseOperator):
.. code-block:: python

bash_command = "set -e; python3 script.py '{{ next_execution_date }}'"

.. warning::

Care should be taken with "user" input or when using Jinja templates in the
``bash_command``, as this bash operator does not perform any escaping or
sanitization of the command.

This applies mostly to using "dag_run" conf, as that can be submitted via
users in the Web UI. Most of the default template variables are not at
risk.

For example, do **not** do this:

.. code-block:: python

bash_task = BashOperator(
task_id="bash_task",
bash_command='echo "Here is the message: \'{{ dag_run.conf["message"] if dag_run else "" }}\'"',
)

Instead, you should pass this via the ``env`` kwarg and use double-quotes
inside the bash_command, as below:

.. code-block:: python

bash_task = BashOperator(
task_id="bash_task",
bash_command='echo "here is the message: \'$message\'"',
env={'message': '{{ dag_run.conf["message"] if dag_run else "" }}'},
)

"""
template_fields = ('bash_command', 'env')
template_ext = ('.sh', '.bash',)
Expand Down
31 changes: 31 additions & 0 deletions docs/howto/operator/bash.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,37 @@ You can use :ref:`Jinja templates <jinja-templating>` to parameterize the
:start-after: [START howto_operator_bash_template]
:end-before: [END howto_operator_bash_template]


.. warning::

Care should be taken with "user" input or when using Jinja templates in the
``bash_command``, as this bash operator does not perform any escaping or
sanitization of the command.

This applies mostly to using "dag_run" conf, as that can be submitted via
users in the Web UI. Most of the default template variables are not at
risk.

For example, do **not** do this:

.. code-block:: python

bash_task = BashOperator(
task_id="bash_task",
bash_command='echo "Here is the message: \'{{ dag_run.conf["message"] if dag_run else "" }}\'"',
)

Instead, you should pass this via the ``env`` kwarg and use double-quotes
inside the bash_command, as below:

.. code-block:: python

bash_task = BashOperator(
task_id="bash_task",
bash_command='echo "here is the message: \\'$message\\'"',
env={'message': '{{ dag_run.conf["message"] if dag_run else "" }}'},
)

Troubleshooting
---------------

Expand Down