ipdb no longer works with airflow test command #26352

GuruComposer · 2022-09-12T16:54:07Z

GuruComposer
Sep 12, 2022

Apache Airflow version

2.3.4

What happened

I used to be able to use ipdb to debug DAGs by running airflow tasks test <dag_name> <dag_id>, and hitting an ipdb breakpoint (ipdb.set_trace()).

This no longer works. I get a strange type error:

    return self.dispatch_line(frame)
  File "/usr/local/lib/python3.9/bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "/usr/local/lib/python3.9/pdb.py", line 262, in user_line
    self.interaction(frame, None)
  File "/home/astro/.local/lib/python3.9/site-packages/IPython/core/debugger.py", line 336, in interaction
    OldPdb.interaction(self, frame, traceback)
  File "/usr/local/lib/python3.9/pdb.py", line 357, in interaction
    self._cmdloop()
  File "/usr/local/lib/python3.9/pdb.py", line 322, in _cmdloop
    self.cmdloop()
  File "/usr/local/lib/python3.9/cmd.py", line 126, in cmdloop
    line = input(self.prompt)
TypeError: an integer is required (got type NoneType)```



### What you think should happen instead

I should get the ipdb shell.

### How to reproduce

1. Add ipdb breakpoint anywhere in airflow task.
import ipdb; ipdb.set_trace()

2. Run that task:
Run `airflow tasks test <dag_name> <dag_id>`, and 

### Operating System

Debian GNU/Linux

### Versions of Apache Airflow Providers

2.3.4

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Answered by ashb

Sep 30, 2022

I have reproduced this error, and re-created the issue (as I can't move back from discussion to issue, nor-reopen then existing one) #26802

View full answer

potiuk · 2022-09-12T16:54:09Z

boring-cyborg[bot]
bot Sep 12, 2022

Thanks for opening your first issue here! Be sure to follow the issue template!

3 replies

potiuk Sep 12, 2022
Collaborator

This is at most a discussion. I can use Ipdb with no problems (just tested it). So it must be something in your environment.

I suggest you to look for similar problems and trace it back (there a number of similar errors reported when you just google it).

Also I would suggest you the "bisecting" approach (unfortunately it is something that only you can do). Try to incrementally change different things (lower version of airflow, downgrading/upgrading suspected libraries in order to pin-point the culprit. That's what I would do (unfortunately ipdb works for me so I have way of doing it).

It would also be great, when you find the environmental issue - if you come back and report what it was, this might be helpful for others who would hit the same problem.

GuruComposer Sep 29, 2022
Author

I just tested ipdb in an airflow task with the following code with the latest Open Source Airflow 2.4.0, and it does not work. I ran airflow tasks test my_test_dag test_task <date_stamp>.

from datetime import datetime, timedelta

from airflow.models import DAG
from airflow.operators.python import PythonOperator

import logging
import ipdb


def _test_task():
    ipdb.set_trace()
    logging.info('Hello World')

with DAG(
    'my_test_dag',
    default_args={
        'depends_on_past': False,
        'email': ['airflow@example.com'],
        'email_on_failure': False,
        'email_on_retry': False
    },
    description='A simple test DAG',
    schedule_interval=timedelta(minutes=30),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['test_dag'],
) as dag:

    test_task = PythonOperator(
        task_id='test_task',
        python_callable=_test_task
    )

    test_task

Here are the logs:

/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py:134 RemovedInAirflow3Warning: Calling `DAG.create_dagrun()` without an explicit data interval is deprecated
[2022-09-29 14:26:07,388] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: my_test_dag.test_task __***_temporary_run_2022-09-29T14:26:07.322490+00:00__ [None]>
[2022-09-29 14:26:07,404] {taskinstance.py:1165} INFO - Dependencies all met for <TaskInstance: my_test_dag.test_task __***_temporary_run_2022-09-29T14:26:07.322490+00:00__ [None]>
[2022-09-29 14:26:07,404] {taskinstance.py:1362} INFO -
--------------------------------------------------------------------------------
[2022-09-29 14:26:07,405] {taskinstance.py:1363} INFO - Starting attempt 1 of 1
[2022-09-29 14:26:07,405] {taskinstance.py:1364} INFO -
--------------------------------------------------------------------------------
[2022-09-29 14:26:07,408] {taskinstance.py:1383} INFO - Executing <Task(PythonOperator): test_task> on 2022-09-28T00:00:00+00:00
[2022-09-29 14:26:07,668] {taskinstance.py:1592} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=***@example.com
AIRFLOW_CTX_DAG_OWNER=***
AIRFLOW_CTX_DAG_ID=my_test_dag
AIRFLOW_CTX_TASK_ID=test_task
AIRFLOW_CTX_EXECUTION_DATE=2022-09-28T00:00:00+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=__***_temporary_run_2022-09-29T14:26:07.322490+00:00__
> /opt/***/dags/test_ipdb.py(12)_test_task()
     11     ipdb.set_trace()
---> 12     logging.info('Hello World')
     13

[2022-09-29 14:26:07,979] {taskinstance.py:1851} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/***/.local/lib/python3.7/site-packages/***/operators/python.py", line 175, in execute
    return_value = self.execute_callable()
  File "/home/***/.local/lib/python3.7/site-packages/***/operators/python.py", line 193, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/***/dags/test_ipdb.py", line 12, in _test_task
    logging.info('Hello World')
  File "/opt/***/dags/test_ipdb.py", line 12, in _test_task
    logging.info('Hello World')
  File "/usr/local/lib/python3.7/bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/local/lib/python3.7/bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "/usr/local/lib/python3.7/pdb.py", line 259, in user_line
    self.interaction(frame, None)
  File "/home/***/.local/lib/python3.7/site-packages/IPython/core/debugger.py", line 442, in interaction
    OldPdb.interaction(self, frame, traceback)
  File "/usr/local/lib/python3.7/pdb.py", line 350, in interaction
    self._cmdloop()
  File "/usr/local/lib/python3.7/pdb.py", line 319, in _cmdloop
    self.cmdloop()
  File "/home/***/.local/lib/python3.7/site-packages/IPython/terminal/debugger.py", line 116, in cmdloop
    line = self.thread_executor.submit(self.pt_app.prompt).result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1037, in prompt
    handle_sigint=handle_sigint,
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 982, in run
    handle_sigint=handle_sigint,
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 885, in run_async
    return await _run_async(f)
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 743, in _run_async
    self._redraw()
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 557, in _redraw
    self.context.copy().run(run_in_context)
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 540, in run_in_context
    self.renderer.render(self, self.layout)
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/renderer.py", line 721, in render
    output.flush()
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/output/plain_text.py", line 57, in flush
    flush_stdout(self.stdout, data)
  File "/home/***/.local/lib/python3.7/site-packages/prompt_toolkit/output/flush_stdout.py", line 32, in flush_stdout
    stdout.buffer.write(data.encode(stdout.encoding or "utf-8", "replace"))
AttributeError: 'NoneType' object has no attribute 'write'
[2022-09-29 14:26:07,984] {taskinstance.py:1406} INFO - Marking task as FAILED. dag_id=my_test_dag, task_id=test_task, execution_date=20220928T000000, start_date=, end_date=20220929T142607
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 99, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 562, in task_test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1753, in run
    mark_success=mark_success, test_mode=test_mode, job_id=job_id, pool=pool, session=session
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1457, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1603, in _execute_task_with_callbacks
    result = self._execute_task(context, task_orig)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1664, in _execute_task
    result = execute_callable(context=context)
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 175, in execute
    return_value = self.execute_callable()
  File "/home/airflow/.local/lib/python3.7/site-packages/airflow/operators/python.py", line 193, in execute_callable
    return self.python_callable(*self.op_args, **self.op_kwargs)
  File "/opt/airflow/dags/test_ipdb.py", line 12, in _test_task
    logging.info('Hello World')
  File "/opt/airflow/dags/test_ipdb.py", line 12, in _test_task
    logging.info('Hello World')
  File "/usr/local/lib/python3.7/bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/local/lib/python3.7/bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "/usr/local/lib/python3.7/pdb.py", line 259, in user_line
    self.interaction(frame, None)
  File "/home/airflow/.local/lib/python3.7/site-packages/IPython/core/debugger.py", line 442, in interaction
    OldPdb.interaction(self, frame, traceback)
  File "/usr/local/lib/python3.7/pdb.py", line 350, in interaction
    self._cmdloop()
  File "/usr/local/lib/python3.7/pdb.py", line 319, in _cmdloop
    self.cmdloop()
  File "/home/airflow/.local/lib/python3.7/site-packages/IPython/terminal/debugger.py", line 116, in cmdloop
    line = self.thread_executor.submit(self.pt_app.prompt).result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/shortcuts/prompt.py", line 1037, in prompt
    handle_sigint=handle_sigint,
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 982, in run
    handle_sigint=handle_sigint,
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 885, in run_async
    return await _run_async(f)
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 743, in _run_async
    self._redraw()
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 557, in _redraw
    self.context.copy().run(run_in_context)
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/application/application.py", line 540, in run_in_context
    self.renderer.render(self, self.layout)
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/renderer.py", line 721, in render
    output.flush()
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/output/plain_text.py", line 57, in flush
    flush_stdout(self.stdout, data)
  File "/home/airflow/.local/lib/python3.7/site-packages/prompt_toolkit/output/flush_stdout.py", line 32, in flush_stdout
    stdout.buffer.write(data.encode(stdout.encoding or "utf-8", "replace"))
AttributeError: 'NoneType' object has no attribute 'write'

If you suspect this is an IPython 7.34.0 bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

GuruComposer Sep 29, 2022
Author

So for some reason ipdb is no longer working in the latest version of Airflow. We should be able to pop into the ipdb shell.

GhostNA · 2022-09-21T18:04:44Z

GhostNA
Sep 21, 2022

Same issue!

9 replies

potiuk Sep 30, 2022
Collaborator

There are no instructions for ipdb specifically (yet). We might add it soon as we are About to take care about mor improvements, to dag development experience. But you can do it in the same way as you are used to for other debugging.

There's is airflow test command that you can likely plug in, you a na also look for DebugExecutor in the docs that has some debug-specific features (single process execution for one).

abhishekbhakat Sep 30, 2022

I'm able to drop into ipdb with export PYTHONBREAKPOINT="ipdb.set_trace" and adding a breakpoint. For example:

from datetime import datetime, timedelta

from airflow.models import DAG
from airflow.operators.python import PythonOperator

import logging
import ipdb

ipdb.set_trace()

def _test_task():
    breakpoint()
    logging.info('Hello World')

with DAG('my_test_dag',
default_args={
'depends_on_past': False,
'email': ['airflow@example.com'],
'email_on_failure': False,
'email_on_retry': False
},
description='A simple test DAG',
schedule_interval=timedelta(minutes=30),
start_date=datetime(2021, 1, 1),
catchup=False,
tags=['test_dag'],
) as dag:

    test_task = PythonOperator(
    task_id='test_task',
    python_callable=_test_task
    )

abhishekbhakat Sep 30, 2022

Seems like this drops into ipdb shell while loading the DAG definition and not while executing the task.

ashb Sep 30, 2022
Collaborator

@abhishekbhakat That's because you've placed the breakpoint at the "top level" of the file -- ipdb.set_trace() is another way of setting a breakpoint.

abhishekbhakat Sep 30, 2022

That way it drops to AttributeError: 'NoneType' object has no attribute 'write'.

ashb · 2022-09-30T14:59:24Z

ashb
Sep 30, 2022
Collaborator

I have reproduced this error, and re-created the issue (as I can't move back from discussion to issue, nor-reopen then existing one) #26802

6 replies

wes-devore-mpulse Jun 7, 2023

Hi, ipdb broke again in 2.6.1

potiuk Jun 8, 2023
Collaborator

@wes-devore-mpulse - Creating a new issue about it, or even-better - fixing it in PR is likely a better idea than commenting on discussion.

wes-devore-mpulse Jun 8, 2023

Here is the issue: #31784. Can anyone take a look at this?

potiuk Jun 8, 2023
Collaborator

Hopefully someone will.

jkryanchou Aug 13, 2024

Here is the issue: #31784. Can anyone take a look at this?

I got the solution from @utkarsharma2's answer in the issue #31784

from IPython.core import debugger; debugger.Pdb().set_trace()

It works. While it cannot tab to complete the variables. I thought it may not be the iterative shell when adding the codes above.

wes-devore-mpulse · 2023-06-07T23:37:51Z

wes-devore-mpulse
Jun 7, 2023

It looks like ipdb broke again in 2.6.1

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ipdb no longer works with airflow test command #26352

{{title}}

Replies: 4 comments 18 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

ipdb no longer works with airflow test command #26352

Apache Airflow version

What happened

Replies: 4 comments · 18 replies

boring-cyborg[bot] bot Sep 12, 2022

potiuk Sep 12, 2022 Collaborator

GuruComposer Sep 29, 2022 Author

GuruComposer Sep 29, 2022 Author

potiuk Sep 30, 2022 Collaborator

ashb Sep 30, 2022 Collaborator

ashb Sep 30, 2022 Collaborator

potiuk Jun 8, 2023 Collaborator

potiuk Jun 8, 2023 Collaborator

Replies: 4 comments 18 replies

boring-cyborg[bot]
bot Sep 12, 2022

potiuk Sep 12, 2022
Collaborator

GuruComposer Sep 29, 2022
Author

GuruComposer Sep 29, 2022
Author

potiuk Sep 30, 2022
Collaborator

ashb Sep 30, 2022
Collaborator

ashb
Sep 30, 2022
Collaborator

potiuk Jun 8, 2023
Collaborator

potiuk Jun 8, 2023
Collaborator