Airflow backfilling can't be disabled

[hafid-d]

Apache Airflow version: 2.0.2

Kubernetes version (if you are using kubernetes) (use kubectl version):

• Cloud provider or hardware configuration:
• OS : Ubuntu 18.04.3
• Install tools: celery = 4.4.7, redis = 3.5.3

What happened:.
Noticed a weird behavior with Airflow backfilling: my previous dags are still queing and running even after doing the following :

• Setting catchup_by_default=False in airflow.cfg
• Setting catchup=False in the DAG definition
• Using LatestOnlyOperator

What you expected to happen:
I expect the old dags not to be running again.

[motherhubbard]

I had this issue also in 2.0.2. Ive just bumped to 2.1.0 and it seems to be fixed in there.

[hafid-d]

@motherhubbard tried using 2.1.0 but still have the issue :-/ did u update anything else?

[eladkal]

Can you please add a reproduce example?

[GergelyKalmar]

I had a similar issue – in my case it was a mistyped cron that seemingly caused backfills. The trigger seemed to work fine but the intervals were wrong (the cron expressions were like 5/10 * * * * *, notice it has one too many stars in it).

[uranusjr]

the cron expressions were like 5/10 * * * * *, notice it has one too many stars in it

Could you open an issue for that? This is an invalid expression (I think) IMO we should prevent this from even being accepted. (I think this is a bug in croniter, but we should be able to perform additional validation in Airflow if they don’t want to fix it.)

[eladkal]

the cron expressions were like 5/10 * * * * *, notice it has one too many stars in it

This is a valid cron exp. The 6th parameter means year.
However there is a bug in croniter it seems they use the 6th parameter wrong and it means seconds: taichino/croniter#76

>>> from croniter import croniter
>>> croniter('5/10 * * * * *')
<croniter.croniter.croniter object at 0x10b07c190>
>>> croniter('5/10 * * * * 1')
<croniter.croniter.croniter object at 0x10af92750>
>>> croniter('5/10 * * * * 59')
<croniter.croniter.croniter object at 0x10b07c190>
>>> croniter('5/10 * * * * 60')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/croniter/croniter.py", line 154, in __init__
    self.expanded, self.nth_weekday_of_month = self.expand(expr_format, hash_id=hash_id)
  File "/usr/local/lib/python3.7/site-packages/croniter/croniter.py", line 759, in expand
    return cls._expand(expr_format, hash_id=hash_id)
  File "/usr/local/lib/python3.7/site-packages/croniter/croniter.py", line 725, in _expand
    expr_format))
croniter.croniter.CroniterBadCronError: [5/10 * * * * 60] is not acceptable, out of range
>>> croniter('5/10 * * * * 2000')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/croniter/croniter.py", line 154, in __init__
    self.expanded, self.nth_weekday_of_month = self.expand(expr_format, hash_id=hash_id)
  File "/usr/local/lib/python3.7/site-packages/croniter/croniter.py", line 759, in expand
    return cls._expand(expr_format, hash_id=hash_id)
  File "/usr/local/lib/python3.7/site-packages/croniter/croniter.py", line 725, in _expand
    expr_format))
croniter.croniter.CroniterBadCronError: [5/10 * * * * 2000] is not acceptable, out of range

[GergelyKalmar]

That is in line with what I observed! What is weird though is that Airflow would schedule a separate job for every second even if backfilling is disabled.

[eladkal]

That is in line with what I observed! What is weird though is that Airflow would schedule a separate job for every second even if backfilling is disabled.

So yeah it's a bug in croniter - but your cron expression is valid one it's just doesn't say what it should :)
I suggest to open an issue with https://github.com/taichino/croniter/issues/

[eladkal]

@GergelyKalmar can you be more specific about what is the bug?

from airflow.models import DAG
from datetime import datetime
from airflow.operators.bash import BashOperator

with DAG(
    dag_id="16107",
    schedule_interval='5/10 * * * * *',
    start_date=datetime(2018, 1, 1),
    catchup=False
) as dag:

    BashOperator(task_id='try', bash_command='echo {{ ds }}')

It behaves as expected:

[GergelyKalmar]

It works as expected when enabling the DAG but when the next scheduled run comes you should see a lot of instances being created (I think one for every second in the given minute). I know it is not exactly backfilling but it kind of looked like it on first sight (just because of the many instances).

I've checked your job and I could reproduce the weird behavior with https://github.com/aws/aws-mwaa-local-runner.

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[github-actions]

This issue has been closed because it has not received response from the issue author.

[hafid-d]

Upgraded to 2.1.2 and still facing the issue even with catchup_by_default = False