Airflow DAG deployment delaying by Hours

[vthorwat]

Hello Team,

We are currently using Airflow 2.0.1 version in our environment. All of a sudden some DAG deployments are taking unusual timeline to deploy the dag ( ~approximately 2 hours).

We have not changed anything on the servers or anything in our regular DAGs. We have our custom logic created for DAG deployment in our environment.

Please find the attached configuration log screenshot for review.

[vthorwat]

We have written a custom script that uses a dag template file and just adds the schedule interval and path of the file to be scheduled. once this file is created according to the template it is placed in the dags directory of airflow

[potiuk]

Looks like it's some change on your system. But it's rathre not airflow related but whatever you have in your deployment. You need to look there.

It's unrelated - you already miss out on a big number (in couple of hundreds) of bug and security fixes in airflow and its providers (not mentioning . currently Airlfow providers are Airflow 2.4 only. You should definitely upgrade (and the more often you upgrade, the better and the easier it is to upgrade, so I recommend upgrade-early, upgrade-often). The longer you wait, the more cost of not-upgrading grows way faster than the cost of upgrading.