RuntimeError "Detected recursive loop when walking DAG directory"

[tatiana]

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Disclaimer: This may be considered a bug or a change of behaviour in Airflow.

Airflow is very strict with symbolic links in the DAGs folder that lead to recursive loops:

airflow/airflow/utils/file.py

Line 243 in 965d752

"Detected recursive loop when walking DAG directory "

However, let's say someone accidentally deploys a "bad" folder (e.g. by using astro deploy --dags). To have Airflow raise a RuntimeError and stop parsing all DAGs, which may be critical to a business, could be considered an overreaction. To raise this exception can have a very high cost for business and is very disruptive to Airflow users, as recently experienced by at least one Astronomer customer.

What you think should happen instead?

I believe by default Airflow should raise warnings if it faces symbolic links that lead to recursive links and ignore them. We could expose a configuration for erroring, if users want to keep the current behaviour.

How to reproduce

In an existing working Airflow project, create a symbolic link that leads to a recursive loop, e.g.:

ln -s `pwd`/my-airflow-project/ `pwd`/my-airflow-project/dags/bla

Try to run a DAG that previously worked:

airflow dags test example_dag  `date -Iseconds`

See the exception RuntimeError: Detected recursive loop for being raised.

Operating System

N/A

Versions of Apache Airflow Providers

N/A

Deployment

Astronomer

Deployment details

ZD 57833

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct