You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/airflow/pages/troubleshooting/index.adoc
+95Lines changed: 95 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -50,3 +50,98 @@ webservers:
50
50
----
51
51
52
52
TIP: Our strong recommendation is to increase the webserver replicas, with each webserver running a single worker, as this removes the risk of running into timeouts or memory issues.
53
+
54
+
== Checking DAG syntax (Upgrading to Airflow 3.x+)
55
+
56
+
DAG files that ran under Airflow 2.x may need to be adjusted to be compatible with Airflow 3.x+.
57
+
The https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#installing-and-using-ruff[documentation] shows how this can be done with the Python `ruff` tool.
58
+
For example, the following DAG was compatible with Airflow 2.x:
59
+
60
+
[source,python]
61
+
----
62
+
import pendulum
63
+
from airflow import DAG
64
+
from airflow.decorators import task
65
+
from airflow.operators.bash import BashOperator
66
+
67
+
@task(task_id="run_this")
68
+
def run_this_func(dag_run=None):
69
+
"""
70
+
Print the payload "message" passed to the DagRun conf attribute.
71
+
72
+
:param dag_run: The DagRun object
73
+
:type dag_run: DagRun
74
+
"""
75
+
print(f"Remotely received value of {dag_run.conf.get('message')} for key=message")
bash_command='echo "Here is the message: $message"',
89
+
env={'message': '{% raw %}{{ dag_run.conf.get("message") }}{% endraw %}'},
90
+
)
91
+
----
92
+
93
+
Assume this DAG lies in the `dags` folder in the current directory.
94
+
Testing this with `ruff` indicates one incompatability with Airflow 3.x and one deprecated operator:
95
+
96
+
[source,bash]
97
+
----
98
+
$ ruff check dags/ --select AIR3 --preview
99
+
dags/dag.py:6:2: AIR311 `airflow.decorators.task` is removed in Airflow 3.0; It still works in Airflow 3.0 but is expected to be removed in a future version.
100
+
|
101
+
4 | from airflow.operators.bash import BashOperator
102
+
5 |
103
+
6 | @task(task_id="run_this")
104
+
| ^^^^ AIR311
105
+
7 | def run_this_func(dag_run=None):
106
+
8 | """
107
+
|
108
+
= help: Use `airflow.sdk.task` instead
109
+
110
+
dags/dag.py:20:5: AIR301 [*] `schedule_interval` is removed in Airflow 3.0
dags/dag.py:25:17: AIR312 `airflow.operators.bash.BashOperator` is deprecated and moved into `standard` provider in Airflow 3.0; It still works in Airflow 3.0 but is expected to be removed in a future version.
122
+
|
123
+
23 | run_this = run_this_func()
124
+
24 |
125
+
25 | bash_task = BashOperator(
126
+
| ^^^^^^^^^^^^ AIR312
127
+
26 | task_id="bash_task",
128
+
27 | bash_command='echo "Here is the message: $message"',
129
+
|
130
+
= help: Install `apache-airflow-providers-standard>=0.0.1` and use `airflow.providers.standard.operators.bash.BashOperator` instead.
131
+
132
+
Found 3 errors.
133
+
[*] 1 fixable with the `--fix` option.
134
+
----
135
+
136
+
== PYTHONPATH with custom DAGs folder using python modules
137
+
138
+
When a custom DAG folder (e.g. `/dags`) is defined with `envOverrides` and some DAGs contain a python module structure, then the variable `PYTHONPATH` should be explicitly defined to contain both this folder and the log config location that is set by the operator. This setting is done automatically by the operator when the default DAGs folder or gitsync are used, but is not done when this is set by the user directly.
139
+
140
+
[source,yaml]
141
+
----
142
+
envOverrides: &envOverrides
143
+
AIRFLOW__CORE__DAGS_FOLDER: "/dags"
144
+
PYTHONPATH: "/stackable/app/log_config:/dags"
145
+
----
146
+
147
+
NOTE: Generally speaking it is https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#configuration-reference[recommended] by Airflow to have the same config everywhere across all components.
0 commit comments