You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/demos/pages/airflow-scheduled-job.adoc
+10-5Lines changed: 10 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -129,7 +129,7 @@ asynchronously - and another to poll the running job to report on its status.
129
129
130
130
image::airflow-scheduled-job/airflow_11.png[]
131
131
132
-
== Patching Airflow to deactivate database initialization
132
+
== Patching Airflow to stress-test DAG parsing using relevant environment variables
133
133
134
134
By default, Airflow runs database intialization routines on start-up.
135
135
These check that an Admin user exists and that the database schema is up-to-date.
@@ -145,13 +145,17 @@ metadata:
145
145
name: airflow
146
146
spec:
147
147
clusterConfig:
148
-
dbInit: false # <1>
148
+
databaseInitialization:
149
+
enabled: false # <1>
149
150
----
150
-
<1> Turn off the initialization routine by setting `dbInit` to `false`
151
+
<1> Turn off the initialization routine by setting `databaseInitialization.enabled` to `false`
151
152
152
-
NOTE: The field `dbInit` is `true` by default to be backwards-compatible.
153
+
NOTE: The field `databaseInitialization.enabled` is `true` by default to be backwards-compatible.
153
154
A fresh Airflow cluster cannot be created with this field set to `false` as this results in missing metadata in the Airflow database.
154
155
156
+
WARNING: Setting `databaseInitialization.enabled` to `false` is an unsupported operation as subsequent updates to a running Airflow cluster can result in broken behaviour due to inconsistent metadata.
157
+
Only set `databaseInitialization.enabled` to `false` if you know what you are doing!
158
+
155
159
The demo also created a third DAG in the ConfigMap, called `dag_factory.py`, which was not mounted to the cluster and therefore does not appear in the UI.
156
160
This DAG can be used to create a number of individual DAGs on-the-fly, thus allowing a certain degree of stress-testing of the DAG scan/register steps (the generated DAGs themselves are trivial and so this approach will not really increase the burden of DAG _parsing_).
157
161
To include this is in the list of DAGs (without removing the existing ones), an extra volumeMount is needed, as shown below.
0 commit comments