You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/modules/demos/pages/airflow-scheduled-job.adoc
+69Lines changed: 69 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -129,6 +129,75 @@ asynchronously - and another to poll the running job to report on its status.
129
129
130
130
image::airflow-scheduled-job/airflow_11.png[]
131
131
132
+
== Patching Airflow to deactivate database initialization
133
+
134
+
By default, Airflow runs database intialization routines on start-up.
135
+
These check that an Admin user exists and that the database schema is up-to-date.
136
+
Since they are idempotent and invoke little overhead, they can be run safely each time for most environments.
137
+
If, however, it makes sense to deactivate this, it can be turned off by patching the running cluster with a resource definition such as this:
138
+
139
+
[source,yaml]
140
+
----
141
+
---
142
+
apiVersion: airflow.stackable.tech/v1alpha1
143
+
kind: AirflowCluster
144
+
metadata:
145
+
name: airflow
146
+
spec:
147
+
clusterConfig:
148
+
dbInit: false # <1>
149
+
----
150
+
<1> Turn off the initialization routine by setting `dbInit` to `false`
151
+
152
+
NOTE: The field `dbInit` is `true` by default to be backwards-compatible.
153
+
A fresh Airflow cluster cannot be created with this field set to `false` as this results in missing metadata in the Airflow database.
154
+
155
+
The demo also created a third DAG in the ConfigMap, called `dag_factory.py`, which was not mounted to the cluster and therefore does not appear in the UI.
156
+
This DAG can be used to create a number of individual DAGs on-the-fly, thus allowing a certain degree of stress-testing of the DAG scan/register steps (the generated DAGs themselves are trivial and so this approach will not really increase the burden of DAG _parsing_).
157
+
To include this is in the list of DAGs (without removing the existing ones), an extra volumeMount is needed, as shown below.
158
+
The patch also sets some environment variables that can used to change the frequency of certain operations. The descriptions can be found here: https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html.
0 commit comments