chore(airflow): Update container images ahead of Stackable Release 24.11

[NickLarsenNZ]

Part of #831.

• Add: 2.10.2 (2.9.3 remains the latest/LTS, and was already added in chore: Bump airflow to 2.9.3 #809).
• Remove: 2.6.x, 2.8.x (already done in chore: Bump airflow to 2.9.3 #809).

Tip

Please add the scheduled-for/20XX-XX label.

### Update tasks
- [x] Update `versions.py` to reflect the agreed upon versions in the spreadsheet (including the removal of old versions).
- [x] Download new constraints file (see `airflow/download_constraints.sh`).
- [x] Update other dependencies if applicable (eg: python, statsd_exporter, etc).
- [ ] Check other operators (getting_started / kuttl) for usage of the versions. Add the PR(s) to the list below.
- [ ] Update the version in demos. Add the PR(s) to the list below.
- [ ] Research memory OOM problems in demo when using Airflow 2.10.2 -> Maybe increase default memory for webservers

### Related Pull Requests
- [ ] https://github.com/stackabletech/docker-images/pull/877
- [ ] https://github.com/stackabletech/airflow-operator/pull/512

### Acceptance
- [x] Can build image locally
- [x] Kuttl smoke tests passes locally
- [ ] Release notes written in a comment below
- [ ] Applicable `release-note` label added to this issue

Testing instructions

# See the latest version at https://pypi.org/project/image-tools-stackabletech/
pip install image-tools-stackabletech==0.0.12

bake --product airflow=x.y.z # where x.y.z is the new version added in this PR

kind load docker-image docker.stackable.tech/stackable/airflow:x.y.z-stackable0.0.0-dev

# Change directory into the airflow-operator repository and update the
# product version in tests/test-definition.yaml
./scripts/run-tests --test-suite smoke-latest # or similar

Please consider updating this template if these instructions are wrong, or
could be made clearer.

[NickLarsenNZ]

I see a new statsd_exporter version available. Will update that in a separate PR.

[NickLarsenNZ]

Smoke tests pass:

--- PASS: kuttl (423.52s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_airflow-2.10.2_openshift-false_executor-kubernetes (423.49s)
PASS

[NickLarsenNZ]

airflow-operator getting_started passes, but will leave it at the current stable version.

diff

diff --git a/docs/modules/airflow/examples/getting_started/code/airflow.yaml b/docs/modules/airflow/examples/getting_started/code/airflow.yaml
index 9389942..9165718 100644
--- a/docs/modules/airflow/examples/getting_started/code/airflow.yaml
+++ b/docs/modules/airflow/examples/getting_started/code/airflow.yaml
@@ -5,7 +5,8 @@ metadata:
   name: airflow
 spec:
   image:
-    productVersion: 2.9.3
+    productVersion: 2.10.2
+    pullPolicy: Never
   clusterConfig:
     loadExamples: true
     exposeConfig: false

[NickLarsenNZ]

The demo does not work with 2.10.2, that will need some work.

In case it helps, @adwk67:

1. Launching the cluster via stackablectl demo install airflow-scheduled-job, but then there is a failed job in the UI (with very little detail in the logs).
2. There appears to be a new button in the UI next to each DAG, which lets you re-parse the DAGs. After hitting that, the Airflow UI is unreachable, and stackablectl stacklet list shows the cluster as Reconciling.

diff

diff --git a/stacks/airflow/airflow.yaml b/stacks/airflow/airflow.yaml
index 5e8b96c..7b12ce0 100644
--- a/stacks/airflow/airflow.yaml
+++ b/stacks/airflow/airflow.yaml
@@ -6,7 +6,7 @@ metadata:
   name: airflow
 spec:
   image:
-    productVersion: 2.9.3
+    productVersion: 2.10.2
   clusterConfig:
     listenerClass: external-unstable
     loadExamples: false
diff --git a/stacks/stacks-v2.yaml b/stacks/stacks-v2.yaml
index 12a55e9..b51ea27 100644
--- a/stacks/stacks-v2.yaml
+++ b/stacks/stacks-v2.yaml
@@ -85,7 +85,7 @@ stacks:
         default: adminadmin
   airflow:
     description: Stack containing Airflow scheduling platform
-    stackableRelease: 24.7
+    stackableRelease: dev
     stackableOperators:
       - commons
       - listener
@@ -95,9 +95,9 @@ stacks:
     labels:
       - airflow
     manifests:
-      - helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/postgresql-airflow.yaml
-      - helmChart: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/_templates/redis-airflow.yaml
-      - plainYaml: https://raw.githubusercontent.com/stackabletech/demos/main/stacks/airflow/airflow.yaml
+      - helmChart: ./stacks/_templates/postgresql-airflow.yaml
+      - helmChart: ./stacks/_templates/redis-airflow.yaml
+      - plainYaml: ./stacks/airflow/airflow.yaml
     supportedNamespaces: []
     resourceRequests:
       cpu: 3400m

[sbernauer]

I would prefer to keep the Issue open until the problem is solved to not forget about it. We don't need to ship Airflow 2.10 with SDP 24.11, although I would prefer investing the time (at a min. 1-2 days) to properly research the problem

[sbernauer]

Gave it a very quick try (installed the airflow-scheduled-job demo) and updated to 2.10.2. Everything worked perfectly fine, even the DAG re-parsing. At some point of time the webserver OOMed. Did yours too?
Maybe we only need to give Airflow a bit more memory and everything works