Skip to content

Conversation

@lfrancke
Copy link
Member

@lfrancke lfrancke commented Jun 11, 2025

Description

The 10.4.3 Kubernetes provider shipped with Airflow 3.0.1 has a problem.
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/10.5.0/changelog.html#id1
It uses a busybox sidecar container to start tasks using the kubernetes executor
Two problems with that:

  1. The securityContext says "runAsNonRoot" but the busybox image defaults to root so it fails to be scheduled (at least on a standard OpenShift)
  2. The busybox image might not be available on air-gapped clusters

I ran a few integration tests on OpenShift and they all passed 🥳

Not sure if this requires a CHANGELOG entry?

Definition of Done Checklist

Note

Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant.

Please make sure all these things are done and tick the boxes

  • Changes are OpenShift compatible
  • Add an entry to the CHANGELOG.md file
  • Integration tests ran successfully
TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

bake --product <product> --image-version <stackable-image-version>
kind load docker-image <image-tagged-with-the-major-version> --name=<name-of-your-test-cluster>

See the output of bake to retrieve the image tag for <image-tagged-with-the-major-version>.

@lfrancke lfrancke moved this to Development: In Progress in Stackable Engineering Jun 11, 2025
@lfrancke lfrancke moved this from Development: In Progress to Development: Waiting for Review in Stackable Engineering Jun 11, 2025
@lfrancke lfrancke self-assigned this Jun 11, 2025
@lfrancke lfrancke force-pushed the feat/update-kubernetes branch from 158144b to 584c411 Compare June 11, 2025 20:14
@adwk67 adwk67 self-requested a review June 12, 2025 06:55
@adwk67 adwk67 moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jun 12, 2025
Copy link
Member

@adwk67 adwk67 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Didn't run on Openshift (as you've tested that already), btu ran selected other tests (all smoke and mount-* tests) successfully. Can confirm that the command is now passed as a json string and not via an initcontainer:

...
  containers:
  - args:
    - python
    - -m
    - airflow.sdk.execution_time.execute_workload
    - --json-string
    - '{"token":"eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIwMTk3NjMyMC01ZWMyLTc3NWQtYWZjNy02YjYyMjgxNGM3MTgiLCJhdWQiOiJ1cm46YWlyZmxvdy5hcGFjaGUub3JnOnRhc2siLCJuYmYiOjE3NDk3MTQ3NjksImV4cCI6MTc0OTcxNTM2OSwiaWF0IjoxNzQ5NzE0NzY5fQ.bvlHrcgSoFSQ9lUp7tVjOGP42EKwz7-2nX_69zK0ep8AqsJKqVIDRHW7lQOXOUwi-OD3e3e_HkYBIQuXvMhgaw","ti":{"id":"01976320-5ec2-775d-afc7-6b622814c718","task_id":"spark_pi_monitor","dag_id":"sparkapp_dag","run_id":"manual__2025-06-12T07:52:35.241768+00:00","try_number":1,"map_index":-1,"pool_slots":1,"queue":"default","priority_weight":1,"parent_context_carrier":{},"context_carrier":{},"queued_dttm":null},"dag_rel_path":"pyspark_pi.py","bundle_info":{"name":"dags-folder","version":null},"log_path":"dag_id=sparkapp_dag/run_id=manual__2025-06-12T07:52:35.241768+00:00/task_id=spark_pi_monitor/attempt=1.log","type":"ExecuteTask"}'
...
    image: oci.stackable.tech/sdp/airflow:3.0.1-stackable0.0.0-dev
    imagePullPolicy: IfNotPresent
    name: base
...

@lfrancke lfrancke added this pull request to the merge queue Jun 12, 2025
Merged via the queue into main with commit 01c6b05 Jun 12, 2025
3 checks passed
@lfrancke lfrancke deleted the feat/update-kubernetes branch June 12, 2025 08:16
@adwk67 adwk67 moved this from Development: In Review to Development: Done in Stackable Engineering Jun 17, 2025
@lfrancke lfrancke moved this from Development: Done to Done in Stackable Engineering Jun 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

3 participants