Worker logs not reachable from webserver while running

[ebrard]

Official Helm Chart version

1.7.0 (latest released)

Apache Airflow version

2.2.2

Kubernetes Version

1.23

Helm Chart configuration

No response

Docker Image customisations

No response

What happened

When a task runs on a worker, the web server cannot reach the worker to read the log from.

Considering the workers are deployed as a deployment and not as a statefulset, I suspect that a hostname and a subdomain should be used somehow in the worker template, which is currently not the case.

What you think should happen instead

The web server is able to reach any worker using the headless kubernetes service which is created.

How to reproduce

Any deployment of the helm should have this issue I think.

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

I don't think this is in "all cases" - I think you have a specific problem that manifests this way - because the workers would usually use hostname_callable (https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#hostname-callable) to get their internal name. I suspect what you observe might be the in-famous instabilito of DNS in k8s (which happens quite often adn in some cases it might be even completely broken).

I thnk you need to get deeper into seeing what is your problem (follow the hostname_callable lead). It's likely that you will find the root cause by doing it and likely that it might result in a new feature, but I think more details and digging is need on what your problem is.

For now I am converting it into a discussion. We can always convert it back to issue or you will be able to open a new one.

Also I suggest to try whatever problem you have with latest airflow version 2.4.3 - there have been some improvements in som parts of the hostname_callable handling that might give better diagnostics into what the real problem is.