Using --celery-hostname causes workers to reserve, but never execute tasks

[CreeperBeatz]

Apache Airflow Provider(s)

celery

Versions of Apache Airflow Providers

3.14.1

Apache Airflow version

3.1.5

Operating System

WSL Ubuntu 22.04

Deployment

Virtualenv installation

Deployment details

Installed with UV:

uv pip install "apache-airflow[celery]==3.1.5" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.1.5/constraints-3.12.txt"

What happened

When starting a Celery worker with a custom --celery-hostname, the worker successfully connects to the broker and reserves tasks, but never executes them.

The task stays in reserved state with:

acknowledged: False
worker_pid: None
time_start: None

Removing --celery-hostname and using the default hostname makes everything work correctly.

Output when using --celery-hostname:

(.venv) me@DESKTOP-MAG5124:~/airflow$ celery -A airflow.providers.celery.executors.celery_executor_utils.app inspect reserved
->  default@DESKTOP-MAG5124: OK
    * {'id': '923f016a-a3da-42c2-bf57-4f4193e11721', 'name': 'execute_workload', 'args': ['{"token":"token_here","task_id":"get_airflow_variables","dag_id":"dirbg_crawler","run_id":"manual__2025-12-22T07:57:45+00:00","try_number":2,"map_index":-1,"pool_slots":1,"queue":"default","priority_weight":2,"parent_context_carrier":{},"context_carrier":{}},"dag_rel_path":"dirbg_crawler.py","bundle_info":{"name":"dags-folder","version":null},"log_path":"dag_id=dirbg_crawler/run_id=manual__2025-12-22T07:57:45+00:00/task_id=get_airflow_variables/attempt=2.log","type":"ExecuteTask"}'], 'kwargs': {}, 'type': 'execute_workload', 'hostname': 'default@DESKTOP-MAG5124', 'time_start': None, 'acknowledged': False, 'delivery_info': {'exchange': '', 'routing_key': 'default', 'priority': 0, 'redelivered': False}, 'worker_pid': None}

1 node online.

(.venv) me@DESKTOP-MAG5124:~/airflow$ celery -A airflow.providers.celery.executors.celery_executor_utils.app inspect active_queues
->  default@DESKTOP-MAG5124: OK
    * {'name': 'default', 'exchange': {'name': 'default', 'type': 'direct', 'arguments': None, 'durable': True, 'passive': False, 'auto_delete': False, 'delivery_mode': None, 'no_declare': False}, 'routing_key': 'default', 'queue_arguments': None, 'binding_arguments': None, 'consumer_arguments': None, 'durable': True, 'exclusive': False, 'auto_delete': False, 'no_ack': False, 'alias': None, 'bindings': [], 'no_declare': None, 'expires': None, 'message_ttl': None, 'max_length': None, 'max_length_bytes': None, 'max_priority': None}

1 node online.

Output when NOT using --celery-hostname:

Note: The task is already executing.

(.venv) dani@DESKTOP-MAG5124:~/airflow$ celery -A airflow.providers.celery.executors.celery_executor_utils.app inspect reserved
->  celery@DESKTOP-MAG5124: OK
    - empty -

What you think should happen instead

Worker should execute reserved tasks regardless of custom hostname.

How to reproduce

This does NOT work - task reserved but never executed

airflow celery worker \
  --queues my_queue \
  --concurrency 1 \
  --celery-hostname "myworker@%h"

This is valid even if you set the --celery-hostname to "celery@%h" which should be the default option.

This WORKS

airflow celery worker \
  --queues my_queue \
  --concurrency 1

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[CreeperBeatz]

I just tested downgrading to Celery Providers 3.13.1, and the issue DOES NOT exist there. This is something that happened in the 3.14.* release of the celery providers.

[brunocmacedo]

After a couple hours trying to find what was the issue after 3.1.5 upgrade I found this and can confirm that I have the same issue with apache-airflow-providers-celery 3.14.1. Falling back to 3.13.1 solved the issue for now.

As a reference, here is the airflow-worker definition in my docker compose. In my case i need to set celery-hostname due to the three replicas being used.

airflow-worker:
        <<: *airflow-common
        command: celery worker --celery-hostname=%h
        healthcheck:
            test:
                - "CMD-SHELL"
                - 'celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}" || celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
            interval: 30s
            timeout: 10s
            retries: 5
            start_period: 30s
        environment:
            <<: *airflow-common-env
            # Required to handle warm shutdown of the celery workers properly
            # See https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
            DUMB_INIT_SETSID: "0"
        restart: always
        depends_on:
            <<: *airflow-common-depends-on
            airflow-apiserver:
                condition: service_healthy
            airflow-init:
                condition: service_completed_successfully
        deploy:
            replicas: 3 # Number of worker replicas to run, increase this to create multiple worker containers allowing more parallel task execution
        mem_swappiness: 0 # Disable swap to prevent slowdowns