Unable to read remote logs with celery executor

[lohithasree]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

We migrated from Local Executor to Celery Executor where using RabbitMQ as a broker.

With Local executor, remote logging is working as expected. Able to read logs

Airflow Version: 2.6.1
Celery Version: 5.3.0

apache-airflow-providers-common-sql==1.1.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-google==8.3.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-postgres==5.2.0
apache-airflow-providers-celery==3.2.0

Airflow Config for Remote Logging:

base_log_folder = /opt/airflow/logs
remote_logging = True
remote_log_conn_id = google_cloud_default
delete_local_logs = True
google_key_path = 
remote_base_log_folder = gs://bucket-name/logs
remote_task_handler_kwargs =
encrypt_s3_logs = False
logging_level = INFO
celery_logging_level =
fab_logging_level = WARNING
logging_config_class =
colored_console_log = False
colored_log_format = [%(blue)s%(asctime)s%(reset)s] {%(blue)s%(filename)s:%(reset)s%(lineno)d} %(log_color)s%(levelname)s%(reset)s - %(log_color)s%(message)s%(reset)s
colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s
simple_log_format = %(asctime)s %(levelname)s - %(message)s
dag_processor_log_target = file
dag_processor_log_format = [%(asctime)s] [SOURCE:DAG_PROCESSOR] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s
log_formatter_class = airflow.utils.log.timezone_aware.TimezoneAware
secret_mask_adapter =
task_log_prefix_template =
log_filename_template = dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{% if ti.map_index >= 0 %}map_index={{ ti.map_index }}/{% endif %}attempt={{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location = /opt/airflow/logs/dag_processor_manager/dag_processor_manager.log
task_log_reader = task
extra_logger_names =
worker_log_server_port = 8793
trigger_log_server_port = 8794
file_task_handler_new_folder_permissions = 0o775
file_task_handler_new_file_permissions = 0o664

when we trigger dags, dags are going into queue state and then failed state by stating unable to read the remote logs like below error.

Celery Config:

celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_prefetch_multiplier = 1
worker_enable_remote_control = True
broker_url = amqps://username:password@rabbitmq1-dev.project-name.dev.gcp.cloud.xx:5671/host_name;amqps://username:password@rabbitmq2-dev.project-name.dev.dev.gcp.cloud.xx:5671/host_name;amqps://username:password@rabbitmq3-dev.project-name.dev.gcp.cloud.xx:5671/host_name
flower_host = 0.0.0.0
flower_url_prefix = flower
flower_port = 5555
flower_basic_auth = QW3Q3uWPJqloOhEWv:pvmGvSATFxBMuD4El
sync_parallelism = 0
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
ssl_active = True
ssl_key = /opt/airflow/cert/ddsdev-airflow.uk.hsbc.key
ssl_cert = /opt/airflow/cert/ddsdev-airflow.uk.hsbc.crt
ssl_cacert =
pool = prefork
operation_timeout = 1.0
task_track_started = True
task_publish_max_retries = 3
worker_precheck = False
result_backend = db+postgresql://username:password@localhost:5432/database

Accessing Airflow webserver and Flower UI using NGINX reverse proxy

***** Unable to read remote log from gs://**********/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log *** 404 GET https://storage.googleapis.com/download/storage/v1/b/*********/o/logs%2Fdag_id%3DDDS_catalog_update%2Frun_id%3Dmanual__2023-06-20T15%3A22%3A58.216561%2B00%3A00%2Ftask_id%3Dinsert_dds_ct_start_ts%2Fattempt%3D1.log?alt=media: No such object: *********88/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>) *** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.**

We have verified the object path manually, Its existing. But from airflow we are getting error like above.

We have verified all the permissions and config.

Can you please help us to resolve.

What you think should happen instead

I expected I could read the task log on UI

How to reproduce

While I run any DAGs, the log always shows up this.

***** Unable to read remote log from gs://**********/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log *** 404 GET https://storage.googleapis.com/download/storage/v1/b/*********/o/logs%2Fdag_id%3DDDS_catalog_update%2Frun_id%3Dmanual__2023-06-20T15%3A22%3A58.216561%2B00%3A00%2Ftask_id%3Dinsert_dds_ct_start_ts%2Fattempt%3D1.log?alt=media: No such object: *********88/logs/dag_id=DDS_catalog_update/run_id=manual__2023-06-20T15:22:58.216561+00:00/task_id=insert_dds_ct_start_ts/attempt=1.log: ('Request failed with status code', 404, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>) *** Could not read served logs: Request URL is missing an 'http://' or 'https://' protocol.**

The log files have already exists on worker and store on GCS.

Operating System

RHEL 7

Versions of Apache Airflow Providers

No response

Deployment

Virtualenv installation

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[hussein-awala]

This might not be related to remote logging since the task fails immediately after being sent to the worker. I believe there could be an issue with the parsing of DAG files or the availability of these files in the workers' pods/containers/processes. Have you checked the workers' logs?

I convert the issue to discussion, and we can reopen it if needed.