S3TaskHandler doesn't seem to run the close method and doesn't push the logs to S3

[chyzouakia]

Apache Airflow Provider(s)

amazon

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.29.0

Apache Airflow version

2.10.5

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Deployment details

Deployment through the Helm chart https://airflow-helm.github.io/charts version 8.9.0.
Airflow image : slim-2.10.5-python3.10
My deployment creates a Service Account to allow to connect to our AWS S3 to push the logs there.

Configuration for logging :
AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://{{ bucket }}/airflow/logs"
AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_remote_logging"

What happened

It seems the close() method in S3TaskHandler is not called. I don't have any log stored in S3.
I am able to read a file from S3 :

I am able to push a log from the worker manually by running this code :

from airflow.providers.amazon.aws.hooks.s3 import S3Hook
s3_hook=S3Hook(aws_conn_id="s3_remote_logging")
import os
import pathlib
log = pathlib.Path([LOCAL_PATH]).read_text()
s3_hook.load_string(log, key=[S3_KEY], replace=True, encrypt=False)

What you think should happen instead

When closing the task, The Task Handler should have pushed the logs to my S3 bucket.

How to reproduce

• Run the Airflow image slim-2.10.5-python3.10 with the amazon provider package
• Configure Airflow to push the logs to S3 (the connection should be created) :
  • AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER: "s3://{{ bucket }}/airflow/logs"
  • AIRFLOW__LOGGING__REMOTE_LOGGING: "True"
  • AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID: "s3_remote_logging"
• Run a task and see if the code pushes the logs to S3.

Anything else

This problem is occurring on all my tasks now. Please let me know if I misconfigured something. Thanks !

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

My guess is something wrong with the chart, maybe the variables are not passed as you think they are - but you should check there - this chart is not managed by us. This works if you configure it locally in general (a lot of people would complain if not) - so something must be problematic in the configuration of the chart, or you have some typo. Generally speaking you should find information in the local logs (writtent in the place where the chart configures them to be written) and you should look there for clues.