Task logging does not show in the task log

[Haapalaj]

Apache Airflow version

3.1.5

If "Other Airflow 3 version" selected, which one?

No response

What happened?

According to this document:
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/logging-monitoring/logging-tasks.html

Logging inside python code executed in a task, this should work and has been working e.g. with airflow version 2.11.0:

import logging
logger = logging.getLogger(__name__)
logger.info("This is a log message")

The airflow LOGGING_LEVEL is set to INFO.

But only if giving e.g.
logger.log(55, 'The log msg content')
Then it would show up in the task log.
Seems like the logging level is somehow set wrong.

What you think should happen instead?

If logging like this inside python code executed by the airflow task, the log entries should be seen in the task log output:

import logging
logger = logging.getLogger(__name__)
logger.info("This is a log message")

How to reproduce

Create a DAG and a task and add logging to it as described:

import logging
logger = logging.getLogger(__name__)
logger.info("This is a log message")

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

airflow 3.1.5 (Image apache/airflow:3.1.5-python3.12
apache-airflow-providers-standard==1.10.1
apache-airflow-providers-snowflake==6.8.0
apache-airflow-providers-amazon==9.18.1
apache-airflow-providers-google==19.2.0
apache-airflow-providers-microsoft-azure==12.10.0
apache-airflow-providers-odbc==4.11.0
apache-airflow-providers-postgres==6.5.1
apache-airflow-providers-databricks==7.8.1
apache-airflow-task-sdk==1.1.5
apache-airflow-providers-fab==3.1.0

python3.12

Deployment

Official Apache Airflow Helm Chart

Deployment details

Docker compose

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jroachgolf84]

Can you share a snip of your code where you've written the logging statements?

[Haapalaj]

Here is an example code to see where the logging fails on an Airflow task level implementation:

An custom operator having the task level logging. Both init and execute phase logging has been working with the version 2.x Airflow, but not with 3.x

# An custom operator with logging
import logging
from airflow.sdk import BaseOperator

log = logging.getLogger(__name__)

class MyOperator(BaseOperator):
    def __init__(self, myparams,
                 *args, **kwargs):
        self.myparams = myparams
        super(MyOperator, self).__init__(*args, **kwargs)
        log.info("Init myoperator")
    def execute(self, context):
        log.info("Execute myoperator")

How the custom operator is used.

# Then creating DAG with task like:
from airflow.sdk import DAG
dag = DAG(dag_id="MyDag")
task = MyOperator(task_id="MyTask", dag=dag)

[A1shalia]

I'm new to open source contributions and would be willing to take on this task.

[jroachgolf84]

@A1shalia, do you have an idea of what a fix would look like here? Were you able to reproduce this error?

[rajatmohan22]

A1shalia, do let me know if you need any help.
@jroachgolf84 I was actually able to reproduce the error.

I think.. (not very sure)
in Airflow 3.1.5, user loggers created with logging.getLogger(name) default to level NOTSET and it rely on the parent logger levels. They may not propagate to the airflow.task handlers, or their effective level may be too high, causing the info messages to be filtered out

[rajatmohan22]

A fix would probably look like something like:

Atleast a starting point:

To ensure user loggers propagate or have an explicit level during task execution. we can do this by configuring the root logger level and ensuring propagation to task handlers when a task is about to start

[Ankurdeewan]

From a quick look, this seems to stem from the logging isolation introduced in Airflow 3 / Task SDK, where only task-scoped loggers (self.log) are guaranteed to be captured. Module-level loggers that worked in 2.x can now fail silently during task execution.

While this is arguably expected behavior, the silent log loss is pretty surprising from a DX standpoint, especially for users migrating from 2.x. At minimum, it feels like we should guide users when this happens (e.g. a runtime warning or clearer documentation), rather than dropping logs with no signal.

I’m happy to take a stab at improving this, if that direction sounds reasonable.

[Haapalaj]

From a quick look, this seems to stem from the logging isolation introduced in Airflow 3 / Task SDK, where only task-scoped loggers (self.log) are guaranteed to be captured. Module-level loggers that worked in 2.x can now fail silently during task execution.

While this is arguably expected behavior, the silent log loss is pretty surprising from a DX standpoint, especially for users migrating from 2.x. At minimum, it feels like we should guide users when this happens (e.g. a runtime warning or clearer documentation), rather than dropping logs with no signal.

I’m happy to take a stab at improving this, if that direction sounds reasonable.

Thanks!
So would it be that for task logging, either use:

from airflow.utils.log.logging_mixin import LoggingMixin
LOG = LoggingMixin().log

Or
self.log ?

I would prefer to have option to use the same way as with the Airflow 2.x

[Ankurdeewan]

In Airflow 3.x, task execution runs in an isolated context via the Task SDK, and only task-scoped loggers are guaranteed to be captured in task logs.

For operators, the supported and recommended way to log is using self.log. This logger is explicitly tied to the task instance and is correctly routed to task logs.

LoggingMixin can be used for non-operator helper or utility classes, but it is not a replacement for task-scoped logging inside operators.

Module-level loggers created via logging.getLogger(name), which worked in Airflow 2.x, are no longer guaranteed to be captured during task execution in Airflow 3.x due to this isolation.

I agree that the silent loss of such logs can be confusing for users migrating from 2.x, and this behavior likely needs clearer documentation or guidance, but functionally self.log is the correct approach going forward.