airflow does not obey AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS=False and still masks passwords

[sreyan32]

Apache Airflow version

Other Airflow 2/3 version (please specify below)

If "Other Airflow 2/3 version" selected, which one?

2.10.2

What happened?

I am using the AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS = False using an -e enviroment variable in an Airflow Podman image.

But I see that this configuration is not respected and I still am getting masked passwords in my Airflow task logs.

How can I fully disable the masking for the task logs ?

What you think should happen instead?

The configuration AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS = False should be respected and all masking should be disabled in the Airflow task logs.

How to reproduce

Start Airflow with

podman run -d --name airflow --network airflow-net --cpus 8 --memory 8192m -e AIRFLOW__CORE__EXECUTOR=LocalExecutor -e AIRFLOW_UID=50000 -e AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS=False -e AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow -v doc-dbt:/opt/airflow/doc-dbt -v airflow-dags:/opt/airflow/dags -v airflow-logs:/opt/airflow/logs -v airflow-plugins:/opt/airflow/plugins -p 8080:8080 apache/airflow:2.10.5 webserver

Create a connection for snowflake_test with a password.

Use the following DAG to test:

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from airflow.models.connection import Connection
from cryptography.hazmat.primitives import serialization
import base64

# Define the Python function
def python_test():
    # Fetch the connection from Airflow secrets
    conn = Connection.get_connection_from_secrets("snowflake_test")
    
    # Extract extra fields
    extra_dict = conn.extra_dejson
    private_key_content = extra_dict.get("private_key_content")  # safer than self._get_field
        
    print(f"private_key_content: {private_key_content}")

    if conn.password:
        print(f"conn.test:{conn.password}")

        passphrase = conn.password.strip().encode()
        
    private_key_pem = base64.b64decode(private_key_content)
        
    
    from typing import Any
    def default_backend() -> Any:
        from cryptography.hazmat.backends.openssl.backend import backend
        return backend

    p_key = serialization.load_pem_private_key(private_key_pem, password=passphrase,backend=default_backend())


# Define the DAG
with DAG(
    dag_id="snowflake_private_key_dag",
    start_date=days_ago(1),
    schedule_interval=None,  # Run on demand
    catchup=False,
    tags=["example", "snowflake"]
) as dag:

    python_task = PythonOperator(
        task_id="print_private_key",
        python_callable=python_test
    )

Operating System

Windows (Podman Fedora Core OS)

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

Using Podman with Fedora Core OS VM.

podman run -d --name airflow --network airflow-net --cpus 8 --memory 8192m -e AIRFLOW__CORE__EXECUTOR=LocalExecutor -e AIRFLOW_UID=50000 -e AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS=False -e AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow -v doc-dbt:/opt/airflow/doc-dbt -v airflow-dags:/opt/airflow/dags -v airflow-logs:/opt/airflow/logs -v airflow-plugins:/opt/airflow/plugins -p 8080:8080 apache/airflow:2.10.2 webserver

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

This is not possible. Airflow will always mask passwords - if you are using "airflow" as a password (which you should not) it will be masked. The description is quite clear about it (it's only about sensitive connection extra keys not passwords).

Hide sensitive Variables or Connection extra json keys from UI and task logs when set to True

Since Airflow 3 you can set minimum length of secret min_length_masked_secret so upgrading to Airflow 3 is a possiblity.

Other than that you can likely follow the path of custom logging and adapt configuration of logger to remove secrets masker that is configured as logging filter (see the custom logging config and alirflow_local_settings.py).

[potiuk]

Also please next time open discussion not an issue - this is something you have quesiton about not an airflow issue.. I am converting it to a discussion now.