Add config option [secrets]backends_order #45931

moiseenkov · 2025-01-22T13:03:26Z

Introduce a new configuration option for specifying secret backends load order:

[secrets]backends_order = custom,environment_variable,metastore

The default value represents current behavior, thus nothing will change for existing users.

airflow/config_templates/config.yml

moiseenkov · 2025-01-23T08:29:35Z

@eladkal , please take a look at the updates.

potiuk · 2025-01-25T19:46:16Z

I was initially against making it configurable, but seeing the simplicity and flexibility, I am in.

potiuk · 2025-01-25T19:46:32Z

@eladkal ?

VladaZakharova · 2025-02-06T10:29:00Z

hi there!
@potiuk
Can we merge this one please?

VladaZakharova · 2025-02-13T10:40:54Z

Hi @potiuk @eladkal ! Are there some other changes we need to make here? Or we can merge this one?

eladkal · 2025-02-13T15:05:12Z

We are on feature freeze for Airflow 3.
https://lists.apache.org/thread/r26htzl0w3th7pw0l1y31g6s14qbtwwt

potiuk · 2025-02-15T20:43:55Z

Yeah. I think that might be 3.1

eladkal

prevent accidental merge. We are on feature freeze for Airflow 3.
PR can not be merged till main branch is for 3.1

eladkal · 2025-03-31T13:10:09Z

airflow-core/src/airflow/config_templates/config.yml

+    backends_order:
+      description: |
+        Comma-separated list of secret backends. These backends will be used in the order they are specified.
+        Please note that the `environment_variable` and `metastore` are required values and cannot be removed
+        from the list. Supported values are:
+
+        * ``custom``: Custom secret backend specified in the ``secrets[backend]`` configuration option.
+        * ``environment_variable``: Standard environment variable backend
+          ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+        * ``metastore``: Standard metastore backend ``airflow.secrets.metastore.MetastoreBackend``.
+      version_added: 3.0.0
+      type: string
+      example: ~
+      default: "custom,environment_variable,metastore"


i am a bit concerned making it just a "hidden" setting.
I think we better to come up with a way to expose the chosen order in the UI. that way DAG authors can verify and use it for debug for questions like (why I see wrong value in variable).

Also, correct me if I am wrong the order affect only read, not write. maybe the setting should be backends_read_order?

Hello @eladkal ! I don't see any info regarding the secrets section the UI is using right now. So maybe we could skip it for now? What do you think?
Regarding the naming. As stated in the doc it is responsible for secrets backends search ordering. So maybe backends_search_order is more convenient?

right now it's not configurable so the Airflow docs is good enough but after this PR is accepted the Airflow docs can't help you... You'll need to check the deployment to understand the priority order.
In many deployments dag authors don't have access to the deployment code.

My thoughts on the UI is an improvement (I think crucial but that is just my own perspective) I welcome others to share their thoughts. This feature is already targeting 3.1+ so regardless if we do the UI part or not we can't merge it now. We need to wait for main branch to become 3.1 (probably only after we cut RC1 for Airflow 3(

Hi there!
I see that we are close to release Af3.1, so maybe we can merge this change?
Thanks :)

potiuk · 2025-07-01T08:48:38Z

This looks good to me - but I think it might be worth to raise a devlist discussion for it @VladaZakharova -> there were past discussions about changing the sequence of resolving configurations, and I know people have strong opinion about "fixed" vs. "confifurable" sequence - and there are arguments pro / against each of those options.

I think it would be good to raise a discussion asking what peopel think about it and try to reach consensus.

eladkal · 2025-07-01T10:45:42Z

I think it would be good to raise a discussion asking what peopel think about it and try to reach consensus.

I agree. I am not comfortable with making this change without the UI indication / other mechanisem that allows dag authors to see the cluster admin setup for backend order

amoghrajesh · 2025-10-22T12:53:47Z

@moiseenkov looking at it

VladaZakharova · 2025-10-22T12:56:31Z

@moiseenkov looking at it
He is not, because he doesn't work in our team anymore, @Crowiant is the one responsible for it, so that's why he is requesting review from you

amoghrajesh · 2025-10-22T12:59:29Z

@VladaZakharova I meant I am looking at it.

amoghrajesh

I have some comments re the design of the config.

I would also not add the UI portion in the same PR, maybe a orthogonal one would be the place to add it.

amoghrajesh · 2025-10-22T13:16:20Z

airflow-core/src/airflow/configuration.py

+    required_backends = ["environment_variable"] if worker_mode else ["metastore", "environment_variable"]
+    if missing_backends := [b for b in required_backends if b not in backends_order]:


Do you think defining an Enum here would make things cleaner?

class Backend(Enum): ENVIRONMENT_VARIABLE = "environment_variable" METASTORE = "metastore" CUSTOM = "custom"

The hardcoded strings are error prone imo

Yes, I agree, added to the PR.

amoghrajesh · 2025-10-22T13:17:42Z

airflow-core/tests/unit/always/test_secrets.py

+    @conf_vars(
+        {
+            (
+                "secrets",
+                "backend",
+            ): "airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend",
+            ("secrets", "backend_kwargs"): '{"connections_prefix": "/airflow", "profile_name": null}',
+            ("secrets", "backends_order"): "custom,environment_variable,metastore",
+        }
+    )
+    def test_backends_order(self):
+        backends = ensure_secrets_loaded()
+        backend_classes = [backend.__class__.__name__ for backend in backends]
+        assert backend_classes == [
+            "SystemsManagerParameterStoreBackend",
+            "EnvironmentVariablesBackend",
+            "MetastoreBackend",
+        ]
+
+    @pytest.mark.parametrize(
+        "backends_order",
+        [
+            pytest.param("custom,metastore", id="no_environment_variable_backend"),
+            pytest.param("environment_variable", id="no_metastore_backend"),
+            pytest.param("metastore,environment_variable,unsupported", id="unsupported_backend"),
+        ],
+    )
+    def test_backends_order_invalid_cases(self, backends_order):
+        with conf_vars({("secrets", "backends_order"): backends_order}):
+            with pytest.raises(AirflowConfigException):
+                ensure_secrets_loaded()


Needs more test coverage to cover the cases for worker mode too. This only checks for "non workers"

amoghrajesh · 2025-10-22T13:22:49Z

airflow-core/src/airflow/config_templates/config.yml

+    backends_order:
+      description: |
+        Comma-separated list of secret backends. These backends will be used in the order they are specified.
+        Please note that the `environment_variable` and `metastore` are required values and cannot be removed
+        from the list. Supported values are:
+
+        * ``custom``: Custom secret backend specified in the ``secrets[backend]`` configuration option.
+        * ``environment_variable``: Standard environment variable backend
+          ``airflow.secrets.environment_variables.EnvironmentVariablesBackend``.
+        * ``metastore``: Standard metastore backend ``airflow.secrets.metastore.MetastoreBackend``.
+      version_added: 3.0.0
+      type: string
+      example: ~
+      default: "custom,environment_variable,metastore"


On giving this some more thought, I am not very convinced that having the config under "secrets" is the right thing to do.

For someone wanting to configure workers backend, they have to set this:

[secrets] backends_order, [workers] secrets_backend which is not at all intuitive and is confusing.

We should consider having backends_order as a config for workers too, maybe: [workers] backends_order which plays nice with workers backend and the separate one that we have for secrets can be used there.

Yeah, I agree. Added [workers]backends_order as a separate option that could be configured.

Crowiant · 2025-11-03T16:44:29Z

Hello @amoghrajesh Thank you for your review!

I have some comments re the design of the config.

I would also not add the UI portion in the same PR, maybe a orthogonal one would be the place to add it.

Regarding UI: as it was mentioned here: #45931 (comment) UI should be introduced in this PR

Crowiant · 2025-11-17T12:47:52Z

Hello @amoghrajesh Could you please review this PR? I responded to your comments.

VladaZakharova · 2025-11-19T11:35:07Z

hi @amoghrajesh :)
can you please check if we need to add additional changes here in PR? thanks :)

amoghrajesh · 2025-11-20T06:36:18Z

@VladaZakharova @moiseenkov @Crowiant I will take a look at this one soon.

uranusjr · 2025-11-21T07:58:09Z

I wonder if this should just be called backends and have the description explain how the ordering is significant. The implementation is fine to me.

uranusjr · 2025-11-21T07:59:39Z

airflow-core/docs/security/secrets/secrets-backend/index.rst

    secrets_backend =
    secrets_backend_kwargs =
-
+    backends_order =


This doesn’t read right? The key implemented here is not under [workders].

Hello @uranusjr ! Thank you for your review and comments!
No, actually it should be under both sections: [secrets] and [workers]. Because after talking with @amoghrajesh We agreed that implementation should be extended on workers too.

amoghrajesh

We are certainly close on this one, some more work needs to be done to adjust the PR as per the latest codebase. For the UI changes, I am not so sure, so maybe someone else can review that: @potiuk / @eladkal since you had some suggestion there.

amoghrajesh · 2025-11-25T07:28:49Z

airflow-core/docs/security/secrets/secrets-backend/index.rst

 You can provide ``backend_kwargs`` with json and it will be passed as kwargs to the ``__init__`` method of
 your secrets backend.

+``backends_order`` comma-separated list of secret backends. These backends will be used in the order they are specified.


Suggested change

``backends_order`` comma-separated list of secret backends. These backends will be used in the order they are specified.

``backends_order`` is a comma-separated list of secret backends. These backends will be used in the order they are specified.

amoghrajesh · 2025-11-25T07:29:15Z

airflow-core/docs/security/secrets/secrets-backend/index.rst

 You can provide ``secrets_backend_kwargs`` with json and it will be passed as kwargs to the ``__init__`` method of
 your secrets backend for the workers.

+``backends_order`` comma-separated list of secret backends for workers. These backends will be used in the order they are specified.


Suggested change

``backends_order`` comma-separated list of secret backends for workers. These backends will be used in the order they are specified.

``backends_order`` is a comma-separated list of secret backends for workers. These backends will be used in the order they are specified.

amoghrajesh · 2025-11-25T07:31:16Z

airflow-core/src/airflow/configuration.py

 def ensure_secrets_loaded(
-    default_backends: list[str] = DEFAULT_SECRETS_SEARCH_PATH,
+    default_backends: list[str] | None = None,
 ) -> list[BaseSecretsBackend]:
    """
    Ensure that all secrets backends are loaded.

    If the secrets_backend_list contains only 2 default backends, reload it.
    """
    # Check if the secrets_backend_list contains only 2 default backends.

-    # Check if we are loading the backends for worker too by checking if the default_backends is equal
-    # to DEFAULT_SECRETS_SEARCH_PATH.
-    if len(secrets_backend_list) == 2 or default_backends != DEFAULT_SECRETS_SEARCH_PATH:
+    # Check if we are loading the backends for worker too by checking if the default_backends is not None
+    if len(secrets_backend_list) == 2 or default_backends is not None:
        return initialize_secrets_backends(default_backends=default_backends)
    return secrets_backend_list


With: #57744 merged, there is a shared config parser now. You will have to update this in sdk/configuration.py too

Hello @amoghrajesh ! Thank you for mentioning the PR, I updated the code. Please check when you have time!

amoghrajesh · 2025-11-25T07:32:13Z

airflow-core/src/airflow/configuration.py

+        execution_args = (
+            "airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend"
+            if "airflow.sdk.execution_time.secrets.execution_api.ExecutionAPISecretsBackend"
+            in default_backends
+            else None
+        )


We are trying to avoid importing sdk in airflow core when possible

Could you expand a little bit on this? ExecutionAPISecretsBackend exists only in task-sdk and applies for the workers. Where else could it be imported?

moiseenkov requested review from ashb, dstandish and potiuk as code owners January 22, 2025 13:03

boring-cyborg bot added the area:secrets label Jan 22, 2025

eladkal requested changes Jan 22, 2025

View reviewed changes

airflow/config_templates/config.yml Outdated Show resolved Hide resolved

airflow/config_templates/config.yml Show resolved Hide resolved

moiseenkov force-pushed the secrets-backends-order branch from 61b5ab6 to 4489808 Compare January 22, 2025 13:38

moiseenkov requested a review from eladkal January 22, 2025 13:45

moiseenkov force-pushed the secrets-backends-order branch from 4489808 to 58d80f3 Compare January 22, 2025 14:35

potiuk approved these changes Jan 25, 2025

View reviewed changes

raphaelauv mentioned this pull request Mar 11, 2025

Add a way to skip the secret_backend #19251

Open

2 tasks

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from 8c516f8 to 347e2f1 Compare March 31, 2025 13:00

eladkal requested changes Mar 31, 2025

View reviewed changes

eladkal added this to the Arflow 3.1+ milestone Mar 31, 2025

eladkal reviewed Mar 31, 2025

View reviewed changes

eladkal modified the milestones: Airflow 3.1+, Airflow 3.1.0 Apr 21, 2025

Crowiant force-pushed the secrets-backends-order branch from 347e2f1 to 2b750de Compare April 24, 2025 12:21

Crowiant force-pushed the secrets-backends-order branch from 2b750de to 4c0d958 Compare May 23, 2025 14:52

potiuk requested a review from eladkal June 30, 2025 21:02

amoghrajesh reviewed Oct 22, 2025

View reviewed changes

Crowiant force-pushed the secrets-backends-order branch from 8aae4af to 780bbd2 Compare November 3, 2025 16:17

Crowiant requested a review from guan404ming as a code owner November 3, 2025 16:17

Crowiant force-pushed the secrets-backends-order branch 5 times, most recently from 368a70c to 1ce52cc Compare November 5, 2025 10:10

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from 159dd6e to 64d53e7 Compare November 17, 2025 10:08

uranusjr reviewed Nov 21, 2025

View reviewed changes

amoghrajesh reviewed Nov 25, 2025

View reviewed changes

potiuk mentioned this pull request Dec 6, 2025

Fix : 'airflow connections test' failure by adding DB fallback #59038

Closed

Crowiant force-pushed the secrets-backends-order branch from 64d53e7 to 437208f Compare December 17, 2025 13:41

Crowiant requested a review from vatsrahul1001 as a code owner December 17, 2025 13:41

Crowiant force-pushed the secrets-backends-order branch 3 times, most recently from 785af81 to 2015d8b Compare December 22, 2025 17:14

Add config option [secrets]backends_order

a5092ed

Crowiant force-pushed the secrets-backends-order branch from 2015d8b to a5092ed Compare January 8, 2026 21:42

Crowiant requested a review from choo121600 as a code owner January 8, 2026 21:42

		required_backends = ["environment_variable"] if worker_mode else ["metastore", "environment_variable"]
		if missing_backends := [b for b in required_backends if b not in backends_order]:

	``backends_order`` comma-separated list of secret backends. These backends will be used in the order they are specified.
	``backends_order`` is a comma-separated list of secret backends. These backends will be used in the order they are specified.

Add config option [secrets]backends_order #45931

Are you sure you want to change the base?

Add config option [secrets]backends_order #45931

Uh oh!

Conversation

moiseenkov commented Jan 22, 2025

Uh oh!

Uh oh!

Uh oh!

moiseenkov commented Jan 23, 2025

Uh oh!

potiuk commented Jan 25, 2025

Uh oh!

potiuk commented Jan 25, 2025

Uh oh!

VladaZakharova commented Feb 6, 2025

Uh oh!

VladaZakharova commented Feb 13, 2025

Uh oh!

eladkal commented Feb 13, 2025

Uh oh!

potiuk commented Feb 15, 2025

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

potiuk commented Jul 1, 2025

Uh oh!

eladkal commented Jul 1, 2025

Uh oh!

amoghrajesh commented Oct 22, 2025

Uh oh!

VladaZakharova commented Oct 22, 2025

Uh oh!

amoghrajesh commented Oct 22, 2025

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Crowiant commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Crowiant commented Nov 17, 2025

Uh oh!

VladaZakharova commented Nov 19, 2025

Uh oh!

amoghrajesh commented Nov 20, 2025

Uh oh!

uranusjr commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Crowiant commented Nov 3, 2025 •

edited

Loading