How to update a connection password (secret rotation)?

[AchimGaedkeLynker]

I failed to google this question and find an answer specific to 3.x - as the direct access to the ORM is prevented (used to do this in airflow 2.x)

I try to create an airflow task, which:

• Retrieves the most recent password.
• Retrieves the postgres connection using the connection id.
• Compare the passwords
• If the passwords are different update (or delete, create) the connection

And then the downstream tasks will use the updated connection via the connection id.

I get stuck in the last step, how do I update an airflow connection within an airflow DAG run?

One solution might be "obvious": I call a subprocess in the LocalExecutor to use the CLI...

but surely, there must be something better out there.

[potiuk]

For that you should use secrets managers - and use Secrets manager's to update APIs. Generally the idea is that connections are largerly "read only" for Airflow and rotation is managed outside. When your Dag gets a password that has been rotated - it's something that should not happen - your password should be rotated in airflow connection from outside WHEN it is rotated - not when random Dag finds that it should be rotated. It's not only too late but also brittle (which taks is changing it)

[AchimGaedkeLynker]

In short you're calling my proposed solution an Anti-Pattern (in general and specifically for Airflow 3.x). And I agree.

[potiuk]

Yep :)

[AchimGaedkeLynker]

It's not only too late but also brittle (which tasks is changing it)

I do see your point (and even anticipated it).

I am after a pragmatic (maybe substandard) solution as architecture decisions are sometimes not in sync with subsequent decisions in a big organization with outsourced IT support.

My thinking was guided by Airflow 2. I'm actually liking the restrictions Airflow 3 imposes on tasks, but yes, I thought I could do a short-cut.