Update promotion scripts to use SM2A dag launcher role

[smohiudd]

Description

veda-data is currently setup to use SM2A admin credentials to post to the Rest API. Because of the security vulnerabilities of this, we've created a Dag Launcher role in SM2A.

We are currently using basic authentication using either credentials from an admin or dag launcher user.

Now that we're transitioning to using Keycloak with SM2A we can use a machine-to-machine service account user to authenticate in veda-data and then pass that access token to SM2A.

Acceptance Criteria

• veda-data has removed the SM2A admin credentials workflow and is instead using Keycloak client to authenticate and pass access tokens

[smohiudd]

NASA-IMPACT/veda-data#431

This will need to be tested once's we've transitioned both Staging and Production SM2A to Keycloak.

[smohiudd]

We created a airflow-webserver-fab service account user in Keycloak (which used the Airflow Dag Launcer role) and then tried to use client_credentials auth to get a token from the keycloak token url

    response = requests.post(
        token_url,
        data={
            "client_id": client_id,
            "client_secret": client_secret,
            "grant_type": "client_credentials",
        },
    )
    access_token = response.json()["access_token"]

And then posting that token as a Bearer to the Airflow DAG api (/api/v1/dags/) resulted in a 401 Unauthorized error.

This likely failed because by default Airflow uses the FAB auth manager for token validation which means the token must be generated by the FAB Auth manager.

To use Keycloak with the Airflow DAG API, we'll need to "Roll our own API Auth" which does require some customization on the Airflow side and introspection of the Keycloak token.

Blockers

We're on Airflow version 2.10 which is using FAB auth v 1.5. There are limitation with using Keycloak with these versions.

FAB Auth 3.3.0 seems to have better documentation on using Keyclaok with the Airflow API. We'll need to upgrade to Airflow version 3 to get these features - that would introduce breaking changes to our implementation.

Another option if we upgrade Airflow would be to use the Keycloak auth manager which would simplify token generation and validation since we can use FAB auth manager endpoints in Airflow to generate tokens.

[smohiudd]

Here is a ticket about upgrading Airflow to 3.0: NASA-IMPACT/veda-data-airflow#423