feat(airflow): Add OPA support to Airflow (#978)

* feat: Add initial OpaFabAuthManager

* feat(airflow): Call OPA from FabAuthManager

* feat(airflow): Add cache and configuration options for OPA and implement all is_authorized functions

* fix(airflow): Fix call to OPA

* test(airflow): Test OpaFabAuthManager configuration and caching

* chore: Update changelog

* feat(airflow): Add metric opa_cache_limit_reached

* docs(airflow): Add a README to the opa-auth-manager

* chore(airflow): Reformat Python code

* docs(airflow): Fix markdown linter warning

* chore(airflow): Fix Dockerfile linter warning

* docs(airflow): Fix a typo

Author: siegfriedweber

CHANGELOG.md

@@ -6,6 +6,7 @@ All notable changes to this project will be documented in this file.
 
 ### Added
 
+- airflow: Add OPA support to Airflow ([#978]).
 - nifi: Activate `include-hadoop` profile for NiFi version 2.* ([#958]).
 - nifi: Add NiFi hadoop Azure and GCP libraries ([#943]).
 - base: Add containerdebug tool ([#928], [#959]).
@@ -45,6 +46,7 @@ All notable changes to this project will be documented in this file.
 [#959]: https://github.com/stackabletech/docker-images/pull/959
 [#935]: https://github.com/stackabletech/docker-images/pull/935
 [#962]: https://github.com/stackabletech/docker-images/pull/962
+[#978]: https://github.com/stackabletech/docker-images/pull/978
 [#980]: https://github.com/stackabletech/docker-images/pull/980
 [#981]: https://github.com/stackabletech/docker-images/pull/981
 [#982]: https://github.com/stackabletech/docker-images/pull/982

airflow/Dockerfile

@@ -9,6 +9,19 @@ FROM oci.stackable.tech/sdp/git-sync/git-sync:${GIT_SYNC} AS gitsync-image
 
 FROM stackable/image/statsd_exporter AS statsd_exporter-builder
 
+FROM python:3.12-bookworm AS opa-auth-manager-builder
+
+COPY airflow/opa-auth-manager/ /tmp/opa-auth-manager
+
+WORKDIR /tmp/opa-auth-manager
+
+RUN <<EOF
+pip install --no-cache-dir poetry
+poetry build
+poetry install
+poetry run pytest
+EOF
+
 FROM stackable/image/vector AS airflow-build-image
 
 ARG PRODUCT
@@ -17,6 +30,7 @@ ARG PYTHON
 ARG TARGETARCH
 
 COPY airflow/constraints-${PRODUCT}-python${PYTHON}.txt /tmp/constraints.txt
+COPY --from=opa-auth-manager-builder /tmp/opa-auth-manager/dist/opa_auth_manager-0.1.0-py3-none-any.whl /tmp/
 
 # The mysql provider is currently excluded.
 # Requires implementation of https://github.com/apache/airflow/blob/2.2.5/scripts/docker/install_mysql.sh
@@ -57,6 +71,8 @@ pip install --no-cache-dir s3fs==2024.9.0 cyclonedx-bom==5.0.0
 # Needed for OIDC
 pip install --no-cache-dir Flask_OIDC==2.2.0 Flask-OpenID==1.3.1
 
+pip install --no-cache-dir /tmp/opa_auth_manager-0.1.0-py3-none-any.whl
+
 # Create the SBOM for Airflow
 # Important: All `pip install` commands must be above this line, otherwise the SBOM will be incomplete
 cyclonedx-py environment --schema-version 1.5 --outfile /tmp/sbom.json

airflow/opa-auth-manager/.gitignore

@@ -0,0 +1,2 @@
+*.pytest_cache/
+dist/

airflow/opa-auth-manager/README.md

@@ -0,0 +1,12 @@
+# Airflow OPA auth manager
+
+Auth manager for Airflow which delegates the authorization to an Open Policy
+Agent
+
+[Poetry](https://python-poetry.org/) is used to build the project:
+
+    poetry build
+
+The unit tests can be run as follows:
+
+    poetry run pytest