Merge pull request #249701 from jonburchel/2023-08-29-add-2-managed-airflow-docs

2023 08 29 add 2 managed airflow docs

Author: v-dirichards

articles/data-factory/TOC.yml

@@ -1320,6 +1320,12 @@ items:
       href: how-does-managed-airflow-work.md
     - name: How to change the Airflow password
       href: password-change-airflow.md
+    - name: Enable Azure Key Vault for Airflow
+      href: enable-azure-key-vault-for-managed-airflow.md
+    - name: Add Kubernetes secret to access a private container registry
+      href: kubernetes-secret-pull-image-from-private-container-registry.md
+    - name: Rest APIs for managing the Airflow integrated runtime
+      href: rest-apis-for-airflow-integrated-runtime.md
   - name: Pricing
     href: airflow-pricing.md
 - name: Reference

articles/data-factory/enable-azure-key-vault-for-managed-airflow.md

@@ -0,0 +1,112 @@
+---
+title: Enable Azure Key Vault for airflow
+titleSuffix: Azure Data Factory
+description: This article explains how to enable Azure Key Vault as the secret backend for a Managed Airflow instance.
+ms.service: data-factory
+ms.topic: how-to
+author: nabhishek
+ms.author: abnarain
+ms.date: 08/29/2023
+---
+
+# Enable Azure Key Vault for Managed Airflow
+
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+> [!NOTE]
+> Managed Airflow for Azure Data Factory relies on the open source Apache Airflow application. Documentation and more tutorials for Airflow can be found on the Apache Airflow [Documentation](https://airflow.apache.org/docs/) or [Community](https://airflow.apache.org/community/) pages.
+
+Apache Airflow provides a range of backends for storing sensitive information like variables and connections, including Azure Key Vault. This guide shows you how to configure Azure Key Vault as the secret backend for Apache Airflow, enabling you to store and manage your sensitive information in a secure and centralized manner.
+
+## Prerequisites 
+
+- **Azure subscription** - If you don't have an Azure subscription, create a [free Azure account](https://azure.microsoft.com/free/) before you begin.
+- **Azure storage account** - If you don't have a storage account, see [Create an Azure storage account](/azure/storage/common/storage-account-create?tabs=azure-portal) for steps to create one. Ensure the storage account allows access only from selected networks.
+- **Azure Data Factory pipeline** - You can follow any of the tutorials and create a new data factory pipeline in case you don't already have one or create one with one select in [Get started and try out your first data factory pipeline](quickstart-get-started.md).
+- **Azure Key Vault** - You can follow [this tutorial to create a new Azure Key Vault](/azure/key-vault/general/quick-create-portal) if you don’t have one.
+- **Service Principal** - You'll need to [create a new service principal](/azure/active-directory/develop/howto-create-service-principal-portal) or use an existing one and grant it permission to access Azure Key Vault (example - grant the **key-vault-contributor role** to the SPN for the key vault, so the SPN can manage it). Additionally, you'll need to get the service principal **Client ID** and **Client Secret** (API Key) to add them as environment variables, as described later in this article.
+
+## Permissions
+
+Assign your SPN the following roles in your key vault from the [Built-in roles](/azure/role-based-access-control/built-in-roles).
+
+- Key Vault Contributor
+- Key Vault Secrets User
+
+## Enable the Azure Key Vault backend for a Managed Airflow instance
+
+Follow these steps to enable the Azure Key Vault as the secret backend for your Managed Airflow instance.
+
+1. Navigate to the [Managed Airflow instance's integrated runtime (IR) environment](how-does-managed-airflow-work.md).
+1. Install the [**apache-airflow-providers-microsoft-azure**](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/index.html) for the **Airflow requirements** during your initial Airflow environment setup.
+
+   :::image type="content" source="media/enable-azure-key-vault-for-managed-airflow/airflow-environment-setup.png" alt-text="Screenshot showing the Airflow Environment Setup window highlighting the Airflow requirements."  lightbox="media/enable-azure-key-vault-for-managed-airflow/airflow-environment-setup.png":::
+
+1. Add the following settings for the **Airflow configuration overrides** in integrated runtime properties:
+
+   - **AIRFLOW__SECRETS__BACKEND**: "airflow.providers.microsoft.azure.secrets.key_vault.AzureKeyVaultBackend"
+   - **AIRFLOW__SECRETS__BACKEND_KWARGS**: "{"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables", "vault_url": **\<your keyvault uri\>**}”
+
+   :::image type="content" source="media/enable-azure-key-vault-for-managed-airflow/airflow-configuration-overrides.png" alt-text="Screenshot showing the configuration of the Airflow configuration overrides setting in the Airflow environment setup."  lightbox="media/enable-azure-key-vault-for-managed-airflow/airflow-configuration-overrides.png":::
+
+1. Add the following for the **Environment variables** configuration in the Airflow integrated runtime properties:
+
+   - **AZURE_CLIENT_ID** = \<Client ID of SPN\>
+   - **AZURE_TENANT_ID** = \<Tenant Id\>
+   - **AZURE_CLIENT_SECRET** = \<Client Secret of SPN\>
+
+   :::image type="content" source="media/enable-azure-key-vault-for-managed-airflow/environment-variables.png" alt-text="Screenshot showing the Environment variables section of the Airflow integrated runtime properties."  lightbox="media/enable-azure-key-vault-for-managed-airflow/environment-variables.png":::
+
+1. Then you can use variables and connections and they will automatically be stored in Azure Key Vault. The name of connections and variables need to follow AIRFLOW__SECRETS__BACKEND_KWARGS as defined previously. For more information, refer to [Azure-key-vault as secret backend](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/secrets-backends/azure-key-vault.html).
+
+## Sample DAG using Azure Key Vault as the backend
+
+1. Create a new Python file **adf.py** with the following contents:
+
+   ```python
+   from datetime import datetime, timedelta
+   from airflow.operators.python_operator import PythonOperator
+   from textwrap import dedent
+   from airflow.models import Variable
+   from airflow import DAG
+   import logging
+
+   def retrieve_variable_from_akv():
+       variable_value = Variable.get("sample-variable")
+       logger = logging.getLogger(__name__)
+       logger.info(variable_value)
+
+   with DAG(
+      "tutorial",
+      default_args={
+          "depends_on_past": False,
+          "email": ["[email protected]"],
+          "email_on_failure": False,
+          "email_on_retry": False,
+          "retries": 1,
+          "retry_delay": timedelta(minutes=5),
+       },
+      description="A simple tutorial DAG",
+      schedule_interval=timedelta(days=1),
+      start_date=datetime(2021, 1, 1),
+      catchup=False,
+      tags=["example"],
+   ) as dag:
+
+       get_variable_task = PythonOperator(
+           task_id="get_variable",
+           python_callable=retrieve_variable_from_akv,
+       )
+   
+   get_variable_task
+   ```
+
+1. Store variables for connections in Azure Key Vault. Refer to [Store credentials in Azure Key Vault](store-credentials-in-key-vault.md)
+
+   :::image type="content" source="media/enable-azure-key-vault-for-managed-airflow/secrets-configuration.png" alt-text="Screenshot showing the configuration of secrets in Azure Key Vault."  lightbox="media/enable-azure-key-vault-for-managed-airflow/secrets-configuration.png":::
+
+## Next steps
+
+- [Run an existing pipeline with Managed Airflow](tutorial-run-existing-pipeline-with-airflow.md)
+- [Managed Airflow pricing](airflow-pricing.md)
+- [How to change the password for Managed Airflow environments](password-change-airflow.md)

articles/data-factory/index.yml

@@ -138,7 +138,7 @@ landingContent:
       - linkListType: tutorial
         links:
           - text: Capture changed data with a CDC resource
-            url: how-to-change-data-capture-resource.md          
+            url: how-to-change-data-capture-resource.md            
   - title: Managed Airflow
     linkLists:
       - linkListType: concept

articles/data-factory/kubernetes-secret-pull-image-from-private-container-registry.md

@@ -0,0 +1,52 @@
+---
+title: Add a Kubernetes secret to pull an image from a private container registry
+titleSuffix: Azure Data Factory
+description: This article explains how to add a Kubernetes secret to pull a custom image from a private container registry with Managed Airflow in Data Factory for Microsoft Fabric.
+ms.service: data-factory
+ms.topic: how-to
+author: nabhishek
+ms.author: abnarain
+ms.date: 08/30/2023
+---
+
+# Add a Kubernetes secret to access a private container registry
+
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+> [!NOTE]
+> Managed Airflow for Azure Data Factory relies on the open source Apache Airflow application. Documentation and more tutorials for Airflow can be found on the Apache Airflow [Documentation](https://airflow.apache.org/docs/) or [Community](https://airflow.apache.org/community/) pages.
+
+This article explains how to add a Kubernetes secret to pull a custom image from a private container registry with Managed Airflow in Data Factory for Microsoft Fabric.
+
+## Prerequisites
+
+- **Azure subscription** - If you don't have an Azure subscription, create a [free Azure account](https://azure.microsoft.com/free/) before you begin.
+- **Azure storage account** - If you don't have a storage account, see [Create an Azure storage account](/azure/storage/common/storage-account-create?tabs=azure-portal) for steps to create one. Ensure the storage account allows access only from selected networks.
+- **Azure Data Factory pipeline** - You can follow any of the tutorials and create a new data factory pipeline in case you don't already have one or create one with one select in [Get started and try out your first data factory pipeline](quickstart-get-started.md).
+- **Azure Container Registry** - Configure an [Azure Container Registry](/azure/container-registry/container-registry-get-started-portal?tabs=azure-cli) with the custom Docker image you want to use in the DAG. For more information on push and pull container images, see [Push & pull container image - Azure Container Registry](/azure/container-registry/container-registry-get-started-docker-cli?tabs=azure-cli).
+
+### Step 1: Create a new Managed Airflow environment
+
+Open the Azure Data Factory Studio and select the **Manage** tab from the left toolbar, then select **Apache Airflow** under **Workflow Orchestration Manager**. Finally, select **+ New** to create a new Managed Airflow environment.
+
+:::image type="content" source="media/kubernetes-secret-pull-image-from-private-container-registry/create-new-airflow-environment.png" alt-text="Screenshot showing the steps to create a new Managed Airflow environment in the Azure Data Factory Studio." lightbox="media/kubernetes-secret-pull-image-from-private-container-registry/create-new-airflow-environment.png":::
+
+### Step 2: Add a Kubernetes secret
+
+In the Airflow environment setup window, scroll to the bottom and expand the **Advanced** section, then select **+ New** under **Kubernetes secrets**.
+
+:::image type="content" source="media/kubernetes-secret-pull-image-from-private-container-registry/add-kubernetes-secret.png" alt-text="Screenshot showing the Airflow environment setup window with the Advanced section expanded to show the Kubernetes secrets section." lightbox="media/kubernetes-secret-pull-image-from-private-container-registry/add-kubernetes-secret.png":::
+
+### Step 3: Configure authentication
+
+Provide the required field **Secret name**, select **Private registry auth** for the **Secret type**, and enter the other required fields. The **Registry server URL** should be the URL of your private container registry, for example, ```\registry_name\>.azurecr.io```.
+
+:::image type="content" source="media/kubernetes-secret-pull-image-from-private-container-registry/create-airflow-secret.png" alt-text="Screenshot showing the Create airflow secret window and its fields." lightbox="media/kubernetes-secret-pull-image-from-private-container-registry/create-airflow-secret.png" :::
+
+Once you provide the required fields, select **Apply** to add the secret.
+
+## Next steps
+
+- [Run an existing pipeline with Managed Airflow](tutorial-run-existing-pipeline-with-airflow.md)
+- [Managed Airflow pricing](airflow-pricing.md)
+- [How to change the password for Managed Airflow environments](password-change-airflow.md)