Merge pull request #224903 from jonburchel/2023-01-24-airflow-docs-pages-11-16-of-original-word-doc

Airflow docs

Author: Court72

articles/data-factory/TOC.yml

@@ -1291,6 +1291,24 @@ items:
     - name: Diagnose connectivity in Azure-SSIS IR
       href: ssis-integration-runtime-diagnose-connectivity-faq.md
       displayName: timeout, troubleshooting
+- name: Managed Airflow
+  items:
+  - name: Tutorials
+    items:
+    - name: Run an existing pipeline with Airflow
+      href: tutorial-run-existing-pipeline-with-airflow.md
+    - name: Refresh a Power BI dataset with Airflow
+      href: tutorial-refresh-power-bi-dataset-with-airflow.md
+  - name: Concepts
+    href: concept-managed-airflow.md
+  - name: How-to
+    items:
+    - name: How does Managed Airflow work
+      href: how-does-managed-airflow-work.md
+    - name: How to change the Airflow password
+      href: password-change-airflow.md
+  - name: Pricing
+    href: airflow-pricing.md
 - name: Reference
   items:
   - name: Data flow script

articles/data-factory/airflow-pricing.md

@@ -0,0 +1,28 @@
+---
+title: Managed Airflow pricing
+description: This article describes the pricing for Managed Airflow.
+author: nabhishek
+ms.service: data-factory
+ms.subservice: pricing
+ms.topic: conceptual
+ms.date: 01/24/2023
+ms.author: abnarain
+---
+
+# Managed Airflow pricing
+
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+This article describes the pricing for Managed Airflow usage within data factory.
+
+## Pricing details
+
+Managed Airflow supports either small (D2v4) or large (D4v4) node sizing.  Small can support up to 50 DAGs simultaneously, and large can support up to 1000 DAGs. The following table describes pricing for each option:
+
+:::image type="content" source="media/airflow-pricing/airflow-pricing.png" alt-text="Shows a screenshot of a table of pricing options for Managed Airflow configuration.":::
+
+## Next steps
+
+- [Run an existing pipeline with Managed Airflow](tutorial-run-existing-pipeline-with-airflow.md)
+- [Refresh a Power BI dataset with Managed Airflow](tutorial-refresh-power-bi-dataset-with-airflow.md)
+- [Changing password for Airflow environments](password-change-airflow.md)

articles/data-factory/concept-managed-airflow.md

@@ -0,0 +1,83 @@
+---
+title: What is Managed Airflow?
+titleSuffix: Azure Data Factory
+description: Learn about when to use Managed Airflow, basic concepts and supported regions.
+ms.service: data-factory
+ms.topic: conceptual
+author: nabhishek
+ms.author: abnarain
+ms.date: 01/20/2023
+ms.custom: references_regions
+---
+
+# What is Azure Data Factory Managed Airflow?
+
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+> [!NOTE]
+> This feature is in public preview. For questions or feature suggestions, please send an email to mailto:[email protected] with the details.
+
+> [!NOTE]
+> Managed Airflow for Azure Data Factory relies on the open source Apache Airflow application. Documentation and more tutorials for Airflow can be found on the Apache Airflow [Documentation](https://airflow.apache.org/docs/) or [Community](https://airflow.apache.org/community/) pages.
+
+Azure Data Factory offers serverless pipelines for data process orchestration, data movement with 100+ managed connectors, and visual transformations with the mapping data flow.
+
+Managed Airflow in Azure Data Factory is a managed orchestration service for [Apache Airflow](https://airflow.apache.org/) that simplifies the creation and management of Airflow environments on which you can operate end-to-end data pipelines at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as "workflows." With Managed Airflow in Azure Data Factory, you can use Airflow and Python to create data workflows without managing the underlying infrastructure for scalability, availability, and security.  
+
+:::image type="content" source="media/concept-managed-airflow/data-integration.png" alt-text="Screenshot shows data integration.":::
+
+## When to use Managed Airflow?
+
+Azure Data Factory offers [Pipelines](concepts-pipelines-activities.md) to visually orchestrate data processes (UI-based authoring). While Managed Airflow, offers Airflow based python DAGs (python code-centric authoring) for defining the data orchestration process. If you have the Airflow background, or are currently using Apace Airflow, you may prefer to use the Managed Airflow instead of the pipelines. On the contrary, if you wouldn't like to write/ manage python-based DAGs for data process orchestration, you may prefer to use pipelines.  
+
+With Managed Airflow, Azure Data Factory now offers multi-orchestration capabilities spanning across visual, code-centric, OSS orchestration requirements.
+
+## Features
+
+- **Automatic Airflow setup** – Quickly set up Apache Airflow by choosing an [Apache Airflow version](concept-managed-airflow.md#supported-apache-airflow-versions) when you create a Managed Airflow environment. ADF Managed Airflow sets up Apache Airflow for you using the same Apache Airflow user interface and open-source code you can download on the Internet.
+- **Automatic scaling** – Automatically scale Apache Airflow Workers by setting the minimum and maximum number of Workers that run in your environment. ADF Managed Airflow monitors the Workers in your environment. It uses its autoscaling component to add Workers to meet demand until it reaches the maximum number of Workers you defined.
+- **Built-in authentication** – Enable Azure Active Directory (Azure AD) role-based authentication and authorization for your Airflow Web server by defining Azure AD RBAC's access control policies.  
+- **Built-in security** – Metadata is also automatically encrypted by Azure-managed keys, so your environment is secure by default. Additionally, it supports double encryption with a Customer-Managed Key (CMK).  
+- **Streamlined upgrades and patches** – Azure Data Factory Managed Airflow provide new versions of Apache Airflow periodically. The ADF Managed Airflow team will auto-update and patch the minor versions. 
+- **Workflow monitoring** – View Airflow logs and Airflow metrics in Azure Monitor to identify Airflow task delays or workflow errors without needing additional third-party tools. Managed Airflow automatically sends environment metrics, and if enabled, Airflow logs to Azure Monitor. 
+- **Azure integration** – Azure Data Factory Managed Airflow supports open-source integrations with Azure Data Factory pipelines, Azure Batch, Azure Cosmos DB, Azure Key Vault, ACI, ADLS Gen2, Azure Kusto, as well as hundreds of built-in and community-created operators and sensors.
+
+## Architecture
+   :::image type="content" source="media/concept-managed-airflow/architecture.png" alt-text="Screenshot shows architecture in Managed Airflow.":::
+
+## Region availability (public preview)
+
+* EastUs
+* SouthCentralUs
+* WestUs
+* UKSouth  
+* NorthEurope  
+* WestEurope  
+* SouthEastAsia
+* EastUS2
+* WestUS2
+* GermanyWestCentral
+* AustraliaEast
+
+> [!NOTE]
+> By GA, all ADF regions will be supported. The Airflow environment region is defaulted to the Data Factory region and is not configurable, so ensure you use a Data Factory in the above supported region to be able to access the Managed Airflow preview.  
+
+## Supported Apache Airflow versions
+
+* 1.10.14
+* 2.2.4
+
+## Integrations
+
+Apache Airflow integrates with Microsoft Azure services through microsoft.azure [provider](https://airflow.apache.org/docs/apache-airflow-providers-microsoft-azure/stable/index.html).  
+
+You can install any provider package by editing the airflow environment from the Azure Data Factory UI. It takes around a couple of minutes to install the package.
+
+   :::image type="content" source="media/concept-managed-airflow/airflow-integration.png" lightbox="media/concept-managed-airflow/airflow-integration.png" alt-text="Screenshot shows airflow integration.":::
+
+## Next steps
+
+- [Run an existing pipeline with Managed Airflow](tutorial-run-existing-pipeline-with-airflow.md)
+- [Refresh a Power BI dataset with Managed Airflow](tutorial-refresh-power-bi-dataset-with-airflow.md)
+- [Managed Airflow pricing](airflow-pricing.md)
+- [How to change the password for Managed Airflow environments](password-change-airflow.md)

articles/data-factory/how-does-managed-airflow-work.md

@@ -0,0 +1,130 @@
+---
+title: How does Managed Airflow work?
+titleSuffix: Azure Data Factory
+description: This article explains how to create a Managed Airflow instance and use DAG to make it work.
+ms.service: data-factory
+ms.topic: conceptual
+author: nabhishek
+ms.author: abnarain
+ms.date: 01/20/2023
+---
+
+# How does Azure Data Factory Managed Airflow work? 
+
+[!INCLUDE[appliesto-adf-xxx-md](includes/appliesto-adf-xxx-md.md)]
+
+> [!NOTE]
+> Managed Airflow for Azure Data Factory relies on the open source Apache Airflow application. Documentation and more tutorials for Airflow can be found on the Apache Airflow [Documentation](https://airflow.apache.org/docs/) or [Community](https://airflow.apache.org/community/) pages.
+
+Azure Data Factory Managed Airflow orchestrates your workflows using Directed Acyclic Graphs (DAGs) written in Python. You must provide your DAGs and plugins in Azure Blob Storage. Airflow requirements or library dependencies can be installed during the creation of the new Managed Airflow environment or by editing an existing Managed Airflow environment. Then run and monitor your DAGs by launching the Airflow UI from ADF using a command line interface (CLI) or a software development kit (SDK).
+
+## Create a Managed Airflow environment
+The following steps setup and configure your Managed Airflow environment.
+
+### Prerequisites
+**Azure subscription**: If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/) before you begin.
+    Create or select an existing Data Factory in the region where the managed airflow preview is supported.
+
+### Steps to create the environment
+1. Create new Managed Airflow environment.  
+Go to **Manage** hub -> **Airflow (Preview)** -> **+New** to create a new Airflow environment
+
+   :::image type="content" source="media/how-does-managed-airflow-work/create-new-airflow.png" alt-text="Screenshot that shows how to create a new Managed Apache Airflow environment.":::
+
+1.  Provide the details (Airflow config)
+
+   :::image type="content" source="media/how-does-managed-airflow-work/airflow-environment-details.png" alt-text="Screenshot that shows some Managed Airflow environment details.":::
+
+   > [!IMPORTANT]
+   > When using **Basic** authentication, remember the username and password specified in this screen. It will be needed to login later in the Managed Airflow UI. The default option is **AAD** and it does not require creating username/ password for your Airflow environment, but instead uses the logged in user**s credential to Azure Data Factory to login/ monitor DAGs.
+1. **Environment variables** a simple key value store within Airflow to store and retrieve arbitrary content or settings.
+1. **Requirements** can be used to pre-install python libraries. You can update these later as well.
+
+## Import DAGs
+
+The following steps describe how to import DAGs into Managed Airflow.
+
+### Prerequisite
+
+You will need to upload a sample DAG onto an accessible Storage account.
+
+> [!NOTE]
+> Blob Storage behind VNet are not supported during the preview. We will be adding the support shortly.
+
+[Sample Apache Airflow v2.x DAG](https://airflow.apache.org/docs/apache-airflow/stable/tutorial/fundamentals.html).
+[Sample Apache Airflow v1.10 DAG](https://airflow.apache.org/docs/apache-airflow/1.10.11/_modules/airflow/example_dags/tutorial.html).
+
+1. Copy-paste the content (either v2.x or v1.10 based on the Airflow environment that you have setup) into a new file called as **tutorial.py**.
+
+   Upload the **tutorial.py** to a blob storage. ([How to upload a file into blob](/storage/blobs/storage-quickstart-blobs-portal.md))
+
+   > [!NOTE]
+   > You will need to select a directory path from a blob storage account that contains folders named **dags** and **plugins** to import those into the Airflow environment. **Plugins** are not mandatory. You can also have a container named **dags** and upload all Airflow files within it.  
+
+1. Click on **Airflow (Preview)** under **Manage** hub. Then hover over the earlier created **Airflow** environment and click on **Import files** to Import all DAGs and dependencies into the Airflow Environment.
+
+   :::image type="content" source="media/how-does-managed-airflow-work/import-files.png" alt-text="Screenshot shows import files in manage hub.":::
+
+1. Create a new Linked Service to the accessible storage account mentioned in the prerequisite (or use an existing one if you already have your own DAGs).
+
+   :::image type="content" source="media/how-does-managed-airflow-work/create-new-linked-service.png" alt-text="Screenshot that shows how to create a new linked service.":::
+
+1. Use the storage account where you uploaded the DAG (check prerequisite). Test connection, then click **Create**.
+
+   :::image type="content" source="media/how-does-managed-airflow-work/linked-service-details.png" alt-text="Screenshot shows some linked service details.":::
+
+1. Browse and select **airflow** if using the sample SAS URL or select the folder that contains **dags** folder with DAG files.
+
+   > [!NOTE]
+   > You can import DAGs and their dependencies through this interface. You will need to select a directory path from a blob storage account that contains folders named **dags** and **plugins** to import those into the Airflow environment. **Plugins** are not mandatory.
+
+   :::image type="content" source="media/how-does-managed-airflow-work/browse-storage.png" alt-text="Screenshot shows browse storage in import files.":::
+
+   :::image type="content" source="media/how-does-managed-airflow-work/browse.png" alt-text="Screenshot that shows browse in airflow.":::
+
+   :::image type="content" source="media/how-does-managed-airflow-work/import-in-import-files.png" alt-text="Screenshot shows import in import files.":::
+
+   :::image type="content" source="media/how-does-managed-airflow-work/import-dags.png" alt-text="Screenshot shows import dags.":::
+
+> [!NOTE]
+> Importing DAGs could take a couple of minutes during **Preview**. The notification center (bell icon in ADF UI) can be used to track the import status updates.
+
+## Troubleshooting import DAG issues
+
+* Problem: DAG import is taking over 5 minutes 
+Mitigation: Reduce the size of the imported DAGs with a single import. One way to achieve this is by creating multiple DAG folders with lesser DAGs across multiple containers.  
+
+* Problem: Imported DAGs do not show up when you login into the Airflow UI.  
+Mitigation: Login into the Airflow UI and see if there are any DAG parsing errors. This could happen if the DAG files contains any incompatible code. You will find the exact line numbers and the files which have the issue through the Airflow UI.
+
+    :::image type="content" source="media/how-does-managed-airflow-work/import-dag-issues.png" alt-text="Screenshot shows import dag issues.":::
+
+
+## Monitor DAG runs
+
+To monitor the Airflow DAGs, login into Airflow UI with the earlier created username and password.
+
+1. Click on the Airflow environment created.
+
+   :::image type="content" source="media/how-does-managed-airflow-work/airflow-environment-monitor-dag.png" alt-text="Screenshot that shows the Airflow environment created.":::
+
+1. Login using the username-password provided during the Airflow Integration Runtime creation. ([You can reset the username or password by editing the Airflow Integration runtime]() if needed)
+
+   :::image type="content" source="media/how-does-managed-airflow-work/login-in-dags.png" alt-text="Screenshot that shows login using the username-password provided during the Airflow Integration Runtime creation.":::
+
+## Remove DAGs from the Airflow environment
+
+If you are using Airflow version 1.x, delete DAGs that are deployed on any Airflow environment (IR), you need to delete the DAGs in two different places.
+
+1. Delete the DAG from Airflow UI 
+1. Delete the DAG in ADF UI
+
+> [!NOTE]
+> This is the current experience during the Public Preview, and we will be improving this experience. 
+
+## Next steps
+
+* [Run an existing pipeline with Managed Airflow](tutorial-run-existing-pipeline-with-airflow.md)
+* [Refresh a Power BI dataset with Managed Airflow](tutorial-refresh-power-bi-dataset-with-airflow.md)
+* [Managed Airflow pricing](airflow-pricing.md)
+* [How to change the password for Managed Airflow environments](password-change-airflow.md)

articles/data-factory/index.yml

@@ -15,14 +15,22 @@ metadata:
   ms.custom: contperf-fy21q4
 
 landingContent:
+  - title: What's New in Azure Data Factory
+    linkLists:
+      - linkListType: overview
+        links:
+          - text: What's New in Azure Data Factory
+            url: ./whats-new.md
+          - text: Change data capture
+            url: ./concepts-change-data-capture.md
+          - text: Managed Airflow
+            url: ./concept-managed-airflow.md
   - title: About Azure Data Factory
     linkLists:
       - linkListType: overview
         links:
           - text: Introduction to Azure Data Factory
             url: ./introduction.md
-          - text: What's New in Azure Data Factory
-            url: ./whats-new.md
       - linkListType: reference
         links:
           - text: Whitepapers
@@ -111,6 +119,22 @@ landingContent:
             url: ./data-transformation-functions.md 
           - text: Change Data Capture (CDC)
             url: ./concepts-change-data-capture.md
+  - title: Managed Airflow
+    linkLists:
+      - linkListType: concept
+        links:
+          - text: Managed Airflow
+            url: concept-managed-airflow.md
+      - linkListType: how-to-guide
+        links:
+          - text: How does Managed Airflow work?
+            url: how-does-managed-airflow-work.md
+      - linkListType: tutorial
+        links:
+          - text: Run an existing pipeline with Managed Airflow
+            url: tutorial-run-existing-pipeline-with-airflow.md
+          - text: Refresh a Power BI dataset with Managed Airflow
+            url: tutorial-refresh-power-bi-dataset-with-airflow.md
   - title: SAP knowledge center
     linkLists:
       - linkListType: concept