You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/concept-managed-airflow.md
+20-18Lines changed: 20 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,7 +22,8 @@ ms.custom: references_regions
22
22
23
23
Azure Data Factory offers serverless pipelines for data process orchestration, data movement with 100+ managed connectors, and visual transformations with the mapping data flow.
24
24
25
-
Managed Airflow in Azure Data Factory is a managed orchestration service for [Apache Airflow](https://airflow.apache.org/) that simplifies the creation and management of Airflow environments on which you can operate end-to-end data pipelines at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as "workflows." With Managed Airflow in Azure Data Factory, you can use Airflow and Python to create data workflows without managing the underlying infrastructure for scalability, availability, and security.
25
+
Azure Data Factory's Managed Airflow service is a simple and efficient way to create and manage [Apache Airflow](https://airflow.apache.org) environments, enabling you to run data pipelines at scale with ease.
26
+
[Apache Airflow](https://airflow.apache.org) is an open-source platform used to programmatically create, schedule, and monitor complex data workflows. It allows you to define a set of tasks, called operators, that can be combined into directed acyclic graphs (DAGs) to represent data pipelines. Airflow enables you to execute these DAGs on a schedule or in response to an event, monitor the progress of workflows, and provide visibility into the state of each task. It is widely used in data engineering and data science to orchestrate data pipelines, and is known for its flexibility, extensibility, and ease of use.
26
27
27
28
:::image type="content" source="media/concept-managed-airflow/data-integration.png" alt-text="Screenshot shows data integration.":::
28
29
@@ -34,29 +35,30 @@ With Managed Airflow, Azure Data Factory now offers multi-orchestration capabili
34
35
35
36
## Features
36
37
37
-
-**Automatic Airflow setup** – Quickly set up Apache Airflow by choosing an [Apache Airflow version](concept-managed-airflow.md#supported-apache-airflow-versions) when you create a Managed Airflow environment. ADF Managed Airflow sets up Apache Airflow for you using the same Apache Airflow user interface and open-source code you can download on the Internet.
38
-
-**Automatic scaling** – Automatically scale Apache Airflow Workers by setting the minimum and maximum number of Workers that run in your environment. ADF Managed Airflow monitors the Workers in your environment. It uses its autoscaling component to add Workers to meet demand until it reaches the maximum number of Workers you defined.
39
-
-**Built-in authentication** – Enable Azure Active Directory (Azure AD) role-based authentication and authorization for your Airflow Web server by defining Azure AD RBAC's access control policies.
40
-
-**Built-in security** – Metadata is also automatically encrypted by Azure-managed keys, so your environment is secure by default. Additionally, it supports double encryption with a Customer-Managed Key (CMK).
41
-
-**Streamlined upgrades and patches** – Azure Data Factory Managed Airflow provide new versions of Apache Airflow periodically. The ADF Managed Airflow team will auto-update and patch the minor versions.
42
-
-**Workflow monitoring** – View Airflow logs and Airflow metrics in Azure Monitor to identify Airflow task delays or workflow errors without needing additional third-party tools. Managed Airflow automatically sends environment metrics, and if enabled, Airflow logs to Azure Monitor.
43
-
-**Azure integration** – Azure Data Factory Managed Airflow supports open-source integrations with Azure Data Factory pipelines, Azure Batch, Azure Cosmos DB, Azure Key Vault, ACI, ADLS Gen2, Azure Kusto, as well as hundreds of built-in and community-created operators and sensors.
38
+
Managed Airflow in Azure Data Factory offers a range of powerful features, including:
39
+
40
+
-**Fast and simple deployment** – You can quickly and easily set up Apache Airflow by selecting an [Apache Airflow version](concept-managed-airflow.md#supported-apache-airflow-versions) when you create a Managed Airflow.
41
+
-**Cloud scale** – Managed Airflow automatically scales Apache Airflow nodes when required based on range specification (min, max).
42
+
-**Azure Active Directory integration** – You can enable [Azure AD RBAC](concepts-roles-permissions.md) against your Airflow environment for a single sign on experience that is secured by Azure Active Directory.
43
+
-**Managed Virtual Network integration** (coming soon) – You can access your data source via private endpoints or on-premises using ADF Managed Virtual Network that provides extra network isolation.
44
+
-**Metadata encryption** – Managed Airflow automatically encrypts metadata using Azure-managed keys to ensure your environment is secure by default. It also supports double encryption with a [Customer-Managed Key (CMK)](enable-customer-managed-key.md).
45
+
-**Azure Monitoring and alerting** – All the logs generated by Managed Airflow is exported to Azure Monitor. It also provides metrics to track critical conditions and help you notify if the need be.
44
46
45
47
## Architecture
46
48
:::image type="content" source="media/concept-managed-airflow/architecture.png" lightbox="media/concept-managed-airflow/architecture.png" alt-text="Screenshot shows architecture in Managed Airflow.":::
0 commit comments