Merge pull request #258148 from MicrosoftDocs/main

11/8/2023 AM Publish

Author: Taojunshen

articles/data-factory/ci-cd-pattern-with-airflow.md

@@ -33,10 +33,10 @@ Continuous Deployment (CD) is an extension of CI that takes the automation one s
 **CI Pipeline with Dev IR:** 
 
 When a pull request (PR) is made from a feature branch to the Dev branch, it triggers a PR pipeline. This pipeline is designed to efficiently perform quality checks on your feature branches, ensuring code integrity and reliability. The following types of checks can be included in the pipeline: 
-1. **Python Dependencies Testing**: These tests install and verify the correctness of Python dependencies to ensure that the project's dependencies are properly configured. 
-2. **Code Analysis and Linting:** Tools for static code analysis and linting are applied to evaluate code quality and adherence to coding standards. 
-3. **Airflow DAG’s Tests:** These tests execute validation tests, including tests for the DAG definition and unit tests designed for Airflow DAGs. 
-4. **Unit Tests for Airflow custom operators, hooks, sensors and triggers**  
+- **Python Dependencies Testing**: These tests install and verify the correctness of Python dependencies to ensure that the project's dependencies are properly configured. 
+- **Code Analysis and Linting:** Tools for static code analysis and linting are applied to evaluate code quality and adherence to coding standards. 
+- **Airflow DAG’s Tests:** These tests execute validation tests, including tests for the DAG definition and unit tests designed for Airflow DAGs. 
+- **Unit Tests for Airflow custom operators, hooks, sensors and triggers**  
 If any of these checks fail, the pipeline terminates, signaling that the developer needs to address the issues identified. 
 
 #### Git-sync with Prod IR: Map your Managed Airflow environment with your Git repository’s Production branch. 
@@ -50,9 +50,9 @@ Once the Feature branch successfully merges with the Dev branch, you can create
 
 It allows you to continuously deploy the DAGs/ code into Managed Airflow environment.  
 
-1. **Fail-fast approach:** Without the integration of CI/CD process, the first time you know DAG contains errors is likely when it's pushed to GitHub, synchronized with managed airflow and throws an Import Error. Meanwhile the other developer can unknowingly pull the faulty code from the repository, potentially leading to inefficiencies down the line. 
+- **Fail-fast approach:** Without the integration of CI/CD process, the first time you know DAG contains errors is likely when it's pushed to GitHub, synchronized with managed airflow and throws an Import Error. Meanwhile the other developer can unknowingly pull the faulty code from the repository, potentially leading to inefficiencies down the line. 
 
-2. **Code quality improvement:** Neglecting fundamental checks like syntax verification, necessary imports, and checks for other best coding practices, can increase the likelihood of delivering subpar code. 
+- **Code quality improvement:** Neglecting fundamental checks like syntax verification, necessary imports, and checks for other best coding practices, can increase the likelihood of delivering subpar code. 
 
 ## Deployment Patterns in Azure Managed Airflow: 
 
@@ -66,13 +66,13 @@ It allows you to continuously deploy the DAGs/ code into Managed Airflow environ
 
 ### Advantages: 
 
-1. **No Local Development Environment Required:** Managed Airflow handles the underlying infrastructure, updates, and maintenance, reducing the operational overhead of managing Airflow clusters. The service allows you to focus on building and managing workflows rather than managing infrastructure. 
+- **No Local Development Environment Required:** Managed Airflow handles the underlying infrastructure, updates, and maintenance, reducing the operational overhead of managing Airflow clusters. The service allows you to focus on building and managing workflows rather than managing infrastructure. 
 
-2. **Scalability:** Managed Airflow provides auto scaling capability to scale resources as needed, ensuring that your data pipelines can handle increasing workloads or bursts of activity without manual intervention. 
+- **Scalability:** Managed Airflow provides auto scaling capability to scale resources as needed, ensuring that your data pipelines can handle increasing workloads or bursts of activity without manual intervention. 
 
-3. **Monitoring and Logging:** Managed Airflow includes Diagnostic logs and monitoring, making it easier to track the execution of your workflows, diagnose issues, and optimize performance. 
+- **Monitoring and Logging:** Managed Airflow includes Diagnostic logs and monitoring, making it easier to track the execution of your workflows, diagnose issues, and optimize performance. 
 
-4. **Git Integration**: Managed Airflow supports Git-sync feature, allowing you to store your DAGs in Git repository, making it easier to manage changes and collaborate with the team.  
+- **Git Integration**: Managed Airflow supports Git-sync feature, allowing you to store your DAGs in Git repository, making it easier to manage changes and collaborate with the team.  
 
 ### Workflow: 
 
@@ -114,7 +114,7 @@ Synchronize your GitHub repository’s branch with Azure Managed Airflow Service
 
 Learn more about how to use Azure Managed Airflow's [Git-sync feature](airflow-sync-github-repository.md).
 
-3. **Utilize Managed Airflow Service as Production environment:** 
+- **Utilize Managed Airflow Service as Production environment:** 
 
 You can raise a Pull Request (PR) to the branch that is sync with the Managed Airflow Service after successfully developing and testing data pipelines on local development setup. Once the branch is merged you can utilize the Managed Airflow service's features like auto-scaling and monitoring and logging at production level. 
 
@@ -197,14 +197,15 @@ jobs:
 
 **Step 5:** In the tests folder, create the tests for Airflow DAGs. Following are the few examples: 
 
-1. At the least, it's crucial to conduct initial testing using `import_errors` to ensure the DAG's integrity and correctness.   
+* At the least, it's crucial to conduct initial testing using `import_errors` to ensure the DAG's integrity and correctness.   
 This test ensures: 
 
 - **Your DAG does not contain cyclicity:** Cyclicity, where a task forms a loop or circular dependency within  the workflow, can lead to unexpected and infinite execution loops. 
 
 - **There are no import errors:** Import errors can arise due to issues like missing dependencies, incorrect module paths, or coding errors.  
 
 - **Tasks are defined correctly:** Confirm that the tasks within your DAG are correctly defined.
+
 ```python
 @pytest.fixture() 
 
@@ -217,9 +218,9 @@ def test_no_import_errors(dagbag):
     """ 
     assert not dagbag.import_errors 
 ```
- 
 
-1. Test to ensure specific Dag IDs to be present in your feature branch before merging it into the development (dev) branch. 
+* Test to ensure specific Dag IDs to be present in your feature branch before merging it into the development (dev) branch. 
+
 ```python
 def test_expected_dags(dagbag): 
     """ 
@@ -234,7 +235,8 @@ def test_expected_dags(dagbag):
         assert dag_id == dag.dag_id 
 ```
 
-2. Test to ensure only approved tags are associated with your DAGs. This test helps to enforce the approved tag usage. 
+* Test to ensure only approved tags are associated with your DAGs. This test helps to enforce the approved tag usage. 
+
 ```python
 def test_requires_approved_tag(dagbag): 
     """ 
@@ -252,7 +254,7 @@ def test_requires_approved_tag(dagbag):
 
 **Step 6:** Now, when you raise pull request to dev branch, GitHub Actions triggers the CI pipeline, to run all the tests. 
 
-#### For information: 
+#### For More Information: 
 
 - [https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/dagbag.html](https://airflow.apache.org/docs/apache-airflow/stable/_modules/airflow/models/dagbag.html) 
 

articles/data-factory/concept-managed-airflow.md

@@ -66,9 +66,7 @@ Managed Airflow in Azure Data Factory offers a range of powerful features, inclu
 
 ## Supported Apache Airflow versions
 
-* 1.10.14
-* 2.2.2
-* 2.4.3
+* 2.6.3
 
 > [!NOTE]
 > Changing the Airflow version within an existing IR is not supported. Instead, the recommended solution is to create a new Airflow IR with the desired version
@@ -84,11 +82,9 @@ You can install any provider package by editing the airflow environment from the
 ## Limitations
 
 * Managed Airflow in other regions is available by GA.
-* Data Sources connecting through airflow should be publicly accessible. 
-* Blob Storage behind VNet is not supported during the public preview.
+* Data Sources connecting through airflow should be accessible through public endpoint (network).
 * DAGs that are inside a Blob Storage in VNet/behind Firewall is currently not supported.
 * Azure Key Vault isn't supported in LinkedServices to import dags.
-* Airflow supports officially Blob Storage and ADLS with some limitations.
 
 ## Next steps
 

articles/migrate/common-questions-discovery-assessment.md

@@ -64,7 +64,7 @@ To ensure performance data is collected, check:
 - If Azure Migrate connection status for all SQL instances is 'Connected' in the discovered SQL instance section.
 - If all of the performance counters are missing, ensure that outbound connections on ports 443 (HTTPS) are allowed.
 
-If any of the performance counters are missing, Azure SQL assessment recommends the smallest Azure SQL configuration for that instance/database.
+If any of the performance counters are missing, Azure SQL assessment falls back to As on-premises sizing and recommends an Azure SQL configuration based on the allocated cores, memory and total database size on-premises.
 
 ## Why is confidence rating not available for Azure App Service assessments?
 
@@ -147,7 +147,7 @@ The readiness for your web apps is computed by running series of technical check
 
 ## Why is my web app marked as Ready with conditions or Not ready in my Azure App Service assessment?
 
-This can happen when one or more technical checks fail for a given web app. You may select the readiness status for the web app to find out details and remediation for failed checks.
+This can happen when one or more technical checks fail for a given web app. You can select the readiness status for the web app to find out details and remediation for failed checks.
 
 ## Why is the readiness for all my SQL instances marked as unknown?
 
@@ -294,7 +294,7 @@ Using the 95th percentile value ensures that outliers are ignored. Outliers migh
 
 Import-based Azure VM assessments are assessments created with machines that are imported into Azure Migrate using a CSV file. Only four fields are mandatory to import: Server name, cores, memory, and operating system. Here are some things to note:
 
- - The readiness criteria is less stringent in import-based assessments on the boot type parameter. If the boot type isn't provided, it's assumed the machine has BIOS boot type, and the machine isn't marked as **Conditionally Ready**. In assessments with discovery source as appliance, the readiness is marked as **Conditionally Ready** if the boot type is missing. This difference in readiness calculation is because users may not have all information on the machines in the early stages of migration planning when import-based assessments are done.
+ - The readiness criteria is less stringent in import-based assessments on the boot type parameter. If the boot type isn't provided, it's assumed the machine has BIOS boot type, and the machine isn't marked as **Conditionally Ready**. In assessments with discovery source as appliance, the readiness is marked as **Conditionally Ready** if the boot type is missing. This difference in readiness calculation is because users might not have all information on the machines in the early stages of migration planning when import-based assessments are done.
  - Performance-based import assessments use the utilization value provided by the user for right-sizing calculations. Since the utilization value is provided by the user, the **Performance history** and **Percentile utilization** options are disabled in the assessment properties. In assessments with discovery source as appliance, the chosen percentile value is picked from the performance data collected by the appliance.
 
 ## Why is the suggested migration tool in import-based AVS assessment marked as unknown?