You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Continuous integrationis the practice of testing each change made to your codebase automatically and as early as possible. Continuous deliveryfollows the testing that happens duringcontinuous integration and pushes changes to a staging or production system.
17
+
Continuous integrationis the practice of testing each change made to your codebase automatically and as early as possible. Continuous deliveryfollows the testing that happens duringcontinuous integration and pushes changes to a staging or production system.
18
18
19
19
In Azure Data Factory, continuous integration and delivery (CI/CD) means moving Data Factory pipelines from one environment (development, test, production) to another. Azure Data Factory utilizes [Azure Resource Manager templates](../azure-resource-manager/templates/overview.md) to store the configuration of your various ADF entities (pipelines, datasets, data flows, and so on). There are two suggested methods to promote a data factory to another environment:
20
20
21
-
-Automated deployment using Data Factory's integration with [Azure Pipelines](/azure/devops/pipelines/get-started/what-is-azure-pipelines)
22
-
-Manually upload a Resource Manager template using Data Factory UX integration with Azure Resource Manager.
21
+
- Automated deployment using Data Factory's integration with [Azure Pipelines](/azure/devops/pipelines/get-started/what-is-azure-pipelines)
22
+
- Manually upload a Resource Manager template using Data Factory UX integration with Azure Resource Manager.
@@ -30,17 +30,17 @@ In Azure Data Factory, continuous integration and delivery (CI/CD) means moving
30
30
31
31
Below is a sample overview of the CI/CD lifecycle in an Azure data factory that's configured with Azure Repos Git. For more information on how to configure a Git repository, see [Source control in Azure Data Factory](source-control.md).
32
32
33
-
1.A development data factory is created and configured with Azure Repos Git. All developers should have permission to author Data Factory resources like pipelines and datasets.
33
+
1. A development data factory is created and configured with Azure Repos Git. All developers should have permission to author Data Factory resources like pipelines and datasets.
34
34
35
-
1.A developer [creates a feature branch](source-control.md#creating-feature-branches) to make a change. They debug their pipeline runs with their most recent changes. For more information on how to debug a pipeline run, see [Iterative development and debugging with Azure Data Factory](iterative-development-debugging.md).
35
+
1. A developer [creates a feature branch](source-control.md#creating-feature-branches) to make a change. They debug their pipeline runs with their most recent changes. For more information on how to debug a pipeline run, see [Iterative development and debugging with Azure Data Factory](iterative-development-debugging.md).
36
36
37
-
1.After a developer is satisfied with their changes, they create a pull request from their feature branch to the main or collaboration branch to get their changes reviewed by peers.
37
+
1. After a developer is satisfied with their changes, they create a pull request from their feature branch to the main or collaboration branch to get their changes reviewed by peers.
38
38
39
-
1.After a pull request is approved and changes are merged in the main branch, the changes get published to the development factory.
39
+
1. After a pull request is approved and changes are merged in the main branch, the changes get published to the development factory.
40
40
41
-
1.When the team is ready to deploy the changes to a test or UAT (User Acceptance Testing) factory, the team goes to their Azure Pipelines release and deploys the desired version of the development factory to UAT. This deployment takes place as part of an Azure Pipelines task and uses Resource Manager template parameters to apply the appropriate configuration.
41
+
1. When the team is ready to deploy the changes to a test or UAT (User Acceptance Testing) factory, the team goes to their Azure Pipelines release and deploys the desired version of the development factory to UAT. This deployment takes place as part of an Azure Pipelines task and uses Resource Manager template parameters to apply the appropriate configuration.
42
42
43
-
1.After the changes have been verified in the test factory, deploy to the production factory by using the next task of the pipelines release.
43
+
1. After the changes have been verified in the test factory, deploy to the production factory by using the next task of the pipelines release.
44
44
45
45
> [!NOTE]
46
46
> Only the development factory is associated with a git repository. The test and production factories shouldn't have a git repository associated with them and should only be updated via an Azure DevOps pipeline or via a Resource Management template.
@@ -53,7 +53,7 @@ The below image highlights the different steps of this lifecycle.
53
53
54
54
If you're using Git integration with your data factory and have a CI/CD pipeline that moves your changes from development into test and then to production, we recommend these best practices:
55
55
56
-
-**Git integration**. Configure only your development data factory with Git integration. Changes to test and production are deployed via CI/CD and don't need Git integration.
56
+
-**Git integration**. Configure only your development data factory with Git integration. Changes to test and production are deployed via CI/CD and don't need Git integration.
57
57
58
58
-**Pre- and post-deployment script**. Before the Resource Manager deployment step in CI/CD, you need to complete certain tasks, like stopping and restarting triggers and performing cleanup. We recommend that you use PowerShell scripts before and after the deployment task. For more information, see [Update active triggers](continuous-integration-delivery-automate-azure-pipelines.md#updating-active-triggers). The data factory team has [provided a script](continuous-integration-delivery-sample-script.md) to use located at the bottom of this page.
59
59
@@ -66,48 +66,44 @@ If you're using Git integration with your data factory and have a CI/CD pipeline
66
66
>[!WARNING]
67
67
>If you do not use latest versions of PowerShell and Data Factory module, you may run into deserialization errors while running the commands.
68
68
69
-
-**Integration runtimes and sharing**. Integration runtimes don't change often and are similar across all stages in your CI/CD. So Data Factory expects you to have the same name, type and sub-type of integration runtime across all stages of CI/CD. If you want to share integration runtimes across all stages, consider using a ternary factory just to contain the shared integration runtimes. You can use this shared factory in all of your environments as a linked integration runtime type.
69
+
-**Integration runtimes and sharing**. Integration runtimes don't change often and are similar across all stages in your CI/CD. So Data Factory expects you to have the same name, type and sub-type of integration runtime across all stages of CI/CD. If you want to share integration runtimes across all stages, consider using a ternary factory just to contain the shared integration runtimes. You can use this shared factory in all of your environments as a linked integration runtime type.
70
70
71
-
>[!Note]
72
-
>The integration runtime sharing is only available for self-hosted integration runtimes. Azure-SSIS integration runtimes don't support sharing.
71
+
>[!Note]
72
+
>The integration runtime sharing is only available for self-hosted integration runtimes. Azure-SSIS integration runtimes don't support sharing.
73
73
74
-
-**Managed private endpoint deployment**. If a private endpoint already exists in a factory and you try to deploy an ARM template that contains a private endpoint with the same name but with modified properties, the deployment will fail. In other words, you can successfully deploy a private endpoint as long as it has the same properties as the one that already exists in the factory. If any property is different between environments, you can override it by parameterizing that property and providing the respective value during deployment.
74
+
-**Managed private endpoint deployment**. If a private endpoint already exists in a factory and you try to deploy an ARM template that contains a private endpoint with the same name but with modified properties, the deployment will fail. In other words, you can successfully deploy a private endpoint as long as it has the same properties as the one that already exists in the factory. If any property is different between environments, you can override it by parameterizing that property and providing the respective value during deployment.
75
75
76
-
-**Key Vault**. When you use linked services whose connection information is stored in Azure Key Vault, it is recommended to keep separate key vaults for different environments. You can also configure separate permission levels for each key vault. For example, you might not want your team members to have permissions to production secrets. If you follow this approach, we recommend that you to keep the same secret names across all stages. If you keep the same secret names, you don't need to parameterize each connection string across CI/CD environments because the only thing that changes is the key vault name, which is a separate parameter.
76
+
-**Key Vault**. When you use linked services whose connection information is stored in Azure Key Vault, it is recommended to keep separate key vaults for different environments. You can also configure separate permission levels for each key vault. For example, you might not want your team members to have permissions to production secrets. If you follow this approach, we recommend that you to keep the same secret names across all stages. If you keep the same secret names, you don't need to parameterize each connection string across CI/CD environments because the only thing that changes is the key vault name, which is a separate parameter.
77
77
78
-
-**Resource naming**. Due to ARM template constraints, issues in deployment may arise if your resources contain spaces in the name. The Azure Data Factory team recommends using '_' or '-' characters instead of spaces for resources. For example, 'Pipeline_1' would be a preferable name over 'Pipeline 1'.
79
-
80
-
-**Altering repository**. ADF manages GIT repository content automatically. Altering or adding manually unrelated files or folder into anywhere in ADF Git repository data folder could cause resource loading errors. For example, presence of *.bak* files can cause ADF CI/CD error, so they should be removed for ADF to load.
78
+
-**Resource naming**. Due to ARM template constraints, issues in deployment may arise if your resources contain spaces in the name. The Azure Data Factory team recommends using '_' or '-' characters instead of spaces for resources. For example, 'Pipeline_1' would be a preferable name over 'Pipeline 1'.
79
+
80
+
-**Altering repository**. ADF manages GIT repository content automatically. Altering or adding manually unrelated files or folder into anywhere in ADF Git repository data folder could cause resource loading errors. For example, presence of *.bak* files can cause ADF CI/CD error, so they should be removed for ADF to load.
81
81
82
82
-**Exposure control and feature flags**. When working in a team, there are instances where you may merge changes, but don't want them to be run in elevated environments such as PROD and QA. To handle this scenario, the ADF team recommends [the DevOps concept of using feature flags](/devops/operate/progressive-experimentation-feature-flags). In ADF, you can combine [global parameters](author-global-parameters.md) and the [if condition activity](control-flow-if-condition-activity.md) to hide sets of logic based upon these environment flags.
83
83
84
-
To learn how to set up a feature flag, see the below video tutorial:
84
+
To learn how to set up a feature flag, see the below video tutorial:
- By design, Data Factory doesn't allow cherry-picking of commits or selective publishing of resources. Publishes will include all changes made in the data factory.
91
91
92
-
- Data factory entities depend on each other. For example, triggers depend on pipelines, and pipelines depend on datasets and other pipelines. Selective publishing of a subset of resources could lead to unexpected behaviors and errors.
93
-
- On rare occasions when you need selective publishing, consider using a hotfix. For more information, see [Hotfix production environment](continuous-integration-delivery-hotfix-environment.md).
92
+
- Data factory entities depend on each other. For example, triggers depend on pipelines, and pipelines depend on datasets and other pipelines. Selective publishing of a subset of resources could lead to unexpected behaviors and errors.
93
+
- On rare occasions when you need selective publishing, consider using a hotfix. For more information, see [Hotfix production environment](continuous-integration-delivery-hotfix-environment.md).
94
94
95
95
- The Azure Data Factory team doesn’t recommend assigning Azure RBAC controls to individual entities (pipelines, datasets, etc.) in a data factory. For example, if a developer has access to a pipeline or a dataset, they should be able to access all pipelines or datasets in the data factory. If you feel that you need to implement many Azure roles within a data factory, look at deploying a second data factory.
96
96
97
-
-You can't publish from private branches.
97
+
- You can't publish from private branches.
98
98
99
-
-You can't currently host projects on Bitbucket.
99
+
- You can't currently host projects on Bitbucket.
100
100
101
-
-You can't currently export and import alerts and matrices as parameters.
101
+
- You can't currently export and import alerts and matrices as parameters.
102
102
103
-
-In the code repository under the *adf_publish*branch, a folder named 'PartialArmTemplates' is currently added beside the 'linkedTemplates' folder, 'ARMTemplateForFactory.json' and 'ARMTemplateParametersForFactory.json' files as part of publishing with source control.
103
+
-Partial ARM templates in your publish branch are no longer supported as of November 1, 2021. If your project utilized this feature, please switch to a supported mechanism for deployments, using: ```ARMTemplateForFactory.json``` or ```linkedTemplates``` files.
104
104
105
105
:::image type="content" source="media/continuous-integration-delivery/partial-arm-templates-folder.png" alt-text="Diagram of 'PartialArmTemplates' folder.":::
106
106
107
-
We will no longer be publishing 'PartialArmTemplates' to the *adf_publish* branch starting 1-November 2021.
108
-
109
-
**No action is required unless you are using 'PartialArmTemplates'. Otherwise, switch to any supported mechanism for deployments using: 'ARMTemplateForFactory.json' or 'linkedTemplates' files.**
| Public Cloud: `*.servicebus.windows.net` <br> Azure Government: `*.servicebus.usgovcloudapi.net` <br> China: `*.servicebus.chinacloudapi.cn`| 443 | Required by the self-hosted integration runtime for interactive authoring. It's not required if self-contained interactive authoring is enabled.|
13
-
| Public Cloud: `{datafactory}.{region}.datafactory.azure.net`<br> or `*.frontend.clouddatahub.net` <br> Azure Government: `{datafactory}.{region}.datafactory.azure.us` <br> China: `{datafactory}.{region}.datafactory.azure.cn`| 443 | Required by the self-hosted integration runtime to connect to the Data Factory service. <br>For new created Data Factory in public cloud, find the FQDN from your Self-hosted Integration Runtime key, which is in format {datafactory}.{region}.datafactory.azure.net. For old Data factory and Azure Synapse Analytics, if you don't see the FQDN in your Self-hosted Integration key, use *.frontend.clouddatahub.net instead. |
14
-
|`download.microsoft.com`| 443 | Required by the self-hosted integration runtime for downloading the updates. If you have disabled auto-update, you can skip configuring this domain. |
12
+
| Public Cloud: `*.servicebus.windows.net` <br> Azure Government: `*.servicebus.usgovcloudapi.net` <br> China: `*.servicebus.chinacloudapi.cn`| 443 | Required by the self-hosted integration runtime for interactive authoring. Not required if self-contained interactive authoring is enabled.|
13
+
| Public Cloud: `{datafactory}.{region}.datafactory.azure.net`<br> or `*.frontend.clouddatahub.net` <br> Azure Government: `{datafactory}.{region}.datafactory.azure.us` <br> China: `{datafactory}.{region}.datafactory.azure.cn`| 443 | Required by the self-hosted integration runtime to connect to the Data Factory service. <br>For new created Data Factory in public cloud, find the fully qualified domain name (FQDN) from your Self-hosted Integration Runtime key, which is in format {data factory}.{region}.datafactory.azure.net. For old Data factory and any version of Azure Synapse Analytics, if you don't see the FQDN in your Self-hosted Integration key, use *.frontend.clouddatahub.net instead. |
14
+
|`download.microsoft.com`| 443 | Required by the self-hosted integration runtime for downloading the updates. If you disabled autoupdate, you can skip configuring this domain. |
15
15
| Key Vault URL | 443 | Required by Azure Key Vault if you store the credential in Key Vault. |
0 commit comments