You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/apply-dataops.md
+14-14Lines changed: 14 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -34,15 +34,15 @@ Specifically, once you bring your own GitHub or Azure DevOps repository into dat
34
34
35
35
All artifacts in Azure Data Factory, whether they're pipelines, linked services, triggers, etc. have corresponding “code” representations in JSON behind the visual UI integration. These artifacts act in compliance with [Azure Resource Manager templates](/azure-resource-manager/templates/overview.md) standards. You can find the code by clicking on the bracket icon on the top right of the canvas. Sample JSON “code” would look like this:
36
36
37
-
:::image type="content" source="media/applying-dataops/view-json-button.png" alt-text="Screenshot showing the View JSON button on the pipeline UI.":::
37
+
:::image type="content" source="media/apply-dataops/view-json-button.png" alt-text="Screenshot showing the View JSON button on the pipeline UI.":::
38
38
39
-
:::image type="content" source="media/applying-dataops/json-example.png" alt-text="Screenshot showing example JSON for a pipeline.":::
39
+
:::image type="content" source="media/apply-dataops/json-example.png" alt-text="Screenshot showing example JSON for a pipeline.":::
40
40
41
41
### Live mode and Git version control
42
42
43
43
Every factory has one single source of truth: pipelines, linked services, and trigger definitions stored within the service. This source of truth is what the pipeline runs execute and what determines the behaviors of triggers. If you are in live mode, every time you publish, you directly modify the single source of truth. The following image shows what the **Publish All** button looks like in live mode.
44
44
45
-
:::image type="content" source="media/applying-dataops/publish-button-live-mode.png" alt-text="Screenshot showing the Publish All button in live mode.":::
45
+
:::image type="content" source="media/apply-dataops/publish-button-live-mode.png" alt-text="Screenshot showing the Publish All button in live mode.":::
46
46
47
47
Live mode can be convenient for single person working on side projects, as it allows developers to see immediate effects of their code changes. However, it's discouraged for a team of developers working on production-level work projects. The dangers include fat fingers, accidental deletions of critical resources, publishing of untested codes, etc., just to name a few.
48
48
When working on mission critical projects and platforms, consider bringing in a Git repository and use the Git mode in data factory to streamline the development process. [Version control](source-control.md#version-control) and gated check-in capabilities of the Git mode helps you prevent most, if not all, of the accidents associated with touching live mode directly.
@@ -60,7 +60,7 @@ When you create a new data factory from the Azure portal, the default Git repo i
60
60
61
61
From the Azure portal, select the repo type and enter the repo and branch names to create a new factory natively integrated with Git.
62
62
63
-
:::image type="content" source="media/applying-dataops/create-data-factory-configure-git.png" alt-text="Screenshot showing the Create Azure Data Factory UI with the Git configuration settings displayed.":::
63
+
:::image type="content" source="media/apply-dataops/create-data-factory-configure-git.png" alt-text="Screenshot showing the Create Azure Data Factory UI with the Git configuration settings displayed.":::
64
64
65
65
#### Enforcing use of Git with Azure Policy in your organization
66
66
@@ -70,11 +70,11 @@ The use of Git in your Azure Data Factory projects is a highly recommended best
70
70
71
71
After you create your data factory, you can also connect to your repo through the Azure Data Factory Studio. In the **Manage** tab, you'll see the option to configure your repo and repo settings.
72
72
73
-
:::image type="content" lightbox="media/applying-dataops/data-factory-studio-git-configuration.png" source="media/applying-dataops/data-factory-studio-git-configuration.png" alt-text="Screenshot showing the Azure Data Factory Studio on the Manage tab with the Git Configuration section selected.":::
73
+
:::image type="content" lightbox="media/apply-dataops/data-factory-studio-git-configuration.png" source="media/apply-dataops/data-factory-studio-git-configuration.png" alt-text="Screenshot showing the Azure Data Factory Studio on the Manage tab with the Git Configuration section selected.":::
74
74
75
75
Through a guided process, you're directed through a series of steps to help you easily configure and connect to your repository of choice. Once fully set up, you can start to work collaboratively and save your resources to your repo.
76
76
77
-
:::image type="content" source="media/applying-dataops/configure-repository.png" alt-text="Screenshot showing the repository configuration page.":::
77
+
:::image type="content" source="media/apply-dataops/configure-repository.png" alt-text="Screenshot showing the repository configuration page.":::
78
78
79
79
### Continuous integration and continuous delivery (CI/CD)
80
80
@@ -154,7 +154,7 @@ To get started with the [Azure Data Factory utilities npm package](https://www.n
154
154
155
155
After you've merged your branch back to the main collaboration branch in your Git repository, you can manually publish your changes to the live Azure Data Factory service. The service provides UI control over publishing from non-development factories with the **Disable publish (from ADF Studio)** option.
156
156
157
-
:::image type="content" source="media/applying-dataops/disable-publish-option.png" alt-text="Screenshot showing the Git repository edit page and the Disable publish (from ADF Studio) button.":::
157
+
:::image type="content" source="media/apply-dataops/disable-publish-option.png" alt-text="Screenshot showing the Git repository edit page and the Disable publish (from ADF Studio) button.":::
158
158
159
159
### Selective deployment
160
160
@@ -168,11 +168,11 @@ Once you've cherry picked the changes and merged to the main collaboration pipel
168
168
169
169
Unit testing is an important part of the process of developing new pipelines or editing existing data factory artifacts, which focuses on testing components of the code. Data Factory allows for individual unit testing at both the pipeline and data flow artifact level by using the pipeline [debug feature](iterative-development-debugging.md?tabs=data-factory#debugging-a-pipeline).
170
170
171
-
:::image type="content" lightbox="media/applying-dataops/pipeline-debugging.png" source="media/applying-dataops/pipeline-debugging.png" alt-text="Screenshot showing the pipeline editor canvas with the debug option.":::
171
+
:::image type="content" lightbox="media/apply-dataops/pipeline-debugging.png" source="media/apply-dataops/pipeline-debugging.png" alt-text="Screenshot showing the pipeline editor canvas with the debug option.":::
172
172
173
173
When developing data flows, you'll be able to gain insights into each individual transformation and code change by using the [data preview feature](concepts-data-flow-debug-mode.md?tabs=data-factory) to achieve unit testing before deploying your changes to production.
174
174
175
-
:::image type="content" lightbox="media/applying-dataops/data-preview-feature.png" source="media/applying-dataops/data-preview-feature.png" alt-text="Screenshot showing the data preview feature.":::
175
+
:::image type="content" lightbox="media/apply-dataops/data-preview-feature.png" source="media/apply-dataops/data-preview-feature.png" alt-text="Screenshot showing the data preview feature.":::
176
176
177
177
The service provides live and interactive feedback of your pipeline activities in the UI when debugging and unit testing in Azure Data Factory.
178
178
@@ -186,7 +186,7 @@ Customers can also run **TEST** pipelines with **PowerShell** or **AZ CLI** as p
186
186
187
187
A key strength of data factory lies in its parameterization of data sets. This feature empowers customers to run same pipelines with different data sets to make sure their new development meets all source and destination requirements.
188
188
189
-
:::image type="content" source="media/applying-dataops/test-explorer.png" alt-text="Screenshot showing the Test Explorer for Azure Data Factory.":::
189
+
:::image type="content" source="media/apply-dataops/test-explorer.png" alt-text="Screenshot showing the Test Explorer for Azure Data Factory.":::
190
190
191
191
### Other CI/CD frameworks for Azure Data Factory
192
192
@@ -196,21 +196,21 @@ As described previously, built-in Git integration is available natively through
196
196
197
197
An important aspect of effective DataOps is data governance. For data integration ETL tools, providing data lineage and artifact relationships can provide important information for a data engineer to understand the impact of downstream changes. Data factory provides built-in related artifact views that constitute your factory implementation.
198
198
199
-
:::image type="content" source="media/applying-dataops/related-artifacts.png" alt-text="Screenshot showing data factory related artifacts for a sample dataset.":::
199
+
:::image type="content" source="media/apply-dataops/related-artifacts.png" alt-text="Screenshot showing data factory related artifacts for a sample dataset.":::
200
200
201
201
Native integration with Microsoft Purview further provides lineage, impact analysis, and data cataloging.
202
202
203
203
[Microsoft Purview](https://azure.microsoft.com/products/purview/) provides a unified data governance solution to help manage and govern your on-premises, multicloud, and software as a service (SaaS) data. It allows you to easily create a holistic, up-to-date map of your data landscape with automated data discovery, sensitive data classification, and end-to-end data lineage. These features enable data consumers to access valuable, trustworthy data management.
204
204
205
-
:::image type="content" source="media/applying-dataops/data-lineage.png" alt-text="Screenshot showing the data lineage tracking possible with Microsoft Purview.":::
205
+
:::image type="content" source="media/apply-dataops/data-lineage.png" alt-text="Screenshot showing the data lineage tracking possible with Microsoft Purview.":::
206
206
207
207
With native integration into your Purview Data Catalog, data factory enables easy search and discovery of data assets to use in your data integration pipelines across the full breadth of your organization’s data estate.
208
208
209
-
:::image type="content" lightbox="media/applying-dataops/purview-data-catalog.png" source="media/applying-dataops/purview-data-catalog.png" alt-text="Screenshot showing the Microsoft Purview Data Catalog.":::
209
+
:::image type="content" lightbox="media/apply-dataops/purview-data-catalog.png" source="media/apply-dataops/purview-data-catalog.png" alt-text="Screenshot showing the Microsoft Purview Data Catalog.":::
210
210
211
211
You can use the main search bar from the Azure Data Factory Studio to find data assets in your Purview catalog.
212
212
213
-
:::image type="content" lightbox="media/applying-dataops/purview-search.png" source="media/applying-dataops/purview-search.png" alt-text="Screenshot showing Purview results from a search in the Azure Data Factory Studio search bar.":::
213
+
:::image type="content" lightbox="media/apply-dataops/purview-search.png" source="media/apply-dataops/purview-search.png" alt-text="Screenshot showing Purview results from a search in the Azure Data Factory Studio search bar.":::
0 commit comments