You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/data-factory/frequently-asked-questions.yml
+7-7Lines changed: 7 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -49,7 +49,7 @@ sections:
49
49
- Looping containers:
50
50
* The foreach activity will iterate over a specified collection of activities in a loop.
51
51
- Trigger-based flows:
52
-
- Pipelines can be triggered on demand, by wall-clock time, or in response to driven by event grid topics
52
+
- Pipelines can be triggered on demand, by wall-clock time, or in response to driven by Event Grid topics
53
53
- Delta flows:
54
54
- Parameters can be used to define your high-water mark for delta copy while moving dimension or reference tables from a relational store, either on-premises or in the cloud, to load the data into the lake.
55
55
@@ -234,7 +234,7 @@ sections:
234
234
### I need help troubleshooting my data flow logic. What info do I need to provide to get help?
235
235
236
236
When Microsoft provides help or troubleshooting with data flows, please provide the ADF pipeline support files.
237
-
This Zip file contains the code-behind script from your data flow graph. From the ADF UI, click **...** next to pipeline, and then click **Download support files**.
237
+
This Zip file contains the code-behind script from your data flow graph. From the ADF UI, select **...** next to pipeline, and then select **Download support files**.
238
238
239
239
### How do I access data by using the other 90 dataset types in Data Factory?
240
240
@@ -244,23 +244,23 @@ sections:
244
244
245
245
### Is the self-hosted integration runtime available for data flows?
246
246
247
-
Self-hosted IR is an ADF pipeline construct that you can use with the Copy Activity to acquire or move data to and from on-prem or VM-based data sources and sinks. The virtual machines that you use for a self-hosted IR can also be placed inside of the same VNET as your protected data stores for access to those data stores from ADF. With data flows, you'll achieve these same end-results using the Azure IR with managed VNET instead.
247
+
Self-hosted IR is an ADF pipeline construct that you can use with the Copy Activity to acquire or move data to and from on-premises or VM-based data sources and sinks. The virtual machines that you use for a self-hosted IR can also be placed inside of the same VNET as your protected data stores for access to those data stores from ADF. With data flows, you'll achieve these same end-results using the Azure IR with managed VNET instead.
248
248
249
249
### Does the data flow compute engine serve multiple tenants?
250
250
251
251
Clusters are never shared. We guarantee isolation for each job run in production runs. In case of debug scenario one person gets one cluster, and all debugs will go to that cluster which are initiated by that user.
252
252
253
-
### Is there a way to write attributes in cosmos db in the same order as specified in the sink in ADF data flow?
253
+
### Is there a way to write attributes in Cosmos DB in the same order as specified in the sink in ADF data flow?
254
254
255
-
For cosmos DB, the underlying format of each document is a JSON object which is an unordered set of name/value pairs, so the order cannot be reserved.
255
+
For Cosmos DB, the underlying format of each document is a JSON object which is an unordered set of name/value pairs, so the order cannot be reserved.
256
256
257
257
### Why a user is unable to use data preview in the data flows?
258
258
259
259
You should check permissions for custom role. There are multiple actions involved in the dataflow data preview. You start by checking network traffic while debugging on your browser. Please follow all of the actions, for details, please refer to [Resource provider.](../role-based-access-control/resource-provider-operations.md#microsoftdatafactory)
260
260
261
261
### In ADF, can I calculate value for a new column from existing column from mapping?
262
262
263
-
You can use derive transformation in mapping data flow to create a new column on the logic you want. When creating a derived column, you can either generate a new column or update an existing one. In the Column textbox, enter in the column you are creating. To override an existing column in your schema, you can use the column dropdown. To build the derived column's expression, click on the Enter expression textbox. You can either start typing your expression or open up the expression builder to construct your logic.
263
+
You can use derive transformation in mapping data flow to create a new column on the logic you want. When creating a derived column, you can either generate a new column or update an existing one. In the Column textbox, enter in the column you are creating. To override an existing column in your schema, you can use the column dropdown. To build the derived column's expression, select on the Enter expression textbox. You can either start typing your expression or open up the expression builder to construct your logic.
264
264
265
265
### Why mapping data flow preview failing with Gateway timeout?
266
266
@@ -283,7 +283,7 @@ sections:
283
283
Data factory is available in following [regions.](https://azure.microsoft.com/global-infrastructure/services/?products=data-factory)
284
284
The Power Query feature is available in all data flow regions. If the feature is not available in your region, please check with support.
285
285
286
-
### What is the difference between mapping data flow and Power query actvity (data wrangling)?
286
+
### What is the difference between mapping data flow and Power query activity (data wrangling)?
287
287
288
288
Mapping data flows provide a way to transform data at scale without any coding required. You can design a data transformation job in the data flow canvas by constructing a series of transformations. Start with any number of source transformations followed by data transformation steps. Complete your data flow with a sink to land your results in a destination. Mapping data flow is great at mapping and transforming data with both known and unknown schemas in the sinks and sources.
Copy file name to clipboardExpand all lines: articles/data-factory/plan-manage-costs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -145,7 +145,7 @@ Budgets can be created with filters for specific resources or services in Azure
145
145
146
146
## Export cost data
147
147
148
-
You can also [export your cost data](../cost-management-billing/costs/tutorial-export-acm-data.md?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) to a storage account. This is helpful when you need or others to do additional data analysis for costs. For example, a finance teams can analyze the data using Excel or Power BI. You can export your costs on a daily, weekly, or monthly schedule and set a custom date range. Exporting cost data is the recommended way to retrieve cost datasets.
148
+
You can also [export your cost data](../cost-management-billing/costs/tutorial-export-acm-data.md?WT.mc_id=costmanagementcontent_docsacmhorizontal_-inproduct-learn) to a storage account. This is helpful when you need or others to do additional data analysis for costs. For example, finance teams can analyze the data using Excel or Power BI. You can export your costs on a daily, weekly, or monthly schedule and set a custom date range. Exporting cost data is the recommended way to retrieve cost datasets.
Copy file name to clipboardExpand all lines: articles/data-factory/pricing-examples-copy-transform-azure-databricks.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,11 +24,11 @@ Refer to the [Azure Pricing Calculator](https://azure.microsoft.com/pricing/calc
24
24
25
25
To accomplish the scenario, you need to create a pipeline with the following items:
26
26
27
-
1. One copy activity with an input dataset for the data to be copied from AWS S3, and an output dataset for the data on Azure storage.
28
-
2. One Azure Databricks activity for the data transformation.
29
-
3. One schedule trigger to execute the pipeline every hour. When you want to run a pipeline, you can either [trigger it immediately or schedule it](concepts-pipeline-execution-triggers.md). In addition to the pipeline itself, each trigger instance counts as a single Activity run.
27
+
- One copy activity with an input dataset for the data to be copied from AWS S3, and an output dataset for the data on Azure storage.
28
+
- One Azure Databricks activity for the data transformation.
29
+
- One schedule trigger to execute the pipeline every hour. When you want to run a pipeline, you can either [trigger it immediately or schedule it](concepts-pipeline-execution-triggers.md). In addition to the pipeline itself, each trigger instance counts as a single Activity run.
30
30
31
-
:::image type="content" source="media/pricing-concepts/scenario2.png" alt-text="Diagram shows a pipeline with a schedule trigger. In the pipeline, copy activity flows to an input dataset, an output dataset, and a DataBricks activity, which runs on Azure Databricks. The input dataset flows to an A W S S3 linked service. The output dataset flows to an Azure Storage linked service.":::
31
+
:::image type="content" source="media/pricing-concepts/scenario2.png" alt-text="Diagram shows a pipeline with a schedule trigger. In the pipeline, copy activity flows to an input dataset, an output dataset, and a DataBricks activity, which runs on Azure Databricks. The input dataset flows to an AWS S3 linked service. The output dataset flows to an Azure Storage linked service.":::
32
32
33
33
## Costs estimation
34
34
@@ -42,9 +42,9 @@ To accomplish the scenario, you need to create a pipeline with the following ite
42
42
43
43
**Total scenario pricing for 30 days: $122.03**
44
44
45
-
:::image type="content" source="media/pricing-concepts/scenario-2-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for a copy data and transform with Azure Databricks scenario.":::
45
+
:::image type="content" source="media/pricing-concepts/scenario-2-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for a copy data and transform with Azure Databricks scenario." lightbox="media/pricing-concepts/scenario-2-pricing-calculator.png":::
46
46
47
-
## Next Steps
47
+
## Next steps
48
48
49
49
-[Pricing example: Copy data from AWS S3 to Azure Blob storage hourly for 30 days](pricing-examples-s3-to-blob.md)
50
50
-[Pricing example: Copy data and transform with dynamic parameters hourly for 30 days](pricing-examples-copy-transform-dynamic-parameters.md)
Copy file name to clipboardExpand all lines: articles/data-factory/pricing-examples-copy-transform-dynamic-parameters.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,12 +24,12 @@ Refer to the [Azure Pricing Calculator](https://azure.microsoft.com/pricing/calc
24
24
25
25
To accomplish the scenario, you need to create a pipeline with the following items:
26
26
27
-
1. One copy activity with an input dataset for the data to be copied from AWS S3, an output dataset for the data on Azure storage.
28
-
2. One Lookup activity for passing parameters dynamically to the transformation script.
29
-
3. One Azure Databricks activity for the data transformation.
30
-
4. One schedule trigger to execute the pipeline every hour. When you want to run a pipeline, you can either [trigger it immediately or schedule it](concepts-pipeline-execution-triggers.md). In addition to the pipeline itself, each trigger instance counts as a single Activity run.
27
+
- One copy activity with an input dataset for the data to be copied from AWS S3, an output dataset for the data on Azure storage.
28
+
- One Lookup activity for passing parameters dynamically to the transformation script.
29
+
- One Azure Databricks activity for the data transformation.
30
+
- One schedule trigger to execute the pipeline every hour. When you want to run a pipeline, you can either [trigger it immediately or schedule it](concepts-pipeline-execution-triggers.md). In addition to the pipeline itself, each trigger instance counts as a single Activity run.
31
31
32
-
:::image type="content" source="media/pricing-concepts/scenario3.png" alt-text="Diagram shows a pipeline with a schedule trigger. In the pipeline, copy activity flows to an input dataset, an output dataset, and lookup activity that flows to a DataBricks activity, which runs on Azure Databricks. The input dataset flows to an A W S S3 linked service. The output dataset flows to an Azure Storage linked service.":::
32
+
:::image type="content" source="media/pricing-concepts/scenario3.png" alt-text="Diagram shows a pipeline with a schedule trigger. In the pipeline, copy activity flows to an input dataset, an output dataset, and lookup activity that flows to a DataBricks activity, which runs on Azure Databricks. The input dataset flows to an AWS S3 linked service. The output dataset flows to an Azure Storage linked service.":::
33
33
34
34
## Costs estimation
35
35
@@ -44,9 +44,9 @@ To accomplish the scenario, you need to create a pipeline with the following ite
44
44
45
45
**Total scenario pricing for 30 days: $122.09**
46
46
47
-
:::image type="content" source="media/pricing-concepts/scenario-3-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for a copy data and transform with dynamic parameters scenario.":::
47
+
:::image type="content" source="media/pricing-concepts/scenario-3-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for a copy data and transform with dynamic parameters scenario." lightbox="media/pricing-concepts/scenario-3-pricing-calculator.png":::
48
48
49
-
## Next Steps
49
+
## Next steps
50
50
51
51
-[Pricing example: Copy data from AWS S3 to Azure Blob storage hourly for 30 days](pricing-examples-s3-to-blob.md)
52
52
-[Pricing example: Copy data and transform with Azure Databricks hourly for 30 days](pricing-examples-copy-transform-azure-databricks.md)
Copy file name to clipboardExpand all lines: articles/data-factory/pricing-examples-data-integration-managed-vnet.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -41,9 +41,9 @@ To accomplish the scenario, you need to create two pipelines with the following
41
41
42
42
**Total scenario pricing for 30 days: $129.02**
43
43
44
-
:::image type="content" source="media/pricing-concepts/scenario-5-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for data integration with Managed VNET.":::
44
+
:::image type="content" source="media/pricing-concepts/scenario-5-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for data integration with Managed VNET." lightbox="media/pricing-concepts/scenario-5-pricing-calculator.png":::
45
45
46
-
## Next Steps
46
+
## Next steps
47
47
48
48
-[Pricing example: Copy data from AWS S3 to Azure Blob storage hourly for 30 days](pricing-examples-s3-to-blob.md)
49
49
-[Pricing example: Copy data and transform with Azure Databricks hourly for 30 days](pricing-examples-copy-transform-azure-databricks.md)
Copy file name to clipboardExpand all lines: articles/data-factory/pricing-examples-get-delta-data-from-sap-ecc.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -30,7 +30,7 @@ To accomplish the scenario, you need to create a pipeline with the following ite
30
30
31
31
## Costs estimation
32
32
33
-
In order to load data from SAP ECC via SAP CDC connector in Mapping Data Flow, you need to install your Self-Hosted Integration Runtime on an on-premise machine, or VM to directly connect to your SAP ECC system. Given that, you'll be charged on both Self-Hosted Integration Runtime with $0.10/hour and Mapping Data Flow with its vCore-hour price unit.
33
+
In order to load data from SAP ECC via SAP CDC connector in Mapping Data Flow, you need to install your Self-Hosted Integration Runtime on an on-premises machine, or VM to directly connect to your SAP ECC system. Given that, you'll be charged on both Self-Hosted Integration Runtime with $0.10/hour and Mapping Data Flow with its vCore-hour price unit.
34
34
35
35
Assuming every time it requires 15 minutes to complete the job, the cost estimations are as below.
36
36
@@ -44,9 +44,9 @@ Assuming every time it requires 15 minutes to complete the job, the cost estimat
44
44
45
45
**Total scenario pricing for 30 days: $17.21**
46
46
47
-
:::image type="content" source="media/pricing-concepts/scenario-6-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for getting delta data from SAP ECC via SAP CDC in mapping data flows.":::
47
+
:::image type="content" source="media/pricing-concepts/scenario-6-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for getting delta data from SAP ECC via SAP CDC in mapping data flows." lightbox="media/pricing-concepts/scenario-6-pricing-calculator.png":::
48
48
49
-
## Next Steps
49
+
## Next steps
50
50
51
51
-[Pricing example: Copy data from AWS S3 to Azure Blob storage hourly for 30 days](pricing-examples-s3-to-blob.md)
52
52
-[Pricing example: Copy data and transform with Azure Databricks hourly for 30 days](pricing-examples-copy-transform-azure-databricks.md)
Copy file name to clipboardExpand all lines: articles/data-factory/pricing-examples-s3-to-blob.md
+4-5Lines changed: 4 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -24,9 +24,8 @@ Refer to the [Azure Pricing Calculator](https://azure.microsoft.com/pricing/calc
24
24
25
25
To accomplish the scenario, you need to create a pipeline with the following items:
26
26
27
-
1. I'll copy data from AWS S3 to Azure Blob storage, and this will move 10 GB of data from S3 to blob storage. I estimate it will run for 2-3 hours, and I plan to set DIU as Auto.
28
-
29
-
3. A schedule trigger to execute the pipeline every hour for 8 hours every day. When you want to run a pipeline, you can either [trigger it immediately or schedule it](concepts-pipeline-execution-triggers.md). In addition to the pipeline itself, each trigger instance counts as a single Activity run.
27
+
- I'll copy data from AWS S3 to Azure Blob storage, and this will move 10 GB of data from S3 to blob storage. I estimate it will run for 2-3 hours, and I plan to set DIU as Auto.
28
+
- A schedule trigger to execute the pipeline every hour for 8 hours every day. When you want to run a pipeline, you can either [trigger it immediately or schedule it](concepts-pipeline-execution-triggers.md). In addition to the pipeline itself, each trigger instance counts as a single Activity run.
30
29
31
30
:::image type="content" source="media/pricing-concepts/scenario1.png" alt-text="Diagram shows a pipeline with a schedule trigger.":::
32
31
@@ -42,9 +41,9 @@ To accomplish the scenario, you need to create a pipeline with the following ite
42
41
43
42
**Total scenario pricing for 30 days: $122.00**
44
43
45
-
:::image type="content" source="media/pricing-concepts/scenario-1-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for an hourly pipeline run.":::
44
+
:::image type="content" source="media/pricing-concepts/scenario-1-pricing-calculator.png" alt-text="Screenshot of the pricing calculator configured for an hourly pipeline run." lightbox="media/pricing-concepts/scenario-1-pricing-calculator.png":::
46
45
47
-
## Next Steps
46
+
## Next steps
48
47
49
48
-[Pricing example: Copy data and transform with Azure Databricks hourly for 30 days](pricing-examples-copy-transform-azure-databricks.md)
50
49
-[Pricing example: Copy data and transform with dynamic parameters hourly for 30 days](pricing-examples-copy-transform-dynamic-parameters.md)
0 commit comments