Skip to content

Commit bf6c4f1

Browse files
authored
Improved for tone
1 parent e92ed27 commit bf6c4f1

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

articles/data-factory/concepts-integration-runtime-performance.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,13 @@ A best practice is to start small and scale up to meet your performance needs.
5151

5252
## Custom shuffle partition
5353

54-
Dataflow divides the data into partitions and transforms it using different processes. If the data size in a partition is more than the process can hold in memory, the process fails with OOM(out of memory) errors. If dataflow contains huge amounts of data having joins/aggregations, you may want to try changing shuffle partitions in incremental way. You can set it from 50 up to 2000, to avoid OOM errors. **Compute Custom properties** in dataflow runtime, is a way to control your compute requirements. Property name is **Shuffle partitions** and it is integer type. This customization should only be used in known scenarios, otherwise it can cause unnecessary dataflow failures.
54+
Dataflow divides the data into partitions and transforms it using different processes. If the data size in a partition is more than the process can hold in memory, the process fails with OOM(out of memory) errors. If dataflow contains huge amounts of data having joins/aggregations, you may want to try changing shuffle partitions in incremental way. You can set it from 50 up to 2000, to avoid OOM errors. **Compute Custom properties** in dataflow runtime, is a way to control your compute requirements. Property name is **Shuffle partitions** and it's integer type. This customization should only be used in known scenarios, otherwise it can cause unnecessary dataflow failures.
5555

56-
While increasing the shuffle partitions, make sure data is spread across well. A rough number is to have approximately 1.5 GB of data per partition, If data is skewed, increasing the "Shuffle partitions" will not be helpful. For example, if you have 500 GB of data, having a value between 400 to 500 should work. Default limit for shuffle partitions is 200 which works well for ~300 GB of data.
56+
While increasing the shuffle partitions, make sure data is spread across well. A rough number is to have approximately 1.5 GB of data per partition. If data is skewed, increasing the "Shuffle partitions" won't be helpful. For example, if you have 500 GB of data, having a value between 400 to 500 should work. Default limit for shuffle partitions is 200 which works well for ~300 GB of data.
5757

58-
Here are the steps on how it is set in a custom integration runtime. You cannot set it for autoresolve integrtaion runtime.
58+
Here are the steps on how it's set in a custom integration runtime. You can't set it for autoresolve integrtaion runtime.
5959

60-
1. From ADF portal under Manage, select a custom itegration run time and you will go to edit mode.
60+
1. From ADF portal under Manage, select a custom itegration run time and you'll go to edit mode.
6161
2. Under dataflow run time tab, go to Compute Cusotm Properties section.
6262
3. Select Shuffle Partitions under Property name, input value of your choice, like 250, 500 etc.
6363

0 commit comments

Comments
 (0)