Skip to content

Commit 3e12137

Browse files
authored
Added clarity
1 parent 4693fdc commit 3e12137

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

articles/data-factory/concepts-integration-runtime-performance.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,11 @@ A best practice is to start small and scale up to meet your performance needs.
5353

5454
Dataflow divides the data into partitions and transforms it using different processes. If the data size in a partition is more than the process can hold in memory, the process fails with OOM(out of memory) errors. If dataflow contains huge amounts of data having joins/aggregations, you may want to try changing shuffle partitions in incremental way. You can set it from 50 up to 2000, to avoid OOM errors. **Compute Custom properties** in dataflow runtime, is a way to control your compute requirements. Property name is **Shuffle partitions** and it's integer type. This customization should only be used in known scenarios, otherwise it can cause unnecessary dataflow failures.
5555

56-
While increasing the shuffle partitions, make sure data is spread across well. A rough number is to have approximately 1.5 GB of data per partition. If data is skewed, increasing the "Shuffle partitions" won't be helpful. For example, if you have 500 GB of data, having a value between 400 to 500 should work. Default limit for shuffle partitions is 200 which works well for ~300 GB of data.
56+
While increasing the shuffle partitions, make sure data is spread across well. A rough number is to have approximately 1.5 GB of data per partition. If data is skewed, increasing the "Shuffle partitions" won't be helpful. For example, if you have 500 GB of data, having a value between 400 to 500 should work. Default limit for shuffle partitions is 200 that works well for approximately 300 GB of data.
5757

5858
Here are the steps on how it's set in a custom integration runtime. You can't set it for autoresolve integrtaion runtime.
5959

60-
1. From ADF portal under Manage, select a custom itegration run time and you'll go to edit mode.
60+
1. From ADF portal under Manage, select a custom itegration run time and you go to edit mode.
6161
2. Under dataflow run time tab, go to Compute Cusotm Properties section.
6262
3. Select Shuffle Partitions under Property name, input value of your choice, like 250, 500 etc.
6363

0 commit comments

Comments
 (0)