Skip to content

Commit f8858fb

Browse files
khdowniekhdownie
committed
nconnect for NFS Azure file shares
1 parent 65123db commit f8858fb

File tree

1 file changed

+154
-0
lines changed

1 file changed

+154
-0
lines changed

‎articles/storage/files/nfs-nconnect-performance.md

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
---
2+
title: Improve NFS Azure file share performance with nconnect
3+
description: Learn how using nconnect with Linux clients can improve the performance of NFS Azure file shares at scale.
4+
author: khdownie
5+
ms.service: storage
6+
ms.topic: conceptual
7+
ms.date: 03/17/2023
8+
ms.author: kendownie
9+
ms.subservice: files
10+
---
11+
12+
# Improve NFS Azure file share performance with `nconnect`
13+
14+
`Nconnect` is a client-side Linux mount option that increases performance at scale by allowing you to use more TCP connections between the client and the Azure Premium Files service for NFSv4.1, while maintaining the resiliency of platform as a service (PaaS).
15+
16+
## Applies to
17+
| File share type | SMB | NFS |
18+
|-|:-:|:-:|
19+
| Standard file shares (GPv2), LRS/ZRS | ![No, this article doesn't apply to standard SMB Azure file shares LRS/ZRS.](../media/icons/no-icon.png) | ![NFS shares are only available in premium Azure file shares.](../media/icons/no-icon.png) |
20+
| Standard file shares (GPv2), GRS/GZRS | ![No, this article doesn't apply to standard SMB Azure file shares GRS/GZRS.](../media/icons/no-icon.png) | ![NFS is only available in premium Azure file shares.](../media/icons/no-icon.png) |
21+
| Premium file shares (FileStorage), LRS/ZRS | ![No, this article doesn't apply to premium SMB Azure file shares.](../media/icons/no-icon.png) | ![Yes, this article applies to premium NFS Azure file shares.](../media/icons/yes-icon.png) |
22+
23+
## Benefits of `nconnect`
24+
25+
With `nconnect` you can increase performance at scale using fewer client machines, lowering total cost of ownership (TCO). `Nconnect` accomplishes this by leveraging multiple TCP channels on one or more NICs, using single or multiple clients. Without `nconnect`, you'd need roughly 20 client machines in order to achieve the bandwidth scale limits (10 GiB/s) offered by the largest premium Azure file share provisioning size. With `nconnect`, you can achieve those limits using only 6-7 clients. That’s almost a 70% reduction in computing cost, while providing significant improvements to IOPS and throughput at scale (see table).
26+
27+
| **Metric (operation)** | **I/O size** | **Performance improvement** |
28+
|------------------------|---------------|-----------------------------|
29+
| IOPS (write) | 64K, 1024K | 3x |
30+
| IOPS (read) | All I/O sizes | 2-4x |
31+
| Throughput (write) | 64K, 1024K | 3x |
32+
| Throughput (read) | All I/O sizes | 2-4x |
33+
34+
35+
## Prerequisites
36+
37+
- The latest Linux distributions fully support `nconnect`. For older Linux distributions, ensure that the Linux kernel version is 5.3 or higher.
38+
- Per-mount configuration is only supported when a single file share is used per storage account over a private endpoint.
39+
40+
## Performance impact of `nconnect`
41+
42+
We achieved the following performance results when using the `nconnect` mount option with NFS Azure file shares on Linux clients at scale. For more information, see [performance test configuration](#performance-test-configuration).
43+
44+
## Recommendations
45+
46+
Following these recommendations will help you get the best results from `nconnect`.
47+
48+
### Set `nconnect=4`
49+
While Azure Files supports setting `nconnect` up to the maximum setting of 16, we recommend configuring the mount options with the optimal setting of `nconnect=4`. Currently, there are no gains beyond 4 channels for the Azure Files implementation of `nconnect`. In fact, exceeding 4 channels to a single Azure file share from a single client might adversely effect performance due to TCP network saturation.
50+
51+
### Size virtual machines carefully
52+
Depending on your workload requirements, it’s important to correctly size the client machines to avoid being restricted by their [expected network bandwidth](../../virtual-network/virtual-machine-network-throughput.md#expected-network-throughput). Note that you don't need multiple NICs in order to achieve the expected network throughput. While it's common to use [general purpose VMs](../../virtual-machines/sizes-general.md) with Azure Files, a variety of VM types are available depending on your workload needs and region availability. For more information, see [Azure VM Selector](https://azure.microsoft.com/pricing/vm-selector/).
53+
54+
### Keep queue depth less than or equal to 64
55+
Queue depth is the number of pending I/O requests that a storage resource can service. We don't recommend exceeding the optimal queue depth of 64. If you do, you won't see any additional performance gains. For more information, see [Queue depth](understand-performance.md#queue-depth).
56+
57+
### `Nconnect` per-mount configuration
58+
If a workload requires you to mount multiple shares with one or more storage accounts with different `nconnect` settings from a single client, we can't guarantee that those settings will persist when mounting over the public endpoint. Per-mount configuration is only supported when a single Azure file share is used per storage account over the private endpoint as described in Scenario 2.
59+
60+
#### Scenario 1: (not supported) `nconnect` per-mount configuration over public endpoint
61+
62+
- StorageAccount.file.core.windows.net = 52.239.238.8
63+
- StorageAccount2.file.core.windows.net = 52.239.238.7
64+
- `Mount StorageAccount.file.core.windows.net:/FileShare1 nconnect=4`
65+
- `Mount StorageAccount.file.core.windows.net:/FileShare2`
66+
- `Mount StorageAccount2.file.core.windows.net:/FileShare1`
67+
68+
> [!NOTE]
69+
> Even if the storage account resolves to different IP address, we can't guarantee that address will persist because public endpoints aren't static addresses.
70+
71+
#### Scenario 2: (supported) `nconnect` per-mount configuration over private endpoint with multiple storage accounts
72+
73+
- StorageAccount.file.core.windows.net = 10.10.10.10
74+
- StorageAccount2.file.core.windows.net = 10.10.10.11
75+
- `Mount StorageAccount.file.core.windows.net:/FileShare1 nconnect=4`
76+
- `Mount StorageAccount2.file.core.windows.net:/FileShare1`
77+
78+
#### Scenario 3: (not supported) `nconnect` per-mount configuration over private endpoint with multiple shares on single storage account
79+
80+
- StorageAccount.file.core.windows.net = 10.10.10.10
81+
- `Mount StorageAccount.file.core.windows.net:/FileShare1 nconnect=4`
82+
- `Mount StorageAccount.file.core.windows.net:/FileShare2`
83+
- `Mount StorageAccount.file.core.windows.net:/FileShare3`
84+
85+
## Performance test configuration
86+
87+
We used the following resources and benchmarking tools to achieve and measure the results outlined in this article.
88+
89+
- **Single client:** Azure Virtual Machine ([DSv4-Series](../../virtual-machines/dv4-dsv4-series.md#dsv4-series)) with single NIC
90+
- **OS:** Linux (Ubuntu 20.40)
91+
- **NFS storage:** Azure Files premium file share (provisioned 30 TiB, set `nconnect=4`)
92+
93+
| **Size** | **vCPU** | **Memory** | **Temp storage (SSD)** | **Max data disks** | **Max NICs** | **Expected network bandwidth** |
94+
|-----------------|-----------|------------|------------------------|--------------------|--------------|--------------------------------|
95+
| Standard_D16_v4 | 16 | 64 GiB | Remote storage only | 32 | 8 | 12,500 Mbps |
96+
97+
### Benchmarking tools and tests
98+
99+
We used Flexible I/O Tester (FIO), a free, open-source disk I/O tool used both for benchmark and stress/hardware verification. To install FIO, follow the Binary Packages section in the [FIO README file](https://github.com/axboe/fio#readme) to install for the platform of your choice.
100+
101+
While these tests focus on random I/O access patterns, you'll get similar results when using sequential I/O.
102+
103+
#### High IOPS: 100% reads
104+
105+
- 4k I/O size - random read - 64 queue depth
106+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=4k --iodepth=64 --filesize=4G --rw=randread --group_reporting --ramp_time=300`
107+
108+
- 8k I/O size - random read - 64 queue depth
109+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=8k --iodepth=64 --filesize=4G --rw=randread --group_reporting --ramp_time=300`
110+
111+
#### High throughput: 100% reads
112+
113+
- 64k I/O size - random read - 64 queue depth
114+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=64k --iodepth=64 --filesize=4G --rw=randread --group_reporting --ramp_time=300`
115+
116+
- 1024k I/O size - 100% random read - 64 queue depth
117+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=1024k --iodepth=64 --filesize=4G --rw=randread --group_reporting --ramp_time=300`
118+
119+
#### High IOPS: 100% writes
120+
121+
- 4k I/O size - 100% random write - 64 queue depth
122+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=4k --iodepth=64 --filesize=4G --rw=randwrite --group_reporting --ramp_time=300`
123+
124+
- 8k I/O size - 100% random write - 64 queue depth
125+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=8k --iodepth=64 --filesize=4G --rw=randwrite --group_reporting --ramp_time=300`
126+
127+
#### High throughput: 100% writes
128+
129+
- 64k I/O size - 100% random write - 64 queue depth
130+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=64k --iodepth=64 --filesize=4G --rw=randwrite --group_reporting --ramp_time=300`
131+
132+
- 1024k I/O size - 100% random write - 64 queue depth
133+
`fio --ioengine=libaio --direct=1 --nrfiles=4 --numjobs=1 --runtime=1800 --time_based --bs=1024k --iodepth=64 --filesize=4G --rw=randwrite --group_reporting --ramp_time=300`
134+
135+
## Performance considerations
136+
137+
When using the `nconnect` mount option, you should closely evaluate workloads that have the following characteristics:
138+
139+
- Latency sensitive write workloads that are single threaded and/or use a low queue depth (less than 16)
140+
- Latency sensitive read workloads that are single threaded and/or use a low queue depth in combination with smaller I/O sizes
141+
142+
Not all workloads require high-scale IOPS or throughout performance. If you're running smaller scale workloads, `nconnect` might not make sense for your workload.
143+
144+
Use the following tables for reference.
145+
146+
147+
148+
149+
150+
151+
## See also
152+
- For mounting instructions, see [Mount NFS file Share to Linux](storage-files-how-to-mount-nfs-shares).
153+
- For additional mount options, see [Linux NFS man page](https://linux.die.net/man/5/nfs).
154+
- For information on latency, IOPS, throughput, and other performance concepts, see [Understand Azure Files performance](understand-performance.md).

0 commit comments

Comments
 (0)