hpc-cache: add NAS troubleshooting article

Author: ekpgh

articles/hpc-cache/hpc-cache-prereqs.md

@@ -4,7 +4,7 @@ description: Prerequisites for using Azure HPC Cache
 author: ekpgh
 ms.service: hpc-cache
 ms.topic: conceptual
-ms.date: 02/12/2020
+ms.date: 02/20/2020
 ms.author: rohogue
 ---
 
@@ -90,7 +90,9 @@ If using an NFS storage system (for example, an on-premises hardware NAS system)
 > [!NOTE]
 > Storage target creation will fail if the cache has insufficient access to the NFS storage system.
 
-* **Network connectivity:** The Azure HPC Cache needs high-bandwidth network access between the cache subnet and the NFS system's data center. [ExpressRoute](https://docs.microsoft.com/azure/expressroute/) or similar access is recommended. If using a VPN, you might need to configure it to clamp TCP MSS at 1350 to make sure large packets are not blocked.
+More information is included in [Troubleshoot NAS configuration and NFS storage target issues](troubleshoot-nas.md).
+
+* **Network connectivity:** The Azure HPC Cache needs high-bandwidth network access between the cache subnet and the NFS system's data center. [ExpressRoute](https://docs.microsoft.com/azure/expressroute/) or similar access is recommended. If using a VPN, you might need to configure it to clamp TCP MSS at 1350 to make sure large packets are not blocked. Read [VPN packet size restrictions](troubleshoot-nas.md#adjust-vpn-packet-size-restrictions) for additional help troubleshooting VPN settings.
 
 * **Port access:** The cache needs access to specific TCP/UDP ports on your storage system. Different types of storage have different port requirements.
 
@@ -104,6 +106,8 @@ If using an NFS storage system (for example, an on-premises hardware NAS system)
     rpcinfo -p <storage_IP> |egrep "100000\s+4\s+tcp|100005\s+3\s+tcp|100003\s+3\s+tcp|100024\s+1\s+tcp|100021\s+4\s+tcp"| awk '{print $4 "/" $3 " " $5}'|column -t
     ```
 
+  Make sure that all of the ports returned by the ``rpcinfo`` query allow unrestricted traffic from the Azure HPC Cache's subnet.
+
   * In addition to the ports returned by the `rpcinfo` command, make sure that these commonly used ports allow inbound and outbound traffic:
 
     | Protocol | Port  | Service  |
@@ -116,17 +120,21 @@ If using an NFS storage system (for example, an on-premises hardware NAS system)
 
   * Check firewall settings to be sure that they allow traffic on all of these required ports. Be sure to check firewalls used in Azure as well as on-premises firewalls in your data center.
 
-* **Directory access:** Enable the `showmount` command on the storage system. Azure HPC Cache uses this command to check that your storage target configuration points to a valid export, and also to make sure that multiple mounts don't access the same subdirectories (which risks file collisions).
+* **Directory access:** Enable the `showmount` command on the storage system. Azure HPC Cache uses this command to check that your storage target configuration points to a valid export, and also to make sure that multiple mounts don't access the same subdirectories (a risk for file collision).
 
   > [!NOTE]
   > If your NFS storage system uses NetApp's ONTAP 9.2 operating system, **do not enable `showmount`**. [Contact Microsoft Service and Support](hpc-cache-support-ticket.md) for help.
 
+  Learn more about directory listing access in the NFS storage target [troubleshooting article](troubleshoot-nas.md#enable-export-listing).
+
 * **Root access:** The cache connects to the back-end system as user ID 0. Check these settings on your storage system:
   
   * Enable `no_root_squash`. This option ensures that the remote root user can access files owned by root.
 
   * Check export policies to make sure they do not include restrictions on root access from the cache's subnet.
 
+  * If your storage has any exports that are subdirectories of another export, make sure the cache has root access to the lowest segment of the path. Read [Root access on directory paths](troubleshoot-nas.md#allow-root-access-on-directory-paths) in the NFS storage target troubleshooting article for details.
+
 * NFS back-end storage must be a compatible hardware/software platform. Contact the Azure HPC Cache team for details.
 
 ## Next steps

articles/hpc-cache/index.yml

@@ -59,7 +59,11 @@ landingContent:
             url: hpc-cache-edit-storage.md
           - text: Work around firewall settings to create Blob storage targets
             url: hpc-cache-blob-firewall-fix.md
+          - text: Troubleshoot NFS storage target creation
+            url: troubleshoot-nas.md
       - linkListType: concept
         links:
           - text: Recover from a regional outage
             url: hpc-region-recovery.md
+          - text: Use Azure NetApp Files storage targets
+            url: hpc-cache-netapp.md

articles/hpc-cache/toc.yml

@@ -46,6 +46,8 @@
     href: hpc-cache-support-ticket.md
   - name: Work around Blob storage account firewall settings
     href: hpc-cache-blob-firewall-fix.md
+  - name: Troubleshoot NFS storage target creation
+    href: troubleshoot-nas.md
   - name: Recover from a regional outage
     href: hpc-region-recovery.md
   - name: Use Azure NetApp Files with Azure HPC Cache

articles/hpc-cache/troubleshoot-nas.md

@@ -0,0 +1,133 @@
+---
+title: Troubleshoot Azure HPC Cache NFS storage targets
+description: Tips to avoid and fix configuration errors and other problems that can cause failure when creating an NFS storage target
+author: ekpgh
+ms.service: hpc-cache
+ms.topic: conceptual
+ms.date: 02/20/2020
+ms.author: rohogue
+---
+
+# Troubleshoot NAS configuration and NFS storage target issues
+
+This article gives solutions for some common configuration errors and other issues that might prevent Azure HPC Cache from adding an NFS storage system as a storage target.
+
+If your problem is not included here, please [open a support ticket](hpc-cache-support-ticket.md) so that Microsoft Service and Support can investigate and solve the problem.
+
+Before using this guide, read and check the [prerequisites for NFS storage targets](hpc-cache-prereqs.md#nfs-storage-requirements).
+
+This article includes details about how to check ports and how to enable root access, plus information about other issues that might cause NFS storage target creation to fail.
+
+## Check port settings
+
+Azure HPC Cache needs read/write access to several UDP/TCP ports on the back-end storage system. Make sure these ports are accessible on the NAS system and that traffic is permitted through its firewalls. You might need to work with firewall and network administrators for your data center to verify this configuration.
+
+The ports are different for storage systems from different vendors, so check your system's requirements when setting up a storage target.
+
+In general, the cache needs access to these ports:
+
+| Protocol | Port  | Service  |
+|----------|-------|----------|
+| TCP/UDP  | 111   | rpcbind  |
+| TCP/UDP  | 2049  | NFS      |
+| TCP/UDP  | 4045  | nlockmgr |
+| TCP/UDP  | 4046  | mountd   |
+| TCP/UDP  | 4047  | status   |
+
+To learn the specific ports needed for your system, use the following ``rpcinfo`` command. This command  below lists the ports and formats the relevant results in a table. (Use your system's IP address in place of the *<storage_IP>* term.)
+
+You can issue this command from any Linux client that has NFS infrastructure installed. If you use a client inside the cluster subnet, it also can help verify connectivity between the subnet and the storage system.
+
+```bash
+rpcinfo -p <storage_IP> |egrep "100000\s+4\s+tcp|100005\s+3\s+tcp|100003\s+3\s+tcp|100024\s+1\s+tcp|100021\s+4\s+tcp"| awk '{print $4 "/" $3 " " $5}'|column -t
+```
+
+Make sure that all of the ports returned by the ``rpcinfo`` query allow unrestricted traffic from the Azure HPC Cache's subnet.
+
+Check these settings both on the NAS itself as well as on any firewalls between the storage system and the cache subnet.
+
+## Check root access
+
+Azure HPC Cache needs access to your storage system's exports to create the storage target. Specifically, it mounts the exports as user ID 0.
+
+Different storage systems use different method to enable this access:
+
+* Linux servers generally add ``no_root_squash`` to the exported path in ``/etc/exports``.
+* NetApp and EMC systems typically control access with export rules that are tied to specific IP addresses or networks.
+
+If using export rules, remember that the cache can use multiple different IP addresses from the cache subnet. Allow access from the full range of possible subnet IP addresses.
+
+Work with your NAS storage vendor to enable the right level of access for the cache.
+
+### Allow root access on directory paths
+<!-- linked in prereqs article -->
+
+For NAS systems that export hierarchical directories, Azure HPC Cache needs root access to each export level.
+
+For example, a system might show three exports like these, where the ``/ifs/accounting/payroll`` export is a child of the export ``/ifs/accounting``:
+
+```bash
+/ifs
+/ifs/accounting
+/ifs/accounting/payroll
+```
+
+If you add the ``payroll`` export as an HPC cache storage target, the cache actually mounts ``/ifs/`` and accesses the payroll directory from there. So Azure HPC Cache needs root access to ``/ifs`` in order to access the ``/ifs/accounting/payroll`` export.
+
+This requirement is related to the way the cache indexes files and avoids file collisions. A NAS system with hierarchical exports can give clients different file handles for the same file, based on which export was used. The storage system aliases the file handles internally, but Azure HPC Cache cannot tell which file handles in its index reference the same item. So it is possible that the cache can have different writes cached for the same file, and apply them incorrectly because it does not know that they are the same file.
+
+To avoid this possible file collision for files in multiple exports, Azure HPC Cache automatically mounts the shallowest available export in the path (``/ifs`` in the example) and uses the file handle given from that export. If multiple exports use the same base path, Azure HPC Cache needs root access to that path.
+
+## Enable export listing
+<!-- link in prereqs article -->
+
+The NAS must list its exports when the Azure HPC Cache queries it.
+
+On most NFS storage systems, you can test this by sending the following query from a Linux client: ``showmount -e <storage IP address>``
+
+Use a Linux client from the same virtual network as your cache, if possible.
+
+If that command doesn't list the exports, the cache will have trouble connecting to your storage system. Work with your NAS vendor to enable export listing.
+
+## Adjust VPN packet size restrictions
+<!-- link in prereqs article -->
+
+If you have a VPN between the cache and your NAS device, the VPN might block full-sized 1500-byte Ethernet packets. You might have this problem if large exchanges between the NAS and the Azure HPC Cache instance do not complete, but smaller updates work as expected.
+
+There isn't a simple way to tell whether or not your system has this problem, but here are a few methods to diagnose it.
+
+* Use packet sniffers on both sides of the VPN to detect which packets transfer successfully.
+* If your VPN allows ping commands, you can test sending a full-sized packet.
+
+  Run a ping command over the VPN to the NAS with these options. (Use your storage system's IP address in place of the *<storage_IP>* value.)
+
+   ```bash
+   ping -M do -s 1472 -c 1 <storage_IP>
+   ```
+
+  These are the options in the command:
+
+  * ``-M do`` - Do not fragment
+  * ``-c 1`` - Send only one packet
+  * ``-s 1472`` - Set the size of the payload to 1472 bytes. This is the maximum size payload for a 1500-byte packet after accounting for the Ethernet overhead.
+
+  A successful response looks like this:
+
+  ```bash
+  PING 10.54.54.11 (10.54.54.11) 1472(1500) bytes of data.
+  1480 bytes from 10.54.54.11: icmp_seq=1 ttl=64 time=2.06 ms
+  ```
+
+  If the ping fails with 1472 bytes, you might need to configure MSS clamping on the VPN to make the remote system properly detect the maximum frame size. Read the [VPN Gateway IPsec/IKE parameters documentation](../vpn-gateway/vpn-gateway-about-vpn-devices#ipsec) for more information.
+
+## Check for ACL security style
+
+Some NAS systems use a hybrid security style that combines access control lists (ACLs) with traditional POSIX or UNIX security.
+
+If your system reports its security style as UNIX or POSIX without including the acronym "ACL", this issue does not affect you.
+
+For systems that use ACLs, Azure HPC Cache needs to track additional user-specific values in order to control file access. This is done by enabling an access cache. There isn't a user-facing control to turn on the access cache, but you can open a support ticket to request that it be enabled for the affected storage targets on your cache system.
+
+## Next steps
+
+If you have a problem that was not addressed in this article, [open a support ticket](hpc-cache-support-ticket.md) to get expert help.