testing methodology topic

b-ahibbard · b-ahibbard · commit 1064b965a834 · 2024-10-31T13:01:33.000-04:00
diff --git a/articles/azure-netapp-files/TOC.yml b/articles/azure-netapp-files/TOC.yml
@@ -109,6 +109,8 @@
       items:
         - name: General performance considerations for Azure NetApp Files
           href: azure-netapp-files-performance-considerations.md  
+        - name: Understand workload types
+          href: workload-types.md
         - name: Linux direct I/O best practices
           href: performance-linux-direct-io.md
         - name: Linux filesystem cache best practices
@@ -133,6 +135,8 @@
       items:
       - name: Performance benchmark test recommendations for Azure NetApp Files
         href: azure-netapp-files-performance-metrics-volumes.md 
+      - name: Testing methodology
+        href: testing-methodology.md
       - name: Regular volume performance benchmarks for Linux
         href: performance-benchmarks-linux.md
       - name: Large volume performance benchmarks for Linux
diff --git a/articles/azure-netapp-files/azure-netapp-files-performance-metrics-volumes.md b/articles/azure-netapp-files/azure-netapp-files-performance-metrics-volumes.md
@@ -5,7 +5,7 @@ author: b-hchen
 ms.author: anfdocs
 ms.service: azure-netapp-files
 ms.topic: conceptual
-ms.date: 05/08/2023
+ms.date: 10/31/2024
 ---
 # Performance benchmark test recommendations for Azure NetApp Files
 
@@ -16,9 +16,9 @@ This article provides benchmark testing recommendations for volume performance a
 To understand the performance characteristics of an Azure NetApp Files volume, you can use the open-source tool [FIO](https://github.com/axboe/fio) to run a series of benchmarks to simulate various workloads. FIO can be installed on both Linux and Windows-based operating systems.  It is an excellent tool to get a quick snapshot of both IOPS and throughput for a volume.
 
 > [!IMPORTANT]
-> Azure NetApp Files does *not* recommend using the `dd` utility as a baseline benchmarking tool. You should use an actual application workload, workload simulation, and benchmarking and analyzing tools (for example, Oracle AWR with Oracle, or the IBM equivalent for DB2) to establish and analyze optimal infrastructure performance. Tools such as FIO, vdbench, and iometer have their places in determining virtual machines to storage limits, matching the parameters of the test to the actual application workload mixtures for most useful results. However, it is always best to test with the real-world application. 
+> Azure NetApp Files does *not* recommend using the `dd` utility as a baseline benchmarking tool. You should use an actual application workload, workload simulation, and benchmarking and analyzing tools (for example, Oracle AWR with Oracle, or the IBM equivalent for Db2) to establish and analyze optimal infrastructure performance. Tools such as FIO, vdbench, and iometer have their places in determining virtual machines to storage limits, matching the parameters of the test to the actual application workload mixtures for most useful results. However, it is always best to test with the real-world application. 
  
-### VM instance sizing
+### Virtual machine (VM) instance sizing
 
 For best results, ensure that you are using a virtual machine (VM) instance that is appropriately sized to perform the tests. The following examples use a Standard_D32s_v3 instance. For more information about VM instance sizes, see [Sizes for Windows virtual machines in Azure](/azure/virtual-machines/sizes?toc=%2fazure%2fvirtual-network%2ftoc.json) for Windows-based VMs, and [Sizes for Linux virtual machines in Azure](/azure/virtual-machines/sizes?toc=%2fazure%2fvirtual-machines%2flinux%2ftoc.json) for Linux-based VMs.
 
@@ -48,50 +48,9 @@ Follow the Getting started section in the SSB README file to install for the pla
 
 ### FIO 
 
-Flexible I/O Tester (FIO) is a free and open-source disk I/O tool used both for benchmark and stress/hardware verification. 
+Flexible I/O Tester (FIO) is a free and open-source disk I/O tool used both for benchmark and stress/hardware verification. FIO is available in binary format for both Linux and Windows. 
 
-FIO is available in binary format for both Linux and Windows. 
-
-#### Installation of FIO
-
-Follow the Binary Packages section in the [FIO README file](https://github.com/axboe/fio#readme) to install for the platform of your choice.
-
-#### FIO examples for IOPS 
-
-The FIO examples in this section use the following setup:
-* VM instance size: D32s_v3
-* Capacity pool service level and size: Premium / 50 TiB
-* Volume quota size: 48 TiB
-
-The following examples show the FIO random reads and writes.
-
-##### FIO: 8k block size 100% random reads
-
-`fio --name=8krandomreads --rw=randread --direct=1 --ioengine=libaio --bs=8k --numjobs=4 --iodepth=128 --size=4G --runtime=600 --group_reporting`
-
-##### FIO: 8k block size 100% random writes
-
-`fio --name=8krandomwrites --rw=randwrite --direct=1 --ioengine=libaio --bs=8k --numjobs=4 --iodepth=128  --size=4G --runtime=600 --group_reporting`
-
-##### Benchmark results
-
-For official benchmark results for how FIO performs in Azure NetApp Files, see [Azure NetApp Files performance benchmarks for Linux](performance-benchmarks-linux.md).
-
-#### FIO examples for bandwidth
-
-The examples in this section show the FIO sequential reads and writes.
-
-##### FIO: 64k block size 100% sequential reads
-
-`fio --name=64kseqreads --rw=read --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128  --size=4G --runtime=600 --group_reporting`
-
-##### FIO: 64k block size 100% sequential writes
-
-`fio --name=64kseqwrites --rw=write --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128  --size=4G --runtime=600 --group_reporting`
-
-##### Benchmark results
-
-For official benchmark results for how FIO performs in Azure NetApp Files, see [Azure NetApp Files performance benchmarks for Linux](performance-benchmarks-linux.md).
+For more information, see [Understand Azure NetApp Files testing methodology](testing-methodology.md).
 
 ## Volume metrics
 
@@ -129,3 +88,4 @@ The following example shows a GET URL for viewing logical volume size:
 
 - [Service levels for Azure NetApp Files](azure-netapp-files-service-levels.md)
 - [Performance benchmarks for Linux](performance-benchmarks-linux.md)
+- [Understand Azure NetApp Files testing methodology](testing-methodology.md)
diff --git a/articles/azure-netapp-files/large-volumes.md b/articles/azure-netapp-files/large-volumes.md
@@ -55,10 +55,11 @@ Large volumes allow workloads to extend beyond the current limitations of regula
 | Volume type | Primary use cases | 
 | - | -- |
 | Regular volumes | <ul><li>General file shares</li><li>SAP HANA and databases (Oracle, SQL Server, Db2, and others)</li><li>VDI/Azure VMware Service</li><li>Capacities less than 50 TiB</li></ul> |
-| Large volumes | <ul><li>General file shares</li><li>High file count or high metadata workloads (such as electronic design automation, software development, FSI)</li><li>High capacity workloads (such as AI/ML/LLP, oil & gas, media, healthcare images, backup, and archives)</li><li>Large-scale workloads (many client connections such as FSLogix profiles)</li><li>High performance workloads</li><li>Capacity quotas between 50 TiB and 1 PiB</li></ul> |
+| Large volumes | <ul><li>General file shares</li><li>High file count or high metadata workloads (such as electronic design automation, software development, financial services)</li><li>High capacity workloads (such as AI/ML/LLP, oil & gas, media, healthcare images, backup, and archives)</li><li>Large-scale workloads (many client connections such as FSLogix profiles)</li><li>High performance workloads</li><li>Capacity quotas between 50 TiB and 1 PiB</li></ul> |
 
 ## More information
 
 * [Requirements and considerations for large volumes](large-volumes-requirements-considerations.md)
 * [Storage hierarchy of Azure NetApp Files](azure-netapp-files-understand-storage-hierarchy.md)
 * [Resource limits for Azure NetApp Files](azure-netapp-files-resource-limits.md)
+* [Understand workload types in Azure NetApp Files](workload-types.md)
diff --git a/articles/azure-netapp-files/testing-methodology.md b/articles/azure-netapp-files/testing-methodology.md
@@ -0,0 +1,89 @@
+---
+title: Understand performance testing methodology in Azure NetApp Files
+description: Learn how Azure NetApp Files benchmark tests are conducted. 
+services: azure-netapp-files
+author: b-ahibbard
+ms.service: azure-netapp-files
+ms.topic: conceptual
+ms.date: 10/31/2024
+ms.author: anfdocs
+---
+
+# Understand performance testing methodology in Azure NetApp Files
+
+The benchmark tool used in these tests is called [Flexible I/O Tester (FIO)](https://fio.readthedocs.io/en/latest/fio_doc.html).  
+
+When testing the edges of performance limits for storage, workload generation must be **highly parallelized** to achieve the maximum results possible.  
+
+That means: 
+- one, to many clients 
+- multiple CPUs 
+- multiple threads  
+- performing I/O to multiple files  
+- multi-threaded network connections (such as nconnect) 
+
+The end goal is to push the storage system as far as it can go before operations must begin to wait for other operations to finish. Use of a single client traversing a single network flow, or reading/writing from/to a single file (for instance, using dd or diskspd on a single client) doesn't deliver results indicative of Azure NetApp Files' capability. Instead, these setups show the performance of a single file, which generally trends with line speed and/or the Azure NetApp File [QoS settings](azure-netapp-files-understand-storage-hierarchy.md#qos_types). 
+
+In addition, caching must be minimized as much as possible to achieve accurate, representative results of what the storage can accomplish. However, caching is a very real tool for modern applications to perform at their absolute best. These cover scenarios with some caching and with caching bypassed for random I/O workloads by using randomization of the workload via FIO options (specifically, `randrepeat=0` to prevent caching on the storage and [directio](performance-linux-direct-io.md) to prevent client caching). 
+
+## About Flexible I/O tester 
+
+Flexible I/O tester (FIO) is an open source workload generation tool commonly used for storage benchmarking due to its ease of use and flexibility in defining workload patterns. For information about its use with Azure NetApp Files, see [Performance benchmark test recommendations for Azure NetApp Files](azure-netapp-files-performance-metrics-volumes.md).
+
+### Installation of FIO
+
+Follow the Binary Packages section in the [FIO README file](https://github.com/axboe/fio#readme) to install for the platform of your choice.
+
+### FIO examples for IOPS 
+
+The FIO examples in this section use the following setup:
+* VM instance size: D32s_v3
+* Capacity pool service level and size: Premium / 50 TiB
+* Volume quota size: 48 TiB
+
+The following examples show the FIO random reads and writes.
+
+#### FIO: 8k block size 100% random reads
+
+`fio --name=8krandomreads --rw=randread --direct=1 --ioengine=libaio --bs=8k --numjobs=4 --iodepth=128 --size=4G --runtime=600 --group_reporting`
+
+#### FIO: 8k block size 100% random writes
+
+`fio --name=8krandomwrites --rw=randwrite --direct=1 --ioengine=libaio --bs=8k --numjobs=4 --iodepth=128  --size=4G --runtime=600 --group_reporting`
+
+#### Benchmark results
+
+For official benchmark results for how FIO performs in Azure NetApp Files, see [Azure NetApp Files performance benchmarks for Linux](performance-benchmarks-linux.md).
+
+### FIO examples for bandwidth
+
+The examples in this section show the FIO sequential reads and writes.
+
+#### FIO: 64k block size 100% sequential reads
+
+`fio --name=64kseqreads --rw=read --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128  --size=4G --runtime=600 --group_reporting`
+
+#### FIO: 64k block size 100% sequential writes
+
+`fio --name=64kseqwrites --rw=write --direct=1 --ioengine=libaio --bs=64k --numjobs=4 --iodepth=128  --size=4G --runtime=600 --group_reporting`
+
+#### Benchmark results
+
+For official benchmark results for how FIO performs in Azure NetApp Files, see [Azure NetApp Files performance benchmarks for Linux](performance-benchmarks-linux.md).
+
+## Caching with FIO 
+
+FIO can be run with specific options to control how a performance benchmark reads and writes files. In the benchmarks tests with caching excluded, the FIO flag `randrepeat=0` was used to avoid caching by running a true random workload rather than a repeated pattern. 
+
+**[`randrepeat`](https://fio.readthedocs.io/latest/fio_doc.html#cmdoption-arg-randrepeat)**
+
+By default, when `randrepeat` isn't defined, the FIO tool sets the value to "true," meaning that the data produced in the files isn't truly random. Thus, filesystem caches aren't utilized to improve overall performance of the workload.  
+
+In the original benchmarks for Azure NetApp Files, `randrepeat` wasn't defined, so some filesystem caching was implemented. In the new tests, this option is set to “0” (or, false) to ensure there is adequate randomness in the data to avoid filesystem caches in the Azure NetApp Files service. This results in slightly lower overall numbers, but is a more accurate representation of what the storage itself is capable of when caching is bypassed. 
+
+## Next steps 
+
+* [Performance benchmark test recommendations for Azure NetApp Files](azure-netapp-files-performance-metrics-volumes.md)
+* [Azure NetApp Files regular volume performance benchmarks for Linux](performance-benchmarks-linux.md)
+* [Azure NetApp Files large volume performance benchmarks for Linux](performance-large-volumes-linux.md)
+
diff --git a/articles/azure-netapp-files/workload-types.md b/articles/azure-netapp-files/workload-types.md
@@ -0,0 +1,38 @@
+---
+title: Understand workload types in Azure NetApp Files 
+description: Choose the correct volume type depending on your Azure NetApp Files workload.   
+services: azure-netapp-files
+author: b-ahibbard
+ms.service: azure-netapp-files
+ms.topic: conceptual
+ms.date: 10/31/2024
+ms.author: anfdocs
+---
+
+# Understand workload types in Azure NetApp Files
+
+When considering use cases for cloud storage, industry silos can often be broken down into workload types, since there can be commonalities across industries in specific workloads. For instance, a media workload can have a similar workload profile to an AI/ML training set with heavy sequential reads and writes.  
+
+Azure NetApp Files is well suited for any type of workload from the low to high I/O, low to high throughput – from home directory to electronic design automation (EDA). Learn about the different workload types and develop an understanding of which Azure NetApp Files [volumes types](azure-netapp-files-understand-storage-hierarchy.md) are best suited for those workloads. 
+
+For more information, see [Understand large volumes in Azure NetApp Files](large-volumes.md)
+
+## Workload types 
+
+* **Specific offset, streaming random read/write workloads:** OLTP Databases are typical here. A signature of an OLTP workload is a dependence on random read to find the desired file offset (such as a database table row) and write performance against a small number of files. With this type of workload, tens of thousands to hundreds of thousands of I/O operations are common. Application vendors and database administrators typically have specific latency targets for these workloads.  Azure NetApp Files regular volumes are generally best suited for this workload. 
+
+* **Whole file streaming workloads:** Examples include post-production media rendering of media repositories, high-performance computing suites such as those seen in computer-aided engineering/design suites (for example, computational fluid dynamics), oil and gas suites, and machine learning fine-tuning frameworks. A hallmark of this type of workload is larger files read or written in a continuous manner. For these workloads, storage throughput is the most critical attribute as it has the biggest impact on time to completion. Latency sensitivity is common here as workloads typically use a fixed amount of concurrency, thus throughput is determined by latency. Workloads typical of post-production are latency sensitive to the degree that framerate is only achieved when specific latency values are met. Both Azure NetApp Files regular volumes and Azure NetApp Files large volumes are appropriate for these workloads, with large volumes providing [more capacity](azure-netapp-files-resource-limits) and [higher file count possibilities](maxfiles-concept.md).  
+
+* **Metadata rich, high file count workloads:** Examples include software development, EDA, and financial services (FSI) applications. A typical signature of these types of workloads is millions of smaller files being created, statted alone or statted then read or written. In high file count workload, remote procedure calls (RPC) other than read and write typically represent the majority of I/O. I/O rate (I/OPS) is typically the most important attribute for these workloads. Latency is often less important as concurrency might be controlled by scaling out at the application. Some customers have latency expectations of 1 ms, while others might expect 10 ms. As long as the I/O rate is achieved, so is satisfaction. This type of workload is ideally suited for _Azure NetApp Files large volumes_.  
+
+For more information on EDA workloads in Azure NetApp Files, see [Benefits of using Azure NetApp Files for Electronic Design Automation](solutions-benefits-azure-netapp-files-electronic-design-automation.md).
+
+## More information 
+
+* [General performance considerations for Azure NetApp Files](azure-netapp-files-performance-considerations.md)
+* [Performance benchmark test recommendations for Azure NetApp Files](azure-netapp-files-performance-metrics-volumes.md)
+* [Azure NetApp Files regular volume performance benchmarks for Linux](performance-benchmarks-linux.md)
+* [Azure NetApp Files large volume performance benchmarks for Linux](performance-large-volumes-linux.md)
+* [Oracle database performance on Azure NetApp Files single volumes](performance-oracle-single-volumes.md)
+* [Oracle database performance on Azure NetApp Files multiple volumes](performance-oracle-multiple-volumes.md)
+* [Azure NetApp Files datastore performance benchmarks for Azure VMware Solution](performance-azure-vmware-solution-datastore.md)
