Made it read the files backwards.

Author: kcantrel

Management-Utilities/warm_performance_tier/README.md

@@ -2,13 +2,21 @@
 
 ## Introduction
 This sample provides a script that can be used to warm a FSx for ONTAP
-volume. In other words, it ensures all the blocks for a volume are in
-the "performance tier" as opposed to the "capacity tier." It does that
-simply by reading every byte of every file in the volume. Doing that
-causes any block that is currently in the capacity tier to be pulled
+volume. In other words, it ensures that all the blocks for a volume are in
+the "performance tier" as opposed to the "capacity tier." It does that by
+simply reading every byte of every file in the volume. Doing that
+causes all blocks that are currently in the capacity tier to be pulled
 into the performance tier before being returned to the reader. At that point,
-assuming the tiering policy is not set to 'all', all the data should remain
-in the performance tier until the policy tiers it back out to the capacity tier.
+assuming, the tiering policy is not set to 'all', all the data should remain
+in the performance tier until ONTAP tiers it back based on the volume's
+tiering policy.
+
+Note that Data ONTAP will not store data in the performance
+tier from the capacity tier if it detects that the data is being read
+sequentially. This is to keep things like backups and virus scans from
+filling up the performance tier. Because of that, this script will
+read files in "reverse" order. Meaning it will read the last block of
+the file first, then the second to last block, and so on. 
 
 To speed up the process, the script will spawn multiple threads to process
 the volume. It will spawn a separate thread for each directory
@@ -19,13 +27,12 @@ will spawn -x reader threads **per** directory thread. So for example, if you ha
 directory threads and 10 reader threads, you could have up to 40 reader
 threads running at one time.
 
-You can specify a FSx for ONTAP file system data endpoint and a volume
-name to the script and it will attempt to mount the volume if it isn't already.
-If it does mount it, it will mount it read-only and unmount
-when it is done. The script assumes that the junction path is the same
-as the volume name. If this isn't the case, then just mount the volume
-before running the script and provide the path to the mount point
-with the -d option.
+Since the goal of this script is to force all the data that is currently
+in the capacity tier to the performance tier you should ensure you have
+enough free space in your performance tier to hold all the data in the volume.
+You can use the `volume show-footprint` ONTAP command to see how much space
+is currently in the capacity tier. You can then use `storage aggregate show`
+to see how much space is available in the performance tier.
 
 ## Set Up
 The script is meant to be run on a Linux based host that is able to NFS
@@ -34,21 +41,38 @@ any user that has read access to the files in the volume can run it.
 Otherwise, the script needs to be run as 'root' so it can mount the
 volume before reading the files.
 
-Once the volume has been mounted, then the script should be run by
-a user that has read access to all the files in the volume.
+If the 'root' user can't read the files in the volume, then you should use 'root'
+to mount the volume first and then run the script from a user ID that can read the contents
+of all the files in the volume. 
 
-Of course it only makes sense to have the tiering policy on the volume
-set to something other than "all", otherwise the script will be ineffective.
+Make sure you have set the tiering policy on the volume set to something
+other than "all" or "snapshot-only", otherwise the script will be ineffective.
 
 # Running The Script
-To run this script you just need to change the UNIX permissions on the file to be executable, then run it as a command:
+There are two main ways to run the script. The first is to just provide
+the script with a directory to start from. The script will then read
+every file in that directory and all subdirectories. The second way
+is to provide the script with the FSx for ONTAP file system data endpoint
+and the volume name. The script will then attempt to mount the volume
+if it isn't already mounted. If it does mount it, it will mount it read-only
+and unmount when it is done.
+
+In order to mount the volume the script assumes that the junction path is the same
+as the volume name. If this isn't the case, then just mount the volume first
+before running the script and provide the path to the mount point
+with the -d option.
+
+To run this script you just need to change the UNIX permissions on
+the file to be executable, then run it as a command:
 ```
 chmod +x warm_performance_tier
  ./warm_performance_tier -d /path/to/mount/point
 ```
-The above example will force the script to read every file in the /path/to/mount/point directory.
+The above example will force the script to read every file in the /path/to/mount/point
+directory and any directory under it.
 
-If you want to script to ensure the volume is mounted, you can specify the filesystem endpoint and volume name:
+If you want the script to ensure the volume is mounted, you can specify
+the file system data endpoint and volume name:
 ```
 ./warm_performance_tier -f fsxfileserver.us-west-2.amazonaws.com -v myvolume
 ```

Management-Utilities/warm_performance_tier/warm_performance_tier

@@ -90,6 +90,29 @@ isMounted () {
   return 1
 }
 
+################################################################################
+# This function is used to read a file and discard the output. It reads the file
+# in blocks of 4MB in reverse order. It is done this way so ONTAP doesn't
+# assume this is a backup process and not promote the data to the performance
+# tier.
+################################################################################
+readFile () {
+  local file=$1
+  local blockSize=$((4*1024*1024))
+
+  fileSize=$(stat -c "%s" "$file")
+  fileBlocks=$(($fileSize/$blockSize))
+  while [ $fileBlocks -ge 0 ]; do
+    if dd if="$file" of=/dev/null bs=$blockSize count=1 skip=$fileBlocks > /dev/null 2>&1; then
+      :
+    else
+      return 1
+    fi
+    let fileBlocks-=1
+  done
+  return 0
+}
+
 ################################################################################
 # Read all the files in the passed directory.
 ################################################################################
@@ -122,7 +145,7 @@ processDirectory () {
       if [ "$verbose" == "true" ]; then
         echo "$BASHPID $(date +%s) - Reading $1/$file and discarding output. Number of threads: $((${#runningPIDs[*]}+1))"
       fi
-      cat "$file" > /dev/null 2>&1 || echo "$BASHPID $(date +%s) - Warning, cannot read '${file}'." &
+      readFile "$file" || echo "$BASHPID $(date +%s) - Warning, cannot read '${file}'." 1>&2 &
       #
       # Record the PID of the process mostly so we can keep up with how many are running.
       runningPIDs[$!]=$!