Merge pull request #276894 from normesta/azcopy

Updating reference articles based on new AzCopy release

Author: Jill Grant

articles/storage/common/storage-ref-azcopy-bench.md

@@ -4,7 +4,7 @@ description: This article provides reference information for the azcopy bench co
 author: normesta
 ms.service: azure-storage
 ms.topic: reference
-ms.date: 03/29/2024
+ms.date: 05/31/2024
 ms.author: normesta
 ms.subservice: storage-common-concepts
 ms.reviewer: zezha-msft
@@ -45,24 +45,40 @@ azcopy bench [destination] [flags]
 
 ## Examples
 
-Run an upload benchmark with default parameters (suitable for benchmarking networks up to 1 Gbps):
+Run an upload benchmark with default parameters (suitable for benchmarking networks up to 1 Gbps).
 
 `azcopy bench "https://[account].blob.core.windows.net/[container]?<SAS>"`
 
-Run a benchmark test that uploads 100 files, each 2 GiB in size: (suitable for benchmarking on a fast network, e.g. 10 Gbps):'
+Run an upload benchmark with a specified block size of 2 MiB and check the length of files after the transfer.
+
+`azcopy bench "https://[account].blob.core.windows.net/[container]?<SAS>" --block-size-mb 2 --check-length`
+
+Run a benchmark test that uploads 500 files. Each file is 500 MiB in size, and the log level is set to display only errors.
+
+`azcopy bench "https://[account].blob.core.windows.net/[container]?<SAS>" --file-count 500 --size-per-file 500M --log-level ERROR`
+
+Run a benchmark test that uploads 100 files. Each file is 2 GiB in size. This is suitable for benchmarking on a fast network (For example: 10 Gbps).
 
 `azcopy bench "https://[account].blob.core.windows.net/[container]?<SAS>" --file-count 100 --size-per-file 2G`
 
-Same as above, but use 50,000 files, each 8 MiB in size and compute their MD5 hashes (in the same way that the --put-md5 flag does this
-in the copy command). The purpose of --put-md5 when benchmarking is to test whether MD5 computation affects throughput for the selected file count and size:
+The next example is the same as above, but with 50,000 files. Each file 8 MiB in size. This example also computes the MD5 hashes of each file similar to the way that the --put-md5 flag computes the MD5 in the azcopy copy command. The purpose of --put-md5 when benchmarking is to test whether MD5 computation affects throughput for the 
+selected file count and size.
 
 `azcopy bench --mode='Upload' "https://[account].blob.core.windows.net/[container]?<SAS>" --file-count 50000 --size-per-file 8M --put-md5`
 
-Run a benchmark test that downloads existing files from a target
+Run a benchmark test that uploads 1000 files and creates folders to divide up the data. Each file is 100 KiB in size.
+
+`azcopy bench "https://[account].blob.core.windows.net/[container]?<SAS>" --file-count 1000 --size-per-file 100K --number-of-folders 5`
+ 
+Run a benchmark test that downloads existing files from a target.
 
 `azcopy bench --mode='Download' "https://[account].blob.core.windows.net/[container]?<SAS?"`
 
-Run an upload that doesn't delete the transferred files. (These files can then serve as the payload for a download test)
+Run a download benchmark with the default parameters and cap the transfer rate at 500 Mbps.
+
+`azcopy bench --mode=Download "https://[account].blob.core.windows.net/[container]?<SAS>" --cap-mbps 500`
+
+Run an upload that does not delete the transferred files. These files can then serve as the payload for a download test.
 
 `azcopy bench "https://[account].blob.core.windows.net/[container]?<SAS>" --file-count 100 --delete-test-data=false`
 

articles/storage/common/storage-ref-azcopy-copy.md

@@ -4,7 +4,7 @@ description: This article provides reference information for the azcopy copy com
 author: normesta
 ms.service: azure-storage
 ms.topic: reference
-ms.date: 03/29/2024
+ms.date: 05/31/2024
 ms.author: normesta
 ms.subservice: storage-common-concepts
 ms.reviewer: zezha-msft
@@ -144,7 +144,7 @@ Download a subset of containers within a storage account by using a wildcard sym
 
 `azcopy cp "https://[srcaccount].blob.core.windows.net/[container*name]" "/path/to/dir" --recursive`
 
-Download all the versions of a blob from Azure Storage to local directory. Ensure that source is a valid blob, destination is a local folder and `versionidsFile` which takes in a path to the file where each version is written on a separate line. All the specified versions get downloaded in the destination folder specified.
+Download all the versions of a blob from Azure Storage listed in a text file (For example: `versionidsFile`) to local directory. Ensure that source is a valid blob, destination is a local folder and `versionidsFile` is a text file where each version is written on a separate line. All the specified versions get downloaded in the destination folder specified.
 
 `azcopy cp "https://[srcaccount].blob.core.windows.net/[containername]/[blobname]" "/path/to/dir" --list-of-versions="/another/path/to/dir/[versionidsFile]"`
 
@@ -220,6 +220,16 @@ Copy a subset of buckets by using a wildcard symbol (*) in the bucket name from
 
 `azcopy cp "https://storage.cloud.google.com/[bucket*name]/" "https://[destaccount].blob.core.windows.net/?[SAS]" --recursive=true`
 
+To copy files changed before or after the AzCopy job has started, AzCopy provides date and time in the job log in ISO8601 format (search for 'ISO 8601 START TIME' in the job log) that can be used with the `--include-after` and `--include-before` flags, see examples below. This is helpful for incremental copies.
+
+Copy a subset of files modified on or after the given date and time (in ISO8601 format) in a container by using the `include-after` flag.
+
+`azcopy cp "https://[srcaccount].blob.core.windows.net/[containername]?[SAS]" "https://[dstaccount].blob.core.windows.net/[containername]?[SAS]" --include-after='2020-08-19T15:04:00Z''"`
+
+Copy a subset of files modified on or before the given date and time (in ISO8601 format) in a container by using the `include-before` flag.
+
+`azcopy cp "https://[srcaccount].blob.core.windows.net/[containername]?[SAS]" "https://[dstaccount].blob.core.windows.net/[containername]?[SAS]" --include-before='2020-08-19T15:04:00Z'"`
+
 ## Options
 
 `--as-subdir` True by default.    Places folder sources as subdirectories under the destination. (default true)
@@ -250,18 +260,20 @@ Copy a subset of buckets by using a wildcard symbol (*) in the bucket name from
 
 `--cpk-by-name`    (string)    Client provided key by name that gives clients making requests against Azure Blob storage an option to provide an encryption key on a per-request basis. Provided key name is fetched from Azure Key Vault and is used to encrypt the data
 
-`--cpk-by-value`    Client provided key by name that lets clients making requests against Azure Blob storage an option to provide an encryption key on a per-request basis. Provided key and its hash is fetched from environment variables
+`--cpk-by-value`    False by default. Client provided key by name that lets clients making requests against Azure Blob storage an option to provide an encryption key on a per-request basis. Provided key and its hash is fetched from environment variables
 
 `--decompress`    Automatically decompress files when downloading, if their content-encoding indicates that they're compressed. The supported content-encoding values are 'gzip' and 'deflate'. File extensions of '.gz'/'.gzip' or '.zz' aren't necessary, but is removed if present.
 
 `--disable-auto-decoding`    False by default to enable automatic decoding of illegal chars on Windows. Can be set to true to disable automatic decoding.
 
-`--dry-run`    Prints the file paths that would be copied by this command. This flag doesn't copy the actual files. The --overwrite flag has no effect. If you set the --overwrite flag to false, files in the source directory are listed even if those files exist in the destination directory.
+`--dry-run`    False by default. Prints the file paths that would be copied by this command. This flag doesn't copy the actual files. The --overwrite flag has no effect. If you set the --overwrite flag to false, files in the source directory are listed even if those files exist in the destination directory.
 
 `--exclude-attributes`    (string)    (Windows only) Exclude files whose attributes match the attribute list. For example: A;S;R
 
 `--exclude-blob-type`    (string)    Optionally specifies the type of blob (BlockBlob/ PageBlob/ AppendBlob) to exclude when copying blobs from the container or the account. Use of this flag isn't applicable for copying data from non azure-service to service. More than one blob should be separated by ';'.
 
+`--exclude-container`    (string)    Exclude these containers when transferring from account to account only. Multiple containers can be separated with ';'.
+
 `--exclude-path`    (string)    Exclude these paths when copying. This option doesn't support wildcard characters (*). Checks relative path prefix(For example: myFolder;myFolder/subDirName/file.pdf). When used in combination with account traversal, paths don't include the container name.
 
 `--exclude-pattern`    (string)    Exclude these files when copying. This option supports wildcard characters (*)
@@ -292,8 +304,6 @@ Copy a subset of buckets by using a wildcard symbol (*) in the bucket name from
 
 `--list-of-versions`    (string)    Specifies a file where each version ID is listed on a separate line. Ensure that the source must point to a single blob and all the version IDs specified in the file using this flag must belong to the source blob only. AzCopy will download the specified versions in the destination folder provided.
 
-`--log-level` (string)    Define the log verbosity for the log file, available levels: INFO(all requests/responses), WARNING(slow responses), ERROR(only failed requests), and NONE(no output logs). (default 'INFO'). (default "INFO")
-
 `--metadata` (string)    Upload to Azure Storage with these key-value pairs as metadata.
 
 `--no-guess-mime-type`    Prevents AzCopy from detecting the content-type based on the extension or content of the file.
@@ -310,6 +320,8 @@ Copy a subset of buckets by using a wildcard symbol (*) in the bucket name from
 
 `--preserve-permissions`    False by default. Preserves ACLs between aware resources (Windows and Azure Files, or Azure Data Lake Storage Gen2 to Azure Data Lake Storage Gen2). For accounts that have a hierarchical namespace, your security principal must be the owning user of the target container or it must be assigned the Storage Blob Data Owner role, scoped to the target container, storage account, parent resource group, or subscription. For downloads, you'll also need the `--backup` flag to restore permissions where the new Owner won't be the user running AzCopy. This flag applies to both files and folders, unless a file-only filter is specified (for example, include-pattern).
 
+`--preserve-posix-properties`    False by default. Preserves property info gleaned from `stat` or `statx` into object metadata.
+
 `--preserve-smb-info`    For SMB-aware locations, flag is set to true by default. Preserves SMB property info (last write time, creation time, attribute bits) between SMB-aware resources (Windows and Azure Files). Only the attribute bits supported by Azure Files are transferred; any others are ignored. This flag applies to both files and folders, unless a file-only filter is specified (for example, include-pattern). The info transferred for folders is the same as that for files, except for `Last Write Time` which is never preserved for folders. (default true)
 
 `--preserve-symlinks`   If enabled, symlink destinations are preserved as the blob content, rather than uploading the file or folder on the other end of the symlink.
@@ -326,15 +338,19 @@ Copy a subset of buckets by using a wildcard symbol (*) in the bucket name from
 
 `--s2s-preserve-access-tier`    Preserve access tier during service to service copy. Refer to [Azure Blob storage: hot, cool, and archive access tiers](/azure/storage/blobs/storage-blob-storage-tiers) to ensure destination storage account supports setting access tier. In the cases that setting access tier isn't supported, make sure to use s2sPreserveAccessTier=false to bypass copying access tier. (default true).  (default true)
 
-`--s2s-preserve-blob-tags`    Preserve index tags during service to service transfer from one blob storage to another
+`--s2s-preserve-blob-tags`    False by default. Preserve index tags during service to service transfer from one blob storage to another
 
 `--s2s-preserve-properties`    Preserve full properties during service to service copy. For AWS S3 and Azure File non-single file source, the list operation doesn't return full properties of objects and files. To 
 preserve full properties, AzCopy needs to send one more request per object or file. (default true)
 
+`--trailing-dot`  Enabled by default to treat file share related operations in a safe manner. Available options: `Enable`, `Disable`. Choose `Disable` to go back to legacy (potentially unsafe) treatment of trailing dot files where the file service will trim any trailing dots in paths. This can result in potential data corruption if the transfer contains two paths that differ only by a trailing dot (For example `mypath` and `mypath.`). If this flag is set to `Disable` and AzCopy encounters a trailing dot file, it will warn customers in the scanning log but will not attempt to abort the operation. If the destination does not support trailing dot files (Windows or Blob Storage), AzCopy will fail if the trailing dot file is the root of the transfer and skip any trailing dot paths encountered during enumeration.
+
 ## Options inherited from parent commands
 
 `--cap-mbps`    (float)    Caps the transfer rate, in megabits per second. Moment-by-moment throughput might vary slightly from the cap. If this option is set to zero, or it's omitted, the throughput isn't capped.
 
+`--log-level` (string)    Define the log verbosity for the log file, available levels: INFO(all requests/responses), WARNING(slow responses), ERROR(only failed requests), and NONE(no output logs). (default 'INFO'). (default "INFO")
+
 `--output-type`    (string)    Format of the command's output. The choices include: text, json. The default value is 'text'. (default "text")
 
 `--trusted-microsoft-suffixes`    (string)    Specifies additional domain suffixes where Microsoft Entra login tokens can be sent.  The default is '*.core.windows.net;*.core.chinacloudapi.cn;*.core.cloudapi.de;*.core.usgovcloudapi.net;*.storage.azure.net'. Any listed here are added to the default. For security, you should only put Microsoft Azure domains here. Separate multiple entries with semi-colons.

articles/storage/common/storage-ref-azcopy-list.md

@@ -4,7 +4,7 @@ description: This article provides reference information for the azcopy list com
 author: normesta
 ms.service: azure-storage
 ms.topic: reference
-ms.date: 05/26/2022
+ms.date: 05/31/2024
 ms.author: normesta
 ms.subservice: storage-common-concepts
 ms.reviewer: zezha-msft
@@ -38,13 +38,17 @@ azcopy list [containerURL] --properties [semicolon(;) separated list of attribut
 
 `-h`, `--help`    Help for list
 
-`--machine-readable`    Lists file sizes in bytes.
+`--location`  Optionally specifies the location. For Example: `Blob`,`File`, `BlobFS`.
 
-`--mega-units`    Displays units in orders of 1000, not 1024.
+`--machine-readable`    False by default.  Lists file sizes in bytes.
 
-`--properties`    (string)    delimiter (;) separated values of properties required in list output.
+`--mega-units`    False by default. Displays units in orders of 1000, not 1024.
 
-`--running-tally`    Counts the total number of files and their sizes.
+`--properties`     Properties to be displayed in list output. Possible properties include: `LastModifiedTime`, `VersionId`, `BlobType`, `BlobAccessTier`, `ContentType`, `ContentEncoding`, `ContentMD5`, `LeaseState`, `LeaseDuration`, `LeaseStatus`, `ArchiveStatus`. Delimiter (;) should be used to separate multiple values of properties (For example: 'LastModifiedTime;VersionId;BlobType').
+
+`--running-tally`    False by default. Counts the total number of files and their sizes.
+
+`--trailing-dot`  Enabled by default to treat file share related operations in a safe manner. Available options: `Enable`, `Disable`. Choose `Disable` to go back to legacy (potentially unsafe) treatment of trailing dot files where the file service will trim any trailing dots in paths. This can result in potential data corruption if the transfer contains two paths that differ only by a trailing dot (For example `mypath` and `mypath.`). If this flag is set to `Disable` and AzCopy encounters a trailing dot file, it will warn customers in the scanning log but will not attempt to abort the operation. If the destination does not support trailing dot files (Windows or Blob Storage), AzCopy will fail if the trailing dot file is the root of the transfer and skip any trailing dot paths encountered during enumeration.
 
 ## Options inherited from parent commands