Merge pull request #223242 from normesta/cost-management

Access tiers best practices

Author: Maggiemouse1

articles/storage/blobs/TOC.yml

@@ -202,14 +202,16 @@ items:
         href: ../common/storage-redundancy.md?toc=/azure/storage/blobs/toc.json&bc=/azure/storage/blobs/breadcrumb/toc.json
       - name: Customer-managed failover for disaster recovery
         href: ../common/storage-disaster-recovery-guidance.md?toc=/azure/storage/blobs/toc.json&bc=/azure/storage/blobs/breadcrumb/toc.json
-    - name: Blob access tiers and lifecycle management
+    - name: Access tiers and lifecycle management
       items:
-      - name: Overview
+      - name: Access tiers overview
         href: access-tiers-overview.md
-      - name: Blob rehydration from Archive tier
-        href: archive-rehydrate-overview.md
+      - name: Access tiers best practices
+        href: access-tiers-best-practices.md
       - name: Lifecycle management policies
         href: lifecycle-management-overview.md
+      - name: Blob rehydration from archive tier
+        href: archive-rehydrate-overview.md
     - name: Object replication
       href: object-replication-overview.md
     - name: Performance and scalability
@@ -342,6 +344,8 @@ items:
           href: blob-inventory-how-to.md
         - name: Calculate blob count and total size
           href: calculate-blob-count-size.md
+        - name: Convert append and page blobs to block blobs
+          href: convert-append-and-page-blobs-to-block-blobs.md
       - name: Authorize access to blob data
         items:
         - name: Authorization options for users

articles/storage/blobs/access-tiers-best-practices.md

@@ -0,0 +1,77 @@
+---
+title: Best practices for using blob access tiers
+titleSuffix: Azure Storage
+description: Learn about best practice guidelines that help you use access tiers to optimize performance and reduce costs.
+author: normesta
+
+ms.author: normesta
+ms.date: 01/20/2023
+ms.service: storage
+ms.subservice: blobs
+ms.topic: conceptual
+---
+
+# Best practices for using blob access tiers
+
+This article provides best practice guidelines that help you use access tiers to optimize performance and reduce costs. To learn more about access tiers, see [Hot, cool, and archive access tiers for blob data](access-tiers-overview.md?tabs=azure-portal). 
+
+## Choose the most cost-efficient access tiers
+
+You can reduce costs by placing blob data into the most cost-efficient access tiers. Choose from three tiers that are designed to optimize your costs around data use. For example, the hot tier has a higher storage cost but lower read cost. Therefore, if you plan to access data frequently, the hot tier might be the most cost-efficient choice. If you plan to read data less frequently, the cool or archive tier might make the most sense because it raises the cost of reading data while reducing the cost of storing data.
+
+To identify the most optimal access tier, try to estimate what percentage of the data will be read on a monthly basis. The following chart shows the impact on monthly spending given various read percentages. 
+
+> [!div class="mx-imgBorder"]
+> ![Chart that shows a bar for each tier which represents the monthly cost based on percentage read pattern](./media/access-tiers-best-practices/read-pattern-access-tiers.png)
+
+To model and analyze the cost of using cool versus archive storage, see [Archive versus cool](archive-cost-estimation.md#archive-versus-cool). You can apply similar modeling techniques to compare the cost of hot to cool or archive.
+
+## Migrate data directly to the most cost-efficient access tiers
+
+Choosing the most optimal tier up front can reduce costs. If you change the tier of a block blob that you've already uploaded, then you'll pay the cost of writing to the initial tier when you first upload the blob, and then pay the cost of writing to the desired tier. If you change tiers by using a lifecycle management policy, then that policy will require a day to take effect and a day to complete execution. You'll also incur the capacity cost of storing data in the initial tier prior to the tier change.
+
+- For guidance about how to upload to a specific access tier, see [Set a blob's access tier](access-tiers-online-manage.md). 
+
+- For offline data movement to the desired tier, see [Azure Data Box](/products/databox/).
+
+## Move data into the most cost-efficient access tiers
+
+After data is uploaded, you should periodically analyze your containers and blobs to understand how they are stored, organized, and used in production. Then, use lifecycle management policies to move data to the most cost-efficient tiers. For example, data that has not been accessed for more than 30 days might be more cost efficient if placed into the cool tier. Consider archiving data that has not been accessed for over 180 days. 
+
+To gather telemetry, enable [blob inventory reports](blob-inventory.md) and enable [last access time tracking](lifecycle-management-policy-configure.md#optionally-enable-access-time-tracking). Analyze use patterns based on the last access time by using tools such as Azure Synapse or Azure Databricks. To learn about ways to analyze your data, see any of these articles:
+
+- [Tutorial: Analyze blob inventory reports](storage-blob-inventory-report-analytics.md)
+
+- [Calculate blob count and total size per container using Azure Storage inventory](calculate-blob-count-size.md)
+
+- [How to calculate Container Level Statistics in Azure Blob Storage with Azure Databricks](https://techcommunity.microsoft.com/t5/azure-paas-blog/how-to-calculate-container-level-statistics-in-azure-blob/ba-p/3614650)
+
+## Tier append and page blobs
+
+Your analysis might reveal append or page blobs that are not actively used. For example, you might have log files (append blobs) that are no longer being read or written to, but you'd like to store them for compliance reasons. Similarly, you might want to back up disks or disk snapshots (page blobs). You can move these blobs into cooler tiers as well.  However, you must first convert them to block blobs. 
+
+For information about how to convert append and page blobs to block blobs, see [Convert append blobs and page blobs to block blobs](convert-append-and-page-blobs-to-block-blobs.md).
+
+## Pack small files before moving data to cooler tiers
+
+Each read or write operation incurs a cost. To reduce the cost of reading and writing data, consider packing small files into larger ones by using file formats such as TAR or ZIP. Fewer files reduce the number of operations required to transfer data. 
+
+The following chart shows the relative impact of packing files for the cool tier. The read cost assumes a monthly read percentage of 30%.
+
+> [!div class="mx-imgBorder"]
+> ![Chart that shows the impact on costs when you pack small files before uploading to the cool access tier.](./media/access-tiers-best-practices/packing-impact-cool.png)
+
+The following chart shows the relative impact of packing files for the archive tier. The read cost assumes a monthly read percentage of 30%.
+
+> [!div class="mx-imgBorder"]
+> ![Chart that shows the impact on costs when you pack small files before uploading to the archive access tier.](./media/access-tiers-best-practices/packing-impact-archive.png)
+
+> [!TIP]
+> To facilitate search and read scenarios, consider creating an index that maps packed file paths with original file paths, and then storing these indexes as block blobs in the hot tier.
+
+## Next steps
+
+- [Set a blob's access tier](access-tiers-online-manage.md)
+- [Archive a blob](archive-blob.md)
+- [Optimize costs by automatically managing the data lifecycle](lifecycle-management-overview.md)
+- [Estimate the cost of archiving data](archive-cost-estimation.md)

articles/storage/blobs/convert-append-and-page-blobs-to-block-blobs.md

@@ -0,0 +1,118 @@
+---
+title: Convert append and page blobs into block blobs (Azure Storage)
+titleSuffix: Azure Storage
+description: Learn how to convert an append blob or a page blob into a block blob in Azure Blob Storage.
+author: normesta
+
+ms.service: storage
+ms.topic: how-to
+ms.date: 01/20/2023
+ms.author: normesta
+ms.reviewer: fryu
+ms.subservice: blobs
+ms.devlang: powershell, azurecli
+ms.custom: devx-track-azurepowershell, devx-track-azurecli
+---
+
+# Convert append blobs and page blobs into block blobs
+
+To convert blobs, copy them to a new location by using PowerShell, Azure CLI, or AzCopy. You'll use command parameters to ensure that the destination blob is a block blob. All metadata from the source blob is copied to the destination blob.
+
+## Convert append and page blobs
+
+### [PowerShell](#tab/azure-powershell)
+
+1. Open a Windows PowerShell command window.
+
+2. Sign in to your Azure subscription with the [Connect-AzAccount](/powershell/module/az.accounts/connect-azaccount) command and follow the on-screen directions.
+
+   ```powershell
+   Connect-AzAccount
+   ```
+
+3. If your identity is associated with more than one subscription, then set your active subscription to subscription of the storage account which contains the append or page blobs.
+
+   ```powershell
+   $context = Get-AzSubscription -SubscriptionId '<subscription-id>'
+   Set-AzContext $context
+   ```
+
+   Replace the `<subscription-id>` placeholder value with the ID of your subscription.
+
+4. Create the storage account context by using the [New-AzStorageContext](/powershell/module/az.storage/new-azstoragecontext) command. Include the `-UseConnectedAccount` parameter so that data operations will be performed using your Azure Active Directory (Azure AD) credentials.
+
+   ```powershell
+   $ctx = New-AzStorageContext -StorageAccountName '<storage account name>' -UseConnectedAccount
+   ```
+
+5. Use the [Copy-AzStorageBlob](/powershell/module/az.storage/copy-azstorageblob) command and set the `-DestBlobType` parameter to `Block`.
+
+   ```powershell
+   $containerName = '<source container name>'
+   $srcblobName = '<source append or page blob name>'
+   $destcontainerName = '<destination container name>'
+   $destblobName = '<destination block blob name>'
+   $destTier = '<destination block blob tier>'
+
+   Copy-AzStorageBlob -SrcContainer $containerName -SrcBlob $srcblobName -Context $ctx -DestContainer $destcontainerName -DestBlob $destblobName -DestContext $ctx -DestBlobType Block -StandardBlobTier $destTier
+   ```
+
+   > [!TIP]
+   > The `-StandardBlobTier` parameter is optional. If you omit that parameter, then the destination blob infers its tier from the [default account access tier setting](access-tiers-overview.md#default-account-access-tier-setting). To change the tier after you've created a block blob, see [Change a blob's tier](access-tiers-online-manage.md#change-a-blobs-tier). 
+
+
+### [Azure CLI](#tab/azure-cli)
+
+1. First, open the [Azure Cloud Shell](../../cloud-shell/overview.md), or if you've [installed](/cli/azure/install-azure-cli) the Azure CLI locally, open a command console application such as Windows PowerShell.
+
+   > [!NOTE]
+   > If you're using a locally installed version of the Azure CLI, ensure that you are using version 2.44.0 or later. 
+
+2. If your identity is associated with more than one subscription, then set your active subscription to subscription of storage account which contains the append or page blobs.
+
+   ```azurecli-interactive
+   az account set --subscription <subscription-id>
+   ```
+
+   Replace the `<subscription-id>` placeholder value with the ID of your subscription.
+
+3. Use the [az storage blob copy start](/cli/azure/storage/blob/copy#az-storage-blob-copy-start) command and set the `--destination-blob-type` parameter to `blockBlob`. 
+
+   ```azurecli
+   containerName = '<source container name>'
+   srcblobName = '<source append or page blob name>'
+   destcontainerName = '<destination container name>'
+   destblobName = '<destination block blob name>'
+   destTier = '<destination block blob tier>'
+
+   az storage blob copy start --account-name $accountName --destination-blob $destBlobName --destination-container $destcontainerName --destination-blob-type BlockBlob --source-blob $srcblobName --source-container $containerName --tier $destTier
+   ```
+
+   > [!TIP]
+   > The `--tier` parameter is optional. If you omit that parameter, then the destination blob infers its tier from the [default account access tier setting](access-tiers-overview.md#default-account-access-tier-setting). To change the tier after you've created a block blob, see [Change a blob's tier](access-tiers-online-manage.md#change-a-blobs-tier).
+
+   > [!WARNING]
+   > The optional `--metadata` parameter overwrites any existing metadata. Therefore, if you specify metadata by using this parameter, then none of the original metadata from the source blob will be copied to the destination blob.
+
+
+### [AzCopy](#tab/azcopy)
+
+Use the [azcopy copy](../common/storage-ref-azcopy-copy.md) command. Specify the source and destination paths. Set the `blob-type` parameter to `BlockBlob`.
+
+```azcopy
+azcopy copy 'https://<storage-account-name>.<blob or dfs>.core.windows.net/<container-name>/<append-or-page-blob-name>' 'https://<storage-account-name>.<blob or dfs>.core.windows.net/<container-name>/<name-of-new-block-blob>' --blob-type BlockBlob --block-blob-tier <destination-tier>
+```
+
+> [!TIP]
+> The `--block-blob-tier` parameter is optional. If you omit that parameter, then the destination blob infers its tier from the [default account access tier setting](access-tiers-overview.md#default-account-access-tier-setting). To change the tier after you've created a block blob, see [Change a blob's tier](access-tiers-online-manage.md#change-a-blobs-tier). 
+
+> [!WARNING]
+> The optional `--metadata` parameter overwrites any existing metadata. Therefore, if you specify metadata by using this parameter, then none of the original metadata from the source blob will be copied to the destination blob.
+
+---
+
+## See also
+
+- [Hot, Cool, and Archive access tiers for blob data](access-tiers-overview.md)
+- [Set a blob's access tier](access-tiers-online-manage.md)
+- [Best practices for using blob access tiers](access-tiers-best-practices.md)