You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: azure-local/manage/refs-deduplication-and-compression.md
+8-5Lines changed: 8 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: Learn how to use ReFS deduplication and compression in Azure Local
4
4
author: alkohli
5
5
ms.author: alkohli
6
6
ms.topic: how-to
7
-
ms.date: 12/10/2024
7
+
ms.date: 01/16/2025
8
8
---
9
9
10
10
# Optimize storage with ReFS deduplication and compression in Azure Local
@@ -15,7 +15,7 @@ This article describes the Resilient File System (ReFS) deduplication and compre
15
15
16
16
## What is ReFS deduplication and compression?
17
17
18
-
ReFS deduplication and compression is a storage optimization feature designed specifically for activeworkloads, such as [Azure virtual desktop infrastructure (VDI) on Azure Local](../deploy/virtual-desktop-infrastructure.md). This feature helps optimize storage usage and reduce storage cost.
18
+
ReFS deduplication and compression is a storage optimization feature that helps optimize storage usage and reduce storage cost. Use deduplication specifically for active, performance-sensitive, or read-heavy workloads, such as [Azure virtual desktop infrastructure (VDI) on Azure Local](../deploy/virtual-desktop-infrastructure.md). For less performance-intensive workloads, you can use a combination of deduplication and compression or only compression.
19
19
20
20
This feature uses [ReFS block cloning](/windows-server/storage/refs/block-cloning) to reduce data movement and enable metadata only operations. The feature operates at the data block level and uses fixed block size depending on the system size. The compression engine generates a heatmap to identify if a block should be eligible for compression, optimizing for CPU usage.
21
21
@@ -26,7 +26,7 @@ You can run ReFS deduplication and compression as a one-time job or automate it
26
26
Here are the benefits of using ReFS deduplication and compression:
27
27
28
28
-**Storage savings for active workloads.** Designed for active workloads, such as VDI, ensuring efficient performance in demanding environments.
29
-
-**Multiple modes.** Operates in three modes: deduplication only, compression only, and deduplication and compression (default mode), allowing optimization based on your needs.
29
+
-**Multiple modes.** Operates in three modes: deduplication only (default mode), compression only, and deduplication and compression, allowing optimization based on your needs.
30
30
-**Incremental deduplication.** Deduplicates only new or changed data as opposed to scanning the entire volume every time, optimizing job duration and reducing impact on system performance.
31
31
32
32
## Prerequisites
@@ -41,6 +41,9 @@ Before you begin, make sure that the following prerequisites are completed:
41
41
42
42
You can use ReFS deduplication and compression via Windows Admin Center or PowerShell. PowerShell allows both manual and automated jobs, whereas Windows Admin Center supports only scheduled jobs. Regardless of the method, you can customize job settings and utilize file change tracking for quicker subsequent runs.
43
43
44
+
> [!NOTE]
45
+
> We recommend using only deduplication for workloads where performance is a consideration, rather than using compression or a combination of both.
46
+
44
47
### Enable and run ReFS deduplication and compression
45
48
46
49
# [Windows Admin Center](#tab/windowsadmincenter)
@@ -97,8 +100,8 @@ Follow these steps to enable ReFS deduplication and compression via PowerShell:
97
100
98
101
where:
99
102
`Type` is a required parameter and can take one of the following values:
100
-
- **Dedup**: Enables deduplication only.
101
-
- **DedupAndCompress**: Enables both deduplication and compression. This is the default option.
103
+
- **Dedup**: Enables deduplication only. This is the default option.
104
+
- **DedupAndCompress**: Enables both deduplication and compression.
102
105
- **Compress**: Enables compression only.
103
106
104
107
If you want to change the `Type` parameter, you must first [disable ReFS deduplication and compression](#disable-refs-deduplication-and-compression-on-a-volume) and then enable it again with the new `Type` parameter.
Copy file name to clipboardExpand all lines: azure-managed-lustre/amlfs-overview.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ description: Use Azure Managed Lustre to quickly create an Azure-based Lustre fi
4
4
ms.topic: overview
5
5
author: pauljewellmsft
6
6
ms.author: pauljewell
7
-
ms.date: 11/11/2024
7
+
ms.date: 01/17/2025
8
8
ms.reviewer: mayabishop
9
9
ms.custom: references_regions
10
10
@@ -29,7 +29,7 @@ You can also use your Azure Managed Lustre file system with your Azure Kubernete
29
29
30
30
All data stored in Azure is encrypted at rest using Azure managed keys by default. If you want to manage the keys used to encrypt the data stored in your Azure Managed Lustre cluster, follow the instructions in [Server-side encryption of Azure disk storage](/azure/virtual-machines/disk-encryption).
31
31
32
-
All information in an Azure Managed Lustre file system also is protected by VM host encryption on the managed disks that hold your data, even if you add a customer-managed key for the Lustre disks. Adding a customer-managed key gives an extra level of security for customers with high security needs. For more information, see [Server-side encryption of Azure disk storage](/azure/virtual-machines/disk-encryption).
32
+
All information in an Azure Managed Lustre file system is protected by virtual machine (VM) host encryption on the managed disks that hold your data, even if you add a customer-managed key for the Lustre disks. Adding a customer-managed key gives an extra level of security for customers with high security needs. For more information, see [Server-side encryption of Azure disk storage](/azure/virtual-machines/disk-encryption).
33
33
34
34
> [!NOTE]
35
35
> Azure Managed Lustre doesn't store customer data outside the region in which you deploy the service instance.
@@ -58,9 +58,9 @@ If you want to use an Azure Managed Lustre storage system with your Kubernetes c
58
58
59
59
Kubernetes can simplify configuring and deploying virtual client endpoints for your Azure Managed Lustre workload, automating setup tasks such as:
60
60
61
-
* Creating Azure Virtual Machine Scale Sets used by Azure Kubernetes Service (AKS) to run the pods.
62
-
* Loading the correct Lustre client software on VM instances.
63
-
* Specifying the Azure Managed Lustre mount point, and propagating that information to the client pods.
61
+
- Creating Azure Virtual Machine Scale Sets used by Azure Kubernetes Service (AKS) to run the pods.
62
+
- Loading the correct Lustre client software on VM instances.
63
+
- Specifying the Azure Managed Lustre mount point, and propagating that information to the client pods.
64
64
65
65
The Azure Lustre CSI driver for Kubernetes can automate installing the client software and mounting drives. The driver provides a CSI controller plugin as a deployment with two replicas by default, and a CSI node plugin, as a DaemonSet. You can change the number of replicas.
Copy file name to clipboardExpand all lines: azure-managed-lustre/amlfs-prerequisites.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
title: Prerequisites for Azure Managed Lustre file systems
3
3
description: Learn about network and storage prerequisites to complete before you create an Azure Managed Lustre file system.
4
4
ms.topic: overview
5
-
ms.date: 05/14/2024
5
+
ms.date: 01/17/2025
6
6
author: pauljewellmsft
7
7
ms.author: pauljewell
8
8
ms.reviewer: mayabishop
@@ -56,7 +56,7 @@ By default, no specific changes need to be made to enable Azure Managed Lustre.
56
56
| DNS access | Use the default Azure-based DNS server. |
57
57
| Access between hosts on the Azure Managed Lustre subnet | Allow inbound and outbound access between hosts within the Azure Managed Lustre subnet. As an example, access to TCP port 22 (SSH) is necessary for cluster deployment. |
58
58
| Azure cloud service access | Configure your network security group to permit the Azure Managed Lustre file system to access Azure cloud services from within the Azure Managed Lustre subnet.<br><br>Add an outbound security rule with the following properties:<br>- **Port**: Any<br>- **Protocol**: Any<br>- **Source**: Virtual Network<br>- **Destination**: "AzureCloud" service tag<br>- **Action**: Allow<br><br>Note: Configuring the Azure cloud service also enables the necessary configuration of the Azure Queue service.<br><br>For more information, see [Virtual network service tags](/azure/virtual-network/service-tags-overview). |
59
-
| Lustre access<br>(TCP ports 988, 1019-1023) | Your network security group must allow inbound and outbound traffic for TCP port 988 and TCP port range 1019-1023. These rules need to be allowed between hosts on the Azure Managed Lustre subnet, as well as between any client subnets and the Azure Managed Lustre subnet. No other services can reserve or use these ports on your Lustre clients. The default rules `65000 AllowVnetInBound` and `65000 AllowVnetOutBound` meet this requirement. |
59
+
| Lustre access<br>(TCP ports 988, 1019-1023) | Your network security group must allow inbound and outbound traffic for TCP port 988 and TCP port range 1019-1023. These rules need to be allowed between hosts on the Azure Managed Lustre subnet, and between any client subnets and the Azure Managed Lustre subnet. No other services can reserve or use these ports on your Lustre clients. The default rules `65000 AllowVnetInBound` and `65000 AllowVnetOutBound` meet this requirement. |
60
60
61
61
62
62
For detailed guidance about configuring a network security group for Azure Managed Lustre file systems, see [Create and configure a network security group](configure-network-security-group.md#create-and-configure-a-network-security-group).
@@ -138,7 +138,7 @@ You must have two separate blob containers in the same storage account, which ar
138
138
-**Logging container**: A second container for import/export logs in the storage account. You must store the logs in a different container from the data container.
139
139
140
140
> [!NOTE]
141
-
> You can add files to the file system later from clients. However, files added to the original blob container after you create the file system won't be imported to the Azure Managed Lustre file system unless you [create an import job](create-import-job.md).
141
+
> You can add files to the file system later from clients. However, files added to the original blob container after you create the file system aren't imported to the Azure Managed Lustre file system unless you [create an import job](create-import-job.md).
Copy file name to clipboardExpand all lines: azure-managed-lustre/blob-integration.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ When you import data from a blob container to an Azure Managed Lustre file syste
26
26
27
27
You can prefetch the contents of blobs using Lustre's `lfs hsm_restore` command from a mounted client with sudo capabilities. To learn more, see [Restore data from Blob Storage](#restore-data-from-blob-storage).
28
28
29
-
Azure Managed Lustre works with storage accounts that have hierarchical namespace enabled and storage accounts with a non-hierarchical, or flat, namespace. The following minor differences apply:
29
+
Azure Managed Lustre works with storage accounts that have hierarchical namespace enabled and storage accounts with a nonhierarchical, or flat, namespace. The following minor differences apply:
30
30
31
31
- For a storage account with hierarchical namespace enabled, Azure Managed Lustre reads POSIX attributes from the blob header.
32
32
- For a storage account that *does not* have hierarchical namespace enabled, Azure Managed Lustre reads POSIX attributes from the blob metadata. A separate, empty file with the same name as your blob container contents is created to hold the metadata. This file is a sibling to the actual data directory in the Azure Managed Lustre file system.
@@ -50,9 +50,9 @@ For an import job, you can specify import prefixes when you create the job. From
50
50
Keep the following considerations in mind when specifying import prefixes:
51
51
52
52
- The default import prefix is `/`. This default behavior imports the contents of the entire blob container.
53
-
- If you specify multiple prefixes, the prefixes must be non-overlapping. For example, if you specify `/data` and `/data2`, the import job fails because the prefixes overlap.
53
+
- If you specify multiple prefixes, the prefixes must not overlap. For example, if you specify `/data` and `/data2`, the import job fails because the prefixes overlap.
54
54
- If the blob container is in a storage account with hierarchical namespace enabled, you can think of the prefix as a file path. Items under the path are included in the Azure Managed Lustre file system.
55
-
- If the blob container is in a storage account with a non-hierarchical (or flat) namespace, you can think of the import prefix as a search string that is compared with the beginning of the blob name. If the name of a blob in the container starts with the string you specified as the import prefix, that file is made accessible in the file system. Lustre is a hierarchical file system, and `/` characters in blob names become directory delimiters when stored in Lustre.
55
+
- If the blob container is in a storage account with a nonhierarchical (or flat) namespace, you can think of the import prefix as a search string that is compared with the beginning of the blob name. If the name of a blob in the container starts with the string you specified as the import prefix, that file is made accessible in the file system. Lustre is a hierarchical file system, and `/` characters in blob names become directory delimiters when stored in Lustre.
56
56
57
57
### Conflict resolution mode
58
58
@@ -81,7 +81,7 @@ When importing data from a blob container, you can specify the error tolerance.
81
81
82
82
The following error tolerance options are available for import jobs:
83
83
84
-
-**Do not allow errors** (default): The import job fails immediately if any error occurs during the import. This is the default behavior.
84
+
-**Do not allow errors** (default): The import job fails immediately if any error occurs during the import. This behavior is the default.
85
85
-**Allow errors**: The import job continues if an error occurs, and the error is logged. After the import job completes, you can view errors in the logging container.
This command tells the metadata server to asynchronously process a restoration request. The command line doesn't wait for the restore to complete, which means that the command line has the potential to queue up a large number of entries for restore at the metadata server. This approach can overwhelm the metadata server and degrade performance for restores.
108
108
109
-
To avoid this potential performance issue, you can create a basic script that attempts to walk the path and issues restore requests in batches of a specified size. To achieve reasonable performance and avoid overwhelming the metadata server, it's recommended to use batch sizes anywhere from 1,000 to 5,000 requests.
109
+
To avoid this potential performance issue, you can create a basic script that attempts to walk the path and issues restore requests in batches of a specified size. To achieve reasonable performance and avoid overwhelming the metadata server, we recommend using batch sizes anywhere from 1,000 to 5,000 requests.
110
110
111
111
### Example: Create a batch restore script
112
112
@@ -202,7 +202,7 @@ When you export files from your Azure Managed Lustre system, not all files are c
202
202
203
203
In active file systems, changes to files during the export job can result in failure status. This failure status lets you know that not all data in the file system could be exported to Blob Storage. In this situation, you can retry the export by [creating a new export job](export-with-archive-jobs.md#create-an-export-job). The new job copies only the files that weren't copied in the previous job.
204
204
205
-
In file systems with a lot of activity, retries might fail multiple times because files are frequently changing. To verify that a file has been successfully exported to Blob Storage, check the timestamp on the corresponding blob. After the job completes, you can also view the logging container configured at deployment time to see detailed information about the export job. The logging container provides diagnostic information about which files failed, and why they failed.
205
+
In file systems with a lot of activity, retries might fail multiple times because files are frequently changing. To verify that a file was successfully exported to Blob Storage, check the timestamp on the corresponding blob. After the job completes, you can also view the logging container configured at deployment time to see detailed information about the export job. The logging container provides diagnostic information about which files failed, and why they failed.
206
206
207
207
If you're preparing to decommission a cluster and want to perform a final export to Blob Storage, you should make sure that all I/O activities are halted before initiating the export job. This approach helps to guarantee that all data is exported by avoiding errors due to file system activity.
0 commit comments