You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/storage/blobs/blobfuse2-configuration.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ author: jimmart-dev
6
6
ms.service: storage
7
7
ms.subservice: blobs
8
8
ms.topic: how-to
9
-
ms.date: 08/02/2022
9
+
ms.date: 09/29/2022
10
10
ms.author: jammart
11
11
ms.reviewer: tamram
12
12
---
@@ -43,7 +43,7 @@ Using a configuration file is the preferred method, but the other methods can be
43
43
44
44
## Configuration file
45
45
46
-
Creating a configuration file is the preferred method of establishing settings for BlobFuse2. Once you have provided the desired settings in the file, reference the configuration file when using the `blobfuse2 mount` or other commands. Example:
46
+
Creating a configuration file is the preferred method of establishing settings for BlobFuse2. Once you have specified the desired settings in the file, reference the configuration file when using the `blobfuse2 mount` or other commands. Example:
47
47
48
48
````bash
49
49
blobfuse2 mount ./mount --config-file=./config.yaml
Copy file name to clipboardExpand all lines: articles/storage/blobs/blobfuse2-how-to-deploy.md
+42-20Lines changed: 42 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,23 +1,23 @@
1
1
---
2
-
title: How to mount an Azure blob storage container on Linux with BlobFuse2 (preview) | Microsoft Docs
2
+
title: How to mount an Azure Blob Storage container on Linux with BlobFuse2 (preview) | Microsoft Docs
3
3
titleSuffix: Azure Blob Storage
4
-
description: How to mount an Azure blob storage container on Linux with BlobFuse2 (preview).
4
+
description: How to mount an Azure Blob Storage container on Linux with BlobFuse2 (preview).
5
5
author: jammart
6
6
ms.service: storage
7
7
ms.subservice: blobs
8
8
ms.topic: how-to
9
-
ms.date: 09/26/2022
9
+
ms.date: 10/01/2022
10
10
ms.author: jammart
11
11
ms.reviewer: tamram
12
12
---
13
13
14
-
# How to mount an Azure blob storage container on Linux with BlobFuse2 (preview)
14
+
# How to mount an Azure Blob Storage container on Linux with BlobFuse2 (preview)
15
15
16
-
[BlobFuse2](blobfuse2-what-is.md) is a virtual file system driver for Azure Blob storage. BlobFuse2 allows you to access your existing Azure block blob data in your storage account through the Linux file system. For more details see [What is BlobFuse2? (preview)](blobfuse2-what-is.md).
16
+
[BlobFuse2](blobfuse2-what-is.md) is a virtual file system driver for Azure Blob Storage. BlobFuse2 allows you to access your existing Azure block blob data in your storage account through the Linux file system. For more details see [What is BlobFuse2? (preview)](blobfuse2-what-is.md).
17
17
18
18
> [!IMPORTANT]
19
19
> BlobFuse2 is the next generation of BlobFuse and is currently in preview.
20
-
> This preview version is provided without a service level agreement, and is not recommended for production workloads. Certain features might not be supported or might have constrained capabilities.
20
+
> This preview version is provided without a service level agreement, and might not be suitable for production workloads. Certain features might not be supported or might have constrained capabilities.
21
21
> For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
22
22
>
23
23
> If you need to use BlobFuse in a production environment, BlobFuse v1 is generally available (GA). For information about the GA version, see:
@@ -36,10 +36,10 @@ This guide shows you how to install and configure BlobFuse2, mount an Azure blob
36
36
37
37
There are 2 basic options for installing BlobFuse2:
### Option 1: Install the BlobFuse2 Binary (preferred)
43
43
44
44
For supported distributions see [the BlobFuse2 releases page](https://github.com/Azure/azure-storage-fuse/releases).
45
45
For libfuse support information, refer to [the BlobFuse2 README](https://github.com/Azure/azure-storage-fuse/blob/main/README.md#distinctive-features-compared-to-blobfuse-v1x).
@@ -52,7 +52,7 @@ lsb_release -a
52
52
53
53
If there are no binaries available for your distribution, you can [build the binaries from source code](https://github.com/MicrosoftDocs/azure-docs-pr/pull/203174#option-2-build-from-source).
54
54
55
-
#### Install the BlobFuse2 binaries
55
+
#### Install the BlobFuse2 binariesFstream
56
56
57
57
To install BlobFuse2:
58
58
@@ -107,15 +107,15 @@ To build the BlobFuse2 binaries from source:
107
107
108
108
## Configure BlobFuse2
109
109
110
-
You can configure BlobFuse2 with a variety of settings. Some of the common settings used include:
110
+
You can configure BlobFuse2 with a variety of settings. Some of the typical settings used include:
111
111
112
112
- Logging location and options
113
-
- Temporary cache file path
113
+
- Temporary file pathfor caching
114
114
- Information about the Azure storage account and blob container to be mounted
115
115
116
-
The settings can be configured in a yaml configuration file, using environment variables, or as parameters passed to the BlobFuse2 commands. The preferred method is to use the yaml configuration file.
116
+
The settings can be configured in a yaml configuration file, using environment variables, or as parameters passed to the BlobFuse2 commands. The preferred method is to use the configuration file.
117
117
118
-
For details about all of the configuration parameters for BlobFuse2, consult the complete reference material for each:
118
+
For details about each of the configuration parameters for BlobFuse2 and how to specify them, consult the references below:
@@ -124,21 +124,43 @@ For details about all of the configuration parameters for BlobFuse2, consult the
124
124
125
125
The basic steps forconfiguring BlobFuse2in preparation for mounting are:
126
126
127
-
1. [Configure a temporary path forcaching or streaming](#configure-a-temporary-path-for-caching)
127
+
1. [Configure caching](#configure-caching)
128
128
1. [Create an empty directory for mounting the blob container](#create-an-empty-directory-for-mounting-the-blob-container)
129
129
1. [Authorize access to your storage account](#authorize-access-to-your-storage-account)
130
130
131
-
### Configure a temporary path for caching
131
+
### Configure caching
132
132
133
-
BlobFuse2 provides native-like performance by requiring a temporary path in the file system to buffer and cache any open files. For this temporary path, choose the most performant disk available, or use a ramdisk for the best performance.
133
+
BlobFuse2 provides native-like performance by using local file-caching techniques. The caching configuration and behavior varies, depending on whether you are streaming large files or accessing smaller files.
134
+
135
+
#### Configure caching for streaming large files
136
+
137
+
BlobFuse2 supports streaming forboth read and write operations as an alternative to disk caching for files. In streaming mode, BlobFuse2 caches blocks of large filesin memory forboth reading and writing. The configuration settings related to caching for streaming are under the `stream:` settingsin your configuration file as follows:
138
+
139
+
```yml
140
+
stream:
141
+
block-size-mb:
142
+
For read only mode, the size of each block to be cached in memory while streaming (in MB)
143
+
For read/write mode: the size of newly created blocks
144
+
max-buffers: The total number of buffers to store blocks in
145
+
buffer-size-mb: The size for each buffer
146
+
```
147
+
148
+
See [the sample streaming configuration file](https://github.com/Azure/azure-storage-fuse/blob/main/sampleStreamingConfig.yaml) to get started quickly with some settings for a basic streaming scenario.
149
+
150
+
#### Configure caching for smaller files
151
+
152
+
Smaller files are cached to a temporary path specified under `file_cache:`in the configuration file as follows:
153
+
154
+
```yml
155
+
file_cache:
156
+
path: <path to local disk cache>
157
+
```
134
158
135
159
> [!NOTE]
136
-
> BlobFuse2 stores all open file contents in the temporary path. Make sure to have enough space to accommodate all open files.
160
+
> BlobFuse2 stores all open file contents in the temporary path. Make sure to have enough space to contain all open files.
137
161
>
138
162
139
-
#### Choose a caching disk option
140
-
141
-
There are 3 common options for configuring the temporary path for caching:
163
+
There are 3 common options for configuring the temporary path for file caching:
142
164
143
165
- [Use a local high-performing disk](#use-a-local-high-performing-disk)
Copy file name to clipboardExpand all lines: articles/storage/blobs/blobfuse2-what-is.md
+20-7Lines changed: 20 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ author: jammart
6
6
ms.service: storage
7
7
ms.subservice: blobs
8
8
ms.topic: how-to
9
-
ms.date: 09/26/2022
9
+
ms.date: 10/01/2022
10
10
ms.author: jammart
11
11
ms.reviewer: tamram
12
12
---
@@ -47,8 +47,9 @@ A full list of BlobFuse2 features is in the [BlobFuse2 README](https://github.co
47
47
48
48
- Mount an Azure storage blob container or Data Lake Storage Gen2 file system on Linux
49
49
- Use basic file system operations, such as mkdir, opendir, readdir, rmdir, open, read, create, write, close, unlink, truncate, stat, and rename
50
-
- Local caching to improve subsequent access times
50
+
- Local file caching to improve subsequent access times
51
51
- Streaming to support reading and writing large files
52
+
- Gain insights into mount activities and resource usage using BlobFuse2 Health Monitor
52
53
- Parallel downloads and uploads to improve access time for large files
53
54
- Multiple mounts to the same container for read-only workloads
54
55
@@ -59,6 +60,7 @@ Blobfuse2 has more feature support and improved performance in multiple user sce
59
60
- Improved caching
60
61
- More management support through new Azure CLI commands
61
62
- Additional logging support
63
+
- The addition of write-streaming for large files (read-streaming was previous supported)
62
64
- Gain insights into mount activities and resource usage using BlobFuse2 Health Monitor
63
65
- Compatibility and upgrade options for existing BlobFuse v1 users
64
66
- Version checking and upgrade prompting
@@ -88,7 +90,7 @@ In many ways, BlobFuse2-mounted storage can be used just like the native Linux f
88
90
89
91
However, there are some key differences in the way BlobFuse2 behaves:
90
92
91
-
-**Readdir count of hardlinks**:
93
+
-**Readdir count of hard links**:
92
94
93
95
For performance reasons, BlobFuse2 does not correctly report the hard links inside a directory. The number of hard links for empty directories is returned as 2. The number for non-empty directories is always returned as 3, regardless of the actual number of hard links.
94
96
@@ -116,14 +118,25 @@ However, there are some key differences in the way BlobFuse2 behaves:
116
118
117
119
BlobFuse2 doesn't support extended-attributes (x-attrs) operations.
118
120
121
+
-**Write-streaming**:
122
+
123
+
Concurrent streaming of read and write operations on large file data can produce unpredictable results. Simultaneously writing to the same blob from different threads is not supported.
124
+
119
125
### Data integrity
120
126
121
-
When a file is written to, the data is first persisted into cache on a local disk. The data is written to blob storage only after the file handle is closed. If there's an issue attempting to persist the data to blob storage, you receive an error message.
127
+
The file caching behavior plays an important role in the integrity of the data being read and written to a Blob Storage file system mount. Streaming mode is recommended for use with large files, which supports streaming for both read and write operations. BlobFuse2 caches blocks of streaming files in memory. For smaller files that do not consist of blocks, the entire file is stored in memory. File cache is the second mode and is recommended for workloads that do not contain large files. Where files are stored on disk in their entirety.
128
+
129
+
BlobFuse2 supports both read and write operations. Continuous synchronization of data written to storage by using other APIs or other mounts of BlobFuse2 isn't guaranteed. For data integrity, it's recommended that multiple sources don't modify the same blob, especially at the same time. If one or more applications attempt to write to the same file simultaneously, the results could be unexpected. Depending on the timing of multiple write operations and the freshness of the cache for each, the result could be that the last writer wins and previous writes are lost, or generally that the updated file isn't in the desired state.
130
+
131
+
#### File caching on disk
132
+
133
+
When a file is written to, the data is first persisted into cache on a local disk. The data is written to blob storage only after the file handle is closed. If there's an issue attempting to persist the data to blob storage, you will receive an error message.
134
+
135
+
#### Streaming
122
136
123
-
BlobFuse2 supports both read and write operations. Continuous synchronization of data written to storage by using other APIs or other mounts of BlobFuse2 aren't guaranteed. For data integrity, it's recommended that multiple sources don't modify the same blob, especially at the same time. If one or more applications attempt to write to the same file simultaneously, the results can be unexpected. Depending on the timing of multiple write operations and the freshness of the cache for each, the result could be that the last writer wins and previous writes are lost, or generally that the updated file isn't in the desired state.
137
+
For streaming during both read and write operations, blocks of data are cached in memory as they are read or updated. Updates are flushed to Azure Storage when a file is closed or when the buffer is filled with dirty blocks.
124
138
125
-
> [!WARNING]
126
-
> In cases where multiple file handles are open to the same file, simultaneous write operations could result in data loss.
139
+
Reading the same blob from multiple simultaneous threads is supported. However, simultaneous write operations could result in unexpected file data outcomes, including data loss. Performing simultaneous read operations and a single write operation is supported, but the data being read from some threads might not be current.
0 commit comments