Skip to content

Commit d7a9b76

Browse files
authored
Merge pull request #100299 from kgremban/jan7-blobstorage
Blob storage github issues
2 parents 837f7d5 + d67628b commit d7a9b76

File tree

2 files changed

+75
-81
lines changed

2 files changed

+75
-81
lines changed

articles/iot-edge/how-to-deploy-blob.md

Lines changed: 21 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,12 @@
11
---
22
title: Deploy blob storage on module to your device - Azure IoT Edge
33
description: Deploy an Azure Blob Storage module to your IoT Edge device to store data at the edge.
4-
author: arduppal
5-
ms.author: arduppal
4+
author: kgremban
5+
ms.author: kgremban
66
ms.date: 12/13/2019
77
ms.topic: conceptual
88
ms.service: iot-edge
99
ms.reviewer: arduppal
10-
manager: brymat
11-
1210
---
1311

1412
# Deploy the Azure Blob Storage on IoT Edge module to your device
@@ -86,10 +84,10 @@ A deployment manifest is a JSON document that describes which modules to deploy,
8684
- Replace `<storage mount>` according to your container operating system. Provide the name of a [volume](https://docs.docker.com/storage/volumes/) or the absolute path to a directory on your IoT Edge device where you want the blob module to store its data. The storage mount maps a location on your device that you provide to a set location in the module.
8785

8886
- For Linux containers, the format is *\<storage path or volume>:/blobroot*. For example
89-
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:/blobroot**
87+
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:/blobroot**
9088
- use [bind mount](https://docs.docker.com/storage/bind-mounts/): **/srv/containerdata:/blobroot**. Make sure to follow the steps to [grant directory access to the container user](how-to-store-data-blob.md#granting-directory-access-to-container-user-on-linux)
9189
- For Windows containers, the format is *\<storage path or volume>:C:/BlobRoot*. For example
92-
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:C:/blobroot**.
90+
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:C:/blobroot**.
9391
- use [bind mount](https://docs.docker.com/storage/bind-mounts/): **C:/ContainerData:C:/BlobRoot**.
9492
- Instead of using your local drive, you can map your SMB network location, for more information see [using SMB share as your local storage](how-to-store-data-blob.md#using-smb-share-as-your-local-storage)
9593

@@ -104,26 +102,23 @@ A deployment manifest is a JSON document that describes which modules to deploy,
104102

105103
```json
106104
{
107-
"properties.desired": {
108-
"deviceAutoDeleteProperties": {
109-
"deleteOn": <true, false>,
110-
"deleteAfterMinutes": <timeToLiveInMinutes>,
111-
"retainWhileUploading":<true,false>
105+
"deviceAutoDeleteProperties": {
106+
"deleteOn": <true, false>,
107+
"deleteAfterMinutes": <timeToLiveInMinutes>,
108+
"retainWhileUploading": <true,false>
109+
},
110+
"deviceToCloudUploadProperties": {
111+
"uploadOn": <true, false>,
112+
"uploadOrder": "<NewestFirst, OldestFirst>",
113+
"cloudStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=<your Azure Storage Account Name>;AccountKey=<your Azure Storage Account Key>; EndpointSuffix=<your end point suffix>",
114+
"storageContainersForUpload": {
115+
"<source container name1>": {
116+
"target": "<target container name1>"
117+
}
112118
},
113-
"deviceToCloudUploadProperties": {
114-
"uploadOn": <true, false>,
115-
"uploadOrder": "<NewestFirst, OldestFirst>",
116-
"cloudStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=<your Azure Storage Account Name>;AccountKey=<your Azure Storage Account Key>; EndpointSuffix=<your end point suffix>",
117-
"storageContainersForUpload": {
118-
"<source container name1>": {
119-
"target": "<target container name1>"
120-
}
121-
},
122-
"deleteAfterUpload":<true,false>
123-
}
119+
"deleteAfterUpload": <true,false>
124120
}
125121
}
126-
127122
```
128123

129124
For information on configuring deviceToCloudUploadProperties and deviceAutoDeleteProperties after your module has been deployed, see [Edit the Module Twin](https://github.com/Microsoft/vscode-azure-iot-toolkit/wiki/Edit-Module-Twin). For more information about desired properties, see [Define or update desired properties](module-composition.md#define-or-update-desired-properties).
@@ -168,7 +163,7 @@ Azure IoT Edge provides templates in Visual Studio Code to help you develop edge
168163
| Select folder | Choose the location on your development machine for Visual Studio Code to create the solution files. |
169164
| Provide a solution name | Enter a descriptive name for your solution or accept the default **EdgeSolution**. |
170165
| Select module template | Choose **Existing Module (Enter full image URL)**. |
171-
| Provide a module name | Enter an all-lowercase name for your module, like **azureblobstorageoniotedge**.<br /><br />It's important to use a lowercase name for the Azure Blob Storage on IoT Edge module. IoT Edge is case-sensitive when referring to modules, and the Storage SDK defaults to lowercase. |
166+
| Provide a module name | Enter an all-lowercase name for your module, like **azureblobstorageoniotedge**.<br/><br/>It's important to use a lowercase name for the Azure Blob Storage on IoT Edge module. IoT Edge is case-sensitive when referring to modules, and the Storage SDK defaults to lowercase. |
172167
| Provide Docker image for the module | Provide the image URI: **mcr.microsoft.com/azure-blob-storage:latest** |
173168

174169
Visual Studio Code takes the information you provided, creates an IoT Edge solution, and then loads it in a new window. The solution template creates a deployment manifest template that includes your blob storage module image, but you need to configure the module's create options.
@@ -201,10 +196,10 @@ Azure IoT Edge provides templates in Visual Studio Code to help you develop edge
201196
1. Replace `<storage mount>` according to your container operating system. Provide the name of a [volume](https://docs.docker.com/storage/volumes/) or the absolute path to a directory on your IoT Edge device where you want the blob module to store its data. The storage mount maps a location on your device that you provide to a set location in the module.
202197

203198
- For Linux containers, the format is *\<storage path or volume>:/blobroot*. For example
204-
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:/blobroot**
199+
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:/blobroot**
205200
- use [bind mount](https://docs.docker.com/storage/bind-mounts/): **/srv/containerdata:/blobroot**. Make sure to follow the steps to [grant directory access to the container user](how-to-store-data-blob.md#granting-directory-access-to-container-user-on-linux)
206201
- For Windows containers, the format is *\<storage path or volume>:C:/BlobRoot*. For example
207-
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:C:/blobroot**.
202+
- use [volume mount](https://docs.docker.com/storage/volumes/): **my-volume:C:/blobroot**.
208203
- use [bind mount](https://docs.docker.com/storage/bind-mounts/): **C:/ContainerData:C:/BlobRoot**.
209204
- Instead of using your local drive, you can map your SMB network location, for more information see [using SMB share as your local storage](how-to-store-data-blob.md#using-smb-share-as-your-local-storage)
210205

articles/iot-edge/how-to-store-data-blob.md

Lines changed: 54 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
---
22
title: Store block blobs on devices - Azure IoT Edge | Microsoft Docs
33
description: Understand tiering and time-to-live features, see supported blob storage operations, and connect to your blob storage account.
4-
author: arduppal
5-
manager: brymat
6-
ms.author: arduppal
4+
author: kgremban
5+
ms.author: kgremban
76
ms.reviewer: arduppal
87
ms.date: 12/13/2019
98
ms.topic: conceptual
@@ -16,6 +15,7 @@ services: iot-edge
1615
Azure Blob Storage on IoT Edge provides a [block blob](https://docs.microsoft.com/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-block-blobs) and [append blob](https://docs.microsoft.com/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-append-blobs) storage solution at the edge. A blob storage module on your IoT Edge device behaves like an Azure blob service, except the blobs are stored locally on your IoT Edge device. You can access your blobs using the same Azure storage SDK methods or blob API calls that you're already used to. This article explains the concepts related to Azure Blob Storage on IoT Edge container that runs a blob service on your IoT Edge device.
1716

1817
This module is useful in scenarios:
18+
1919
* where data needs to be stored locally until it can be processed or transferred to the cloud. This data can be videos, images, finance data, hospital data, or any other unstructured data.
2020
* when devices are located in a place with limited connectivity.
2121
* when you want to efficiently process the data locally to get low latency access to the data, such that you can respond to emergencies as quickly as possible.
@@ -28,38 +28,37 @@ This module comes with **deviceToCloudUpload** and **deviceAutoDelete** features
2828

2929
**deviceToCloudUpload** is a configurable functionality. This function automatically uploads the data from your local blob storage to Azure with intermittent internet connectivity support. It allows you to:
3030

31-
- Turn ON/OFF the deviceToCloudUpload feature.
32-
- Choose the order in which the data is copied to Azure like NewestFirst or OldestFirst.
33-
- Specify the Azure Storage account to which you want your data uploaded.
34-
- Specify the containers you want to upload to Azure. This module allows you to specify both source and target container names.
35-
- Choose the ability to delete the blobs immediately, after upload to cloud storage is finished
36-
- Do full blob upload (using `Put Blob` operation) and block level upload (using `Put Block`, `Put Block List` and `Append Block` operations).
31+
* Turn ON/OFF the deviceToCloudUpload feature.
32+
* Choose the order in which the data is copied to Azure like NewestFirst or OldestFirst.
33+
* Specify the Azure Storage account to which you want your data uploaded.
34+
* Specify the containers you want to upload to Azure. This module allows you to specify both source and target container names.
35+
* Choose the ability to delete the blobs immediately, after upload to cloud storage is finished
36+
* Do full blob upload (using `Put Blob` operation) and block level upload (using `Put Block`, `Put Block List` and `Append Block` operations).
3737

3838
This module uses block level upload, when your blob consists of blocks. Here are some of the common scenarios:
3939

40-
- Your application updates some blocks of a previously uploaded block blob or appends new blocks to an append blob, this module uploads only the updated blocks and not the whole blob.
41-
- The module is uploading blob and internet connection goes away, when the connectivity is back again it uploads only the remaining blocks and not the whole blob.
42-
40+
* Your application updates some blocks of a previously uploaded block blob or appends new blocks to an append blob, this module uploads only the updated blocks and not the whole blob.
41+
* The module is uploading blob and internet connection goes away, when the connectivity is back again it uploads only the remaining blocks and not the whole blob.
42+
4343
If an unexpected process termination (like power failure) happens during a blob upload, all blocks that were due for the upload will be uploaded again once the module comes back online.
4444

4545
**deviceAutoDelete** is a configurable functionality. This function automatically deletes your blobs from the local storage when the specified duration (measured in minutes) expires. It allows you to:
4646

47-
- Turn ON/OFF the deviceAutoDelete feature.
48-
- Specify the time in minutes (deleteAfterMinutes) after which the blobs will be automatically deleted.
49-
- Choose the ability to retain the blob while it's uploading if the deleteAfterMinutes value expires.
50-
47+
* Turn ON/OFF the deviceAutoDelete feature.
48+
* Specify the time in minutes (deleteAfterMinutes) after which the blobs will be automatically deleted.
49+
* Choose the ability to retain the blob while it's uploading if the deleteAfterMinutes value expires.
5150

5251
## Prerequisites
5352

5453
An Azure IoT Edge device:
5554

56-
- You can use your development machine or a virtual machine as an IoT Edge device by following the steps in the quickstart for [Linux](quickstart-linux.md) or [Windows devices](quickstart.md).
55+
* You can use your development machine or a virtual machine as an IoT Edge device by following the steps in the quickstart for [Linux](quickstart-linux.md) or [Windows devices](quickstart.md).
5756

58-
- Refer to [Azure IoT Edge supported systems](support.md#operating-systems) for a list of supported operating systems and architectures. The Azure Blob Storage on IoT Edge module supports following architectures:
59-
- Windows AMD64
60-
- Linux AMD64
61-
- Linux ARM32
62-
- Linux ARM64 (preview)
57+
* Refer to [Azure IoT Edge supported systems](support.md#operating-systems) for a list of supported operating systems and architectures. The Azure Blob Storage on IoT Edge module supports following architectures:
58+
* Windows AMD64
59+
* Linux AMD64
60+
* Linux ARM32
61+
* Linux ARM64 (preview)
6362

6463
Cloud resources:
6564

@@ -166,15 +165,15 @@ The Azure Blob Storage documentation includes quickstart sample code in several
166165

167166
The following quickstart samples use languages that are also supported by IoT Edge, so you could deploy them as IoT Edge modules alongside the blob storage module:
168167

169-
- [.NET](../storage/blobs/storage-quickstart-blobs-dotnet.md)
170-
- [Python](../storage/blobs/storage-quickstart-blobs-python.md)
171-
- We have a known issue while using this SDK because this version of the module does not return blob creation time. Hence few methods like list blobs does not work. As a workaround set explicitly API version on the blob client to '2017-04-17'. <br>Example: `block_blob_service._X_MS_VERSION = '2017-04-17'`
172-
- [Append Blob Sample](https://github.com/Azure/azure-storage-python/blob/master/samples/blob/append_blob_usage.py)
173-
- [Node.js](../storage/blobs/storage-quickstart-blobs-nodejs-v10.md)
174-
- [JS/HTML](../storage/blobs/storage-quickstart-blobs-javascript-client-libraries-v10.md)
175-
- [Ruby](../storage/blobs/storage-quickstart-blobs-ruby.md)
176-
- [Go](../storage/blobs/storage-quickstart-blobs-go.md)
177-
- [PHP](../storage/blobs/storage-quickstart-blobs-php.md)
168+
* [.NET](../storage/blobs/storage-quickstart-blobs-dotnet.md)
169+
* [Python](../storage/blobs/storage-quickstart-blobs-python.md)
170+
* Versions before V2.1 of the Python SDK have a known issue where the module does not return blob creation time. Because of that issue, some methods like list blobs does not work. As a workaround, explicitly set the API version on the blob client to '2017-04-17'. Example: `block_blob_service._X_MS_VERSION = '2017-04-17'`
171+
* [Append Blob Sample](https://github.com/Azure/azure-storage-python/blob/master/samples/blob/append_blob_usage.py)
172+
* [Node.js](../storage/blobs/storage-quickstart-blobs-nodejs-v10.md)
173+
* [JS/HTML](../storage/blobs/storage-quickstart-blobs-javascript-client-libraries-v10.md)
174+
* [Ruby](../storage/blobs/storage-quickstart-blobs-ruby.md)
175+
* [Go](../storage/blobs/storage-quickstart-blobs-go.md)
176+
* [PHP](../storage/blobs/storage-quickstart-blobs-php.md)
178177

179178
## Connect to your local storage with Azure Storage Explorer
180179

@@ -206,65 +205,65 @@ Because not all Azure Blob Storage operations are supported by Azure Blob Storag
206205

207206
Supported:
208207

209-
- List containers
208+
* List containers
210209

211210
Unsupported:
212211

213-
- Get and set blob service properties
214-
- Preflight blob request
215-
- Get blob service stats
216-
- Get account information
212+
* Get and set blob service properties
213+
* Preflight blob request
214+
* Get blob service stats
215+
* Get account information
217216

218217
### Containers
219218

220219
Supported:
221220

222-
- Create and delete container
223-
- Get container properties and metadata
224-
- List blobs
225-
- Get and set container ACL
226-
- Set container metadata
221+
* Create and delete container
222+
* Get container properties and metadata
223+
* List blobs
224+
* Get and set container ACL
225+
* Set container metadata
227226

228227
Unsupported:
229228

230-
- Lease container
229+
* Lease container
231230

232231
### Blobs
233232

234233
Supported:
235234

236-
- Put, get, and delete blob
237-
- Get and set blob properties
238-
- Get and set blob metadata
235+
* Put, get, and delete blob
236+
* Get and set blob properties
237+
* Get and set blob metadata
239238

240239
Unsupported:
241240

242-
- Lease blob
243-
- Snapshot blob
244-
- Copy and abort copy blob
245-
- Undelete blob
246-
- Set blob tier
241+
* Lease blob
242+
* Snapshot blob
243+
* Copy and abort copy blob
244+
* Undelete blob
245+
* Set blob tier
247246

248247
### Block blobs
249248

250249
Supported:
251250

252-
- Put block
253-
- Put and get block list
251+
* Put block
252+
* Put and get block list
254253

255254
Unsupported:
256255

257-
- Put block from URL
256+
* Put block from URL
258257

259258
### Append blobs
260259

261260
Supported:
262261

263-
- Append block
262+
* Append block
264263

265264
Unsupported:
266265

267-
- Append block from URL
266+
* Append block from URL
268267

269268
## Event Grid on IoT Edge Integration
270269
> [!CAUTION]

0 commit comments

Comments
 (0)