Skip to content

Commit 3cf75a4

Browse files
v-albemiv-albemi
authored
Merge pull request #113100 from xyh1/patch-112
Blob Index docs fork
2 parents 719b729 + 479024b commit 3cf75a4

10 files changed

+655
-8
lines changed

‎articles/storage/blobs/TOC.yml

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -175,8 +175,10 @@
175175
href: ../common/storage-plan-manage-costs.md
176176
- name: Concurrency
177177
href: ../common/storage-concurrency.md?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
178-
- name: Search and understand blob data
178+
- name: Find, search, and understand blob data
179179
items:
180+
- name: Use Blob Index to manage and find blob data
181+
href: storage-manage-find-blobs.md
180182
- name: Use AI to understand blob data
181183
href: ../../search/search-blob-ai-integration.md?toc=%2fazure%2fstorage%2fblobs%2ftoc.json
182184
- name: Full text search for blob data
@@ -398,6 +400,8 @@
398400
href: storage-custom-domain-name.md
399401
- name: Route events to a custom endpoint
400402
href: storage-blob-event-quickstart.md
403+
- name: Utilize Blob Index
404+
href: storage-blob-index-how-to.md
401405
- name: Process change feed logs
402406
href: storage-blob-change-feed-how-to.md
403407
- name: Transfer data
@@ -824,6 +828,3 @@
824828
href: https://stackoverflow.com/questions/tagged/windows-azure-storage
825829
- name: Videos
826830
href: https://azure.microsoft.com/documentation/videos/index/?services=storage
827-
828-
829-

‎articles/storage/blobs/media/storage-blob-index-concepts/blob-index-get-set-tags.png

52.4 KB
Loading

‎articles/storage/blobs/media/storage-blob-index-concepts/blob-index-lifecycle-management-example.png

31.1 KB
Loading

‎articles/storage/blobs/media/storage-blob-index-concepts/blob-index-match-lifecycle-filter-set.png

37.7 KB
Loading

‎articles/storage/blobs/media/storage-blob-index-concepts/blob-index-tag-filter-within-container.png

41.7 KB
Loading

‎articles/storage/blobs/media/storage-blob-index-concepts/blob-index-upload-data-with-tags.png

57.4 KB
Loading

‎articles/storage/blobs/storage-blob-index-how-to.md

Lines changed: 300 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,300 @@
1+
---
2+
title: Utilize Blob Index to manage and find data on Azure Blob storage
3+
description: See examples of how to use Blob Index tags to categorize, manage, and query to discover blob objects.
4+
author: mhopkins-msft
5+
6+
ms.author: mhopkins
7+
ms.date: 04/24/2020
8+
ms.service: storage
9+
ms.subservice: common
10+
ms.topic: conceptual
11+
ms.reviewer: hux
12+
---
13+
14+
# Utilize Blob Index tags (Preview) to manage and find data on Azure Blob storage
15+
16+
Blob Index tags categorize data in your storage account utilizing key-value tag attributes. These tags are automatically indexed and exposed as a queryable multi-dimensional index to easily find data. This article shows you how to set, get, and find data using blob index tags.
17+
18+
To learn more about the Blob Index, see [Manage and find data on Azure Blob Storage with Blob Index (Preview)](storage-manage-find-blobs.md).
19+
20+
> [!NOTE]
21+
> Blob Index is in public preview, and is available in the **France Central** and **France South** regions. To learn more about this feature along with known issues and limitations, see [Manage and find data on Azure Blob Storage with Blob Index (Preview)](storage-manage-find-blobs.md).
22+
23+
## Prerequisites
24+
# [Portal](#tab/azure-portal)
25+
- Subscription registered and approved for access to the Blob Index preview
26+
- Access to [Azure portal](https://portal.azure.com/)
27+
28+
# [.NET](#tab/net)
29+
As Blob Index is in public preview, the .NET storage package is released in the preview NuGet feed. This library is subject to change between now and when it becomes official.
30+
31+
1. In Visual Studio, add the URL `https://azuresdkartifacts.blob.core.windows.net/azure-sdk-for-net/index.json` to your NuGet package sources.
32+
33+
To learn how, see [package sources](https://docs.microsoft.com/nuget/consume-packages/install-use-packages-visual-studio#package-sources).
34+
35+
2. In the NuGet Package Manager, Find the **Azure.Storage.Blobs** package, and install version **12.5.0-dev.20200422.2** to your project. You can also run the command ```Install-Package Azure.Storage.Blobs -Version12.5.0-dev.20200422.2```
36+
37+
To learn how, see [Find and install a package](https://docs.microsoft.com/nuget/consume-packages/install-use-packages-visual-studio#find-and-install-a-package).
38+
39+
3. Add the following using statements to the top of your code file.
40+
```csharp
41+
using Azure;
42+
using Azure.Storage.Blobs;
43+
using Azure.Storage.Blobs.Models;
44+
using Azure.Storage.Blobs.Specialized;
45+
using System;
46+
using System.Collections.Generic;
47+
using System.Threading.Tasks;
48+
```
49+
---
50+
51+
## Upload a new blob with index tags
52+
# [Portal](#tab/azure-portal)
53+
54+
1. In the [Azure portal](https://portal.azure.com/), select your storage account
55+
56+
2. Navigate to the **Containers** option under **Blob Service**, select your container
57+
58+
3. Select the **Upload** button to open the upload blade and browse your local file system to find a file to upload as a block blob.
59+
60+
4. Expand the **Advanced** dropdown and go to the **Blob Index Tags** section
61+
62+
5. Input the key/value blob index tags that you want applied to your data
63+
64+
6. Select the **Upload** button to upload the blob
65+
66+
![Upload data with blob index tags](media/storage-blob-index-concepts/blob-index-upload-data-with-tags.png)
67+
68+
# [.NET](#tab/net)
69+
The following example shows how to create an append blob with tags set during creation.
70+
```csharp
71+
static async Task BlobIndexTagsOnCreate()
72+
{
73+
BlobServiceClient serviceClient = new BlobServiceClient(ConnectionString);
74+
BlobContainerClient container = serviceClient.GetBlobContainerClient("mycontainer");
75+
76+
try
77+
{
78+
// Create a container
79+
await container.CreateIfNotExistsAsync();
80+
81+
// Create an append blob
82+
AppendBlobClient appendBlobWithTags = container.GetAppendBlobClient("myAppendBlob0.logs");
83+
84+
// Blob Index tags to upload
85+
CreateAppendBlobOptions appendOptions = new CreateAppendBlobOptions();
86+
appendOptions.Tags = new Dictionary<string, string>
87+
{
88+
{ "Sealed", "false" },
89+
{ "Content", "logs" },
90+
{ "Date", "2020-04-20" }
91+
};
92+
93+
// Upload data with tags set on creation
94+
await appendBlobWithTags.CreateAsync(appendOptions);
95+
}
96+
finally
97+
{
98+
}
99+
}
100+
```
101+
102+
---
103+
104+
## Get, set, and update blob index tags
105+
# [Portal](#tab/azure-portal)
106+
107+
1. In the [Azure portal](https://portal.azure.com/), select your storage account
108+
109+
2. Navigate to the **Containers** option under **Blob Service**, select your container
110+
111+
3. Select your desired blob from the list of blobs within the selected container
112+
113+
4. The blob overview tab will display your blob's properties including any **Blob Index Tags**
114+
115+
5. You can get, set, modify, or delete any of the key/value index tags for your blob
116+
117+
6. Select the **Save** button to confirm any updates to your blob
118+
119+
![Get, set, update, and delete blob index tags on objects](media/storage-blob-index-concepts/blob-index-get-set-tags.png)
120+
121+
# [.NET](#tab/net)
122+
```csharp
123+
static async Task BlobIndexTagsExample()
124+
{
125+
BlobServiceClient serviceClient = new BlobServiceClient(ConnectionString);
126+
BlobContainerClient container = serviceClient.GetBlobContainerClient("mycontainer");
127+
128+
try
129+
{
130+
// Create a container
131+
await container.CreateIfNotExistsAsync();
132+
133+
// Create a new append blob
134+
AppendBlobClient appendBlob = container.GetAppendBlobClient("myAppendBlob1.logs");
135+
await appendBlob.CreateAsync();
136+
137+
// Set or Update Blob Index tags on existing blob
138+
Dictionary<string, string> tags = new Dictionary<string, string>
139+
{
140+
{ "Project", "Contoso" },
141+
{ "Status", "Unprocessed" },
142+
{ "Sealed", "true" }
143+
};
144+
await appendBlob.SetTagsAsync(tags);
145+
146+
// Get Blob Index tags
147+
Response<IDictionary<string, string>> tagsResponse = await appendBlob.GetTagsAsync();
148+
Console.WriteLine(appendBlob.Name);
149+
foreach (KeyValuePair<string, string> tag in tagsResponse.Value)
150+
{
151+
Console.WriteLine($"{tag.Key}={tag.Value}");
152+
}
153+
154+
// List Blobs with all options returned including Blob Index tags
155+
await foreach (BlobItem blobItem in container.GetBlobsAsync(BlobTraits.All))
156+
{
157+
Console.WriteLine(Environment.NewLine + blobItem.Name);
158+
foreach (KeyValuePair<string, string> tag in blobItem.Tags)
159+
{
160+
Console.WriteLine($"{tag.Key}={tag.Value}");
161+
}
162+
}
163+
164+
// Delete existing Blob Index tags by replacing all tags
165+
Dictionary<string, string> noTags = new Dictionary<string, string>();
166+
await appendBlob.SetTagsAsync(noTags);
167+
168+
}
169+
finally
170+
{
171+
}
172+
}
173+
```
174+
175+
---
176+
177+
## Filter and Find data with blob index tags
178+
179+
# [Portal](#tab/azure-portal)
180+
181+
Within the Azure portal, the Blob Index tags filter automatically applies the `@container` parameter to scope your selected container. If you wish to filter and find tagged data across your entire storage account, please use our REST API, SDKs, or tools.
182+
183+
1. In the [Azure portal](https://portal.azure.com/), select your storage account.
184+
185+
2. Navigate to the **Containers** option under **Blob Service**, select your container
186+
187+
3. Select the **Blob Index tags filter** button to filter within the selected container
188+
189+
4. Enter a Blob Index tag key and tag value
190+
191+
5. Select the **Blob Index tags filter** button to add additional tag filters (up to 10)
192+
193+
![Filter and find tagged objects using blob index tags](media/storage-blob-index-concepts/blob-index-tag-filter-within-container.png)
194+
195+
# [.NET](#tab/net)
196+
```csharp
197+
static async Task FindBlobsByTagsExample()
198+
{
199+
BlobServiceClient serviceClient = new BlobServiceClient(ConnectionString);
200+
BlobContainerClient container1 = serviceClient.GetBlobContainerClient("mycontainer");
201+
BlobContainerClient container2 = serviceClient.GetBlobContainerClient("mycontainer2");
202+
203+
String singleEqualityQuery = @"""Archive"" = 'false'";
204+
String andQuery = @"""Archive"" = 'false' AND ""Priority"" = '01'";
205+
String rangeQuery = @"""Date"" >= '2020-04-20' AND ""Date"" <= '2020-04-30'";
206+
String containerScopedQuery = @"@container = 'mycontainer' AND ""Archive"" = 'false'";
207+
208+
String queryToUse = containerScopedQuery;
209+
210+
try
211+
{
212+
// Create a container
213+
await container1.CreateIfNotExistsAsync();
214+
await container2.CreateIfNotExistsAsync();
215+
216+
// Create append blobs
217+
AppendBlobClient appendBlobWithTags0 = container1.GetAppendBlobClient("myAppendBlob00.logs");
218+
AppendBlobClient appendBlobWithTags1 = container1.GetAppendBlobClient("myAppendBlob01.logs");
219+
AppendBlobClient appendBlobWithTags2 = container1.GetAppendBlobClient("myAppendBlob02.logs");
220+
AppendBlobClient appendBlobWithTags3 = container2.GetAppendBlobClient("myAppendBlob03.logs");
221+
AppendBlobClient appendBlobWithTags4 = container2.GetAppendBlobClient("myAppendBlob04.logs");
222+
AppendBlobClient appendBlobWithTags5 = container2.GetAppendBlobClient("myAppendBlob05.logs");
223+
224+
// Blob Index tags to upload
225+
CreateAppendBlobOptions appendOptions = new CreateAppendBlobOptions();
226+
appendOptions.Tags = new Dictionary<string, string>
227+
{
228+
{ "Archive", "false" },
229+
{ "Priority", "01" },
230+
{ "Date", "2020-04-20" }
231+
};
232+
233+
CreateAppendBlobOptions appendOptions2 = new CreateAppendBlobOptions();
234+
appendOptions2.Tags = new Dictionary<string, string>
235+
{
236+
{ "Archive", "true" },
237+
{ "Priority", "02" },
238+
{ "Date", "2020-04-24" }
239+
};
240+
241+
// Upload data with tags set on creation
242+
await appendBlobWithTags0.CreateAsync(appendOptions);
243+
await appendBlobWithTags1.CreateAsync(appendOptions);
244+
await appendBlobWithTags2.CreateAsync(appendOptions2);
245+
await appendBlobWithTags3.CreateAsync(appendOptions);
246+
await appendBlobWithTags4.CreateAsync(appendOptions2);
247+
await appendBlobWithTags5.CreateAsync(appendOptions2);
248+
249+
// Find Blobs given a tags query
250+
Console.WriteLine("Find Blob by Tags query: " + queryToUse + Environment.NewLine);
251+
252+
List<FilterBlobItem> blobs = new List<FilterBlobItem>();
253+
foreach (Page<FilterBlobItem> page in serviceClient.FindBlobsByTags(queryToUse).AsPages())
254+
{
255+
blobs.AddRange(page.Values);
256+
}
257+
258+
foreach (var filteredBlob in blobs)
259+
{
260+
Console.WriteLine($"BlobIndex result: ContainerName= {filteredBlob.ContainerName}, " +
261+
$"BlobName= {filteredBlob.Name}");
262+
}
263+
264+
}
265+
finally
266+
{
267+
}
268+
}
269+
```
270+
271+
---
272+
273+
## Lifecycle management with blob index tag filters
274+
275+
# [Portal](#tab/azure-portal)
276+
277+
1. In the [Azure portal](https://portal.azure.com/), select your storage account.
278+
279+
2. Navigate to the **Lifecycle Management** option under **Blob Service**
280+
281+
3. Select *Add rule* and then fill out the Action set form fields
282+
283+
4. Select Filter set to add optional filter for Prefix match and Blob Index match
284+
![Add blob index tag filters for lifecycle management](media/storage-blob-index-concepts/blob-index-match-lifecycle-filter-set.png)
285+
286+
5. Select **Review + add** to review the rule settings
287+
![Lifecycle management rule with blob index tags filter example](media/storage-blob-index-concepts/blob-index-lifecycle-management-example.png)
288+
289+
6. Select **Add** to apply the new rule to the lifecycle management policy
290+
291+
# [.NET](#tab/net)
292+
[Lifecycle management](storage-lifecycle-management-concepts.md) policies are applied for each storage account at the control plane level. For .NET, install the [Microsoft Azure Management Storage Library version 16.0.0](https://www.nuget.org/packages/Microsoft.Azure.Management.Storage/) or higher to take advantage of the Blob Index match filter within a lifecycle management rule.
293+
294+
---
295+
296+
## Next steps
297+
298+
Learn more about Blob Index. See [Manage and find data on Azure Blob Storage with Blob Index (Preview)](storage-manage-find-blobs.md )
299+
300+
Learn more about Lifecycle Management. See [Manage the Azure Blob storage lifecycle](storage-lifecycle-management-concepts.md)

‎articles/storage/blobs/storage-blob-properties-metadata.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@ In addition to the data they contain, blobs support system properties and user-d
2121

2222
- **User-defined metadata**: User-defined metadata consists of one or more name-value pairs that you specify for a Blob storage resource. You can use metadata to store additional values with the resource. Metadata values are for your own purposes only, and don't affect how the resource behaves.
2323

24+
> [!NOTE]
25+
> Blob Index tags also provide the ability to store arbitrary user-defined key/value attributes alongside a Blob storage resource. While similar to metadata, only Blob Index tags are automatically indexed and made queryable by the native blob service. Metadata cannot be natively indexed and queried unless you utilize a separate service such as Azure Search.
26+
>
27+
> To learn more about this feature, see [Manage and find data on Azure Blob Storage with Blob Index (Preview)](storage-manage-find-blobs.md).
28+
2429
Retrieving metadata and property values for a Blob storage resource is a two-step process. Before you can read these values, you must explicitly fetch them by calling the `FetchAttributes` or `FetchAttributesAsync` method. The exception to this rule is that the `Exists` and `ExistsAsync` methods call the appropriate `FetchAttributes` method under the covers. When you call one of these methods, you don't need to also call `FetchAttributes`.
2530

2631
> [!IMPORTANT]

0 commit comments

Comments
 (0)