Skip to content

Commit f8aac41

Browse files
authored
Merge pull request #129604 from mhopkins-msft/lifecycle-management
Adding last access time info
2 parents febe9e4 + 8cc9aff commit f8aac41

File tree

6 files changed

+130
-48
lines changed

6 files changed

+130
-48
lines changed
61 KB
Loading
48.5 KB
Loading
48.4 KB
Loading

articles/storage/blobs/storage-lifecycle-management-concepts.md

Lines changed: 130 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,12 @@ description: Learn how to create lifecycle policy rules to transition aging data
44
author: mhopkins-msft
55

66
ms.author: mhopkins
7-
ms.date: 04/24/2020
7+
ms.date: 09/15/2020
88
ms.service: storage
99
ms.subservice: common
1010
ms.topic: conceptual
1111
ms.reviewer: yzheng
12-
ms.custom: devx-track-azurepowershell
12+
ms.custom: "devx-track-azurepowershell, references_regions"
1313
---
1414

1515
# Manage the Azure Blob storage lifecycle
@@ -29,7 +29,7 @@ Consider a scenario where data gets frequent access during the early stages of t
2929

3030
## Availability and pricing
3131

32-
The lifecycle management feature is available in all Azure regions for General Purpose v2 (GPv2) accounts, Blob storage accounts, and Premium Block Blob storage accounts. In the Azure portal, you can upgrade an existing General Purpose (GPv1) account to a GPv2 account. For more information about storage accounts, see [Azure storage account overview](../common/storage-account-overview.md).
32+
The lifecycle management feature is available in all Azure regions for General Purpose v2 (GPv2) accounts, Blob storage accounts, and Premium Block Blob storage accounts. In the Azure portal, you can upgrade an existing General Purpose (GPv1) account to a GPv2 account. For more information about storage accounts, see [Azure storage account overview](../common/storage-account-overview.md).
3333

3434
The lifecycle management feature is free of charge. Customers are charged the regular operation cost for the [Set Blob Tier](https://docs.microsoft.com/rest/api/storageservices/set-blob-tier) API calls. Delete operation is free. For more information about pricing, see [Block Blob pricing](https://azure.microsoft.com/pricing/details/storage/blobs/).
3535

@@ -47,7 +47,7 @@ A policy can be read or written in full. Partial updates are not supported.
4747
> [!NOTE]
4848
> If you enable firewall rules for your storage account, lifecycle management requests may be blocked. You can unblock these requests by providing exceptions for trusted Microsoft services. For more information, see the Exceptions section in [Configure firewalls and virtual networks](https://docs.microsoft.com/azure/storage/common/storage-network-security#exceptions).
4949
50-
This article shows how to manage policy by using the portal and PowerShell methods.
50+
This article shows how to manage policy by using the portal and PowerShell methods.
5151

5252
# [Portal](#tab/azure-portal)
5353

@@ -60,64 +60,78 @@ There are two ways to add a policy through the Azure portal.
6060

6161
1. Sign in to the [Azure portal](https://portal.azure.com).
6262

63-
2. In the Azure portal, search for and select your storage account.
63+
1. In the Azure portal, search for and select your storage account.
6464

65-
3. Under **Blob Service**, select **Lifecycle management** to view or change your rules.
65+
1. Under **Blob service**, select **Lifecycle Management** to view or change your rules.
6666

67-
4. Select the **List view** tab.
67+
1. Select the **List View** tab.
6868

69-
5. Select **Add rule** and then fill out the **Action set** form fields. In the following example, blobs are moved to cool storage if they haven't been modified for 30 days.
69+
1. Select **Add a rule** and name your rule on the **Details** form. You can also set the **Rule scope**, **Blob type**, and **Blob subtype** values. The following example sets the scope to filter blobs. This causes the **Filter set** tab to be added.
7070

71-
![Lifecycle management action set page in Azure portal](media/storage-lifecycle-management-concepts/lifecycle-management-action-set.png)
71+
:::image type="content" source="media/storage-lifecycle-management-concepts/lifecycle-management-details.png" alt-text="Lifecycle management add a rule details page in Azure portal":::
7272

73-
6. Select **Filter set** to add an optional filter. Then, select **Browse** to specify a container and folder by which to filter.
73+
1. Select **Base blobs** to set the conditions for your rule. In the following example, blobs are moved to cool storage if they haven't been modified for 30 days.
7474

75-
![Lifecycle management filter set page in Azure portal](media/storage-lifecycle-management-concepts/lifecycle-management-filter-set-browse.png)
75+
:::image type="content" source="media/storage-lifecycle-management-concepts/lifecycle-management-base-blobs.png" alt-text="Lifecycle management base blobs page in Azure portal":::
7676

77-
8. Select **Review + add** to review the policy settings.
77+
The **Last accessed** option is available in preview in the following regions:
7878

79-
9. Select **Add** to add the new policy.
79+
- France Central
80+
- Canada East
81+
- Canada Central
82+
83+
> [!IMPORTANT]
84+
> The last access time tracking preview is for non-production use only. Production service-level agreements (SLAs) are not currently available.
85+
86+
For more information about the **Last accessed** option, see [Move data based on last accessed date (preview)](#move-data-based-on-last-accessed-date-preview).
87+
88+
1. If you selected **Limit blobs with filters** on the **Details** page, select **Filter set** to add an optional filter. The following example filters on blobs in the *mylifecyclecontainer* container that begin with "log".
89+
90+
:::image type="content" source="media/storage-lifecycle-management-concepts/lifecycle-management-filter-set.png" alt-text="Lifecycle management filter set page in Azure portal":::
91+
92+
1. Select **Add** to add the new policy.
8093

8194
#### Azure portal Code view
8295
1. Sign in to the [Azure portal](https://portal.azure.com).
8396

84-
2. In the Azure portal, search for and select your storage account.
97+
1. In the Azure portal, search for and select your storage account.
8598

86-
3. Under **Blob Service**, select **Lifecycle management** to view or change your policy.
99+
1. Under **Blob service**, select **Lifecycle Management** to view or change your policy.
87100

88-
4. The following JSON is an example of a policy that can be pasted into the **Code view** tab.
101+
1. The following JSON is an example of a policy that can be pasted into the **Code View** tab.
89102

90103
```json
91104
{
92105
"rules": [
93106
{
94-
"name": "ruleFoo",
95107
"enabled": true,
108+
"name": "move-to-cool",
96109
"type": "Lifecycle",
97110
"definition": {
98-
"filters": {
99-
"blobTypes": [ "blockBlob" ],
100-
"prefixMatch": [ "container1/foo" ]
101-
},
102111
"actions": {
103112
"baseBlob": {
104-
"tierToCool": { "daysAfterModificationGreaterThan": 30 },
105-
"tierToArchive": { "daysAfterModificationGreaterThan": 90 },
106-
"delete": { "daysAfterModificationGreaterThan": 2555 }
107-
},
108-
"snapshot": {
109-
"delete": { "daysAfterCreationGreaterThan": 90 }
113+
"tierToCool": {
114+
"daysAfterModificationGreaterThan": 30
115+
}
110116
}
117+
},
118+
"filters": {
119+
"blobTypes": [
120+
"blockBlob"
121+
],
122+
"prefixMatch": [
123+
"mylifecyclecontainer/log"
124+
]
111125
}
112126
}
113127
}
114128
]
115129
}
116130
```
117131

118-
5. Select **Save**.
132+
1. Select **Save**.
119133

120-
6. For more information about this JSON example, see the [Policy](#policy) and [Rules](#rules) sections.
134+
1. For more information about this JSON example, see the [Policy](#policy) and [Rules](#rules) sections.
121135

122136
# [PowerShell](#tab/azure-powershell)
123137

@@ -225,7 +239,7 @@ Each rule within the policy has several parameters:
225239

226240
| Parameter name | Parameter type | Notes | Required |
227241
|----------------|----------------|-------|----------|
228-
| `name` | String |A rule name can include up to 256 alphanumeric characters. Rule name is case-sensitive. It must be unique within a policy. | True |
242+
| `name` | String |A rule name can include up to 256 alphanumeric characters. Rule name is case-sensitive. It must be unique within a policy. | True |
229243
| `enabled` | Boolean | An optional boolean to allow a rule to be temporary disabled. Default value is true if it's not set. | False |
230244
| `type` | An enum value | The current valid type is `Lifecycle`. | True |
231245
| `definition` | An object that defines the lifecycle rule | Each definition is made up of a filter set and an action set. | True |
@@ -236,10 +250,10 @@ Each rule definition includes a filter set and an action set. The [filter set](#
236250

237251
### Sample rule
238252

239-
The following sample rule filters the account to run the actions on objects that exist inside `container1` and start with `foo`.
253+
The following sample rule filters the account to run the actions on objects that exist inside `container1` and start with `foo`.
240254

241255
>[!NOTE]
242-
>- Lifecycle management only supports block blob type.<br>
256+
>- Lifecycle management supports block blob and append blob types.<br>
243257
>- Lifecycle management does not affect system containers like $logs and $web.
244258
245259
- Tier blob to cool tier 30 days after last modification
@@ -283,8 +297,8 @@ Filters include:
283297

284298
| Filter name | Filter type | Notes | Is Required |
285299
|-------------|-------------|-------|-------------|
286-
| blobTypes | An array of predefined enum values. | The current release supports `blockBlob`. | Yes |
287-
| prefixMatch | An array of strings for prefixes to be matched. Each rule can define up to 10 prefixes. A prefix string must start with a container name. For example, if you want to match all blobs under `https://myaccount.blob.core.windows.net/container1/foo/...` for a rule, the prefixMatch is `container1/foo`. | If you don't define prefixMatch, the rule applies to all blobs within the storage account. | No |
300+
| blobTypes | An array of predefined enum values. | The current release supports `blockBlob` and `appendBlob`. Only delete is supported for `appendBlob`, set tier is not supported. | Yes |
301+
| prefixMatch | An array of strings for prefixes to be matched. Each rule can define up to 10 prefixes. A prefix string must start with a container name. For example, if you want to match all blobs under `https://myaccount.blob.core.windows.net/container1/foo/...` for a rule, the prefixMatch is `container1/foo`. | If you don't define prefixMatch, the rule applies to all blobs within the storage account. | No |
288302
| blobIndexMatch | An array of dictionary values consisting of Blob Index tag key and value conditions to be matched. Each rule can define up to 10 Blob Index tag condition. For example, if you want to match all blobs with `Project = Contoso` under `https://myaccount.blob.core.windows.net/` for a rule, the blobIndexMatch is `{"name": "Project","op": "==","value": "Contoso"}`. | If you don't define blobIndexMatch, the rule applies to all blobs within the storage account. | No |
289303

290304
> [!NOTE]
@@ -296,21 +310,23 @@ Actions are applied to the filtered blobs when the run condition is met.
296310

297311
Lifecycle management supports tiering and deletion of blobs and deletion of blob snapshots. Define at least one action for each rule on blobs or blob snapshots.
298312

299-
| Action | Base Blob | Snapshot |
300-
|---------------|---------------------------------------------|---------------|
301-
| tierToCool | Support blobs currently at hot tier | Not supported |
302-
| tierToArchive | Support blobs currently at hot or cool tier | Not supported |
303-
| delete | Supported | Supported |
313+
| Action | Base Blob | Snapshot |
314+
|-----------------------------|---------------------------------------------|---------------|
315+
| tierToCool | Support blobs currently at hot tier | Not supported |
316+
| enableAutoTierToHotFromCool | Support blobs currently at cool tier | Not supported |
317+
| tierToArchive | Support blobs currently at hot or cool tier | Not supported |
318+
| delete | Supported for `blockBlob` and `appendBlob` | Supported |
304319

305320
>[!NOTE]
306321
>If you define more than one action on the same blob, lifecycle management applies the least expensive action to the blob. For example, action `delete` is cheaper than action `tierToArchive`. Action `tierToArchive` is cheaper than action `tierToCool`.
307322
308323
The run conditions are based on age. Base blobs use the last modified time to track age, and blob snapshots use the snapshot creation time to track age.
309324

310-
| Action run condition | Condition value | Description |
311-
|----------------------------------|------------------------------------------|-----------------------------------------|
312-
| daysAfterModificationGreaterThan | Integer value indicating the age in days | The condition for base blob actions |
313-
| daysAfterCreationGreaterThan | Integer value indicating the age in days | The condition for blob snapshot actions |
325+
| Action run condition | Condition value | Description |
326+
|------------------------------------|------------------------------------------|----------------------------------------------------------------------------------|
327+
| daysAfterModificationGreaterThan | Integer value indicating the age in days | The condition for base blob actions |
328+
| daysAfterCreationGreaterThan | Integer value indicating the age in days | The condition for blob snapshot actions |
329+
| daysAfterLastAccessTimeGreaterThan | Integer value indicating the age in days | (preview) The condition for base blob actions when last accessed time is enabled |
314330

315331
## Examples
316332

@@ -344,6 +360,69 @@ This example shows how to transition block blobs prefixed with `container1/foo`
344360
}
345361
```
346362

363+
### Move data based on last accessed date (preview)
364+
365+
You can enable last access time tracking to keep a record of when your blob is last read or written. You can use last access time as a filter to manage tiering and retention of your blob data.
366+
367+
The **Last accessed** option is available in preview in the following regions:
368+
369+
- France Central
370+
- Canada East
371+
- Canada Central
372+
373+
> [!IMPORTANT]
374+
> The last access time tracking preview is for non-production use only. Production service-level agreements (SLAs) are not currently available.
375+
376+
#### How last access time tracking works
377+
378+
When last access time tracking is enabled, the blob property called `LastAccessTime` is updated when a blob is read or written. A [Get Blob](/rest/api/storageservices/get-blob) operation is considered an access operation. [Get Blob Properties](/rest/api/storageservices/get-blob-properties), [Get Blob Metadata](/rest/api/storageservices/get-blob-metadata), and [Get Blob Tags](/rest/api/storageservices/get-blob-tags) are not access operations, and therefore don't update the last access time.
379+
380+
To minimize the impact on read access latency, only the first read of the last 24 hours updates the last access time. Subsequent reads in the same 24-hour period do not update the last access time. If a blob is modified between reads, the last access time is the more recent of the two values.
381+
382+
In the following example, blobs are moved to cool storage if they haven't been accessed for 30 days. The `enableAutoTierToHotFromCool` property is a Boolean value that indicates if a blob should automatically be tiered from cool back to hot if it is accessed again after being tiered to cool.
383+
384+
```json
385+
{
386+
"enabled": true,
387+
"name": "last-accessed-thirty-days-ago",
388+
"type": "Lifecycle",
389+
"definition": {
390+
"actions": {
391+
"baseBlob": {
392+
"enableAutoTierToHotFromCool": true,
393+
"tierToCool": {
394+
"daysAfterLastAccessTimeGreaterThan": 30
395+
}
396+
}
397+
},
398+
"filters": {
399+
"blobTypes": [
400+
"blockBlob"
401+
],
402+
"prefixMatch": [
403+
"mylifecyclecontainer/log"
404+
]
405+
}
406+
}
407+
}
408+
```
409+
410+
#### Storage account support
411+
412+
Last access time tracking is available for the following types of storage accounts:
413+
414+
- General-purpose v2 storage accounts
415+
- Block blob storage accounts
416+
- Blob storage accounts
417+
418+
If your storage account is a general-purpose v1 account, use the Azure portal to upgrade to a general-purpose v2 account.
419+
420+
Storage accounts with a hierarchical namespace enabled for use with Azure Data Lake Storage Gen2 are not yet supported.
421+
422+
#### Pricing and billing
423+
424+
Each last access time update is considered an [other operation](https://azure.microsoft.com/pricing/details/storage/blobs/).
425+
347426
### Archive data after ingest
348427

349428
Some data stays idle in the cloud and is rarely, if ever, accessed once stored. The following lifecycle policy is configured to archive data shortly after it is ingested. This example transitions block blobs in the storage account within container `archivecontainer` into an archive tier. The transition is accomplished by acting on blobs 0 days after last modified time:
@@ -466,13 +545,16 @@ For data that is modified and accessed regularly throughout its lifetime, snapsh
466545

467546
## FAQ
468547

469-
**I created a new policy, why do the actions not run immediately?**
470-
The platform runs the lifecycle policy once a day. Once you configure a policy, it can take up to 24 hours for some actions to run for the first time.
548+
**I created a new policy, why do the actions not run immediately?**
549+
550+
The platform runs the lifecycle policy once a day. Once you configure a policy, it can take up to 24 hours for some actions to run for the first time.
551+
552+
**If I update an existing policy, how long does it take for the actions to run?**
553+
554+
The updated policy takes up to 24 hours to go into effect. Once the policy is in effect, it could take up to 24 hours for the actions to run. Therefore, the policy actions may take up to 48 hours to complete.
471555

472-
**If I update an existing policy, how long does it take for the actions to run?**
473-
The updated policy takes up to 24 hours to go into effect. Once the policy is in effect, it could take up to 24 hours for the actions to run. Therefore, the policy actions may take up to 48 hours to complete.
556+
**I manually rehydrated an archived blob, how do I prevent it from being moved back to the Archive tier temporarily?**
474557

475-
**I manually rehydrated an archived blob, how do I prevent it from being moved back to the Archive tier temporarily?**
476558
When a blob is moved from one access tier to another, its last modification time doesn't change. If you manually rehydrate an archived blob to hot tier, it would be moved back to archive tier by the lifecycle management engine. Disable the rule that affects this blob temporarily to prevent it from being archived again. Re-enable the rule when the blob can be safely moved back to archive tier. You may also copy the blob to another location if it needs to stay in hot or cool tier permanently.
477559

478560
## Next steps

0 commit comments

Comments
 (0)