Skip to content

Commit a6b72a3

Browse files
denreadenrea
authored
Merge pull request #281030 from usanth/docs-editor/personal-data-delete-export-1721171799
Update export instructions
2 parents 421b5fc + 014b538 commit a6b72a3

File tree

2 files changed

+73
-165
lines changed

2 files changed

+73
-165
lines changed

‎articles/devtest-labs/personal-data-delete-export.md

Lines changed: 72 additions & 154 deletions
Original file line numberDiff line numberDiff line change
@@ -17,190 +17,108 @@ DevTest Labs collects two main pieces of personal data from the user. They are:
1717
### User email address
1818
DevTest Labs uses the user email address to send auto shutdown email notifications to lab users. The email notifies users of their machine being shut down. The users can either postpone or skip the shutdown if they wish to do so. You configure the email address at the lab level or at the VM level.
1919

20-
**Setting email at the lab:**
21-
22-
![Setting email at the lab level](./media/personal-data-delete-export/lab-user-email.png)
23-
24-
**Setting email at the VM:**
25-
26-
![Setting email at the VM level](./media/personal-data-delete-export/vm-user-email.png)
27-
28-
### User object ID
29-
DevTest Labs uses the user object ID to show month-over-month cost trends and cost by resource information to lab admins. It allows them to track costs and manage thresholds for their Lab.
30-
31-
**Estimated cost trend for the current calendar month:**
32-
![Estimated cost trend for the current calendar month](./media/personal-data-delete-export/estimated-cost-trend-per-month.png)
33-
34-
**Estimated month-to-date cost by resource:**
35-
![Estimated month-to-date cost by resource](./media/personal-data-delete-export/estimated-month-to-date-cost-by-resource.png)
36-
37-
3820
## Why do we need this personal data?
39-
The DevTest Labs service uses the personal data for operational purposes. This data is critical for the service to deliver key features. If you set a retention policy on the user email address, lab users do not receive timely auto shutdown email notifications after their email address is deleted from our system. Similarly, the lab admin can't view month-over-month cost trends and cost by resource for machines in their labs if the user object IDs are deleted based on a retention policy. Therefore, this data needs to be retained for as long as the user's resource is active in the Lab.
21+
The DevTest Labs service uses the personal data for operational purposes. This data is critical for the service to deliver key features. If you set a retention policy on the user email address, lab users do not receive timely auto shutdown email notifications after their email address is deleted from our system. Therefore, this data needs to be retained for as long as the user's resource is active in the Lab.
4022

41-
## How can I have the system to forget my personal data?
42-
As a lab user, if you like to have this personal data deleted, you can do so by deleting the corresponding resource in the Lab. The DevTest Labs service anonymizes the deleted personal data 30 days after it's deleted by the user.
23+
## How can I have the system forget my personal data?
24+
As a lab user, you can delete your personal by deleting the corresponding resource in the Lab. The DevTest Labs service anonymizes the deleted personal data 30 days after it's deleted by the user.
4325

44-
For example, If you delete your VM, or removed your email address, the DevTest Labs service anonymizes this data 30 days after the resource is deleted. The 30-day retention policy after deletion is to make sure that we provide an accurate month-over-month cost projection to the lab admin.
26+
For example, if you delete your VM, or remove your email address, the DevTest Labs service anonymizes this data 30 days after the resource is deleted. The 30-day retention policy after deletion ensures that DevTest Labs provides an accurate month-over-month cost projection to the lab admin.
4527

4628
## How can I request an export on my personal data?
47-
You can export personal and lab usage data by using the Azure portal or PowerShell. The data is exported as two different CSV files:
48-
49-
- **disks.csv** - contains information about the disks being used by the different VMs
50-
- **virtualmachines.csv** - contains information about the VMs in the lab.
51-
52-
### Azure portal
53-
As a lab user, you can request an export on the personal data that the DevTest Labs service stores. To request for an export, navigate to the **Personal data** option on the **Overview** page of your lab. Select the **Request export** button kicks off the creation of a downloadable excel file in your Lab admin's storage account. You can then contact your lab admin to view this data.
54-
55-
1. Select **Personal data** on the left menu.
56-
57-
![Personal data page](./media/personal-data-delete-export/personal-data-page.png)
58-
2. Select the **resource group** that contains the lab.
59-
60-
![Select resource group](./media/personal-data-delete-export/select-resource-group.png)
61-
3. Select the **storage account** in the resource group.
62-
4. On the **Storage account** page, select **Blobs**.
63-
64-
![Select Blobs tile](./media/personal-data-delete-export/select-blobs-tile.png)
65-
5. Select the container named **labresourceusage** in the list of containers.
66-
67-
![Select blob container](./media/personal-data-delete-export/select-blob-container.png)
68-
6. Select the **folder** named after your lab. You find **csv** files for **disks** and **virtual machines** in your lab in this folder. You can download these csv files, filter the content for the lab user requesting an access, and share it with them.
69-
70-
![Download CSV file](./media/personal-data-delete-export/download-csv-file.png)
29+
You can export personal and lab usage data by using Azure PowerShell. DevTest Labs exports the data as a csv file with the date and time of the export requested in the name.
7130

7231
### Azure PowerShell
7332

7433
```powershell
7534
Param (
76-
[Parameter (Mandatory=$true, HelpMessage="The storage account name where to store usage data")]
77-
[string] $storageAccountName,
78-
79-
[Parameter (Mandatory=$true, HelpMessage="The storage account key")]
80-
[string] $storageKey,
35+
[Parameter (Mandatory=$true, HelpMessage="The resource group name of the storage account")]
36+
[string] $resourceGroupName,
37+
38+
[Parameter (Mandatory=$true, HelpMessage="The subscription id of the storage account and DTL")]
39+
[string] $subscriptionId,
8140
82-
[Parameter (Mandatory=$true, HelpMessage="The DevTest Lab name to get usage data from")]
83-
[string] $labName,
41+
[Parameter (Mandatory=$true, HelpMessage="The storage account name")]
42+
[string] $storageAccountName,
8443
85-
[Parameter (Mandatory=$true, HelpMessage="The DevTest Lab subscription")]
86-
[string] $labSubscription
87-
)
44+
[Parameter (Mandatory=$true, HelpMessage="Expire time of the SAS Token")]
45+
[string] $expiryTime,
8846
89-
#Login
90-
Login-AzureRmAccount
47+
[Parameter (Mandatory=$true, HelpMessage="Date to pull data from")][string] $startTime,
9148
92-
# Set the subscription for the lab
93-
Get-AzureRmSubscription -SubscriptionId $labSubscription | Select-AzureRmSubscription
49+
[Parameter (Mandatory=$true, HelpMessage="Name of the lab to export")]
50+
[string] $labName,
9451
95-
# DTL will create this container in the storage when invoking the action, cannot be changed currently
96-
$containerName = "labresourceusage"
52+
[Parameter (Mandatory=$true, HelpMessage="The desired SKU")]
53+
[string] $desiredSKU,
9754
98-
# Get the storage context
99-
$Ctx = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageKey
100-
$SasToken = New-AzureStorageAccountSASToken -Service Blob, File -ResourceType Container, Service, Object -Permission rwdlacup -Protocol HttpsOnly -Context $Ctx
55+
[Parameter (Mandatory=$true, HelpMessage="Protocol for SAS token generation")]
56+
[string] $protocol,
10157
102-
# Generate the storage blob uri
103-
$blobUri = $Ctx.BlobEndPoint + $SasToken
58+
[Parameter (Mandatory=$true, HelpMessage="Permissions given for SAS token")]
59+
[string] $permissions
10460
105-
# blobStorageAbsoluteSasUri and usageStartDate are required
61+
# Log in
62+
Connect-AzAccount -UseDeviceAuthentication
63+
64+
# Set your subscription
65+
Set-AzContext -SubscriptionId $subscriptionId
66+
67+
68+
# Create a resource group and storage account
69+
New-AzStorageAccount -ResourceGroupName $resourceGroupName `
70+
-Name $storageAccountName `
71+
-Location $location `
72+
-SkuName $desiredSKU
73+
74+
# Get storage account context
75+
$storageAccountContext = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -AccountName $storageAccountName
76+
$storageAccountKeys = Get-AzStorageAccountKey -ResourceGroupName $resourceGroupName -Name $storageAccountName
77+
78+
$Ctx = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKeys[0].Value
79+
80+
# Create blob container
81+
$containerName = "exportlabresources"
82+
New-AzStorageContainer -Name $containerName `
83+
-Context $Ctx `
84+
-Permission Off
85+
86+
# Get SAS token
87+
$sasToken = New-AzStorageContainerSASToken `
88+
-Context $Ctx `
89+
-Name $containerName `
90+
-StartTime (Get-Date) `
91+
-ExpiryTime $expiryTime `
92+
-Permission $permissions `
93+
-Protocol $protocol
94+
95+
# Make blob endpoint
96+
$blobEndpointWithSas = $storageAccountContext.Context.BlobEndPoint + $containerName+ "?" + $sasToken
97+
98+
# Invoke Export Job
10699
$actionParameters = @{
107-
'blobStorageAbsoluteSasUri' = $blobUri
100+
'blobStorageAbsoluteSasUri' = $blobEndpointWithSas
108101
}
109102
110-
$startdate = (Get-Date).AddDays(-7)
111-
112103
$actionParameters.Add('usageStartDate', $startdate.Date.ToString())
113-
114-
# Get the lab resource group
115-
$resourceGroupName = (Find-AzureRmResource -ResourceType 'Microsoft.DevTestLab/labs' | Where-Object { $_.Name -eq $labName}).ResourceGroupName
116-
117-
# Create the lab resource id
118-
$resourceId = "/subscriptions/" + $labSubscription + "/resourceGroups/" + $resourceGroupName + "/providers/Microsoft.DevTestLab/labs/" + $labName + "/"
119-
120-
# !!!!!!! this is the new resource action to get the usage data.
121-
$result = Invoke-AzureRmResourceAction -Action 'exportLabResourceUsage' -ResourceId $resourceId -Parameters $actionParameters -Force
122104
123-
# Finish up cleanly
124-
if ($result.Status -eq "Succeeded") {
125-
Write-Output "Telemetry successfully downloaded for " $labName
126-
return 0
127-
}
128-
else
129-
{
130-
Write-Output "Failed to download lab: " + $labName
131-
Write-Error $result.toString()
132-
return -1
133-
}
105+
$resourceId = "/subscriptions/" + $subscriptionId + "/resourceGroups/" + $resourceGroupName + "/providers/Microsoft.DevTestLab/labs/" + $labName + "/"
106+
107+
$result = Invoke-AzureRmResourceAction -Action 'ExportResourceUsage' -ResourceId $resourceId -Parameters $actionParameters -Force
108+
134109
```
135110

136-
The key components in the above sample are:
111+
The key components in the previous sample are:
137112

138113
- The Invoke-AzureRmResourceAction command.
139-
114+
140115
```
141-
Invoke-AzureRmResourceAction -Action 'exportLabResourceUsage' -ResourceId $resourceId -Parameters $actionParameters -Force
116+
Invoke-AzureRmResourceAction -Action 'ExportResourceUsage' -ResourceId $resourceId -Parameters $actionParameters -Force
142117
```
143118
- Two action parameters
144-
- **blobStorageAbsoluteSasUri** - The storage account URI with the Shared Access Signature (SAS) token. In the PowerShell script, this value could be passed in instead of the storage key.
145-
- **usageStartDate** - The beginning date to pull data, with the end date being the current date on which the action is executed. The granularity is at the day level, so even if you add time information, it will be ignored.
146-
147-
### Exported data - a closer look
148-
Now let’s take a closer look at the exported data. As mentioned earlier, once the data are successfully exported, there will be two CSV files.
149-
150-
The **virtualmachines.csv** contains the following data columns:
151-
152-
| Column name | Description |
153-
| ----------- | ----------- |
154-
| SubscriptionId | The subscription identifier that the lab exists in. |
155-
| LabUId | Unique GUID identifier for the lab. |
156-
| LabName | Name of the lab. |
157-
| LabResourceId | Fully qualified lab resource ID. |
158-
| ResourceGroupName | Name of the resource group that contains the VM |
159-
| ResourceId | Fully qualified resource ID for the VM. |
160-
| ResourceUId | GUID for the VM |
161-
| Name | Virtual machine name. |
162-
| CreatedTime | The date-time at which the VM was created. |
163-
| DeletedDate | The date-time at which the VM was deleted. If it's empty, deletion hasn't occurred, yet. |
164-
| ResourceOwner | Owner of the VM. If the value is empty, then it's either a claimable VM or created by a service principal. |
165-
| PricingTier | Pricing tier of the VM |
166-
| ResourceStatus | Availability state of the VM. Active, if still exists or Inactive, if the VM has been deleted. |
167-
| ComputeResourceId | Fully qualified virtual machine compute resource identifier. |
168-
| Claimable | Set to true if the VM is a claimable VM |
169-
| EnvironmentId | The environment resource identifier within which the Virtual machine was created in. It's empty when the VM wasn't created as part of an environment resource. |
170-
| ExpirationDate | Expiration date for the VM. It's set to empty, if an expiration date hasn't been set.
171-
| GalleryImageReferenceVersion | Version of the VM base image. |
172-
| GalleryImageReferenceOffer | Offer of the VM base image. |
173-
| GalleryImageReferencePublisher | Publisher of the VM base image. |
174-
| GalleryImageReferenceSku | Sku of the VM base image |
175-
| GalleryImageReferenceOsType | OS type of the VM base image |
176-
| CustomImageId | Fully qualified ID of the VM base custom image. |
177-
178-
The data columns contained in **disks.csv** are listed below:
179-
180-
| Column name | Description |
181-
| ----------- | ----------- |
182-
| SubscriptionId | ID of the subscription that contains the lab |
183-
| LabUId | GUID for the lab |
184-
| LabName | Name of the lab |
185-
| LabResourceId | Fully qualified resource ID for the lab |
186-
| ResourceGroupName | Name of the resource group that contains the lab |
187-
| ResourceId | Fully qualified resource ID for the VM. |
188-
| ResourceUId | GUID for the VM |
189-
|Name | The name of the attached disk |
190-
| CreatedTime |The date and time at which the data disk was created. |
191-
| DeletedDate | The date and time at which the data disk was deleted. |
192-
| ResourceStatus | Status of the resource. Active, if the resource exists. Inactive, when deleted. |
193-
| DiskBlobName | Blob name for the data disk. |
194-
| DiskSizeGB | The size of the data disk. |
195-
| DiskType | Type of the data disk. 0 for Standard, 1 for Premium. |
196-
| LeasedByVmId | Resource ID of the VM to which the data disk has been attached. |
197-
198-
199-
> [!NOTE]
200-
> If you are dealing with multiple labs and want to get overall information, the two key columns are the **LabUID** and the **ResourceUId**, which are the unique ids across subscriptions.
201-
202-
The exported data can be manipulated and visualized using tools, like SQL Server, Power BI, etc. This feature is especially useful when you want to report usage of your lab to your management team that may not be using the same Azure subscription as you do.
203-
119+
- **blobStorageAbsoluteSasUri** - The storage account URI with the Shared Access Signature (SAS) token. In the PowerShell script, this value could be passed in instead of the storage key.
120+
- **usageStartDate** - The beginning date to pull data, with the end date being the current date on which the action is executed. The granularity is at the day level, so even if you add time information, it will be ignored.
121+
204122
## Next steps
205123
See the following article:
206124

‎articles/devtest-labs/report-usage-across-multiple-labs-subscriptions.md

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -30,17 +30,7 @@ Before you can export DevTest Labs resource usage, you have to set up an Azure S
3030
> [!NOTE]
3131
> The date parameter doesn't include a time stamp so the data includes everything from midnight based on the time zone where the lab is located.
3232
33-
Once the export is complete, there will be multiple CSV files in the blob storage with the different resource information.
34-
35-
Currently there are two CSV files:
36-
37-
* *virtualmachines.csv* - contains information about the virtual machines in the lab
38-
* *disks.csv* - contains information about the different disks in the lab
39-
40-
These files are stored in the *labresourceusage* blob container. The files are under the lab name, lab unique ID, date executed, and either `full` or the start date of the export request. An example blob structure is:
41-
42-
* `labresourceusage/labname/1111aaaa-bbbb-cccc-dddd-2222eeee/<End>DD26-MM6-2019YYYY/full/virtualmachines.csv`
43-
* `labresourceusage/labname/1111aaaa-bbbb-cccc-dddd-2222eeee/<End>DD-MM-YYYY/26-6-2019/20-6-2019<Start>DD-MM-YYYY/virtualmachines.csv`
33+
Once the export is complete, there will be a CSV file in the blob storage with the different resource information.
4434

4535
## Exporting usage for all labs
4636

0 commit comments

Comments
 (0)