Skip to content

Commit 810ae73

Browse files
rboucherrboucher
committed
private link storage article
1 parent bdbde66 commit 810ae73

File tree

1 file changed

+223
-0
lines changed

1 file changed

+223
-0
lines changed

‎articles/azure-monitor/platform/private-link-storage.md

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
---
2+
title: Customer-owned storage accounts for log ingestion
3+
description: Use your own storage account for ingestion of log data into a Log Analytics workspace in Azure Monitor.
4+
ms.subservice: logs
5+
ms.topic: conceptual
6+
author: bwren
7+
ms.author: bwren
8+
ms.date: 05/20/2020
9+
10+
---
11+
12+
# Customer-owned storage accounts for log ingestion in Azure Monitor
13+
Storage accounts are used by Azure Monitor in the ingestion process of some data types such as [custom logs](data-sources-custom-logs.md) and some [Azure logs](azure-storage-iis-table.md). During the ingestion process, logs are first sent to a storage account and later ingested into Log Analytics or Application Insights. Customers that want control over their data during ingestion can use their own storage accounts instead of the service-managed storage. The use of customer storage account provides customers control over the access, content, encryption, and retention of the logs during ingestion. We refer to this as Bring Your Own Storage, or BYOS.
14+
15+
One scenario that requires this feature is network isolation through Private Links. When using a VNet, network isolation is often a requirement, and access to the public internet is limited. In such cases, accessing Azure Monitor service storage for log ingestion is either completely blocked, or simply considered a bad practice. Logs should instead be ingested through a customer-owned storage account inside the VNet or easily accessible from it.
16+
17+
Another scenario is the encryption of logs with Customer Managed Keys (CMK). Customers can encrypt logged data by using CMK on the clusters that store the logs. The same key can also be used to encrypt logs during the ingestion process.
18+
19+
## Data types supported
20+
Data types that are ingested from a storage account include the following. See [Collect data from Azure diagnostics extension to Azure Monitor Logs](azure-storage-iis-table.md) for more information about the ingestion of these types.
21+
22+
- IIS logs (Blob: wad-iis-logfiles)
23+
- Windows event logs (Table: WADWindowsEventLogsTable)
24+
- Syslog (Table: LinuxsyslogVer2v0)
25+
- Windows ETW logs (Table: WADETWEventTable)
26+
- Service fabric (Table: WADServiceFabricSystemEventTable, WADServiceFabricReliableActorEventTable, WADServiceFabricReliableServiceEventTable)
27+
- Custom logs
28+
- Azure Security Center Watson dump files
29+
30+
## Storage account requirements
31+
The storage account must meet the following requirements:
32+
33+
- Accessible to resources on your VNet that write logs to the storage.
34+
- Must be on the same region as the workspace it’s linked to.
35+
- Explicitly allowed Log Analytics to read logs from the storage account by selecting *allow trusted MS services to access this storage account*.
36+
37+
## Process to configure customer-owned storage
38+
The basic process of using your own storage account for ingestion is as follows:
39+
40+
1. Create a storage account or select an existing account.
41+
2. Link the storage account to a Log Analytics workspace.
42+
3. Manage the storage by reviewing its load and retention to ensure sure it’s functioning as expected.
43+
44+
The only method available to create and remove links is through the REST API. Details on the specific API request required for each process are provided in the sections below.
45+
46+
## API request values
47+
48+
#### dataSourceType
49+
50+
- AzureWatson - Use this value for Azure Security Center Azure Watson dump files.
51+
- CustomLogs – Use this value for the following data types:
52+
- Custom logs
53+
- IIS Logs
54+
- Events (Windows)
55+
- Syslog (Linux)
56+
- ETW Logs
57+
- Service Fabric Events
58+
- Assessment data
59+
60+
#### storage_account_resource_id
61+
This value uses the following structure:
62+
63+
```
64+
subscriptions/{subscriptionId}/resourcesGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{storageAccountName1}
65+
```
66+
67+
68+
69+
## Get current links
70+
71+
### Get linked storage accounts for a specific data source type
72+
73+
#### API request
74+
75+
```
76+
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/linkedStorageAccounts/{dataSourceType}?api-version=2019-08-01-preview
77+
```
78+
79+
#### Response
80+
81+
```json
82+
{
83+
"properties":
84+
{
85+
"dataSourceType": "CustomLogs",
86+
"storageAccountIds ":
87+
[
88+
"<storage_account_resource_id_1>",
89+
"<storage_account_resource_id_2>"
90+
],
91+
},
92+
"id":"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/microsoft. operationalinsights/workspaces/{resourceName}/linkedStorageAccounts/CustomLogs",
93+
"name": "CustomLogs",
94+
"type": "Microsoft.OperationalInsights/workspaces/linkedStorageAccounts"
95+
}
96+
```
97+
98+
### Get all linked storage accounts
99+
100+
#### API request
101+
102+
```
103+
GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/linkedStorageAccounts?api-version=2019-08-01-preview
104+
```
105+
106+
#### Response
107+
108+
```json
109+
{
110+
[
111+
{
112+
"properties":
113+
{
114+
"dataSourceType": "CustomLogs",
115+
"storageAccountIds ":
116+
[
117+
"<storage_account_resource_id_1>",
118+
"<storage_account_resource_id_2>"
119+
],
120+
},
121+
"id":"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/microsoft. operationalinsights/workspaces/{resourceName}/linkedStorageAccounts/CustomLogs",
122+
"name": "CustomLogs",
123+
"type": "Microsoft.OperationalInsights/workspaces/linkedStorageAccounts"
124+
},
125+
{
126+
"properties":
127+
{
128+
"dataSourceType": " AzureWatson "
129+
"storageAccountIds ":
130+
[
131+
"<storage_account_resource_id_3>",
132+
"<storage_account_resource_id_4>"
133+
],
134+
},
135+
"id":"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/microsoft. operationalinsights/workspaces/{resourceName}/linkedStorageAccounts/AzureWatson",
136+
"name": "AzureWatson",
137+
"type": "Microsoft.OperationalInsights/workspaces/linkedStorageAccounts"
138+
}
139+
]
140+
}
141+
```
142+
143+
## Create or modify a link
144+
Once you link a storage account to a workspace, Log Analytics will start using it instead of the storage account owned by the service. You can register a list of storage accounts at the same time, and you can use the same storage account for multiple workspaces.
145+
146+
If your workspace handles both VNet resources and resources outside a VNet, you should make sure it’s not rejecting traffic coming from the internet. Your storage should have the same settings as your workspace and be made available to resources outside your VNet.
147+
148+
### API request
149+
150+
```
151+
PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/linkedStorageAccounts/{dataSourceType}?api-version=2019-08-01-preview
152+
```
153+
154+
### Payload
155+
156+
```json
157+
{
158+
"properties":
159+
{
160+
"storageAccountIds " :
161+
[
162+
"<storage_account_resource_id_1>",
163+
"<storage_account_resource_id_2>"
164+
],
165+
}
166+
}
167+
```
168+
169+
### Response
170+
171+
```json
172+
{
173+
"properties":
174+
{
175+
"dataSourceType": "CustomLogs"
176+
"storageAccountIds ":
177+
[
178+
"<storage_account_resource_id_1>",
179+
"<storage_account_resource_id_2>"
180+
],
181+
},
182+
"id":"/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/microsoft. operationalinsights/workspaces/{resourceName}/linkedStorageAccounts/CustomLogs",
183+
"name": "CustomLogs",
184+
"type": "Microsoft.OperationalInsights/workspaces/linkedStorageAccounts"
185+
}
186+
```
187+
188+
189+
## Unlink a storage account
190+
If you decide to stop using a storage account for ingestion, or replace the workspace you use, you should unlink the storage from the workspace.
191+
192+
Unlinking all storage accounts from a workspace means ingestion will attempt to rely on service-managed storage accounts. If your agents run on a VNet with limited access to the internet, ingestion is expected to fail. The workspace must have a linked storage account that is reachable from your monitored resources.
193+
194+
Before deleting a storage account, you should make sure that all the data it contains has been ingested to the workspace. As a precaution, keep your storage account available after linking an alternative storage. Only delete it once all of its content has been ingested, and you can see new data is written to the newly connected storage account.
195+
196+
197+
### API request
198+
```
199+
DELETE https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/linkedStorageAccounts/{dataSourceType}?api-version=2019-08-01-preview
200+
```
201+
202+
## Replace a storage account
203+
To replace a storage account used for ingestion, first create a link for a new storage account. The logging agents will get the updated configuration and start sending data to the new storage as well.
204+
205+
Next unlink the old storage account so agents will stop writing to the removed account. The ingestion process keeps reading data from this account until it’s all ingested. Don’t delete the storage account until you see all logs were ingested.
206+
207+
Agent configuration will be refreshed after a few minutes, and they will switch to the new storage.
208+
209+
210+
## Manage storage account
211+
212+
### Load
213+
Storage accounts can handle a certain load of read and write requests before they start throttling requests. Throttling affects the time it takes to ingest logs and may results in lost data. If your storage is overloaded, register additional storage accounts and spread the load between them.
214+
215+
### Related charges
216+
Storage accounts are charged by the volume of stored data, types of storage, and type of redundancy. For details see [Block blob pricing](https://azure.microsoft.com/pricing/details/storage/blobs/) and [Table Storage pricing](https://azure.microsoft.com/pricing/details/storage/tables/).
217+
218+
If the registered storage account of your workspace is on another region, you will be charged for egress according to these [Bandwidth Pricing Details](https://azure.microsoft.com/pricing/details/bandwidth/).
219+
220+
221+
222+
## Next steps
223+
- <Rob, need a link here for private link article>

0 commit comments

Comments
 (0)