You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
### Calculate the average size of perf usage reports per computer
374
+
The `Usage` data type can be used to track the ingested data volume by solution or data type. There are other techniques to study ingested data volumes by [computer](https://docs.microsoft.com/azure/azure-monitor/platform/manage-cost-storage#data-volume-by-computer) or [Azure subscription, resource group or resource](https://docs.microsoft.com/azure/azure-monitor/platform/manage-cost-storage#data-volume-by-azure-resource-resource-group-or-subscription).
375
375
376
-
This example calculates the average size of perf usage reports per computer, over the last 3 hours.
377
-
The results are shown in a bar chart.
378
-
```Kusto
376
+
#### Data volume by solution
377
+
378
+
The query used to view the billable data volume by solution over the last month (excluding the last partial day) is:
379
+
380
+
```kusto
379
381
Usage
380
-
| where TimeGenerated > ago(3h)
381
-
| where DataType == "Perf"
382
-
| where QuantityUnit == "MBytes"
383
-
| summarize avg(Quantity) by Computer
384
-
| sort by avg_Quantity desc nulls last
385
-
| render barchart
382
+
| where TimeGenerated > ago(32d)
383
+
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
Note that the clause `where IsBillable = true` filters out data types from certain solutions for which there is no ingestion charge. Also the clause with `TimeGenerated` is only to ensure that the query experience in the Azure portal will look back beyond the default 24 hours. When using the Usage data type, `StartTime`and `EndTime` represent the time buckets for which results are presented.
389
389
390
-
This example calculates and charts the 50th and 95th percentiles of reported **avgLatency**by hour over the last 24 hours.
390
+
#### Data volume by type
391
391
392
-
```Kusto
393
-
Usage
394
-
| where TimeGenerated > ago(24h)
395
-
| summarize percentiles(AvgLatencyInSeconds, 50, 95) by bin(TimeGenerated, 1h)
396
-
| render timechart
392
+
You can drill in further to see data trends for by data type:
393
+
394
+
```kusto
395
+
Usage
396
+
| where TimeGenerated > ago(32d)
397
+
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
This example retrieves **Usage** data from the last day for computer names that contains the string _ContosoFile_. The results are sorted by **TimeGenerated**.
402
+
Or to see a table by solution and type for the last month,
401
403
402
-
```Kusto
403
-
Usage
404
-
| where TimeGenerated > ago(1d)
405
-
| where Computer contains "ContosoFile"
406
-
| sort by TimeGenerated desc nulls last
404
+
```kusto
405
+
Usage
406
+
| where TimeGenerated > ago(32d)
407
+
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
408
+
| where IsBillable == true
409
+
| summarize BillableDataGB = sum(Quantity) / 1000. by Solution, DataType
410
+
| sort by Solution asc, DataType asc
407
411
```
408
412
413
+
> [!NOTE]
414
+
> Some of the fields of the Usage data type, while still in the schema, have been deprecated and will their values are no longer populated.
415
+
> These are **Computer** as well as fields related to ingestion (**TotalBatches**, **BatchesWithinSla**, **BatchesOutsideSla**, **BatchesCapped** and **AverageProcessingTimeMs**.
416
+
409
417
## Updates
410
418
411
419
### Computers Still Missing Updates
@@ -427,4 +435,4 @@ Update
427
435
## Next steps
428
436
429
437
- Refer to the [Kusto language reference](/azure/kusto/query) for details on the language.
430
-
- Walk through a [lesson on writing log queries in Azure Monitor](get-started-queries.md).
438
+
- Walk through a [lesson on writing log queries in Azure Monitor](get-started-queries.md).
Copy file name to clipboardExpand all lines: articles/azure-monitor/platform/manage-cost-storage.md
+39-16Lines changed: 39 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ ms.service: azure-monitor
11
11
ms.workload: na
12
12
ms.tgt_pltfrm: na
13
13
ms.topic: conceptual
14
-
ms.date: 11/05/2019
14
+
ms.date: 03/16/2020
15
15
ms.author: bwren
16
16
ms.subservice:
17
17
---
@@ -107,7 +107,7 @@ To set the default retention for your workspace,
107
107
3. On the pane, move the slider to increase or decrease the number of days and then click **OK**. If you are on the *free* tier, you will not be able to modify the data retention period and you need to upgrade to the paid tier in order to control this setting.
108
108
109
109
110
-
110
+
111
111
The retention can also be [set via Azure Resource Manager](https://docs.microsoft.com/azure/azure-monitor/platform/template-workspace-configuration#configure-a-log-analytics-workspace) using the `retentionInDays` parameter. Additionally, if you set the data retention to 30 days, you can trigger an immediate purge of older data using the `immediatePurgeDataOn30Days` parameter, which may be useful for compliance-related scenarios. This functionality is only exposed via Azure Resource Manager.
112
112
113
113
Two data types -- `Usage` and `AzureActivity` -- are retained for 90 days by default, and there is no charge for for this 90 day retention. These data types are also free from data ingestion charges.
@@ -137,9 +137,9 @@ To set the retention of a particular data type (in this example SecurityEvent) t
137
137
```JSON
138
138
PUT /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/MyResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/MyWorkspaceName/Tables/SecurityEvent?api-version=2017-04-26-preview
139
139
{
140
-
"properties":
141
-
{
142
-
"retentionInDays": 730
140
+
"properties":
141
+
{
142
+
"retentionInDays": 730
143
143
}
144
144
}
145
145
```
@@ -169,15 +169,15 @@ When the daily limit is reached, the collection of billable data types stops for
169
169
170
170
### Identify what daily data limit to define
171
171
172
-
Review [Log Analytics Usage and estimated costs](usage-estimated-costs.md) to understand the data ingestion trend and what is the daily volume cap to define. It should be considered with care, since you won’t be able to monitor your resources after the limit is reached.
172
+
Review [Log Analytics Usage and estimated costs](usage-estimated-costs.md) to understand the data ingestion trend and what is the daily volume cap to define. It should be considered with care, since you won�t be able to monitor your resources after the limit is reached.
173
173
174
174
### Set the Daily Cap
175
175
176
176
The following steps describe how to configure a limit to manage the volume of data that Log Analytics workspace will ingest per day.
177
177
178
178
1. From your workspace, select **Usage and estimated costs** from the left pane.
179
179
2. On the **Usage and estimated costs** page for the selected workspace, click **Data volume management** from the top of the page.
180
-
3. Daily cap is **OFF** by default – click **ON** to enable it, and then set the data volume limit in GB/day.
180
+
3. Daily cap is **OFF** by default � click **ON** to enable it, and then set the data volume limit in GB/day.
181
181
182
182
183
183
@@ -217,10 +217,11 @@ Heartbeat
217
217
| summarize nodes = dcount(Computer) by bin(TimeGenerated, 1d)
218
218
| render timechart
219
219
```
220
-
The get a count of nodes sending data seen can be determined using:
220
+
The get a count of nodes sending data in the last 24 hours use the query:
| summarize TotalVolumeBytes=sum(_BilledSize) by computerName
@@ -242,35 +244,52 @@ union withsource = tt *
242
244
243
245
On the **Usage and Estimated Costs** page, the *Data ingestion per solution* chart shows the total volume of data sent and how much is being sent by each solution. This allows you to determine trends such as whether the overall data usage (or usage by a particular solution) is growing, remaining steady or decreasing.
244
246
247
+
### Data volume for specific events
248
+
249
+
To look at the size of ingested data for a particular set of events, you can query the specific table (in this example `Event`) and then restrict the query to the events of interest (in this example event ID 5145 or 5156):
250
+
251
+
```kusto
252
+
Event
253
+
| where TimeGenerated > startofday(ago(31d)) and TimeGenerated < startofday(now())
254
+
| where EventID == 5145 or EventID == 5156
255
+
| where _IsBillable == true
256
+
| summarize count(), Bytes=sum(_BilledSize) by EventID, bin(TimeGenerated, 1d)
257
+
```
258
+
259
+
Note that the clause `where IsBillable = true` filters out data types from certain solutions for which there is no ingestion charge.
260
+
245
261
### Data volume by solution
246
262
247
-
The query used to view the billable data volume by solution is
263
+
The query used to view the billable data volume by solution over the last month (excluding the last partial day) is:
248
264
249
265
```kusto
250
266
Usage
251
-
| where TimeGenerated > startofday(ago(31d))
267
+
| where TimeGenerated > ago(32d)
268
+
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
Note that the clause `where IsBillable = true` filters out data types from certain solutions for which there is no ingestion charge.
273
+
The clause with `TimeGenerated` is only to ensure that the query experience in the Azure portal will look back beyond the default 24 hours. When using the Usage data type, `StartTime` and `EndTime` represent the time buckets for which results are presented.
257
274
258
275
### Data volume by type
259
276
260
277
You can drill in further to see data trends for by data type:
261
278
262
279
```kusto
263
-
Usage | where TimeGenerated > startofday(ago(31d))| where IsBillable == true
264
-
| where TimeGenerated > startofday(ago(31d))
280
+
Usage
281
+
| where TimeGenerated > ago(32d)
282
+
| where StartTime >= startofday(ago(31d)) and EndTime < startofday(now())
0 commit comments