Skip to content

Commit c230291

Browse files
AnnaMHuffAnnaMHuff
authored
Merge pull request #272386 from halkazwini/nw-read
Add VNet flow logs
2 parents da66f83 + 8619b9c commit c230291

File tree

4 files changed

+351
-184
lines changed

4 files changed

+351
-184
lines changed

‎articles/network-watcher/.openpublishing.redirection.network-watcher.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
{
22
"redirections": [
3+
{
4+
"source_path_from_root": "/articles/network-watcher/network-watcher-read-nsg-flow-logs.md",
5+
"redirect_url": "/azure/network-watcher/flow-logs-read",
6+
"redirect_document_id": true
7+
},
38
{
49
"source_path_from_root": "/articles/network-watcher/network-watcher-connectivity-cli.md",
510
"redirect_url": "/azure/network-watcher/connection-troubleshoot-cli",

‎articles/network-watcher/flow-logs-read.md

Lines changed: 345 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,345 @@
1+
---
2+
title: Read flow logs
3+
titleSuffix: Azure Network Watcher
4+
description: Learn how to use a PowerShell script to parse flow logs that are created hourly and updated every few minutes in Azure Network Watcher.
5+
author: halkazwini
6+
ms.author: halkazwini
7+
ms.service: network-watcher
8+
ms.topic: how-to
9+
ms.date: 04/17/2024
10+
ms.custom: devx-track-azurepowershell
11+
12+
#CustomerIntent: As an Azure administrator, I want to read my flow logs using a PowerShell script so I can see the latest data.
13+
---
14+
15+
# Read flow logs
16+
17+
In this article, you learn how to read portions of Azure Network Watcher flow logs using PowerShell without having to parse the entire log. Flow logs are stored in a storage account in block blobs. Each log is a separate block blob that is generated every hour and updated with the latest data every few minutes. You'll use PowerShell to selectively read the latest events in flow logs that are stored in a storage account. The concepts discussed in this article aren't limited to the PowerShell and are applicable to all languages supported by the Azure Storage APIs.
18+
19+
## Prerequisites
20+
21+
- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
22+
23+
- PowerShell. For more information, see [Install PowerShell on Windows, Linux, and macOS](/powershell/scripting/install/installing-powershell). This article requires the Az PowerShell module. For more information, see [How to install Azure PowerShell](/powershell/azure/install-azure-powershell). To find the installed version, run `Get-Module -ListAvailable Az`.
24+
25+
- NSG flow logs in a region or more. For more information, see [Create NSG flow logs](nsg-flow-logs-portal.md#create-a-flow-log).
26+
27+
- Necessary RBAC permissions for the subscriptions of flow logs and storage account. For more information, see [Network Watcher RBAC permissions](required-rbac-permissions.md).
28+
29+
## Retrieve the blocklist
30+
31+
# [**NSG flow logs**](#tab/nsg)
32+
33+
The following PowerShell script sets up the variables needed to query the NSG flow log blob and list the blocks within the [CloudBlockBlob](/dotnet/api/microsoft.azure.storage.blob.cloudblockblob) block blob. Update the script to contain valid values for your environment.
34+
35+
```powershell
36+
function Get-NSGFlowLogCloudBlockBlob {
37+
[CmdletBinding()]
38+
param (
39+
[string] [Parameter(Mandatory=$true)] $subscriptionId,
40+
[string] [Parameter(Mandatory=$true)] $NSGResourceGroupName,
41+
[string] [Parameter(Mandatory=$true)] $NSGName,
42+
[string] [Parameter(Mandatory=$true)] $storageAccountName,
43+
[string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
44+
[string] [Parameter(Mandatory=$true)] $macAddress,
45+
[datetime] [Parameter(Mandatory=$true)] $logTime
46+
)
47+
48+
process {
49+
# Retrieve the primary storage account key to access the NSG logs
50+
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]
51+
52+
# Setup a new storage context to be used to query the logs
53+
$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $StorageAccountKey
54+
55+
# Container name used by NSG flow logs
56+
$ContainerName = "insights-logs-networksecuritygroupflowevent"
57+
58+
# Name of the blob that contains the NSG flow log
59+
$BlobName = "resourceId=/SUBSCRIPTIONS/${subscriptionId}/RESOURCEGROUPS/${NSGResourceGroupName}/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/${NSGName}/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"
60+
61+
# Gets the storage blog
62+
$Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName
63+
64+
# Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
65+
$CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob
66+
67+
#Return the Cloud Block Blob
68+
$CloudBlockBlob
69+
}
70+
}
71+
72+
function Get-NSGFlowLogBlockList {
73+
[CmdletBinding()]
74+
param (
75+
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
76+
)
77+
process {
78+
# Stores the block list in a variable from the block blob.
79+
$blockList = $CloudBlockBlob.DownloadBlockListAsync()
80+
81+
# Return the Block List
82+
$blockList
83+
}
84+
}
85+
86+
87+
$CloudBlockBlob = Get-NSGFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -NSGResourceGroupName "FLOWLOGSVALIDATIONWESTCENTRALUS" -NSGName "V2VALIDATIONVM-NSG" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "ml-rg" -macAddress "000D3AF87856" -logTime "11/11/2018 03:00"
88+
89+
$blockList = Get-NSGFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
90+
```
91+
92+
# [**VNet flow logs (preview)**](#tab/vnet)
93+
94+
The following PowerShell script sets up the variables needed to query the VNet flow log blob and list the blocks within the [CloudBlockBlob](/dotnet/api/microsoft.azure.storage.blob.cloudblockblob) block blob. Update the script to contain valid values for your environment.
95+
96+
```powershell
97+
function Get-VNetFlowLogCloudBlockBlob {
98+
[CmdletBinding()]
99+
param (
100+
[string] [Parameter(Mandatory=$true)] $subscriptionId,
101+
[string] [Parameter(Mandatory=$true)] $region,
102+
[string] [Parameter(Mandatory=$true)] $VNetFlowLogName,
103+
[string] [Parameter(Mandatory=$true)] $storageAccountName,
104+
[string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
105+
[string] [Parameter(Mandatory=$true)] $macAddress,
106+
[datetime] [Parameter(Mandatory=$true)] $logTime
107+
)
108+
109+
process {
110+
# Retrieve the primary storage account key to access the VNet flow logs
111+
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]
112+
113+
# Setup a new storage context to be used to query the logs
114+
$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $StorageAccountKey
115+
116+
# Container name used by VNet flow logs
117+
$ContainerName = "insights-logs-flowlogflowevent"
118+
119+
# Name of the blob that contains the VNet flow log
120+
$BlobName = "flowLogResourceID=/$($subscriptionId.ToUpper())_NETWORKWATCHERRG/NETWORKWATCHER_$($region.ToUpper())_$($VNetFlowLogName.ToUpper())/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"
121+
122+
# Gets the storage blog
123+
$Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName
124+
125+
# Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
126+
$CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob
127+
128+
#Return the Cloud Block Blob
129+
$CloudBlockBlob
130+
}
131+
}
132+
133+
function Get-VNetFlowLogBlockList {
134+
[CmdletBinding()]
135+
param (
136+
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
137+
)
138+
process {
139+
# Stores the block list in a variable from the block blob.
140+
$blockList = $CloudBlockBlob.DownloadBlockListAsync()
141+
142+
# Return the Block List
143+
$blockList
144+
}
145+
}
146+
147+
$CloudBlockBlob = Get-VNetFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -region "yourVNetFlowLogRegion" -VNetFlowLogName "yourVNetFlowLogName" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "yourStorageAccountRG" -macAddress "0022485D8CF8" -logTime "07/09/2023 03:00"
148+
149+
$blockList = Get-VNetFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
150+
```
151+
152+
---
153+
154+
The `$blockList` variable returns a list of the blocks in the blob. Each block blob contains at least two blocks. The first block has a length of 12 bytes and contains the opening brackets of the JSON log. The other block is the closing brackets and has a length of 2 bytes. The following example log has seven individual entries in it. All new entries in the log are added to the end right before the final block.
155+
156+
```
157+
Name Length Committed
158+
---- ------ ---------
159+
ZDk5MTk5N2FkNGE0MmY5MTk5ZWViYjA0YmZhODRhYzY= 12 True
160+
NzQxNDA5MTRhNDUzMGI2M2Y1MDMyOWZlN2QwNDZiYzQ= 2685 True
161+
ODdjM2UyMWY3NzFhZTU3MmVlMmU5MDNlOWEwNWE3YWY= 2586 True
162+
ZDU2MjA3OGQ2ZDU3MjczMWQ4MTRmYWNhYjAzOGJkMTg= 2688 True
163+
ZmM3ZWJjMGQ0ZDA1ODJlOWMyODhlOWE3MDI1MGJhMTc= 2775 True
164+
ZGVkYTc4MzQzNjEyMzlmZWE5MmRiNjc1OWE5OTc0OTQ= 2676 True
165+
ZmY2MjUzYTIwYWIyOGU1OTA2ZDY1OWYzNmY2NmU4ZTY= 2777 True
166+
Mzk1YzQwM2U0ZWY1ZDRhOWFlMTNhYjQ3OGVhYmUzNjk= 2675 True
167+
ZjAyZTliYWE3OTI1YWZmYjFmMWI0MjJhNzMxZTI4MDM= 2 True
168+
```
169+
170+
171+
## Read the block blob
172+
173+
Next, you read the `$blocklist` variable to retrieve the data. In this example we iterate through the blocklist, read the bytes from each block and story them in an array. Use the [DownloadRangeToByteArray](/dotnet/api/microsoft.azure.storage.blob.cloudblob.downloadrangetobytearray) method to retrieve the data.
174+
175+
# [**NSG flow logs**](#tab/nsg)
176+
177+
```powershell
178+
function Get-NSGFlowLogReadBlock {
179+
[CmdletBinding()]
180+
param (
181+
[System.Array] [Parameter(Mandatory=$true)] $blockList,
182+
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
183+
184+
)
185+
# Set the size of the byte array to the largest block
186+
$maxvalue = ($blocklist | measure Length -Maximum).Maximum
187+
188+
# Create an array to store values in
189+
$valuearray = @()
190+
191+
# Define the starting index to track the current block being read
192+
$index = 0
193+
194+
# Loop through each block in the block list
195+
for($i=0; $i -lt $blocklist.count; $i++)
196+
{
197+
# Create a byte array object to story the bytes from the block
198+
$downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue
199+
200+
# Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
201+
$CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockList[$i].Length)) | Out-Null
202+
203+
# Increment the index by adding the current block length to the previous index
204+
$index = $index + $blockList[$i].Length
205+
206+
# Retrieve the string from the byte array
207+
208+
$value = [System.Text.Encoding]::ASCII.GetString($downloadArray)
209+
210+
# Add the log entry to the value array
211+
$valuearray += $value
212+
}
213+
#Return the Array
214+
$valuearray
215+
}
216+
$valuearray = Get-NSGFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
217+
```
218+
219+
# [**VNet flow logs (preview)**](#tab/vnet)
220+
221+
```powershell
222+
function Get-VNetFlowLogReadBlock {
223+
[CmdletBinding()]
224+
param (
225+
[System.Array] [Parameter(Mandatory=$true)] $blockList,
226+
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
227+
228+
)
229+
$blocklistResult = $blockList.Result
230+
231+
# Set the size of the byte array to the largest block
232+
$maxvalue = ($blocklistResult | Measure-Object Length -Maximum).Maximum
233+
Write-Host "Max value is ${maxvalue}"
234+
235+
# Create an array to store values in
236+
$valuearray = @()
237+
238+
# Define the starting index to track the current block being read
239+
$index = 0
240+
241+
# Loop through each block in the block list
242+
for($i=0; $i -lt $blocklistResult.count; $i++)
243+
{
244+
# Create a byte array object to story the bytes from the block
245+
$downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue
246+
247+
# Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
248+
$CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockListResult[$i].Length)) | Out-Null
249+
250+
# Increment the index by adding the current block length to the previous index
251+
$index = $index + $blockListResult[$i].Length
252+
253+
# Retrieve the string from the byte array
254+
255+
$value = [System.Text.Encoding]::ASCII.GetString($downloadArray)
256+
257+
# Add the log entry to the value array
258+
$valuearray += $value
259+
}
260+
#Return the Array
261+
$valuearray
262+
}
263+
264+
$valuearray = Get-VNetFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
265+
```
266+
267+
---
268+
269+
The `$valuearray` array contains now the string value of each block. To verify the entry, get the second to the last value from the array by running `$valuearray[$valuearray.Length-2]`. You don't need the last value because it's the closing bracket.
270+
271+
The results of this value are shown in the following example:
272+
273+
# [**NSG flow logs**](#tab/nsg)
274+
275+
```json
276+
{
277+
"time": "2017-06-16T20:59:43.7340000Z",
278+
"systemId": "5f4d02d3-a7d0-4ed4-9ce8-c0ae9377951c",
279+
"category": "NetworkSecurityGroupFlowEvent",
280+
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/CONTOSORG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/CONTOSONSG",
281+
"operationName": "NetworkSecurityGroupFlowEvents",
282+
"properties": {"Version":1,"flows":[{"rule":"DefaultRule_AllowInternetOutBound","flows":[{"mac":"000D3A18077E","flowTuples":["1497646722,10.0.0.4,168.62.32.14,44904,443,T,O,A","1497646722,10.0.0.4,52.240.48.24,45218,443,T,O,A","1497646725,10.
283+
0.0.4,168.62.32.14,44910,443,T,O,A","1497646725,10.0.0.4,52.240.48.24,45224,443,T,O,A","1497646728,10.0.0.4,168.62.32.14,44916,443,T,O,A","1497646728,10.0.0.4,52.240.48.24,45230,443,T,O,A","1497646732,10.0.0.4,168.62.32.14,44922,443,T,O,A","14976
284+
46732,10.0.0.4,52.240.48.24,45236,443,T,O,A","1497646735,10.0.0.4,168.62.32.14,44928,443,T,O,A","1497646735,10.0.0.4,52.240.48.24,45242,443,T,O,A","1497646738,10.0.0.4,168.62.32.14,44934,443,T,O,A","1497646738,10.0.0.4,52.240.48.24,45248,443,T,O,
285+
A","1497646742,10.0.0.4,168.62.32.14,44942,443,T,O,A","1497646742,10.0.0.4,52.240.48.24,45256,443,T,O,A","1497646745,10.0.0.4,168.62.32.14,44948,443,T,O,A","1497646745,10.0.0.4,52.240.48.24,45262,443,T,O,A","1497646749,10.0.0.4,168.62.32.14,44954
286+
,443,T,O,A","1497646749,10.0.0.4,52.240.48.24,45268,443,T,O,A","1497646753,10.0.0.4,168.62.32.14,44960,443,T,O,A","1497646753,10.0.0.4,52.240.48.24,45274,443,T,O,A","1497646756,10.0.0.4,168.62.32.14,44966,443,T,O,A","1497646756,10.0.0.4,52.240.48
287+
.24,45280,443,T,O,A","1497646759,10.0.0.4,168.62.32.14,44972,443,T,O,A","1497646759,10.0.0.4,52.240.48.24,45286,443,T,O,A","1497646763,10.0.0.4,168.62.32.14,44978,443,T,O,A","1497646763,10.0.0.4,52.240.48.24,45292,443,T,O,A","1497646766,10.0.0.4,
288+
168.62.32.14,44984,443,T,O,A","1497646766,10.0.0.4,52.240.48.24,45298,443,T,O,A","1497646769,10.0.0.4,168.62.32.14,44990,443,T,O,A","1497646769,10.0.0.4,52.240.48.24,45304,443,T,O,A","1497646773,10.0.0.4,168.62.32.14,44996,443,T,O,A","1497646773,
289+
10.0.0.4,52.240.48.24,45310,443,T,O,A","1497646776,10.0.0.4,168.62.32.14,45002,443,T,O,A","1497646776,10.0.0.4,52.240.48.24,45316,443,T,O,A","1497646779,10.0.0.4,168.62.32.14,45008,443,T,O,A","1497646779,10.0.0.4,52.240.48.24,45322,443,T,O,A"]}]}
290+
,{"rule":"DefaultRule_DenyAllInBound","flows":[]},{"rule":"UserRule_ssh-rule","flows":[]},{"rule":"UserRule_web-rule","flows":[{"mac":"000D3A18077E","flowTuples":["1497646738,13.82.225.93,10.0.0.4,1180,80,T,I,A","1497646750,13.82.225.93,10.0.0.4,
291+
1184,80,T,I,A","1497646768,13.82.225.93,10.0.0.4,1181,80,T,I,A","1497646780,13.82.225.93,10.0.0.4,1336,80,T,I,A"]}]}]}
292+
}
293+
```
294+
295+
# [**VNet flow logs (preview)**](#tab/vnet)
296+
297+
```json
298+
{
299+
"time": "2023-07-09T03:59:30.2837112Z",
300+
"flowLogVersion": 4,
301+
"flowLogGUID": "c4de7bdb-291a-4315-84c2-ba1ecd0296dd",
302+
"macAddress": "0022485D8CF8",
303+
"category": "FlowLogFlowEvent",
304+
"flowLogResourceID": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/NETWORKWATCHERRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKWATCHERS/NETWORKWATCHER_WESTCENTRALUS/FLOWLOGS/CONTOSOVNETWCUSFLOWLOG",
305+
"targetResourceID": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/Contoso-westcentralus-RG/providers/Microsoft.Network/virtualNetworks/ContosoVnetWcus",
306+
"operationName": "FlowLogFlowEvent",
307+
"flowRecords": {
308+
"flows": [
309+
{
310+
"aclID": "db903ae8-908e-491b-b12b-afaafab9d9ed",
311+
"flowGroups": [
312+
{
313+
"rule": "BlockHighRiskTCPPortsFromInternet_456b4993-6e57-4e46-aa4d-81767afff09c",
314+
"flowTuples": [
315+
"1688875131557,45.119.212.87,192.168.0.4,53018,3389,6,I,D,NX,0,0,0,0"
316+
]
317+
},
318+
{
319+
"rule": "Internet_4b9ac3d8-dc7b-4b9e-8702-9e9c25b52451",
320+
"flowTuples": [
321+
"1688875103311,35.203.210.145,192.168.0.4,56688,52113,6,I,D,NX,0,0,0,0",
322+
"1688875119073,162.216.150.87,192.168.0.4,50111,9920,6,I,D,NX,0,0,0,0",
323+
"1688875119910,205.210.31.253,192.168.0.4,54699,1801,6,I,D,NX,0,0,0,0",
324+
"1688875121510,35.203.210.49,192.168.0.4,49250,33013,6,I,D,NX,0,0,0,0",
325+
"1688875121684,162.216.149.206,192.168.0.4,49776,1290,6,I,D,NX,0,0,0,0",
326+
"1688875124012,91.148.190.134,192.168.0.4,57963,40544,6,I,D,NX,0,0,0,0",
327+
"1688875138568,35.203.211.204,192.168.0.4,51309,46956,6,I,D,NX,0,0,0,0",
328+
"1688875142490,205.210.31.18,192.168.0.4,54140,30303,6,I,D,NX,0,0,0,0",
329+
"1688875147864,194.26.135.247,192.168.0.4,53583,20232,6,I,D,NX,0,0,0,0"
330+
]
331+
}
332+
]
333+
}
334+
]
335+
}
336+
}
337+
```
338+
339+
---
340+
341+
## Related content
342+
343+
- [Traffic analytics overview](./traffic-analytics.md)
344+
- [Log Analytics tutorial](../azure-monitor/logs/log-analytics-tutorial.md?toc=/azure/network-watcher/toc.json&bc=/azure/network-watcher/breadcrumb/toc.json)
345+
- [Azure Blob storage bindings for Azure Functions overview](../azure-functions/functions-bindings-storage-blob.md?toc=/azure/network-watcher/toc.json&bc=/azure/network-watcher/breadcrumb/toc.json)

0 commit comments

Comments
 (0)