Skip to content

Commit 9f1e205

Browse files
halkazwinihalkazwini
committed
add vnet flow logs
1 parent e534792 commit 9f1e205

File tree

1 file changed

+181
-21
lines changed

1 file changed

+181
-21
lines changed

‎articles/network-watcher/network-watcher-read-nsg-flow-logs.md

Lines changed: 181 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,32 +1,33 @@
11
---
2-
title: Read NSG flow logs
3-
description: Learn how to use Azure PowerShell to parse network security group flow logs, which are created hourly and updated every few minutes in Azure Network Watcher.
4-
services: network-watcher
2+
title: Read flow logs
3+
description: Learn how to use Azure PowerShell to parse flow logs that are created hourly and updated every few minutes in Azure Network Watcher.
54
author: halkazwini
5+
ms.author: halkazwini
66
ms.service: network-watcher
77
ms.topic: how-to
8-
ms.date: 02/09/2021
9-
ms.author: halkazwini
10-
ms.custom: devx-track-azurepowershell, engagement-fy23
8+
ms.date: 04/17/2024
9+
ms.custom: devx-track-azurepowershell
1110
---
1211

13-
# Read NSG flow logs
12+
# Read flow logs
1413

15-
Learn how to read NSG flow logs entries with PowerShell.
14+
In this article, you learn how to read portions of Azure Network Watcher flow logs using PowerShell without having to parse the entire log. Flow logs are stored in a storage account in block blobs. Each log is a separate block blob that is generated every hour and updated with the latest data every few minutes. You'll use PowerShell to selectively read the latest events in flow logs that are stored in a storage account. The concepts discussed in this article aren't limited to the PowerShell and are applicable to all languages supported by the Azure Storage APIs.
1615

17-
NSG flow logs are stored in a storage account in [block blobs](/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs). Block blobs are made up of smaller blocks. Each log is a separate block blob that is generated every hour. New logs are generated every hour, the logs are updated with new entries every few minutes with the latest data. In this article you learn how to read portions of the flow logs.
16+
## Prerequisites
1817

19-
## Scenario
18+
- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F).
2019

21-
In the following scenario, you have an example flow log that is stored in a storage account. You learn how to selectively read the latest events in NSG flow logs. In this article you use PowerShell, however, the concepts discussed in the article aren't limited to the programming language, and are applicable to all languages supported by the Azure Storage APIs.
20+
- PowerShell. For more information, see [Install PowerShell on Windows, Linux, and macOS](/powershell/scripting/install/installing-powershell). This article requires the Az PowerShell module. For more information, see [How to install Azure PowerShell](/powershell/azure/install-azure-powershell). To find the installed version, run `Get-Module -ListAvailable Az`.
2221

23-
## Setup
22+
- NSG flow logs in a region or more. For more information, see [Create NSG flow logs](nsg-flow-logs-portal.md#create-a-flow-log).
2423

25-
Before you begin, you must have Network Security Group Flow Logging enabled on one or many Network Security Groups in your account. For instructions on enabling Network Security flow logs, refer to the following article: [Introduction to flow logging for Network Security Groups](nsg-flow-logs-overview.md).
24+
- Necessary RBAC permissions for the subscriptions of flow logs and storage account. For more information, see [Network Watcher RBAC permissions](required-rbac-permissions.md).
2625

2726
## Retrieve the block list
2827

29-
The following PowerShell sets up the variables needed to query the NSG flow log blob and list the blocks within the [CloudBlockBlob](/dotnet/api/microsoft.azure.storage.blob.cloudblockblob) block blob. Update the script to contain valid values for your environment.
28+
# [**NSG flow logs**](#tab/nsg)
29+
30+
The following PowerShell script sets up the variables needed to query the NSG flow log blob and list the blocks within the [CloudBlockBlob](/dotnet/api/microsoft.azure.storage.blob.cloudblockblob) block blob. Update the script to contain valid values for your environment.
3031

3132
```powershell
3233
function Get-NSGFlowLogCloudBlockBlob {
@@ -85,7 +86,69 @@ $CloudBlockBlob = Get-NSGFlowLogCloudBlockBlob -subscriptionId "yourSubscription
8586
$blockList = Get-NSGFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
8687
```
8788

88-
The `$blockList` variable returns a list of the blocks in the blob. Each block blob contains at least two blocks. The first block has a length of `12` bytes, this block contains the opening brackets of the json log. The other block is the closing brackets and has a length of `2` bytes. As you can see the following example log has seven entries in it, each being an individual entry. All new entries in the log are added to the end right before the final block.
89+
# [**VNet flow logs (preview)**](#tab/vnet)
90+
91+
The following PowerShell script sets up the variables needed to query the VNet flow log blob and list the blocks within the [CloudBlockBlob](/dotnet/api/microsoft.azure.storage.blob.cloudblockblob) block blob. Update the script to contain valid values for your environment.
92+
93+
```powershell
94+
function Get-VNetFlowLogCloudBlockBlob {
95+
[CmdletBinding()]
96+
param (
97+
[string] [Parameter(Mandatory=$true)] $subscriptionId,
98+
[string] [Parameter(Mandatory=$true)] $region,
99+
[string] [Parameter(Mandatory=$true)] $VNetFlowLogName,
100+
[string] [Parameter(Mandatory=$true)] $storageAccountName,
101+
[string] [Parameter(Mandatory=$true)] $storageAccountResourceGroup,
102+
[string] [Parameter(Mandatory=$true)] $macAddress,
103+
[datetime] [Parameter(Mandatory=$true)] $logTime
104+
)
105+
106+
process {
107+
# Retrieve the primary storage account key to access the VNet flow logs
108+
$StorageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroup -Name $storageAccountName).Value[0]
109+
110+
# Setup a new storage context to be used to query the logs
111+
$ctx = New-AzStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $StorageAccountKey
112+
113+
# Container name used by VNet flow logs
114+
$ContainerName = "insights-logs-flowlogflowevent"
115+
116+
# Name of the blob that contains the VNet flow log
117+
$BlobName = "flowLogResourceID=/$($subscriptionId.ToUpper())_NETWORKWATCHERRG/NETWORKWATCHER_$($region.ToUpper())_$($VNetFlowLogName.ToUpper())/y=$($logTime.Year)/m=$(($logTime).ToString("MM"))/d=$(($logTime).ToString("dd"))/h=$(($logTime).ToString("HH"))/m=00/macAddress=$($macAddress)/PT1H.json"
118+
119+
# Gets the storage blog
120+
$Blob = Get-AzStorageBlob -Context $ctx -Container $ContainerName -Blob $BlobName
121+
122+
# Gets the block blog of type 'Microsoft.Azure.Storage.Blob.CloudBlob' from the storage blob
123+
$CloudBlockBlob = [Microsoft.Azure.Storage.Blob.CloudBlockBlob] $Blob.ICloudBlob
124+
125+
#Return the Cloud Block Blob
126+
$CloudBlockBlob
127+
}
128+
}
129+
130+
function Get-VNetFlowLogBlockList {
131+
[CmdletBinding()]
132+
param (
133+
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
134+
)
135+
process {
136+
# Stores the block list in a variable from the block blob.
137+
$blockList = $CloudBlockBlob.DownloadBlockListAsync()
138+
139+
# Return the Block List
140+
$blockList
141+
}
142+
}
143+
144+
$CloudBlockBlob = Get-VNetFlowLogCloudBlockBlob -subscriptionId "yourSubscriptionId" -region "yourVNetFlowLogRegion" -VNetFlowLogName "yourVNetFlowLogName" -storageAccountName "yourStorageAccountName" -storageAccountResourceGroup "yourStorageAccountRG" -macAddress "0022485D8CF8" -logTime "07/09/2023 03:00"
145+
146+
$blockList = Get-VNetFlowLogBlockList -CloudBlockBlob $CloudBlockBlob
147+
```
148+
149+
---
150+
151+
The `$blockList` variable returns a list of the blocks in the blob. Each block blob contains at least two blocks. The first block has a length of 12 bytes, this block contains the opening brackets of the JSON log. The other block is the closing brackets and has a length of 2 bytes. The following example log has seven individual entries in it. All new entries in the log are added to the end right before the final block.
89152

90153
```
91154
Name Length Committed
@@ -101,9 +164,12 @@ Mzk1YzQwM2U0ZWY1ZDRhOWFlMTNhYjQ3OGVhYmUzNjk= 2675 True
101164
ZjAyZTliYWE3OTI1YWZmYjFmMWI0MjJhNzMxZTI4MDM= 2 True
102165
```
103166

167+
104168
## Read the block blob
105169

106-
Next you need to read the `$blocklist` variable to retrieve the data. In this example we iterate through the blocklist, read the bytes from each block and story them in an array. Use the [DownloadRangeToByteArray](/dotnet/api/microsoft.azure.storage.blob.cloudblob.downloadrangetobytearray) method to retrieve the data.
170+
Next, you read the `$blocklist` variable to retrieve the data. In this example we iterate through the blocklist, read the bytes from each block and story them in an array. Use the [DownloadRangeToByteArray](/dotnet/api/microsoft.azure.storage.blob.cloudblob.downloadrangetobytearray) method to retrieve the data.
171+
172+
# [**NSG flow logs**](#tab/nsg)
107173

108174
```powershell
109175
function Get-NSGFlowLogReadBlock {
@@ -147,10 +213,62 @@ function Get-NSGFlowLogReadBlock {
147213
$valuearray = Get-NSGFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
148214
```
149215

150-
Now the `$valuearray` array contains the string value of each block. To verify the entry, get the second to the last value from the array by running `$valuearray[$valuearray.Length-2]`. You don't want the last value, because it's the closing bracket.
216+
# [**VNet flow logs (preview)**](#tab/vnet)
217+
218+
```powershell
219+
function Get-VNetFlowLogReadBlock {
220+
[CmdletBinding()]
221+
param (
222+
[System.Array] [Parameter(Mandatory=$true)] $blockList,
223+
[Microsoft.Azure.Storage.Blob.CloudBlockBlob] [Parameter(Mandatory=$true)] $CloudBlockBlob
224+
225+
)
226+
$blocklistResult = $blockList.Result
227+
228+
# Set the size of the byte array to the largest block
229+
$maxvalue = ($blocklistResult | Measure-Object Length -Maximum).Maximum
230+
Write-Host "Max value is ${maxvalue}"
231+
232+
# Create an array to store values in
233+
$valuearray = @()
234+
235+
# Define the starting index to track the current block being read
236+
$index = 0
237+
238+
# Loop through each block in the block list
239+
for($i=0; $i -lt $blocklistResult.count; $i++)
240+
{
241+
# Create a byte array object to story the bytes from the block
242+
$downloadArray = New-Object -TypeName byte[] -ArgumentList $maxvalue
243+
244+
# Download the data into the ByteArray, starting with the current index, for the number of bytes in the current block. Index is increased by 3 when reading to remove preceding comma.
245+
$CloudBlockBlob.DownloadRangeToByteArray($downloadArray,0,$index, $($blockListResult[$i].Length)) | Out-Null
246+
247+
# Increment the index by adding the current block length to the previous index
248+
$index = $index + $blockListResult[$i].Length
249+
250+
# Retrieve the string from the byte array
251+
252+
$value = [System.Text.Encoding]::ASCII.GetString($downloadArray)
253+
254+
# Add the log entry to the value array
255+
$valuearray += $value
256+
}
257+
#Return the Array
258+
$valuearray
259+
}
260+
261+
$valuearray = Get-VNetFlowLogReadBlock -blockList $blockList -CloudBlockBlob $CloudBlockBlob
262+
```
263+
264+
---
265+
266+
The `$valuearray` array contains now the string value of each block. To verify the entry, get the second to the last value from the array by running `$valuearray[$valuearray.Length-2]`. You don't need the last value because it's the closing bracket.
151267

152268
The results of this value are shown in the following example:
153269

270+
# [**NSG flow logs**](#tab/nsg)
271+
154272
```json
155273
{
156274
"time": "2017-06-16T20:59:43.7340000Z",
@@ -171,13 +289,55 @@ A","1497646742,10.0.0.4,168.62.32.14,44942,443,T,O,A","1497646742,10.0.0.4,52.24
171289
}
172290
```
173291

174-
This scenario is an example of how to read entries in NSG flow logs without having to parse the entire log. You can read new entries in the log as they're written by using the block ID or by tracking the length of blocks stored in the block blob. This allows you to read only the new entries.
292+
# [**VNet flow logs (preview)**](#tab/vnet)
175293

176-
## Next steps
294+
```json
295+
{
296+
"time": "2023-07-09T03:59:30.2837112Z",
297+
"flowLogVersion": 4,
298+
"flowLogGUID": "c4de7bdb-291a-4315-84c2-ba1ecd0296dd",
299+
"macAddress": "0022485D8CF8",
300+
"category": "FlowLogFlowEvent",
301+
"flowLogResourceID": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/NETWORKWATCHERRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKWATCHERS/NETWORKWATCHER_WESTCENTRALUS/FLOWLOGS/CONTOSOVNETWCUSFLOWLOG",
302+
"targetResourceID": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/Contoso-westcentralus-RG/providers/Microsoft.Network/virtualNetworks/ContosoVnetWcus",
303+
"operationName": "FlowLogFlowEvent",
304+
"flowRecords": {
305+
"flows": [
306+
{
307+
"aclID": "db903ae8-908e-491b-b12b-afaafab9d9ed",
308+
"flowGroups": [
309+
{
310+
"rule": "BlockHighRiskTCPPortsFromInternet_456b4993-6e57-4e46-aa4d-81767afff09c",
311+
"flowTuples": [
312+
"1688875131557,45.119.212.87,192.168.0.4,53018,3389,6,I,D,NX,0,0,0,0"
313+
]
314+
},
315+
{
316+
"rule": "Internet_4b9ac3d8-dc7b-4b9e-8702-9e9c25b52451",
317+
"flowTuples": [
318+
"1688875103311,35.203.210.145,192.168.0.4,56688,52113,6,I,D,NX,0,0,0,0",
319+
"1688875119073,162.216.150.87,192.168.0.4,50111,9920,6,I,D,NX,0,0,0,0",
320+
"1688875119910,205.210.31.253,192.168.0.4,54699,1801,6,I,D,NX,0,0,0,0",
321+
"1688875121510,35.203.210.49,192.168.0.4,49250,33013,6,I,D,NX,0,0,0,0",
322+
"1688875121684,162.216.149.206,192.168.0.4,49776,1290,6,I,D,NX,0,0,0,0",
323+
"1688875124012,91.148.190.134,192.168.0.4,57963,40544,6,I,D,NX,0,0,0,0",
324+
"1688875138568,35.203.211.204,192.168.0.4,51309,46956,6,I,D,NX,0,0,0,0",
325+
"1688875142490,205.210.31.18,192.168.0.4,54140,30303,6,I,D,NX,0,0,0,0",
326+
"1688875147864,194.26.135.247,192.168.0.4,53583,20232,6,I,D,NX,0,0,0,0"
327+
]
328+
}
329+
]
330+
}
331+
]
332+
}
333+
}
334+
```
335+
336+
---
177337

178338

179-
Visit [Use Elastic Stack](network-watcher-visualize-nsg-flow-logs-open-source-tools.md), [Use Grafana](network-watcher-nsg-grafana.md), and [Use Graylog](network-watcher-analyze-nsg-flow-logs-graylog.md) to learn more about ways to view NSG flow logs. An Open Source Azure Function approach to consuming the blobs directly and emitting to various log analytics consumers may be found here: [Azure Network Watcher NSG Flow Logs Connector](https://github.com/Microsoft/AzureNetworkWatcherNSGFlowLogsConnector).
339+
## Next step
180340

181-
You can use [Azure Traffic Analytics](./traffic-analytics.md) to get insights on your traffic flows. Traffic Analytics uses [Log Analytics](../azure-monitor/logs/log-analytics-tutorial.md) to make your traffic flow queryable.
341+
You can use [Traffic analytics](./traffic-analytics.md) to get insights on your traffic flows. Traffic Analytics uses [Log Analytics](../azure-monitor/logs/log-analytics-tutorial.md) to make your traffic flow queryable.
182342

183343
To learn more about storage blobs visit: [Azure Functions Blob storage bindings](../azure-functions/functions-bindings-storage-blob.md)

0 commit comments

Comments
 (0)