Skip to content

Commit e0bc203

Browse files
itechedititechedit
authored
Merge pull request #58070 from mattreatMSFT/master
Changes for NSG Flow Logging Version 2 release
2 parents ae3e9e4 + 3b04fa1 commit e0bc203

10 files changed

+310
-31
lines changed

‎articles/network-watcher/media/network-watcher-nsg-flow-logging-portal/select-flow-log-version.png

40.3 KB
Loading

‎articles/network-watcher/network-watcher-analyze-nsg-flow-logs-graylog.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ ms.author: mareat
2424

2525
You can have many network security groups in your network with flow logging enabled. Several network security groups with flow logging enabled can make it cumbersome to parse and gain insights from your logs. This article provides a solution to centrally manage these network security group flow logs using Graylog, an open source log management and analysis tool, and Logstash, an open source server-side data processing pipeline.
2626

27+
> [!Warning]
28+
> The following steps work with flow logs version 1. For details, see [Introduction to flow logging for network security groups](network-watcher-nsg-flow-logging-overview.md). The following instructions will not work with version 2 of the log files, without modification.
29+
2730
## Scenario
2831

2932
Network security group flow logs are enabled using Network Watcher. Flow logs flow in to Azure blob storage. A Logstash plugin is used to connect and process flow logs from blob storage and send them to Graylog. Once the flow logs are stored in Graylog, they can be analyzed and visualized into customized dashboards.

‎articles/network-watcher/network-watcher-nsg-flow-logging-cli.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,9 @@ Network Security Group flow logs are a feature of Network Watcher that allows yo
3131

3232
To perform the steps in this article, you need to [install the Azure command-line interface for Mac, Linux, and Windows (CLI)](/cli/azure/install-azure-cli).
3333

34+
> [!NOTE]
35+
> Flow Logs Version 2 are only available in the West Central US Region. Configuration is available through the Azure Portal and REST API. Enabling Version 2 logs in an unsupported region will result in Version 1 logs outputted to your storage account.
36+
3437
## Register Insights provider
3538

3639
In order for flow logging to work successfully, the **Microsoft.Insights** provider must be registered. If you are not sure if the **Microsoft.Insights** provider is registered, run the following script.

‎articles/network-watcher/network-watcher-nsg-flow-logging-overview.md

Lines changed: 181 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ ms.author: jdial
2020

2121
# Introduction to flow logging for network security groups
2222

23-
Network security group (NSG) flow logs are a feature of Network Watcher that allows you to view information about ingress and egress IP traffic through an NSG. Flow logs are written in json format, and show outbound and inbound flows on a per rule basis, the network interface (NIC) the flow applies to, 5-tuple information about the flow (Source/destination IP, source/destination port, and protocol), and if the traffic was allowed or denied.
23+
Network security group (NSG) flow logs are a feature of Network Watcher that allows you to view information about ingress and egress IP traffic through an NSG. Flow logs are written in JSON format, and show outbound and inbound flows on a per rule basis, the network interface (NIC) the flow applies to, 5-tuple information about the flow (Source/destination IP, source/destination port, and protocol), if the traffic was allowed or denied, and in Version 2, throughput information (Bytes and Packets).
24+
2425

2526
![flow logs overview](./media/network-watcher-nsg-flow-logging-overview/figure1.png)
2627

@@ -29,9 +30,12 @@ While flow logs target NSGs, they are not displayed the same as the other logs.
2930
```
3031
https://{storageAccountName}.blob.core.windows.net/insights-logs-networksecuritygroupflowevent/resourceId=/SUBSCRIPTIONS/{subscriptionID}/RESOURCEGROUPS/{resourceGroupName}/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/{nsgName}/y={year}/m={month}/d={day}/h={hour}/m=00/macAddress={macAddress}/PT1H.json
3132
```
32-
33+
3334
The same retention policies seen for other logs apply to flow logs. You can set log retention policy from 1 day to 2147483647 days. If a retention policy is not set, the logs are maintained forever.
3435

36+
You can also analyze flow logs using [traffic analytics](traffic-analytics.md).
37+
38+
3539
## Log file
3640

3741
Flow logs include the following properties:
@@ -55,15 +59,118 @@ Flow logs include the following properties:
5559
* **Destination Port** - The destination Port
5660
* **Protocol** - The protocol of the flow. Valid values are **T** for TCP and **U** for UDP
5761
* **Traffic Flow** - The direction of the traffic flow. Valid values are **I** for inbound and **O** for outbound.
58-
* **Traffic** - Whether traffic was allowed or denied. Valid values are **A** for allowed and **D** for denied.
62+
* **Traffic Decision** - Whether traffic was allowed or denied. Valid values are **A** for allowed and **D** for denied.
63+
* **Flow State - Version 2 Only** - Captures the state of the flow. Possible states are **B**: Begin, when a flow is created. Statistics aren't provided. **C**: Continuing for an ongoing flow. Statistics are provided at 5-minute intervals. **E**: End, when a flow is terminated. Statistics are provided.
64+
* **Packets - Source to destination - Version 2 Only** The total number of TCP or UDP packets sent from source to destination since last update.
65+
* **Bytes sent - Source to destination - Version 2 Only** The total number of TCP or UDP packet bytes sent from source to destination since last update. Packet bytes include the packet header and payload.
66+
* **Packets - Destination to source - Version 2 Only** The total number of TCP or UDP packets sent from destination to source since last update.
67+
* **Bytes sent - Destination to source - Version 2 Only** The total number of TCP and UDP packet bytes sent from destination to source since last update. Packet bytes include packet header and payload.
68+
69+
## NSG flow logs version 2
70+
> [!NOTE]
71+
> Flow Logs Version 2 are only available in the West Central US Region. Configuration is available through the Azure Portal and REST API. Enabling Version 2 logs in an unsupported region will result in Version 1 logs outputted to your storage account.
72+
73+
Version 2 of the logs introduces flow state. You can configure which version of flow logs you receive. To learn how to enable flow logs, see [Enabling NSG flow logging](network-watcher-nsg-flow-logging-portal.md).
74+
75+
Flow state *B* is recorded when a flow is initiated. Flow state *C* and flow state *E* are states that mark the continuation of a flow and flow termination, respectively. Both *C* and *E* states contain traffic bandwidth information.
76+
77+
For continuation *C* and end *E* flow states, byte and packet counts are aggregate counts from the time of the previous flow tuple record. Referencing the previous example conversation, the total number of packets transferred is 1021+52+8005+47 = 9125. The total number of bytes transferred is 588096+29952+4610880+27072 = 5256000.
78+
79+
**Example**: Flow tuples from a TCP conversation between 185.170.185.105:35370 and 10.2.0.4:23:
80+
81+
"1493763938,185.170.185.105,10.2.0.4,35370,23,T,I,A,B,,,,"
82+
"1493695838,185.170.185.105,10.2.0.4,35370,23,T,I,A,C,1021,588096,8005,4610880"
83+
"1493696138,185.170.185.105,10.2.0.4,35370,23,T,I,A,E,52,29952,47,27072"
84+
85+
For continuation *C* and end *E* flow states, byte and packet counts are aggregate counts from the time of the previous flow tuple record. Referencing the previous example conversation, the total number of packets transferred is 1021+52+8005+47 = 9125. The total number of bytes transferred is 588096+29952+4610880+27072 = 5256000.
5986

6087
The text that follows is an example of a flow log. As you can see, there are multiple records that follow the property list described in the preceding section.
6188

89+
## Sample log records
90+
91+
The text that follows is an example of a flow log. As you can see, there are multiple records that follow the property list described in the preceding section.
92+
93+
6294
> [!NOTE]
6395
> Values in the **flowTuples* property are a comma-separated list.
6496
97+
### Version 1 NSG flow log format sample
6598
```json
6699
{
100+
"records": [
101+
{
102+
"time": "2017-02-16T22:00:32.8950000Z",
103+
"systemId": "2c002c16-72f3-4dc5-b391-3444c3527434",
104+
"category": "NetworkSecurityGroupFlowEvent",
105+
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG",
106+
"operationName": "NetworkSecurityGroupFlowEvents",
107+
"properties": {
108+
"Version": 1,
109+
"flows": [
110+
{
111+
"rule": "DefaultRule_DenyAllInBound",
112+
"flows": [
113+
{
114+
"mac": "000D3AF8801A",
115+
"flowTuples": [
116+
"1487282421,42.119.146.95,10.1.0.4,51529,5358,T,I,D"
117+
]
118+
}
119+
]
120+
},
121+
{
122+
"rule": "UserRule_default-allow-rdp",
123+
"flows": [
124+
{
125+
"mac": "000D3AF8801A",
126+
"flowTuples": [
127+
"1487282370,163.28.66.17,10.1.0.4,61771,3389,T,I,A",
128+
"1487282393,5.39.218.34,10.1.0.4,58596,3389,T,I,A",
129+
"1487282393,91.224.160.154,10.1.0.4,61540,3389,T,I,A",
130+
"1487282423,13.76.89.229,10.1.0.4,53163,3389,T,I,A"
131+
]
132+
}
133+
]
134+
}
135+
]
136+
}
137+
},
138+
{
139+
"time": "2017-02-16T22:01:32.8960000Z",
140+
"systemId": "2c002c16-72f3-4dc5-b391-3444c3527434",
141+
"category": "NetworkSecurityGroupFlowEvent",
142+
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG",
143+
"operationName": "NetworkSecurityGroupFlowEvents",
144+
"properties": {
145+
"Version": 1,
146+
"flows": [
147+
{
148+
"rule": "DefaultRule_DenyAllInBound",
149+
"flows": [
150+
{
151+
"mac": "000D3AF8801A",
152+
"flowTuples": [
153+
"1487282481,195.78.210.194,10.1.0.4,53,1732,U,I,D"
154+
]
155+
}
156+
]
157+
},
158+
{
159+
"rule": "UserRule_default-allow-rdp",
160+
"flows": [
161+
{
162+
"mac": "000D3AF8801A",
163+
"flowTuples": [
164+
"1487282435,61.129.251.68,10.1.0.4,57776,3389,T,I,A",
165+
"1487282454,84.25.174.170,10.1.0.4,59085,3389,T,I,A",
166+
"1487282477,77.68.9.50,10.1.0.4,65078,3389,T,I,A"
167+
]
168+
}
169+
]
170+
}
171+
]
172+
}
173+
},
67174
"records":
68175
[
69176

@@ -96,6 +203,77 @@ The text that follows is an example of a flow log. As you can see, there are mul
96203
,
97204
...
98205
```
206+
### Version 2 NSG flow log format sample
207+
```json
208+
{
209+
"records": [
210+
{
211+
"time": "2018-11-13T12:00:35.3899262Z",
212+
"systemId": "a0fca5ce-022c-47b1-9735-89943b42f2fa",
213+
"category": "NetworkSecurityGroupFlowEvent",
214+
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG",
215+
"operationName": "NetworkSecurityGroupFlowEvents",
216+
"properties": {
217+
"Version": 2,
218+
"flows": [
219+
{
220+
"rule": "DefaultRule_DenyAllInBound",
221+
"flows": [
222+
{
223+
"mac": "000D3AF87856",
224+
"flowTuples": [
225+
"1542110402,94.102.49.190,10.5.16.4,28746,443,U,I,D,B,,,,",
226+
"1542110424,176.119.4.10,10.5.16.4,56509,59336,T,I,D,B,,,,",
227+
"1542110432,167.99.86.8,10.5.16.4,48495,8088,T,I,D,B,,,,"
228+
]
229+
}
230+
]
231+
},
232+
{
233+
"rule": "DefaultRule_AllowInternetOutBound",
234+
"flows": [
235+
{
236+
"mac": "000D3AF87856",
237+
"flowTuples": [
238+
"1542110377,10.5.16.4,13.67.143.118,59831,443,T,O,A,B,,,,",
239+
"1542110379,10.5.16.4,13.67.143.117,59932,443,T,O,A,E,1,66,1,66",
240+
"1542110379,10.5.16.4,13.67.143.115,44931,443,T,O,A,C,30,16978,24,14008",
241+
"1542110406,10.5.16.4,40.71.12.225,59929,443,T,O,A,E,15,8489,12,7054"
242+
]
243+
}
244+
]
245+
}
246+
]
247+
}
248+
},
249+
{
250+
"time": "2018-11-13T12:01:35.3918317Z",
251+
"systemId": "a0fca5ce-022c-47b1-9735-89943b42f2fa",
252+
"category": "NetworkSecurityGroupFlowEvent",
253+
"resourceId": "/SUBSCRIPTIONS/00000000-0000-0000-0000-000000000000/RESOURCEGROUPS/FABRIKAMRG/PROVIDERS/MICROSOFT.NETWORK/NETWORKSECURITYGROUPS/FABRIAKMVM1-NSG",
254+
"operationName": "NetworkSecurityGroupFlowEvents",
255+
"properties": {
256+
"Version": 2,
257+
"flows": [
258+
{
259+
"rule": "DefaultRule_DenyAllInBound",
260+
"flows": [
261+
{
262+
"mac": "000D3AF87856",
263+
"flowTuples": [
264+
"1542110437,125.64.94.197,10.5.16.4,59752,18264,T,I,D,B,,,,",
265+
"1542110475,80.211.72.221,10.5.16.4,37433,8088,T,I,D,B,,,,",
266+
"1542110487,46.101.199.124,10.5.16.4,60577,8088,T,I,D,B,,,,",
267+
"1542110490,176.119.4.30,10.5.16.4,57067,52801,T,I,D,B,,,,"
268+
]
269+
}
270+
]
271+
}
272+
]
273+
}
274+
},
275+
...
276+
```
99277

100278
## Next steps
101279

0 commit comments

Comments
 (0)