Skip to content

Commit 44f0014

Browse files
PMEds28PMEds28
authored
Merge pull request #106753 from damendo/master
Inital TA schema
2 parents d87ee4c + ea2eaa7 commit 44f0014

File tree

2 files changed

+115
-0
lines changed

2 files changed

+115
-0
lines changed

‎articles/network-watcher/toc.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,8 @@
118118
href: traffic-analytics-faq.md
119119
- name: Schema and Data Aggregation
120120
href: traffic-analytics-schema.md
121+
- name: Schema update (August 2019)
122+
href: traffic-analytics-schema-update.md
121123
- name: Use Power BI
122124
href: network-watcher-visualize-nsg-flow-logs-power-bi.md
123125
- name: Use Elastic Stack

‎articles/network-watcher/traffic-analytics-schema-update.md

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
---
2+
title: Azure traffic analytics schema update - March 2020 | Microsoft Docs
3+
description: Sample queries with new fields in the Traffic Analytics schema.
4+
services: network-watcher
5+
documentationcenter: na
6+
author: vinigam
7+
manager: agummadi
8+
editor:
9+
10+
ms.service: network-watcher
11+
ms.devlang: na
12+
ms.topic: article
13+
ms.tgt_pltfrm: na
14+
ms.workload: infrastructure-services
15+
ms.date: 03/06/2020
16+
ms.author: vinigam
17+
18+
---
19+
# Sample queries with new fields in Traffic Analytics schema (August 2019 schema update)
20+
21+
The [Traffic Analytics Log schema](https://docs.microsoft.com/azure/network-watcher/traffic-analytics-schema) has been updated to include the following new fields: **SrcPublicIPs_s** , **DestPublicIPs_s**, **NSGRule_s**. In the next few months, the following older fields will be deprecated: **VMIP_s**, **Subscription_g**, **Region_s**, **NSGRules_s**, **Subnet_s**, **VM_s**, **NIC_s**, **PublicIPs_s**, **FlowCount_d**.
22+
The new fields provide information about source and destination IPs and simplify queries.
23+
24+
Below are three examples showing how to replace the old fields with new ones.
25+
26+
## Example 1 - VMIP_s, Subscription_g, Region_s, Subnet_s, VM_s, NIC_s, PublicIPs_s
27+
28+
We don’t have to infer Source and destination cases for Azure and External public flows from FlowDirection_s field for AzurePublic and ExternalPublic flows specifically. In case of an NVA (Network Virtual Appliance), the FlowDirection_s field can be inappropriate to be used as well.
29+
30+
```Old Kusto query
31+
AzureNetworkAnalytics_CL
32+
| where SubType_s == "FlowLog" and FASchemaVersion_s == "1"
33+
| extend isAzureOrExternalPublicFlows = FlowType_s in ("AzurePublic", "ExternalPublic")
34+
| extend SourceAzureVM = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'O', VM_s, "N/A"), VM1_s),
35+
SourceAzureVMIP = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'O', VM_s, "N/A"), SrcIP_s),
36+
SourceAzureVMSubscription = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'O', Subscription_g, "N/A"), Subscription1_g),
37+
SourceAzureRegion = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'O', Region_s, "N/A"), Region1_s),
38+
SourceAzureSubnet = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'O', Subnet_s, "N/A"), Subnet1_s),
39+
SourceAzureNIC = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'O', NIC_s, "N/A"), NIC1_s),
40+
DestAzureVM = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'I', VM_s, "N/A"), VM2_s),
41+
DestAzureVMIP = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'I', VM_s, "N/A"), DestIP_s),
42+
DestAzureVMSubscription = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'I', Subscription_g, "N/A"), Subscription2_g),
43+
DestAzureRegion = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'I', Region_s, "N/A"), Region2_s),
44+
DestAzureSubnet = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'I', Subnet_s, "N/A"), Subnet2_s),
45+
DestAzureNIC = iif(isAzureOrExternalPublicFlows, iif(FlowDirection_s == 'I', NIC_s, "N/A"), NIC2_s),
46+
SourcePublicIPsAggregated = iif(isAzureOrExternalPublicFlows and FlowDirection_s == 'I', PublicIPs_s, "N/A"),
47+
DestPublicIPsAggregated = iif(isAzureOrExternalPublicFlows and FlowDirection_s == 'O', PublicIPs_s, "N/A")
48+
```
49+
50+
51+
```New Kusto query
52+
AzureNetworkAnalytics_CL
53+
| where SubType_s == "FlowLog" and FASchemaVersion_s == "2"
54+
| extend SourceAzureVM = iif(isnotempty(VM1_s), VM1_s, "N/A"),
55+
SourceAzureVMIP = iif(isnotempty(SrcIP_s), SrcIP_s, "N/A"),
56+
SourceAzureVMSubscription = iif(isnotempty(Subscription1_g), Subscription1_g, "N/A"),
57+
SourceAzureRegion = iif(isnotempty(Region1_s), Region1_s, "N/A"),
58+
SourceAzureSubnet = iif(isnotempty(Subnet1_s), Subnet1_s, "N/A"),
59+
SourceAzureNIC = iif(isnotempty(NIC1_s), NIC1_s, "N/A"),
60+
DestAzureVM = iif(isnotempty(VM2_s), VM2_s, "N/A"),
61+
DestAzureVMIP = iif(isnotempty(DestIP_s), DestIP_s, "N/A"),
62+
DestAzureVMSubscription = iif(isnotempty(Subscription2_g), Subscription2_g, "N/A"),
63+
DestAzureRegion = iif(isnotempty(Region2_s), Region2_s, "N/A"),
64+
DestAzureSubnet = iif(isnotempty(Subnet2_s), Subnet2_s, "N/A"),
65+
DestAzureNIC = iif(isnotempty(NIC2_s), NIC2_s, "N/A"),
66+
SourcePublicIPsAggregated = iif(isnotempty(SrcPublicIPs_s), SrcPublicIPs_s, "N/A"),
67+
DestPublicIPsAggregated = iif(isnotempty(DestPublicIPs_s), DestPublicIPs_s, "N/A")
68+
```
69+
70+
71+
## Example 2 - NSGRules_s
72+
73+
Earlier field was of format: <Index value 0)>|<NSG_RULENAME>|<Flow Direction>|<Flow Status>|<FlowCount ProcessedByRule>
74+
75+
Earlier we used to aggregate data across NSG and NSGRules. Now we do not aggregate. So NSGList_s contains only one NSG and NSGRules_s also used to contain only one rule. So we have removed the complicated formatting here and the same can be found in other fields as mentioned below:
76+
77+
```Old Kusto query
78+
AzureNetworkAnalytics_CL
79+
| where SubType_s == "FlowLog" and FASchemaVersion_s == "1"
80+
| extend NSGRuleComponents = split(NSGRules_s, "|")
81+
| extend NSGName = NSGList_s // remains same
82+
| extend NSGRuleName = NSGRuleComponents[1],
83+
FlowDirection = NSGRuleComponents[2],
84+
FlowStatus = NSGRuleComponents[3],
85+
FlowCountProcessedByRule = NSGRuleComponents[4]
86+
| project NSGName, NSGRuleName, FlowDirection, FlowStatus, FlowCountProcessedByRule
87+
```
88+
89+
```New Kusto query
90+
AzureNetworkAnalytics_CL
91+
| where SubType_s == "FlowLog" and FASchemaVersion_s == "2"
92+
| extend NSGRuleComponents = split(NSGRules_s, "|")
93+
| project NSGName = NSGList_s,
94+
NSGRuleName = NSGRule_s ,
95+
FlowDirection = FlowDirection_s,
96+
FlowStatus = FlowStatus_s,
97+
FlowCountProcessedByRule = AllowedInFlows_d + DeniedInFlows_d + AllowedOutFlows_d + DeniedOutFlows_d
98+
```
99+
100+
## Example 3 - FlowCount_d
101+
102+
Since we do not club data across NSG, the FlowCount_d is simply AllowedInFlows_d + DeniedInFlows_d + AllowedOutFlows_d + DeniedOutFlows_d.
103+
Only 1 of the above 4 will be non-zero and rest three will be 0. And it would indicate the status and count in the NIC where the flow was captured.
104+
105+
If the flow was allowed, one of the fields prefixed with “Allowed” will be populated. Else one fields prefixed with “Denied” will be populated.
106+
If the flow was inbound, one of the fields suffixed with "\_d" like “InFlows_d” suffixed field will be populated. Else “OutFlows_d” will be populated.
107+
108+
Depending on above 2 conditions, we know which one out of the 4 will be populated.
109+
110+
111+
## Next Steps
112+
To get answers to frequently asked questions, see [Traffic analytics FAQ](traffic-analytics-faq.md)
113+
To see details about functionality, see [Traffic analytics documentation](traffic-analytics.md)

0 commit comments

Comments
 (0)