Skip to content

Commit 0c8f3bf

Browse files
Merge pull request #234175 from halkazwini/docs-editor/traffic-analytics-schema-1681255871
Convert numbered lists to bulleted
2 parents 12c8b58 + 665b1d3 commit 0c8f3bf

File tree

1 file changed

+16
-14
lines changed

1 file changed

+16
-14
lines changed

articles/network-watcher/traffic-analytics-schema.md

Lines changed: 16 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ ms.service: network-watcher
77
ms.topic: conceptual
88
ms.workload: infrastructure-services
99
ms.date: 04/11/2023
10-
ms.custom: template-concept, engagement-fy23
1110
ms.author: halkazwini
11+
ms.custom: template-concept, engagement-fy23
1212
---
1313

1414
# Schema and data aggregation in Azure Network Watcher traffic analytics
@@ -23,12 +23,12 @@ Traffic analytics is a cloud-based solution that provides visibility into user a
2323

2424
## Data aggregation
2525

26-
1. All flow logs at a network security group between `FlowIntervalStartTime_t` and `FlowIntervalEndTime_t` are captured at one-minute intervals as blobs in a storage account.
27-
2. Default processing interval of traffic analytics is 60 minutes, meaning that every hour, traffic analytics picks blobs from the storage account for aggregation. However, if a processing interval of 10 minutes is selected, traffic analytics will instead pick blobs from the storage account every 10 minutes.
28-
3. Flows that have the same `Source IP`, `Destination IP`, `Destination port`, `NSG name`, `NSG rule`, `Flow Direction`, and `Transport layer protocol` (TCP or UDP) (Note: source port is excluded for aggregation) are clubbed into a single flow by traffic analytics.
29-
4. This single record is decorated (details in the section below) and ingested in Log Analytics by traffic analytics. This process can take up to 1 hour max.
30-
5. `FlowStartTime_t` field indicates the first occurrence of such an aggregated flow (same four-tuple) in the flow log processing interval between `FlowIntervalStartTime_t` and `FlowIntervalEndTime_t`.
31-
6. For any resource in traffic analytics, the flows indicated in the Azure portal are total flows seen by the network security group, but in Log Analytics user sees only the single, reduced record. To see all the flows, use the `blob_id` field, which can be referenced from storage. The total flow count for that record matches the individual flows seen in the blob.
26+
- All flow logs at a network security group between `FlowIntervalStartTime_t` and `FlowIntervalEndTime_t` are captured at one-minute intervals as blobs in a storage account.
27+
- Default processing interval of traffic analytics is 60 minutes, meaning that every hour, traffic analytics picks blobs from the storage account for aggregation. However, if a processing interval of 10 minutes is selected, traffic analytics will instead pick blobs from the storage account every 10 minutes.
28+
- Flows that have the same `Source IP`, `Destination IP`, `Destination port`, `NSG name`, `NSG rule`, `Flow Direction`, and `Transport layer protocol` (TCP or UDP) (Note: source port is excluded for aggregation) are clubbed into a single flow by traffic analytics.
29+
- This single record is decorated (details in the section below) and ingested in Log Analytics by traffic analytics. This process can take up to 1 hour max.
30+
- `FlowStartTime_t` field indicates the first occurrence of such an aggregated flow (same four-tuple) in the flow log processing interval between `FlowIntervalStartTime_t` and `FlowIntervalEndTime_t`.
31+
- For any resource in traffic analytics, the flows indicated in the Azure portal are total flows seen by the network security group, but in Log Analytics user sees only the single, reduced record. To see all the flows, use the `blob_id` field, which can be referenced from storage. The total flow count for that record matches the individual flows seen in the blob.
3232

3333
The following query helps you look at all subnets interacting with non-Azure public IPs in the last 30 days.
3434

@@ -77,7 +77,7 @@ https://{saName}@insights-logs-networksecuritygroupflowevent/resoureId=/SUBSCRIP
7777
7878
```
7979

80-
## Fields used in traffic analytics schema
80+
## Traffic analytics schema
8181

8282
> [!IMPORTANT]
8383
> The traffic analytics schema was updated on August 22, 2019. The new schema provides source and destination IPs separately, removing need to parse the `FlowDirection` field so that queries are simpler. These are changes in the updated schema:
@@ -158,11 +158,11 @@ The following table lists the fields in the schema and what they signify.
158158
| SrcPublicIPs_s | <SOURCE_PUBLIC_IP>\|\<FLOW_STARTED_COUNT>\|\<FLOW_ENDED_COUNT>\|\<OUTBOUND_PACKETS>\|\<INBOUND_PACKETS>\|\<OUTBOUND_BYTES>\|\<INBOUND_BYTES> | Entries separated by bars. |
159159
| DestPublicIPs_s | <DESTINATION_PUBLIC_IP>\|\<FLOW_STARTED_COUNT>\|\<FLOW_ENDED_COUNT>\|\<OUTBOUND_PACKETS>\|\<INBOUND_PACKETS>\|\<OUTBOUND_BYTES>\|\<INBOUND_BYTES> | Entries separated by bars. |
160160

161-
## Public IP Details Schema
161+
## Public IP details schema
162162

163163
Traffic analytics provides WHOIS data and geographic location for all public IPs in your environment. For a malicious IP, traffic analytics provides DNS domain, threat type and thread descriptions as identified by Microsoft security intelligence solutions. IP Details are published to your Log Analytics workspace so you can create custom queries and put alerts on them. You can also access prepopulated queries from the traffic analytics dashboard.
164164

165-
The following table details public ip schema:
165+
The following table details public IP schema:
166166

167167
| Field | Format | Comments |
168168
| ----- | ------ | -------- |
@@ -195,12 +195,11 @@ List of threat types:
195195
| PUA | Potentially Unwanted Application. |
196196
| WatchList | A generic bucket into which indicators are placed when it can't be determined exactly what the threat is or will require manual interpretation. `WatchList` should typically not be used by partners submitting data into the system. |
197197

198-
199198
## Notes
200199

201-
1. In case of `AzurePublic` and `ExternalPublic` flows, customer owned Azure virtual machine IP is populated in `VMIP_s` field, while the Public IP addresses are populated in the `PublicIPs_s` field. For these two flow types, you should use `VMIP_s` and `PublicIPs_s` instead of `SrcIP_s` and `DestIP_s` fields. For AzurePublic and ExternalPublic IP addresses, we aggregate further, so that the number of records ingested to log analytics workspace is minimal. (This field will be deprecated soon and you should be using SrcIP_ and DestIP_s depending on whether the virtual machine was the source or the destination in the flow).
202-
1. Some field names are appended with `_s` or `_d`, which don't signify source and destination but indicate the data types *string* and *decimal* respectively.
203-
1. Based on the IP addresses involved in the flow, we categorize the flows into the following flow types:
200+
- In case of `AzurePublic` and `ExternalPublic` flows, customer owned Azure virtual machine IP is populated in `VMIP_s` field, while the Public IP addresses are populated in the `PublicIPs_s` field. For these two flow types, you should use `VMIP_s` and `PublicIPs_s` instead of `SrcIP_s` and `DestIP_s` fields. For AzurePublic and ExternalPublic IP addresses, we aggregate further, so that the number of records ingested to log analytics workspace is minimal. (This field will be deprecated soon and you should be using SrcIP_ and DestIP_s depending on whether the virtual machine was the source or the destination in the flow).
201+
- Some field names are appended with `_s` or `_d`, which don't signify source and destination but indicate the data types *string* and *decimal* respectively.
202+
- Based on the IP addresses involved in the flow, we categorize the flows into the following flow types:
204203
- `IntraVNet`: Both IP addresses in the flow reside in the same Azure virtual network.
205204
- `InterVNet`: IP addresses in the flow reside in two different Azure virtual networks.
206205
- `S2S` (Site-To-Site): One of the IP addresses belongs to an Azure virtual network, while the other IP address belongs to customer network (Site) connected to the virtual network through VPN gateway or ExpressRoute.
@@ -212,5 +211,8 @@ List of threat types:
212211
- `Unknown`: Unable to map either of the IP addresses in the flow with the customer topology in Azure and on-premises (site).
213212

214213
## Next Steps
214+
215215
- To learn more about traffic analytics, see [Azure Network Watcher Traffic analytics](traffic-analytics.md).
216216
- See [Traffic analytics FAQ](traffic-analytics-faq.yml) for answers to traffic analytics frequently asked questions.
217+
218+

0 commit comments

Comments
 (0)