You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/sentinel/network-normalization-schema.md
+17-51Lines changed: 17 additions & 51 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -28,71 +28,37 @@ The network normalization schema can represent any type of an IP network session
28
28
29
29
## Parsers
30
30
31
-
This section discusses parsers, how to add parsers, and how to filter parser parameters. For more information, see [ASIM parsers](normalization-parsers-overview.md) and [Use ASIM parsers](normalization-about-parsers.md).
31
+
For more information about ASIM parsers, see the [ASIM parsers overview](normalization-parsers-overview.md) and [Use ASIM parsers](normalization-about-parsers.md).
32
32
33
33
### Unifying parsers
34
34
35
-
To use the unifying parsers that unify all of the out-of-the-box parsers, and ensure that your analysis runs across all the configured sources, use the following KQL functions as the table name in your query.
36
-
37
-
Deploy ASIM parsers from the [Microsoft Sentinel GitHub repository](https://aka.ms/DeployASIM).
Aggregative parser that uses *union* to include normalized events from all *network session* sources.
42
-
43
-
- Update this parser if you want to add or remove sources from source-agnostic analytics.
44
-
- Use this function in your source-agnostic queries.
45
-
46
-
#### ASimNetworkSession
47
-
48
-
Similar to the [imNetworkSession](#imnetworksession) function, but without parameter support, so it doesn't force the **Logs** page time picker to use the `custom` value.
49
-
50
-
- Update these parsers if you want to add or remove sources from source-agnostic analytics.
51
-
- Use this function in your source-agnostic queries if you don't plan to use parameters.
52
-
53
-
#### vimNetworkSession\<vendor\>\<product\>
54
-
55
-
Source-specific parsers implement normalization for a specific source.
56
-
57
-
Example: `vimNetworkSessionSysmonLinux`
58
-
59
-
- Add a source-specific parser for a source when there's no out-of-the-box normalizing parser. Update the `im` aggregative parser to include reference to your new parser.
60
-
- Update a source-specific parser to resolve parsing and normalization issues.
61
-
- Use a source-specific parser for source-specific analytics.
62
-
63
-
#### ASimNetworkSession\<vendor\>\<product\>
64
-
65
-
Source-specific parsers implement normalization for a specific source.
66
-
67
-
Unlike the `vim*` functions, the `ASim*` functions don't support parameters.
68
-
69
-
- Add a source-specific parser for a source when there's no out-of-the-box normalizing parser. Update the aggregative `ASim` parser to include reference to your new parser.
70
-
- Update a source-specific parser to resolve parsing and normalization issues.
71
-
- Use an `ASim` source-specific parser for interactive queries when not using parameters.
35
+
To use parsers that unify all ASIM out-of-the-box parsers, and ensure that your analysis runs across all the configured sources, use the `_Im_NetworkSession` filtering parser or the `_ASim_NetworkSession` parameter-less parser.
72
36
37
+
You can also use workspace-deployed `ImNetworkSession` and `ASimNetworkSession` parsers by deploying them from the [Microsoft Sentinel GitHub repository](https://aka.ms/DeployASIM).
73
38
39
+
For more information, see [built-in ASIM parsers and workspace-deployed parsers](normalization-parsers-overview.md#built-in-asim-parsers-and-workspace-deployed-parsers).
74
40
75
41
### Out-of-the-box, source-specific parsers
76
42
77
-
Microsoft Sentinel provides the following built-in, product-specific Network Session parsers:
43
+
Microsoft Sentinel provides the following out-of-the-box, product-specific Network Session parsers:
|**Microsoft Sysmon for Linux**| - Parametrized: `vimNetworkSessionSysmonLinux`<br> - Regular: `ASimNetworkSessionSysmonLinux`|
84
-
| **Windows Events Firewall** | Windows firewall activity as represented by using Windows Events 515x, collected by using either the Log Analytics Agent or the Azure Monitor Agent into either the `Event` table or the `WindowsEvent` table.<br><br> - Parametrized: `vimNetworkSessionMicrosoftWindowsEventFirewall` <br> - Regular: `ASimNetworkSessionMicrosoftWindowsEventFirewall`
|**Sysmon for Linux** (event 3)<br> Collected using the Log Analytics Agent<br> or the Azure Monitor Agent |`_ASim_NetworkSession_LinuxSysmon` (regular)<br><br>`_Im_NetworkSession_LinuxSysmon` (filtering) |`ASimNetworkSessionLinuxSysmon` (regular)<br><br> `vimNetworkSessionLinuxSysmon` (filtering) |
50
+
|**Windows Firewall**<br>Collected using the Log Analytics Agent (Event table) or Azure Monitor Agent (WindowsEvent table). Supports Windows events 5150 to 5159. |`_ASim_NetworkSession_`<br>`MicrosoftWindowsEventFirewall` (regular)<br><br>`_Im_NetworkSession_`<br>`MicrosoftWindowsEventFirewall` (filtering) |`ASimNetworkSession`<br>`MicrosoftWindowsEventFirewall` (regular)<br><br> `vimNetworkSession`<br>`MicrosoftWindowsEventFirewall` (filtering) |
When you implement custom parsers for the Network Session information model, name your KQL functions by using the following syntax:
57
+
When implementing custom parsers for the Network Session information model, name your KQL functions using the following syntax:
90
58
91
59
-`vimNetworkSession<vendor><Product>` for parametrized parsers
92
60
-`ASimNetworkSession<vendor><Product>` for regular parsers
93
61
94
-
Then, add the new parser to `imNetworkSession` or `ASimNetworkSession`, respectively.
95
-
96
62
### Filtering parser parameters
97
63
98
64
The `im` and `vim*` parsers support [filtering parameters](normalization-about-parsers.md#optimized-parsers). While these parsers are optional, they can improve your query performance.
@@ -115,7 +81,7 @@ For example, to filter only web sessions for a specified list of domain names, u
115
81
116
82
```kql
117
83
let torProxies=dynamic(["tor2web.org", "tor2web.com", "torlink.co",...]);
@@ -218,7 +184,7 @@ The following fields are common to all network session activity logging:
218
184
|**SrcGeoLongitude**| Optional | Longitude | The longitude of the geographical coordinate associated with the source IP address.<br><br>Example: `73.211944`|
219
185
|**NetworkApplicationProtocol**| Optional | String | The application layer protocol used by the connection or session. If the [DstPortNumber](#dstportnumber) value is provided, we recommend that you include **NetworkApplicationProtocol** too. If the value isn't available from the source, derive the value from the [DstPortNumber](#dstportnumber) value.<br><br>Example: `FTP`|
220
186
|**NetworkProtocol**| Optional | Enumerated | The IP protocol used by the connection or session as listed in [IANA protocol assignment](https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml), which is typically `TCP`, `UDP`, or `ICMP`.<br><br>Example: `TCP`|
221
-
|**NetworkDirection**| Optional | Enumerated | The direction of the connection or session, into or out of the organization. Supported values include `Inbound`, `Outbound`, and `Listen`. The `Listen` value indicates that a device has started accepting network connections but isn't actually, necessarily, connected.|
187
+
|**NetworkDirection**| Optional | Enumerated | The direction of the connection or session, into or out of the organization. Supported values include `Inbound`, `Outbound`, `Listen`, and `Unknown`. The `Listen` value indicates that a device has started accepting network connections but isn't actually, necessarily, connected.|
222
188
| <aname="networkduration"></a>**NetworkDuration**| Optional | Integer | The amount of time, in milliseconds, for the completion of the network session or connection.<br><br>Example: `1500`|
223
189
|**Duration**| Alias || Alias to [NetworkDuration](#networkduration). |
224
190
|**NetworkIcmpCode**| Optional | Integer | For an ICMP message, the ICMP message type numeric value as described in [RFC 2780](https://datatracker.ietf.org/doc/html/rfc2780) for IPv4 network connections, or in [RFC 4443](https://datatracker.ietf.org/doc/html/rfc4443) for IPv6 network connections. If a [NetworkIcmpType](#networkicmptype) value is provided, this field is mandatory. If the value isn't available from the source, derive the value from the [NetworkIcmpType](#networkicmptype) field instead.<br><br>Example: `34`|
Copy file name to clipboardExpand all lines: articles/sentinel/normalization-about-schemas.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -108,7 +108,7 @@ The following fields are defined by ASIM for all schemas:
108
108
| **EventReportUrl** | Optional | String | A URL provided in the event for a resource that provides more information about the event.|
109
109
| <a name="dvc"></a>**Dvc** | Mandatory | String | A unique identifier of the device on which the event occurred or which reported the event, depending on the schema. <br><br>This field might alias the [DvcFQDN](#dvcfqdn), [DvcId](#dvcid), [DvcHostname](#dvchostname), or [DvcIpAddr](#dvcipaddr) fields. For cloud sources, for which there is no apparent device, use the same value as the [Event Product](#eventproduct) field. |
110
110
| <a name ="dvcipaddr"></a>**DvcIpAddr** | Recommended | IP address | The IP address of the device on which the event occurred or which reported the event, depending on the schema. <br><br>Example: `45.21.42.12`|
111
-
| <a name ="dvchostname"></a>**DvcHostname** | Recommended | Hostname | The hostname of the device on which the event occurred or which reported the event, depending on the schema. <br><br>Example: `ContosoDc.Contoso.Azure`|
111
+
| <a name ="dvchostname"></a>**DvcHostname** | Recommended | Hostname | The hostname of the device on which the event occurred or which reported the event, depending on the schema. <br><br>Example: `ContosoDc`|
112
112
| <a name="dvcdomain"></a>**DvcDomain** | Recommended | String | The domain of the device on which the event occurred or which reported the event, depending on the schema.<br><br>Example: `Contoso`|
113
113
| <a name="dvcdomaintype"></a>**DvcDomainType** | Recommended | Enumerated | The type of [DvcDomain](#dvcdomain), if known. Possible values include:<br>- `Windows`, such as for `contoso\mypc`<br>- `FQDN`, such as for `docs.microsoft.com`<br><br>**Note**: This field is required if the [DvcDomain](#dvcdomain) field is used. |
114
114
| <a name="dvcfqdn"></a>**DvcFQDN** | Optional | String | The hostname of the device on which the event occurred or which reported the event, depending on the schema. <br><br> Example: `Contoso\DESKTOP-1282V4D`<br><br>**Note**: This field supports both traditional FQDN format and Windows domain\hostname format. The [DvcDomainType](#dvcdomaintype) field reflects the format used. |
Copy file name to clipboardExpand all lines: articles/sentinel/normalization-develop-parsers.md
+26Lines changed: 26 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -83,6 +83,32 @@ Filtering in KQL is done using the `where` operator. For example, **Sysmon event
83
83
Event | where Source == "Microsoft-Windows-Sysmon" and EventID == 1
84
84
```
85
85
86
+
#### Filtering by source type using a Watchlist
87
+
88
+
In some cases, the event itself does not contain information that would allow filtering for specific source types.
89
+
90
+
For example, Infoblox DNS events are sent as Syslog messages, and are hard to distinguish from Syslog messages sent from other sources. In such cases, the parser relies on a list of sources that defines the relevant events. This list is maintained in the **ASimSourceType** watchlist.
91
+
92
+
**To use the ASimSourceType watchlist in your parsers**:
93
+
94
+
1. Include the following line at the beginning of your parser:
95
+
96
+
```KQL
97
+
let Sources_by_SourceType=(sourcetype:string){_GetWatchlist('ASimSourceType') | where SearchKey == tostring(sourcetype) | extend Source=column_ifexists('Source','') | where isnotempty(Source)| distinct Source };
98
+
```
99
+
100
+
2. Add a filter that uses the watchlist in the parser filtering section. For example, the Infoblox DNS parser includes the following in the filtering section:
101
+
102
+
```KQL
103
+
| where Computer in (Sources_by_SourceType('InfobloxNIOS'))
104
+
```
105
+
106
+
To use this sample in your parser:
107
+
108
+
* Replace `Computer` with the name of the field that includes the source information for your source. You can keep this as `Computer` for any parsers based on Syslog.
109
+
110
+
* Replace the `InfobloxNIOS` token with a value of your choice for your parser. Inform parser users that they must update the `ASimSourceType` watchlist using your selected value, as well as the list of sources that send events of this type.
111
+
86
112
#### Filtering based on parser parameters
87
113
88
114
When developing [filtering parsers](normalization-about-parsers.md#optimized-parsers), make sure that your parser accepts the filtering parameters for the relevant schema, as documented in the reference article for that schema. Using an existing parser as a starting point ensures that your parser includes the correct function signature. In most cases, the actual filtering code is also similar for filtering parsers for the same schema.
Copy file name to clipboardExpand all lines: articles/sentinel/normalization-manage-parsers.md
+13-1Lines changed: 13 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -27,6 +27,8 @@ You may need to manage the source-specific parsers used by each unifying parser
27
27
28
28
- Use a modified version of a built-in parser.
29
29
30
+
- **Configure a source-specific parser**, for example to define the sources that send information relevant to the parser.
31
+
30
32
This article guides you through managing your parsers, whether using built-in, unifying ASIM parsers or workspace-deployed unifying parsers.
31
33
32
34
> [!IMPORTANT]
@@ -49,7 +51,9 @@ Microsoft Sentinel users cannot edit built-in unifying parsers. Instead, use the
49
51
50
52
You can deploy initial, empty, unifying custom parsers to your Microsoft Sentinel workspace for all supported schemas, or individually for specific schemas. For more information, see [Deploy initial ASIM empty custom unifying parsers](https://aka.ms/ASimDeployEmptyCustomUnifyingParsers) in the Microsoft Sentinel GitHub repository.
51
53
52
-
- **To support excluding built-in source-specific parsers**, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel [GitHub](https://aka.ms/DeployASimExceptionWatchlist) repository.
54
+
- **To support excluding built-in source-specific parsers**, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel [GitHub](https://aka.ms/DeployASimWatchlists) repository.
55
+
56
+
- **To define source type for built-in and custom parsers**, ASIM uses a watchlist. Deploy the watchlist to your Microsoft Sentinel workspace from the Microsoft Sentinel [GitHub](https://aka.ms/DeployASimWatchlists) repository.
53
57
54
58
### Add a custom parser to a built-in unifying parser
55
59
@@ -62,6 +66,8 @@ The syntax of the line to add is different for each schema:
When adding an additional parser to a unifying custom parser that already references parsers, make sure you add a comma at the end of the previous line.
@@ -168,6 +174,12 @@ For example, the following code shows a DNS filtering unifying parser, having re
## Configure the sources relevant to a source-specific parser
178
+
179
+
Some parsers requires you to update the list of sources that are relevant to the parser. For example, a parser that uses Syslog data, may not be able to determine what Syslog events are relevant to the parser. Such a parser may use the ASimSourceType watchlist to determine which sources send information relevant to the parser. For such parses add a record for each relevant source to the watchlist:
180
+
- Set the `SourceType` field to the parser specific value specified in the parser documentation.
181
+
- Set the `Source` field to the identifier of the source used in the events. You may need to query the original table, such as Syslog, to determine the correct value.
182
+
171
183
## <a name="next-steps"></a>Next steps
172
184
173
185
This article discusses managing the Advanced SIEM Information Model (ASIM) parsers.
0 commit comments