|
| 1 | +--- |
| 2 | +title: Stream logs in both the CEF and Syslog format to Microsoft Sentinel |
| 3 | +description: Stream and filter logs in both the CEF and Syslog format to your Microsoft Sentinel workspace. |
| 4 | +author: limwainstein |
| 5 | +ms.topic: how-to |
| 6 | +ms.date: 02/09/2023 |
| 7 | +ms.author: lwainstein |
| 8 | +#Customer intent: As a security operator, I want to stream and filter CEF an Syslog-based logs from my organization to my Microsoft Sentinel workspace, so I can avoid duplication between CEF and Syslog data. |
| 9 | +--- |
| 10 | + |
| 11 | +# Stream logs in both the CEF and Syslog format |
| 12 | + |
| 13 | +This article describes how to stream and filter logs in both the CEF and Syslog format to your Microsoft Sentinel workspace from multiple appliances. This article is useful in the following scenario: |
| 14 | + |
| 15 | +- You're using a Linux log collector to forward both Syslog and CEF events to your Microsoft Sentinel workspaces using the Azure Monitor Agent (AMA). |
| 16 | +- You want to ingest Syslog events in the Syslog table and CEF events in the CommonSecurityLog table. |
| 17 | + |
| 18 | +During this process, you use the AMA and Data Collection Rules (DCRs). With DCRs, you can filter the logs before they're ingested, for quicker upload, efficient analysis, and querying. Data Collection Rules (DCRs) to filter the logs before they're ingested, for quicker upload, efficient analysis, and querying. |
| 19 | + |
| 20 | +> [!IMPORTANT] |
| 21 | +> |
| 22 | +> On **February 28th 2023**, we will introduce [changes to the CommonSecurityLog table schema](https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/upcoming-changes-to-the-commonsecuritylog-table/ba-p/3643232). This means that custom queries will require being reviewed and updated. Out-of-the-box content (detections, hunting queries, workbooks, parsers, etc.) will be updated by Microsoft Sentinel. |
| 23 | +
|
| 24 | +Read more about [CEF](connect-cef-ama.md#what-is-cef-collection) and [Syslog](connect-syslog.md#architecture) collection in Microsoft Sentinel. |
| 25 | + |
| 26 | +## Prerequisites |
| 27 | + |
| 28 | +Before you begin, verify that you have: |
| 29 | + |
| 30 | +- The Microsoft Sentinel solution enabled. |
| 31 | +- A defined Microsoft Sentinel workspace. |
| 32 | +- A Linux machine to collect logs. |
| 33 | + - The Linux machine must have Python 2.7 or 3 installed on the Linux machine. Use the ``python --version`` or ``python3 --version`` command to check. |
| 34 | +- Either the `syslog-ng` or `rsyslog` daemon enabled. |
| 35 | +- To collect events from any system that isn't an Azure virtual machine, ensure that [Azure Arc](../azure-monitor/agents/azure-monitor-agent-manage.md) is installed. |
| 36 | +- To ingest Syslog and CEF logs into Microsoft Sentinel, you can designate and configure a Linux machine that collects the logs from your devices and forwards them to your Microsoft Sentinel workspace. [Configure a log forwarder](connect-cef-ama.md#configure-a-log-forwarder). |
| 37 | + |
| 38 | +## Avoid data ingestion duplication |
| 39 | + |
| 40 | +Using the same facility for both Syslog and CEF messages may result in data ingestion duplication between the CommonSecurityLog and Syslog tables. |
| 41 | + |
| 42 | +To avoid this scenario, use one of these methods: |
| 43 | + |
| 44 | +- **If the source device enables configuration of the target facility**: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF won't also be sent in Syslog. Make sure that each DCR you configure in the next steps uses the relevant facility for CEF or Syslog respectively. |
| 45 | +- **If changing the facility for the source appliance isn't applicable**: Use an ingest time transformation to filter out CEF messages from the Syslog stream to avoid duplication: |
| 46 | + |
| 47 | + ```kusto |
| 48 | + source | |
| 49 | + where ProcessName !contains “\“CEF\”” |
| 50 | + ``` |
| 51 | +## Create a DCR for your CEF logs |
| 52 | +
|
| 53 | +- Create the DCR via the UI: |
| 54 | + 1. [Open the connector page and create the DCR](connect-cef-ama.md#open-the-connector-page-and-create-the-dcr). |
| 55 | + 1. [Define resources (VMs)](connect-cef-ama.md#define-resources-vms). |
| 56 | + 1. [Select the data source type and create the DCR](connect-cef-ama.md#select-the-data-source-type-and-create-the-dcr). |
| 57 | +
|
| 58 | + > [!IMPORTANT] |
| 59 | + > Make sure to **[avoid data ingestion duplication](#avoid-data-ingestion-duplication)** (review the options in this section). |
| 60 | +
|
| 61 | + 1. [Run the installation script](connect-cef-ama.md). |
| 62 | +
|
| 63 | +- Create the DCR via the API: |
| 64 | + 1. [Create the request URL and header](connect-cef-ama.md#request-url-and-header). |
| 65 | + 1. [Create the request body](connect-cef-ama.md#request-body). |
| 66 | +
|
| 67 | + See [examples of facilities and log levels sections](connect-cef-ama.md#examples-of-facilities-and-log-levels-sections). |
| 68 | +
|
| 69 | +## Create a DCR for your Syslog logs |
| 70 | +
|
| 71 | +Create the DCR for your Syslog-based logs using the Azure Monitor [guidelines](../azure-monitor/essentials/data-collection-rule-overview.md) and [structure](../azure-monitor/essentials/data-collection-rule-structure.md). Review the [best practices](../azure-monitor/essentials/data-collection-rule-best-practices.md) if needed. |
| 72 | +
|
| 73 | +## Create a DCR for both Syslog and CEF logs |
| 74 | +
|
| 75 | +1. Run this command to launch the installation script: |
| 76 | + |
| 77 | + ```python |
| 78 | + sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py |
| 79 | + ``` |
| 80 | + The installation script configures the `rsyslog` or `syslog-ng` daemon to use the required protocol and restarts the daemon. |
| 81 | +
|
| 82 | +1. Create the request URL and header: |
| 83 | +
|
| 84 | + ```rest |
| 85 | + GET |
| 86 | + https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2019-11-01-preview |
| 87 | + ``` |
| 88 | +
|
| 89 | +1. Create the request body: |
| 90 | + - Verify that the `streams` field is set to `Microsoft-CommonSecurityLog` and `Microsoft-Syslog` for the CEF/Syslog facility respectively. |
| 91 | + - Add the filter and facility log levels in the `facilityNames` and `logLevels` parameters. |
| 92 | +
|
| 93 | + ```rest |
| 94 | + { |
| 95 | + "properties": { |
| 96 | + "immutableId": "dcr-c7847b758fb0484b88b51c5d907796a6", |
| 97 | + "dataSources": { |
| 98 | + "syslog": [ |
| 99 | + { |
| 100 | + "streams": ["Microsoft-Syslog"], |
| 101 | + "facilityNames": ["auth"], |
| 102 | + "logLevels": [ |
| 103 | + "Info", |
| 104 | + "Notice", |
| 105 | + "Warning", |
| 106 | + "Error", |
| 107 | + "Critical", |
| 108 | + "Alert", |
| 109 | + "Emergency" |
| 110 | + ], |
| 111 | + "name": "sysLogsDataSource--1469397783" |
| 112 | + }, |
| 113 | + { |
| 114 | + "streams": ["Microsoft-CommonSecurityLog"], |
| 115 | + "facilityNames": [ |
| 116 | + "local4" |
| 117 | + ], |
| 118 | + "logLevels": [ |
| 119 | + "Warning" |
| 120 | + ], |
| 121 | + "name": "sysLogsDataSource-1688419672" |
| 122 | + } |
| 123 | + ] |
| 124 | + }, |
| 125 | + "destinations": { |
| 126 | + "logAnalytics": [ |
| 127 | + { |
| 128 | + "workspaceResourceId": "/subscriptions/<sub-id>/resourceGroups/<resourceGroup>/providers/Microsoft.OperationalInsights/workspaces/<WS>", |
| 129 | + "workspaceId": "<WS-ID>", |
| 130 | + "name": "la--591870646" |
| 131 | + } |
| 132 | + ] |
| 133 | + }, |
| 134 | + "dataFlows": [ |
| 135 | + { "streams": ["Microsoft-Syslog", "Microsoft-CommonSecurityLog"], "destinations": ["la--591870646"] } |
| 136 | + ], |
| 137 | + "provisioningState": "Succeeded" |
| 138 | + }, |
| 139 | + "location": "eastus", |
| 140 | + "tags": {}, |
| 141 | + "kind": "Linux", |
| 142 | + "id": "/subscriptions/<sub-id>/resourceGroups/<resourceGroup>/providers/Microsoft.Insights/dataCollectionRules/<DCR-Name>", |
| 143 | + "name": "<DCR-Name>", |
| 144 | + "type": "Microsoft.Insights/dataCollectionRules", |
| 145 | + "etag": "\"6d00bdde-0000-0100-0000-62c177f70000\"", |
| 146 | + "systemData": { |
| 147 | + |
| 148 | + "createdByType": "User", |
| 149 | + "createdAt": "2022-07-03T11:05:27.2454015Z", |
| 150 | + "lastModifiedBy": [email protected], |
| 151 | + "lastModifiedByType": "User", |
| 152 | + "lastModifiedAt": "2022-07-03T11:05:27.2454015Z" |
| 153 | + } |
| 154 | + } |
| 155 | + ``` |
| 156 | +1. After you finish editing the template, use `POST` or `PUT` to deploy it: |
| 157 | +
|
| 158 | + ```rest |
| 159 | + PUT |
| 160 | + https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2019-11-01-preview |
| 161 | + ``` |
| 162 | +
|
| 163 | +See [examples of facilities and log levels sections](connect-cef-ama.md#examples-of-facilities-and-log-levels-sections). |
| 164 | +
|
| 165 | +## Next steps |
| 166 | +
|
| 167 | +In this article, you learned how to stream and filter logs in both the CEF and Syslog format to your Microsoft Sentinel workspace. To learn more about Microsoft Sentinel, see the following articles: |
| 168 | +- Learn how to [get visibility into your data, and potential threats](get-visibility.md). |
| 169 | +- Get started [detecting threats with Microsoft Sentinel](detect-threats-built-in.md). |
| 170 | +- [Use workbooks](monitor-your-data.md) to monitor your data. |
0 commit comments