|
| 1 | +--- |
| 2 | +title: Stream logs in both the CEF and Syslog format to Microsoft Sentinel |
| 3 | +description: Stream and filter logs in both the CEF and Syslog format to your Microsoft Sentinel workspace. |
| 4 | +author: limwainstein |
| 5 | +ms.topic: how-to |
| 6 | +ms.date: 02/09/2023 |
| 7 | +ms.author: lwainstein |
| 8 | +#Customer intent: As a security operator, I want to stream and filter CEF an Syslog-based logs from my organization to my Microsoft Sentinel workspace, so I can avoid duplication between CEF and Syslog data. |
| 9 | +--- |
| 10 | + |
| 11 | +# Stream logs in both the CEF and Syslog format |
| 12 | + |
| 13 | +This article describes how to stream and filter logs in both the CEF and Syslog format to your Microsoft Sentinel workspace from multiple on-premises appliances. This process is relevant when your organization uses different appliances that ingest logs over both CEF and Syslog, and you want to ingest both types of data without duplications. |
| 14 | + |
| 15 | +During this process, you use the Azure Monitor Agent (AMA), which uses Data Collection Rules (DCRs). With DCRs, you can filter the logs before they're ingested, for quicker upload, efficient analysis, and querying. Data Collection Rules (DCRs) to filter the logs before they're ingested, for quicker upload, efficient analysis, and querying. |
| 16 | + |
| 17 | +> [!IMPORTANT] |
| 18 | +> |
| 19 | +> On **February 28th 2023**, we will introduce [changes to the CommonSecurityLog table schema](https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/upcoming-changes-to-the-commonsecuritylog-table/ba-p/3643232). This means that custom queries will require being reviewed and updated. Out-of-the-box content (detections, hunting queries, workbooks, parsers, etc.) will be updated by Microsoft Sentinel. |
| 20 | +
|
| 21 | +Read more about [CEF](connect-cef-ama.md#what-is-cef-collection) and [Syslog](connect-syslog.md#architecture) collection in Microsoft Sentinel. |
| 22 | + |
| 23 | +## Prerequisites |
| 24 | + |
| 25 | +Before you begin, verify that you have: |
| 26 | + |
| 27 | +- The Microsoft Sentinel solution enabled. |
| 28 | +- A defined Microsoft Sentinel workspace. |
| 29 | +- A Linux machine to collect logs. |
| 30 | + - The Linux machine must have Python 2.7 or 3 installed on the Linux machine. Use the ``python --version`` or ``python3 --version`` command to check. |
| 31 | +- Either the `syslog-ng` or `rsyslog` daemon enabled. |
| 32 | +- To collect events from any system that isn't an Azure virtual machine, ensure that [Azure Arc](../azure-monitor/agents/azure-monitor-agent-manage.md) is installed. |
| 33 | +- To ingest Syslog and CEF logs into Microsoft Sentinel, you can designate and configure a Linux machine that collects the logs from your devices and forwards them to your Microsoft Sentinel workspace. [Configure a log forwarder](connect-cef-ama.md#configure-a-log-forwarder) |
| 34 | + |
| 35 | +## Separate your facilities |
| 36 | + |
| 37 | +To avoid data duplication, each DCR you configure in the next steps uses a separate facility for CEF and Syslog. To ensure that data isn't duplicated, make sure that the appliance that sends Syslog data and the appliance that sends CEF data do so on different facilities, for example `local1` ad `local2`. |
| 38 | + |
| 39 | +## Create a DCR for your CEF logs |
| 40 | + |
| 41 | +- Create the DCR: via the UI: |
| 42 | + 1. [Open the connector page and create the DCR](connect-cef-ama.md#open-the-connector-page-and-create-the-dcr). |
| 43 | + 1. [Define resources (VMs)](connect-cef-ama.md#define-resources-vms) |
| 44 | + 1. [Select the data source type and create the DCR](connect-cef-ama.md#select-the-data-source-type-and-create-the-dcr) |
| 45 | + |
| 46 | + > [!NOTE] |
| 47 | + > **Using the same machine to forward both plain Syslog *and* CEF messages** |
| 48 | + > |
| 49 | + > If you plan to use the same log forwarder machine to forward Syslog messages as well as CEF, in order to avoid the duplication of events to the Syslog and CommonSecurityLog tables: |
| 50 | + > |
| 51 | + > On each source machine that sends logs to the forwarder in CEF format, you must edit the Syslog configuration file to remove the facilities that are being used to send CEF messages. This way, the facilities that are sent in CEF won't also be sent in Syslog. |
| 52 | +
|
| 53 | + 1. [Run the installation script](connect-cef-ama.md) |
| 54 | + |
| 55 | +- Create the DCR via the API: |
| 56 | + 1. [Create the request URL and header](connect-cef-ama.md#request-url-and-header) |
| 57 | + 1. [Create the request body](connect-cef-ama.md#request-body) |
| 58 | + |
| 59 | + See [examples of facilities and log levels sections](connect-cef-ama.md#examples-of-facilities-and-log-levels-sections). |
| 60 | + |
| 61 | +## Create a DCR for your Syslog logs |
| 62 | + |
| 63 | +Create the DCR for your Syslog logs using the [guidelines](/azure-monitor/essentials/data-collection-rule-overview) and [structure](/azure-monitor/essentials/data-collection-rule-structure). Review the [best practices](/azure-monitor/essentials/data-collection-rule-best-practices) if needed. |
| 64 | + |
| 65 | +## Create a DCR for both Syslog and CEF logs |
| 66 | + |
| 67 | +1. Run this command to launch the installation script: |
| 68 | + |
| 69 | + ```python |
| 70 | + sudo wget -O Forwarder_AMA_installer.py https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/DataConnectors/Syslog/Forwarder_AMA_installer.py&&sudo python Forwarder_AMA_installer.py |
| 71 | + ``` |
| 72 | + The installation script configures the `rsyslog` or `syslog-ng` daemon to use the required protocol and restarts the daemon. |
| 73 | + |
| 74 | +1. Create the request URL and header: |
| 75 | + |
| 76 | + ```rest |
| 77 | + GET |
| 78 | + https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2019-11-01-preview |
| 79 | + ``` |
| 80 | + |
| 81 | +1. Create the request body: |
| 82 | + - Verify that the `streams` field is set to `Microsoft-CommonSecurityLog` and `Microsoft-Syslog` for the CEF/Syslog facility respectively. |
| 83 | + - Add the filter and facility log levels in the `facilityNames` and `logLevels` parameters. |
| 84 | + |
| 85 | + ```rest |
| 86 | + { |
| 87 | + "properties": { |
| 88 | + "immutableId": "dcr-c7847b758fb0484b88b51c5d907796a6", |
| 89 | + "dataSources": { |
| 90 | + "syslog": [ |
| 91 | + { |
| 92 | + "streams": ["Microsoft-Syslog"], |
| 93 | + "facilityNames": ["auth"], |
| 94 | + "logLevels": [ |
| 95 | + "Info", |
| 96 | + "Notice", |
| 97 | + "Warning", |
| 98 | + "Error", |
| 99 | + "Critical", |
| 100 | + "Alert", |
| 101 | + "Emergency" |
| 102 | + ], |
| 103 | + "name": "sysLogsDataSource--1469397783" |
| 104 | + }, |
| 105 | + { |
| 106 | + "streams": ["Microsoft-CommonSecurityLog"], |
| 107 | + "facilityNames": [ |
| 108 | + "local4" |
| 109 | + ], |
| 110 | + "logLevels": [ |
| 111 | + "Warning" |
| 112 | + ], |
| 113 | + "name": "sysLogsDataSource-1688419672" |
| 114 | + } |
| 115 | + ] |
| 116 | + }, |
| 117 | + "destinations": { |
| 118 | + "logAnalytics": [ |
| 119 | + { |
| 120 | + "workspaceResourceId": "/subscriptions/<sub-id>/resourceGroups/<resourceGroup>/providers/Microsoft.OperationalInsights/workspaces/<WS>", |
| 121 | + "workspaceId": "<WS-ID>", |
| 122 | + "name": "la--591870646" |
| 123 | + } |
| 124 | + ] |
| 125 | + }, |
| 126 | + "dataFlows": [ |
| 127 | + { "streams": ["Microsoft-Syslog", "Microsoft-CommonSecurityLog"], "destinations": ["la--591870646"] } |
| 128 | + ], |
| 129 | + "provisioningState": "Succeeded" |
| 130 | + }, |
| 131 | + "location": "eastus", |
| 132 | + "tags": {}, |
| 133 | + "kind": "Linux", |
| 134 | + "id": "/subscriptions/<sub-id>/resourceGroups/<resourceGroup>/providers/Microsoft.Insights/dataCollectionRules/<DCR-Name>", |
| 135 | + "name": "<DCR-Name>", |
| 136 | + "type": "Microsoft.Insights/dataCollectionRules", |
| 137 | + "etag": "\"6d00bdde-0000-0100-0000-62c177f70000\"", |
| 138 | + "systemData": { |
| 139 | + "createdBy": someuser@microsoft.com, |
| 140 | + "createdByType": "User", |
| 141 | + "createdAt": "2022-07-03T11:05:27.2454015Z", |
| 142 | + "lastModifiedBy": someuser@microsoft.com, |
| 143 | + "lastModifiedByType": "User", |
| 144 | + "lastModifiedAt": "2022-07-03T11:05:27.2454015Z" |
| 145 | + } |
| 146 | + } |
| 147 | + ``` |
| 148 | +1. After you finish editing the template, use `POST` or `PUT` to deploy it: |
| 149 | + |
| 150 | + ```rest |
| 151 | + PUT |
| 152 | +https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Insights/dataCollectionRules/{dataCollectionRuleName}?api-version=2019-11-01-preview |
| 153 | + ``` |
| 154 | + |
| 155 | +See [examples of facilities and log levels sections](connect-cef-ama.md#examples-of-facilities-and-log-levels-sections). |
| 156 | + |
| 157 | +## Next steps |
| 158 | + |
| 159 | +In this article, you learned how to stream and filter logs in both the CEF and Syslog format to your Microsoft Sentinel workspace. To learn more about Microsoft Sentinel, see the following articles: |
| 160 | +- Learn how to [get visibility into your data, and potential threats](get-visibility.md). |
| 161 | +- Get started [detecting threats with Microsoft Sentinel](detect-threats-built-in.md). |
| 162 | +- [Use workbooks](monitor-your-data.md) to monitor your data. |
0 commit comments