You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/sentinel/cef-syslog-ama-overview.md
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ ms.date: 04/22/2024
11
11
12
12
# Syslog via AMA and Common Event Format (CEF) via AMA connectors for Microsoft Sentinel
13
13
14
-
The Syslog via AMA and Common Event Format (CEF) via AMA data connectors for Microsoft Sentinel filter and ingest Syslog messages, including those in Common Event Format (CEF), from Linux machines and from network and security devices and appliances. These connectors install the Azure Monitor Agent (AMA) on any Linux machine from which you want to collect Syslog and/or CEF messages. This machine could be the originator of the messages, or it could be a forwarder that collects messages from other machines, such as network or security devices and appliances. The connector sends the agents instructions based on [Data Collection Rules (DCRs)](../azure-monitor/essentials/data-collection-rule-overview.md) that you define. DCRs specify the systems to monitor and the types of logs or messages to collect, and they define filters to apply to the messages before they're ingested, for better performance and more efficient querying and analysis.
14
+
The Syslog via AMA and Common Event Format (CEF) via AMA data connectors for Microsoft Sentinel filter and ingest Syslog messages, including messages in Common Event Format (CEF), from Linux machines and from network and security devices and appliances. These connectors install the Azure Monitor Agent (AMA) on any Linux machine from which you want to collect Syslog and/or CEF messages. This machine could be the originator of the messages, or it could be a forwarder that collects messages from other machines, such as network or security devices and appliances. The connector sends the agents instructions based on [Data Collection Rules (DCRs)](../azure-monitor/essentials/data-collection-rule-overview.md) that you define. DCRs specify the systems to monitor and the types of logs or messages to collect. They define filters to apply to the messages before they're ingested, for better performance and more efficient querying and analysis.
15
15
16
16
Syslog and CEF are two common formats for logging data from different devices and applications. They help system administrators and security analysts to monitor and troubleshoot the network and identify potential threats or incidents.
17
17
@@ -37,7 +37,7 @@ This diagram shows Syslog messages being collected from a single individual Linu
37
37
38
38
The data ingestion process using the Azure Monitor Agent uses the following components and data flows:
39
39
40
-
-**Log sources:** These are your various Linux VMs in your environment that produce Syslog messages. These messages are collected by the local Syslog daemon on TCP or UDP port 514 (or another port per your preference).
40
+
-**Log sources** are your various Linux VMs in your environment that produce Syslog messages. These messages are collected by the local Syslog daemon on TCP or UDP port 514 (or another port per your preference).
41
41
42
42
- The local **Syslog daemon** (either `rsyslog` or `syslog-ng`) collects the log messages on TCP or UDP port 514 (or another port per your preference). The daemon then sends these logs to the **Azure Monitor Agent** in two different ways, depending on the AMA version:
43
43
- AMA versions **1.28.11** and above receive logs on **TCP port 28330**.
@@ -57,9 +57,9 @@ This diagram shows Syslog and CEF messages being collected from a Linux-based lo
57
57
58
58
The data ingestion process using the Azure Monitor Agent uses the following components and data flows:
59
59
60
-
-**Log sources:** These are your various security devices and appliances in your environment that produce log messages in CEF format, or in plain Syslog. These devices are configured to send their log messages over TCP or UDP port 514 (or another port per your preference), *not* to their local Syslog daemon, but instead to the **Syslog daemon on the Log forwarder**.
60
+
-**Log sources** are your various security devices and appliances in your environment that produce log messages in CEF format, or in plain Syslog. These devices are configured to send their log messages over TCP or UDP port 514 (or another port per your preference), *not* to their local Syslog daemon, but instead to the **Syslog daemon on the Log forwarder**.
61
61
62
-
-**Log forwarder:** This is a dedicated Linux VM that your organization sets up to collect the log messages from your Syslog and CEF log sources. The VM can be on-premises, in Azure, or in another cloud. This log forwarder itself has two components:
62
+
-**Log forwarder** is a dedicated Linux VM that your organization sets up to collect the log messages from your Syslog and CEF log sources. The VM can be on-premises, in Azure, or in another cloud. This log forwarder itself has two components:
63
63
- The **Syslog daemon** (either `rsyslog` or `syslog-ng`) collects the log messages on TCP or UDP port 514 (or another port per your preference). The daemon then sends these logs to the **Azure Monitor Agent** in two different ways, depending on the AMA version:
64
64
- AMA versions **1.28.11** and above receive logs on **TCP port 28330**.
65
65
- Earlier versions of AMA receive logs via Unix domain socket.
@@ -71,34 +71,34 @@ The data ingestion process using the Azure Monitor Agent uses the following comp
71
71
72
72
---
73
73
74
-
## Set up process to collect log messages
74
+
## Setup process to collect log messages
75
75
76
76
From the **Content hub** in Microsoft Sentinel, install the appropriate solution for **Syslog** or **Common Event Format**. This step installs the respective data connectors Syslog via AMA or Common Event Format (CEF) via AMA data connector. For more information, see [Discover and manage Microsoft Sentinel out-of-the-box content](sentinel-solutions-deploy.md).
77
77
78
-
As part of the set up process, create a data collection rule and install the Azure Monitor Agent (AMA) on the log forwarder. Do these tasks either by using the Azure or Microsoft Defender portal or by using the Azure Monitor logs ingestion API.
78
+
As part of the setup process, create a data collection rule and install the Azure Monitor Agent (AMA) on the log forwarder. Do these tasks either by using the Azure or Microsoft Defender portal or by using the Azure Monitor logs ingestion API.
79
79
80
80
- When you configure the data connector for the Microsoft Sentinel in the Azure or Microsoft Defender portal, you can create, manage, and delete DCRs per workspace. The AMA is installed automatically on the VMs that you select in the connector configuration.
81
81
82
-
- Alternatively, send HTTP requests to the Logs Ingestion API. With this setup, you can create, manage, and delete DCRs. This option is more flexible than the portal. For example, with the API, you can filter by specific log levels. In the Azure or Defender portal, you can only select a minimum log level. The downside to using this menthod is that you have to manually install the Azure Monitor Agent on the log forwarder before creating a DCR.
82
+
- Alternatively, send HTTP requests to the Logs Ingestion API. With this setup, you can create, manage, and delete DCRs. This option is more flexible than the portal. For example, with the API, you can filter by specific log levels. In the Azure or Defender portal, you can only select a minimum log level. The downside to using this method is that you have to manually install the Azure Monitor Agent on the log forwarder before creating a DCR.
83
83
84
-
After you create the DCR and AMA is installed, run the "installation" script on the log forwarder to configure the Syslog daemon to listen for messages from other machines, and to open the necessary local ports. Then configure the security devices or applicances.
84
+
After you create the DCR, and AMA is installed, run the "installation" script on the log forwarder. This script configures the Syslog daemon to listen for messages from other machines, and to open the necessary local ports. Then configure the security devices, or appliances as needed.
85
85
86
-
For more information, see the following topics:
86
+
For more information, see the following articles:
87
87
88
88
-[Ingest Syslog and CEF messages to Microsoft Sentinel with the Azure Monitor Agent](connect-cef-syslog-ama.md)
89
89
-[Configure specific devices for Microsoft Sentinel syslog or CEF AMA data connectors](unified-connectors-configure-devices.md)
90
90
91
-
## Data ingestion duplication
91
+
## Data ingestion duplication avoidance
92
92
93
-
Using the same facility for both Syslog and CEF messages may result in data ingestion duplication between the CommonSecurityLog and Syslog tables.
93
+
Using the same facility for both Syslog and CEF messages might result in data ingestion duplication between the CommonSecurityLog and Syslog tables.
94
94
95
95
To avoid this scenario, use one of these methods:
96
96
97
-
-**If the source device enables configuration of the target facility**: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF won't also be sent in Syslog. Make sure that each DCR you configure in the next steps uses the relevant facility for CEF or Syslog respectively.
97
+
-**If the source device enables configuration of the target facility**: On each source machine that sends logs to the log forwarder in CEF format, edit the Syslog configuration file to remove the facilities used to send CEF messages. This way, the facilities sent in CEF aren't also be sent in Syslog. Make sure that each DCR you configure in the next steps uses the relevant facility for CEF or Syslog respectively.
98
98
99
99
To see an example of how to arrange a DCR to ingest both Syslog and CEF messages from the same agent, go to [Syslog and CEF streams in the same DCR](connect-cef-syslog-ama.md#syslog-and-cef-streams-in-the-same-dcr).
100
100
101
-
-**If changing the facility for the source appliance isn't applicable**: Use an ingest time transformation to filter out CEF messages from the Syslog stream to avoid duplication, as shown in the query example below. The data will be sent twice from the collector machine to the workspace.
101
+
-**If changing the facility for the source appliance isn't applicable**: Use an ingest time transformation to filter out CEF messages from the Syslog stream to avoid duplication, as shown in the following query example.
Copy file name to clipboardExpand all lines: articles/sentinel/connect-cef-syslog-ama.md
+6-14Lines changed: 6 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -47,7 +47,7 @@ If you're collecting messages from a log forwarder, the following prerequisites
47
47
48
48
- For space requirements for your log forwarder, refer to the [Azure Monitor Agent Performance Benchmark](../azure-monitor/agents/azure-monitor-agent-performance.md). You can also review [this blog post](https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/designs-for-accomplishing-microsoft-sentinel-scalable-ingestion/ba-p/3741516), which includes designs for scalable ingestion.
49
49
50
-
- Your log sources, security devices and appliances, must be configured to send their log messages to the log forwarder's Syslog daemon instead of to their local Syslog daemon.
50
+
- Your log sources, security devices, and appliances, must be configured to send their log messages to the log forwarder's Syslog daemon instead of to their local Syslog daemon.
51
51
52
52
### Machine security prerequisites
53
53
@@ -60,21 +60,13 @@ If your devices are sending Syslog and CEF logs over TLS because, for example, y
60
60
61
61
## Configure the data connector
62
62
63
-
The setup process for the Syslog via AMA or Common Event Format (CEF) via AMA data connectors includes the following steps that you can do in the Azure or Microsoft Defender portal or by using the Azure Monitor logs ingestion API.
63
+
The setup process for the Syslog via AMA or Common Event Format (CEF) via AMA data connectors includes the following steps:
64
64
65
-
- From the data connector in Microsoft Sentinel, create a data collection rule. As part
66
-
1. Install the Azure Monitor Agent and create a Data Collection Rule (DCR).
67
-
-[Using the Azure or Defender portal](?tabs=syslog%2Cportal#create-a-data-collection-rule)
68
-
-[Using the Azure Monitor Logs Ingestion API](?tabs=syslog%2Capi#install-the-azure-monitor-agent)
65
+
1. Install the Azure Monitor Agent and create a Data Collection Rule (DCR) by using either of the following methods:
66
+
-[Azure or Defender portal](?tabs=syslog%2Cportal#create-data-collection-rule)
1. If you're collecting logs from other machines using a log forwarder, [**run the "installation" script**](#run-the-installation-script) on the log forwarder to configure the Syslog daemon to listen for messages from other machines, and to open the necessary local ports.
70
69
71
-
72
-
Set up the appropriate data connector in one of two ways:
73
-
74
-
- Configure the **Syslog via AMA** or **Common Event Format (CEF) via AMA** data connector in the Azure or Microsoft Defender portal for Microsoft Sentinel. With this setup, you can create, manage, and delete DCRs per workspace. The AMA is installed automatically on the VMs that you select in the connector configuration.
75
-
**—OR—**
76
-
- Send HTTP requests to the Logs Ingestion API. With this setup, you can create, manage, and delete DCRs. This option is more flexible than the portal. For example, with the API, you can filter by specific log levels, where with the UI, you can only select a minimum log level. The downside is that you have to manually install the Azure Monitor Agent on the log forwarder before creating a DCR.
77
-
78
70
Select the appropriate tab for instructions.
79
71
80
72
# [Azure or Defender portal](#tab/portal)
@@ -144,7 +136,7 @@ After you complete all the tabs, review what you entered and create the data col
144
136
145
137
### Install the Azure Monitor Agent
146
138
147
-
Follow the appropriate instructions from the Azure Monitor documentation to install the Azure Monitor Agent on your log forwarder. Remember to use the instructions for Linux, not those for Windows.
139
+
Follow the appropriate instructions from the Azure Monitor documentation to install the Azure Monitor Agent on your log forwarder. Remember to use the instructions for Linux, not for Windows.
148
140
-[Install the AMA using PowerShell](../azure-monitor/agents/azure-monitor-agent-manage.md?tabs=azure-powershell)
149
141
-[Install the AMA using the Azure CLI](../azure-monitor/agents/azure-monitor-agent-manage.md?tabs=azure-cli)
150
142
-[Install the AMA using an Azure Resource Manager template](../azure-monitor/agents/azure-monitor-agent-manage.md?tabs=azure-resource-manager)
0 commit comments