Skip to content

Commit 984ffa3

Browse files
author
Jill Grant
authored
Merge pull request #282845 from yelevin/yelevin/custom-logs-ama
Custom Logs via AMA connector
2 parents 3863140 + baae16c commit 984ffa3

10 files changed

+1147
-12
lines changed

‎articles/sentinel/TOC.yml

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -860,10 +860,16 @@
860860
href: connect-log-forwarder.md
861861
- name: Syslog (raw) sources (legacy)
862862
href: connect-syslog.md
863+
- name: Custom log sources (text files)
864+
items:
865+
- name: Collect logs from text files via AMA
866+
href: connect-custom-logs-ama.md
867+
- name: Custom logs - configure security device
868+
href: unified-connector-custom-device.md
869+
- name: Custom log sources (legacy)
870+
href: connect-custom-logs.md
863871
- name: DNS via AMA
864-
href: connect-dns-ama.md
865-
- name: Custom log sources
866-
href: connect-custom-logs.md
872+
href: connect-dns-ama.md
867873
- name: Logstash plugin with Data Collection Rules
868874
href: connect-logstash-data-connection-rules.md
869875
- name: Logstash plugin (legacy)

‎articles/sentinel/api-dcr-reference.md

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,3 +216,158 @@ https://management.azure.com/subscriptions/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee/
216216
}
217217
```
218218

219+
## Custom logs from text files
220+
221+
The following examples are for DCRs using the AMA to collect custom logs from text files.
222+
223+
### Custom text logs DCR
224+
225+
These examples are of the API request for creating a DCR.
226+
227+
#### Custom text logs DCR creation request body
228+
229+
The following is an example of a DCR creation request for a custom log text file. Replace *`{PLACEHOLDER_VALUES}`* with actual values.
230+
231+
The `outputStream` parameter is required only if the transform changes the schema of the stream.
232+
233+
```json
234+
{
235+
"type": "Microsoft.Insights/dataCollectionRules",
236+
"name": "{DCR_NAME}",
237+
"location": "{WORKSPACE_LOCATION}",
238+
"apiVersion": "2022-06-01",
239+
"properties": {
240+
"streamDeclarations": {
241+
"Custom-Text-{TABLE_NAME}": {
242+
"columns": [
243+
{
244+
"name": "TimeGenerated",
245+
"type": "datetime"
246+
},
247+
{
248+
"name": "RawData",
249+
"type": "string"
250+
},
251+
]
252+
}
253+
},
254+
"dataSources": {
255+
"logFiles": [
256+
{
257+
"streams": [
258+
"Custom-Text-{TABLE_NAME}"
259+
],
260+
"filePatterns": [
261+
"{LOCAL_PATH_FILE_1}","{LOCAL_PATH_FILE_2}"
262+
],
263+
"format": "text",
264+
"name": "Custom-Text-{TABLE_NAME}"
265+
}
266+
],
267+
},
268+
"destinations": {
269+
"logAnalytics": [
270+
{
271+
"workspaceResourceId": "{WORKSPACE_RESOURCE_PATH}",
272+
"workspaceId": "{WORKSPACE_ID}",
273+
"name": "DataCollectionEvent"
274+
}
275+
],
276+
},
277+
"dataFlows": [
278+
{
279+
"streams": [
280+
"Custom-Text-{TABLE_NAME}"
281+
],
282+
"destinations": [
283+
"DataCollectionEvent"
284+
],
285+
"transformKql": "source",
286+
"outputStream": "Custom-{TABLE_NAME}"
287+
}
288+
]
289+
}
290+
}
291+
```
292+
293+
#### Custom text logs DCR creation response
294+
295+
```json
296+
{
297+
"properties": {
298+
"immutableId": "dcr-00112233445566778899aabbccddeeff",
299+
"dataCollectionEndpointId": "/subscriptions/aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb/resourceGroups/Contoso-RG-1/providers/Microsoft.Insights/dataCollectionEndpoints/Microsoft-Sentinel-aaaabbbbccccddddeeeefff",
300+
"streamDeclarations": {
301+
"Custom-Text-ApacheHTTPServer_CL": {
302+
"columns": [
303+
{
304+
"name": "TimeGenerated",
305+
"type": "datetime"
306+
},
307+
{
308+
"name": "RawData",
309+
"type": "string"
310+
}
311+
]
312+
}
313+
},
314+
"dataSources": {
315+
"logFiles": [
316+
{
317+
"streams": [
318+
"Custom-Text-ApacheHTTPServer_CL"
319+
],
320+
"filePatterns": [
321+
"C:\\Server\\bin\\log\\Apache24\\logs\\*.log"
322+
],
323+
"format": "text",
324+
"settings": {
325+
"text": {
326+
"recordStartTimestampFormat": "ISO 8601"
327+
}
328+
},
329+
"name": "Custom-Text-ApacheHTTPServer_CL"
330+
}
331+
]
332+
},
333+
"destinations": {
334+
"logAnalytics": [
335+
{
336+
"workspaceResourceId": "/subscriptions/aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb/resourceGroups/contoso-rg-1/providers/Microsoft.OperationalInsights/workspaces/CyberSOC",
337+
"workspaceId": "cccccccc-3333-4444-5555-dddddddddddd",
338+
"name": "DataCollectionEvent"
339+
}
340+
]
341+
},
342+
"dataFlows": [
343+
{
344+
"streams": [
345+
"Custom-Text-ApacheHTTPServer_CL"
346+
],
347+
"destinations": [
348+
"DataCollectionEvent"
349+
],
350+
"transformKql": "source",
351+
"outputStream": "Custom-ApacheHTTPServer_CL"
352+
}
353+
],
354+
"provisioningState": "Succeeded"
355+
},
356+
"location": "centralus",
357+
"tags": {
358+
"createdBy": "Sentinel"
359+
},
360+
"id": "/subscriptions/aaaaaaaa-0000-1111-2222-bbbbbbbbbbbb/resourceGroups/Contoso-RG-1/providers/Microsoft.Insights/dataCollectionRules/DCR-CustomLogs-01",
361+
"name": "DCR-CustomLogs-01",
362+
"type": "Microsoft.Insights/dataCollectionRules",
363+
"etag": "\"00000000-1111-2222-3333-444444444444\"",
364+
"systemData": {
365+
"createdBy": "[email protected]",
366+
"createdByType": "User",
367+
"createdAt": "2024-08-12T09:29:15.1083961Z",
368+
"lastModifiedBy": "[email protected]",
369+
"lastModifiedByType": "User",
370+
"lastModifiedAt": "2024-08-12T09:29:15.1083961Z"
371+
}
372+
}
373+
```

‎articles/sentinel/connect-cef-syslog-ama.md

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
2-
title: Ingest syslog CEF messages to Microsoft Sentinel - AMA
3-
description: Ingest syslog messages from linux machines, devices, and appliances to Microsoft Sentinel using data connectors based on the Azure Monitor Agent (AMA).
2+
title: Ingest syslog and CEF messages to Microsoft Sentinel - AMA
3+
description: Ingest syslog messages from linux machines and from network and security devices and appliances to Microsoft Sentinel, using data connectors based on the Azure Monitor Agent (AMA).
44
author: yelevin
55
ms.author: yelevin
66
ms.topic: how-to
@@ -22,21 +22,25 @@ This article describes how to use the **Syslog via AMA** and **Common Event Form
2222
2323
## Prerequisites
2424

25-
Before you begin, you must have the resources configured and the appropriate permissions described in this section.
25+
Before you begin, you must have the resources configured and the appropriate permissions assigned, as described in this section.
2626

2727
### Microsoft Sentinel prerequisites
2828

2929
Install the appropriate Microsoft Sentinel solution and make sure you have the permissions to complete the steps in this article.
3030

3131
- Install the appropriate solution from the **Content hub** in Microsoft Sentinel. For more information, see [Discover and manage Microsoft Sentinel out-of-the-box content](sentinel-solutions-deploy.md).
32+
3233
- Identify which data connector the Microsoft Sentinel solution requires — **Syslog via AMA** or **Common Event Format (CEF) via AMA** and whether you need to install the **Syslog** or **Common Event Format** solution. To fulfill this prerequisite,
34+
3335
- In the **Content hub**, select **Manage** on the installed solution and review the data connector listed.
36+
3437
- If either **Syslog via AMA** or **Common Event Format (CEF) via AMA** isn't installed with the solution, identify whether you need to install the **Syslog** or **Common Event Format** solution by finding your appliance or device from one of the following articles:
3538

3639
- [CEF via AMA data connector - Configure specific appliance or device for Microsoft Sentinel data ingestion](unified-connector-cef-device.md)
3740
- [Syslog via AMA data connector - Configure specific appliance or device for Microsoft Sentinel data ingestion](unified-connector-syslog-device.md)
3841

3942
Then install either the **Syslog** or **Common Event Format** solution from the content hub to get the related AMA data connector.
43+
4044
- Have an Azure account with the following Azure role-based access control (Azure RBAC) roles:
4145

4246
| Built-in role | Scope | Reason |
@@ -77,17 +81,17 @@ If your devices are sending syslog and CEF logs over TLS because, for example, y
7781
The setup process for the Syslog via AMA or Common Event Format (CEF) via AMA data connectors includes the following steps:
7882

7983
1. Install the Azure Monitor Agent and create a Data Collection Rule (DCR) by using either of the following methods:
80-
- [Azure or Defender portal](?tabs=syslog%2Cportal#create-data-collection-rule)
84+
- [Azure or Defender portal](?tabs=syslog%2Cportal#create-data-collection-rule-dcr)
8185
- [Azure Monitor Logs Ingestion API](?tabs=syslog%2Capi#install-the-azure-monitor-agent)
8286
1. If you're collecting logs from other machines using a log forwarder, [**run the "installation" script**](#run-the-installation-script) on the log forwarder to configure the syslog daemon to listen for messages from other machines, and to open the necessary local ports.
8387

8488
Select the appropriate tab for instructions.
8589

8690
# [Azure or Defender portal](#tab/portal)
8791

88-
### Create data collection rule
92+
### Create data collection rule (DCR)
8993

90-
To get started, open either the **Syslog via AMA** or **Common Event Format (CEF) via AMA** data connector in Microsoft Sentinel and create a data connector rule.
94+
To get started, open either the **Syslog via AMA** or **Common Event Format (CEF) via AMA** data connector in Microsoft Sentinel and create a data collection rule (DCR).
9195

9296
1. For Microsoft Sentinel in the [Azure portal](https://portal.azure.com), under **Configuration**, select **Data connectors**.<br> For Microsoft Sentinel in the [Defender portal](https://security.microsoft.com/), select **Microsoft Sentinel** > **Configuration** > **Data connectors**.
9397

0 commit comments

Comments
 (0)