Skip to content

Commit 0f2bba8

Browse files
SuhaliKondapalli17SuhaliKondapalli17
authored
Update linux-support-perf.md
Removing AuditD and replacing it with eBPF statistics and Hot Event Sources instead.
1 parent 72fc65d commit 0f2bba8

File tree

1 file changed

+149
-90
lines changed

1 file changed

+149
-90
lines changed

‎defender-endpoint/linux-support-perf.md‎

Lines changed: 149 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -143,127 +143,186 @@ The following steps can be used to troubleshoot and mitigate these issues:
143143
> [!NOTE]
144144
> The application stores statistics in memory and only keeps track of file activity since it was started and real-time protection was enabled. Processes that were launched before or during periods when real time protection was off are not counted. Additionally, only events which triggered scans are counted.
145145
146-
147-
## Troubleshoot performance issues using Microsoft Defender for Endpoint Client Analyzer
146+
## Troubleshoot performance issues using Hot Event Sources
148147

149148
**Applies to:**
150-
- Performance issues of all available Defender for Endpoint components such as AV and EDR
151-
152-
The Microsoft Defender for Endpoint Client Analyzer (MDECA) can collect traces, logs, and diagnostic information in order to troubleshoot performance issues on [onboarded devices](onboard-configure.md) on Linux.
153-
154-
> [!NOTE]
155-
> - The Microsoft Defender for Endpoint Client Analyzer tool is regularly used by Microsoft Customer Support Services (CSS) to collect information such as (but not limited to) IP addresses, PC names that will help troubleshoot issues you may be experiencing with Microsoft Defender for Endpoint. For more information about our privacy statement, see [Microsoft Privacy Statement](https://privacy.microsoft.com/privacystatement).
156-
> - As a general best practice, it is recommended to update the [Microsoft Defender for Endpoint agent to latest available version](mac-whatsnew.md) and confirming that the issue still persists before investigating further.
157-
158-
To run the client analyzer for troubleshooting performance issues, see [Run the client analyzer on macOS and Linux](run-analyzer-macos-linux.md).
159-
160-
## Configure Global Exclusions for better performance
161-
162-
Configure Microsoft Defender for Endpoint on Linux with exclusions for the processes or disk locations that contribute to the performance issues. For more information, see [Configure and validate exclusions for Microsoft Defender for Endpoint on Linux](linux-exclusions.md). IF you still have performance issues, contact support for further instructions and mitigation.
163-
164-
## Troubleshoot AuditD performance issues
165-
166-
**Background:**
149+
- Performance issues in global files and executables.
167150

168-
- Microsoft Defender for Endpoint on Linux OS distributions uses AuditD framework to collect certain types of telemetry events.
151+
Hot event sources is a feature that will specifically show the events which have highest count (highest frequency of occurrence) for generating file events.
169152

170-
- System events captured by rules added to `/etc/audit/rules.d/` will add to audit.log(s) and might affect host auditing and upstream collection.
153+
> [!NOTE]
154+
> These commmands require you to have root permissions. Ensure that sudo can be used.
171155
172-
- Events added by Microsoft Defender for Endpoint on Linux will be tagged with `mdatp` key.
156+
First, check the log level on your machine.
173157

174-
- If the AuditD service is misconfigured or offline, then some events might be missing. To troubleshoot such an issue, refer to: [Troubleshoot missing events or alerts issues for Microsoft Defender for Endpoint on Linux.](linux-support-events.md)
158+
```bash
159+
mdatp health --field log_level
160+
```
161+
If it's not on "debug" you need to change it for a detailed report regarding hot files / executables.
175162

176-
In certain server workloads, two issues might be observed:
163+
```bash
164+
sudo mdatp log level set --level debug
165+
```
166+
```console
167+
Log level configured successfully
168+
```
177169

178-
- `High CPU` resource consumption from `mdatp_audisp_plugin` process.
170+
To collect current statistics (for files),
179171

180-
- `/var/log/audit/audit.log` becoming large or frequently rotating.
172+
```bash
173+
sudo mdatp diagnostic hot-event-sources files
174+
```
175+
The output of which will look similar to the following (JSON);
181176

182-
These issues may occur on servers with many events flooding AuditD. Such issues can arise if there are multiple consumers for AuditD, or too many rules with the combination of Microsoft Defender for Endpoint and third party consumers, or high workload that generates a lot of events. To troubleshoot such issues, begin by [collecting MDEClientAnalyzer logs](run-analyzer-macos-linux.md) on the sample affected server.
177+
```console
178+
{
179+
"startTime": "1729535104539160",
180+
"endTime": "1729535117570766",
181+
"totalEvent": "11373",
182+
"eventSource": [
183+
{
184+
"authCount": "2832",
185+
"csId": "",
186+
"notifyCount": "0",
187+
"path": "/mnt/RamDisk/postgres_data/pg_wal/0000000100000014000000A5",
188+
"pidCount": "1",
189+
"teamId": ""
190+
},
191+
{
192+
"authCount": "632",
193+
"csId": "",
194+
"notifyCount": "0",
195+
"path": "/mnt/RamDisk/postgres_data/base/635594/2601",
196+
"pidCount": "1",
197+
"teamId": ""
198+
}
199+
]
200+
}
201+
```
202+
And similarly output on the console looks like the following (this is just a snippet of the entire output). Here the first row is the count (frequency of occurrence) and the second is the file path.
183203

184-
As a best practice, we recommend to configure AuditD logs to rotate when the maximum file size limit is reached. This configuration prevents AuditD logs from accumulating in a single file, and the rotated log files can be moved out to save disk space. To achieve this, you can set the value for `max_log_file_action` to `rotate` in the [auditd.conf](https://linux.die.net/man/8/auditd.conf) file.
185-
186-
> [!NOTE]
187-
> As a general best practice, it is recommended to update the [Microsoft Defender for Endpoint agent to latest available version](linux-whatsnew.md) and confirming issue still persists before investigating further. That there are additional configurations that can affect AuditD subsystem CPU strain. Specifically, in [auditd.conf](https://linux.die.net/man/8/auditd.conf), the value for `disp_qos` can be set to `lossy` to reduce the high CPU consumption. However, this means that some events may be dropped during peak CPU consumption.
204+
```console
205+
Total Events: 11179 Time: 12s. Throughput: 75.3333 events/sec.
206+
=========== Top 684 Hot Event Sources ===========
207+
count file path
208+
2832 /mnt/RamDisk/postgres_data/pg_wal/0000000100000014000000A5
209+
632 /mnt/RamDisk/postgres_data/base/635594/2601
210+
619 /mnt/RamDisk/postgres_data/base/635597/2601
211+
618 /mnt/RamDisk/postgres_data/base/635596/2601
212+
618 /mnt/RamDisk/postgres_data/base/635595/2601
213+
616 /mnt/RamDisk/postgres_data/base/635597/635610
214+
615 /mnt/RamDisk/postgres_data/base/635596/635602
215+
614 /mnt/RamDisk/postgres_data/base/635595/635606
216+
514 /mnt/RamDisk/postgres_data/base/635594/635598_fsm
217+
496 /mnt/RamDisk/postgres_data/base/635597/635610_fsm
218+
```
188219

189-
### XMDEClientAnalyzer
220+
and similarly for the executables,
190221

191-
When you use [XMDEClientAnalyzer](run-analyzer-macos-linux.md), the following files will display output that provides insights to help you troubleshoot issues.
222+
```bash
223+
sudo mdatp diagnostic hot-event-sources executables
224+
```
192225

193-
- `auditd_info.txt`
194-
- `auditd_log_analysis.txt`
226+
The output of which will look similar to the following (JSON);
195227

196-
#### auditd_info.txt
228+
```console
229+
{
230+
"startTime": "1729534260988396",
231+
"endTime": "1729534280026883",
232+
"totalEvent": "48165",
233+
"eventSource": [
234+
{
235+
"authCount": "8126",
236+
"csId": "",
237+
"notifyCount": "0",
238+
"path": "/usr/lib/postgresql/12/bin/psql",
239+
"pidCount": "2487",
240+
"teamId": ""
241+
},
242+
{
243+
"authCount": "5127",
244+
"csId": "",
245+
"notifyCount": "0",
246+
"path": "/usr/lib/postgresql/12/bin/postgres (deleted)",
247+
"pidCount": "2144",
248+
"teamId": ""
249+
}
250+
]
251+
}
252+
```
253+
Output on the console;
197254

198-
Contains general AuditD configuration and displays the following information:
255+
```console
256+
Total Events: 47382 Time: 18s. Throughput: 157 events/sec.
257+
=========== Top 23 Hot Event Sources ===========
258+
count executable path
259+
8216 /usr/lib/postgresql/12/bin/psql
260+
5721 /usr/lib/postgresql/12/bin/postgres (deleted)
261+
3557 /usr/bin/bash
262+
378 /usr/bin/clamscan
263+
88 /usr/bin/sudo
264+
70 /usr/bin/dash
265+
30 /usr/sbin/zabbix_agent2
266+
10 /usr/bin/grep
267+
8 /usr/bin/gawk
268+
6 /opt/microsoft/mdatp/sbin/wdavdaemonclient
269+
4 /usr/bin/sleep
270+
```
271+
To improve this performance, locate the path with the highest number in `count` row and add a process exclusion (in case of executable) or a file/folder exclusion (in case of file) for it. For more information, see [Configure and validate exclusions for Defender for Endpoint on Linux](linux-exclusions.md).
199272

200-
- What processes are registered as AuditD consumers.
273+
## Troubleshoot performance issues using eBPF Statistics
201274

202-
- `Auditctl -s` output with `enabled=2` (Suggests auditd is in immutable mode (requires restart for any config changes to take effect.)
275+
**Applies to:**
276+
- All file/ process events, specifically for syscall based performance issues.
203277

204-
- `Auditctl -l` output (Shows what rules are currently loaded into the kernel, which might differ from what exists on disk in `/etc/auditd/rules.d/mdatp.rules`. Also shows which rules are related to Microsoft Defender for Endpoint.)
278+
eBPF (extended Berkeley Packet Filter) statistics command gives insights into the top event/process that's generating the most file events, along with their syscall ids.
205279

206-
#### auditd_log_analysis.txt
280+
To collect current statistics using eBPF statistics, run:
207281

208-
Contains important aggregated information that is useful when investigating AuditD performance issues.
282+
```bash
283+
mdatp diagnostic ebpf-statistics
284+
```
209285

210-
- Which component owns the most reported events (Microsoft Defender for Endpoint events will be tagged with `key=mdatp`).
286+
The output is always on the console and would look similar to the following (this is only a snippet of the entire output):
211287

212-
- The top reporting initiators.
288+
```console
289+
Top initiator paths:
290+
/usr/lib/postgresql/12/bin/psql : 902
291+
/usr/bin/clamscan : 349
292+
/usr/sbin/zabbix_agent2 : 27
293+
/usr/lib/postgresql/12/bin/postgres : 10
294+
295+
Top syscall ids:
296+
80 : 9034
297+
57 : 8932
298+
60 : 8929
299+
59 : 4942
300+
112 : 4898
301+
90 : 179
302+
87 : 170
303+
119 : 32
304+
288 : 19
305+
41 : 15
306+
```
213307

214-
- The most common system calls (network or filesystem events, and others).
308+
To improve this performance, locate the one with the highest `count` in the `Top initiator path` row and add an exclusion for it. For more information, see [Configure and validate exclusions for Defender for Endpoint on Linux](linux-exclusions.md).
309+
310+
## Troubleshoot performance issues using Microsoft Defender for Endpoint Client Analyzer
215311

216-
- What file system paths are the noisiest.
312+
**Applies to:**
313+
- Performance issues of all available Defender for Endpoint components such as AV and EDR
217314

218-
**To mitigate most AuditD performance issues, you can implement AuditD exclusion. If the given exclusions do not improve the performance then we can use the rate limiter option. This will reduce the number of events being generated by AuditD altogether.**
315+
The Microsoft Defender for Endpoint Client Analyzer (MDECA) can collect traces, logs, and diagnostic information in order to troubleshoot performance issues on [onboarded devices](onboard-configure.md) on Linux.
219316

220317
> [!NOTE]
221-
> Exclusions should be made only for low threat and high noise initiators or paths. For example, do not exclude /bin/bash which risks creating a large blind spot.
222-
> [Common mistakes to avoid when defining exclusions](common-exclusion-mistakes-microsoft-defender-antivirus.md).
223-
224-
### Exclusion Types
225-
226-
The XMDEClientAnalyzer support tool contains syntax that can be used to add AuditD exclusion configuration rules.
227-
228-
AuditD exclusion – support tool syntax help:
229-
230-
:::image type="content" source="media/auditd-exclusion-support-tool-syntax-help.png" alt-text="Screenshot of the syntax that can be used to add AuditD exclusion configuration rules.":::
231-
232-
**By initiator**
233-
234-
- `-e/ -exe` full binary path > Removes all events by this initiator
235-
236-
**By path**
237-
238-
- `-d / -dir` full path to a directory > Removes filesystem events targeting this directory
239-
240-
Examples:
241-
242-
If `/opt/app/bin/app` writes to `/opt/app/cfg/logs/1234.log`, then you can use the support tool to exclude with various options:
243-
244-
`-e /opt/app/bin/app`
245-
246-
`-d /opt/app/cfg`
247-
248-
`-x /usr/bin/python /etc/usercfg`
249-
250-
`-d /usr/app/bin/`
251-
252-
More examples:
253-
254-
`./mde_support_tool.sh exclude -p <process id>`
255-
256-
`./mde_support_tool.sh exclude -e <process name>`
257-
258-
To exclude more than one item - concatenate the exclusions into one line:
259-
260-
`./mde_support_tool.sh exclude -e <process name> -e <process name 2> -e <process name3>`
318+
> - The Microsoft Defender for Endpoint Client Analyzer tool is regularly used by Microsoft Customer Support Services (CSS) to collect information such as (but not limited to) IP addresses, PC names that will help troubleshoot issues you may be experiencing with Microsoft Defender for Endpoint. For more information about our privacy statement, see [Microsoft Privacy Statement](https://privacy.microsoft.com/privacystatement).
319+
> - As a general best practice, it is recommended to update the [Microsoft Defender for Endpoint agent to latest available version](mac-whatsnew.md) and confirming that the issue still persists before investigating further.
261320
262-
The -x flag is used to exclude access to subdirectories by specific initiators for example:
321+
To run the client analyzer for troubleshooting performance issues, see [Run the client analyzer on macOS and Linux](run-analyzer-macos-linux.md).
263322

264-
`./mde_support_tool.sh exclude -x /usr/sbin/mv /tmp`
323+
## Configure Global Exclusions for better performance
265324

266-
The above will exclude monitoring of /tmp subfolder, when accessed by mv process.
325+
Configure Microsoft Defender for Endpoint on Linux with exclusions for the processes or disk locations that contribute to the performance issues. For more information, see [Configure and validate exclusions for Microsoft Defender for Endpoint on Linux](linux-exclusions.md). IF you still have performance issues, contact support for further instructions and mitigation.
267326

268327
### Rate Limiter
269328

0 commit comments

Comments
 (0)